[Design Doc] In Doc Search UI #5707

dojutsu-user · 2019-05-17T12:36:44Z

Initial design doc for In Doc UI.
Closes #5699

dojutsu-user · 2019-05-19T19:29:40Z

@rtfd/core
I have implemented a short demo.
Here is something our search-as-you-type feature may look and work like.

(Implemented using edge-ngrams, same configurations as mentioned in the design docs)

Should I include this is in design docs?

safwanrahman · 2019-05-19T20:01:32Z

Thanks for the design doc. I think we need a hybrid solution here which is mixed of edge-ngrams, completing suggestor as well as Simple Query String. Moreover, I think we should show search suggestion like search engines like google by using the suggestor.

dojutsu-user · 2019-05-20T09:25:34Z

@safwanrahman
I am having some problems setting up the completion suggester, so I haven't tested the results of it yet.
https://discuss.elastic.co/t/problem-with-completion-suggester/181695
It would be great if you could help me out with this.

Also, some problems might be associated with completion suggester + edge_ngrams
My settings:

PUT test-completion
{
  "settings": {
    "analysis": {
      "filter": {
        "ngram_filter": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 30
        }
      },
      "analyzer": {
        "ngram_analyzer": {
          "type": "custom",
          "tokenizer": "whitespace",
          "filter": [
            "lowercase",
            "ngram_filter"
          ]
        }
      }
    }
  },
  "mappings": {
    "doc": {
      "properties": {
        "content": {
          "type": "text",
          "fields": {
            "autocomplete": {
              "type": "completion",
              "analyzer": "ngram_analyzer",
              "search_analyzer": "whitespace",
              "contexts": [
                {
                  "name": "project",
                  "type": "category",
                  "path": "project"
                },
                {
                  "name": "version",
                  "type": "category",
                  "path": "version"
                }
              ]
            }
          }
        },
        "title": {
          "type": "text",
          "fields": {
            "autocomplete": {
              "type": "completion",
              "analyzer": "ngram_analyzer",
              "search_analyzer": "whitespace"
            }
          }
        },
        "headers": {
          "type": "text"
        },
        "commit": {
          "type": "text"
        },
        "path": {
          "type": "text"
        },
        "project": {
          "type": "keyword"
        },
        "version": {
          "type": "keyword"
        }
      }
    }
  }
}

Edge ngrams itself requires large disk space and using it with completion suggester (with the above settings) took 564.1 MB which was originally 4.7 MB. This might be because I used edge_ngrams in filter. I am not sure about this.
We also have to use contexts with the completion suggester, according to the docs

Adding context mappings increases the index size for completion field. The completion index is entirely heap resident, you can monitor the completion field index size using Indices Stats.

We have to use atleast 2 contexts (for filtering based on project and version), but when multiple contexts are used, those are OR-ed and not AND-ed. For that, there is a little hacky workaround which involves making a pseudo-context (#26407 (comment)). This adds to the size of the index.
As per this comment - #504 (comment), we might not be able to support completion suggester as it keeps everything in-memory.

davidfischer · 2019-05-23T00:01:19Z

Firstly, that short demo looks very cool.

This design doc jumps right into specific implementations of how this will work in Elastic Search and while that's pretty important I think we are missing a few higher level things. For example, there's two parts of the in-doc search UI -- the normal search results page and a new search as you type instant-answer overlay. Are you planning to tackle any improvements to the normal Sphinx search page or exclusively tackling the overlay? Also, how customizable do we want to make this?

The normal Sphinx search page (eg. this page for the RTD docs). This is the page that Sphinx provides out of the box with its built-in implementation. It's pretty impressive for what can be done without a server backend but there is a lot of room for improvement. Read the Docs overrides the search API to hit its own backend and this is where @safwanrahman's improvements were made. There's lots of additional things this could do such as:
- Searching subprojects and maybe eventually related projects. To give an example, I'm thinking of finding results for the requests-toolbelt while searching the requests docs.
- Should any UI improvements be done to the search results page? This might be hard to do in a theme-agnostic way.
- Currently, our custom search doesn't integrate with MkDocs at all.
- Should we expose any facets in this search implementation (I'm thinking of how github issue search exposes facets like is:closed)?
- Projects should probably have a way to disable any modifications we make to the search page or even just have a way to just use the Sphinx defaults. There's currently a way to do this but it isn't particularly well-documented or explained (see Add a flag to disable docsearch #4570).
A search as you type JavaScript overlay. This currently doesn't exist at all unless document authors have added third-party tools like how the requests docs have added Algolia. There's a lot to think about here:
- How will the front-end work in a theme agnostic way?
- Will project maintainers be able to opt-out of this or customize it in any way (either styles or something different)? For example, if the requests docs want to continue to use Algolia, they probably don't need our implementation.
- How will we roll this out to users? For example, we probably want to turn it on for docs.readthedocs.io before anyone else. Maybe we make it opt-in for a beta period.
- I would like to start thinking about some example searches we could think of on specific projects and what the results should be. That way we can ensure that our search as you type implementation is returning desired results.

I went through a lot of things here and we don't need to do all of these things for GSoC! There probably isn't enough time. I probably missed some things too! However, I want to get thinking about how this will fit into the codeline, how we'll roll it out, and how project maintainers will be able to customize things. I don't want to focus exclusively on implementation details.

dojutsu-user · 2019-05-23T09:36:56Z

@davidfischer

Firstly, that short demo looks very cool.

Thank you.

Are you planning to tackle any improvements to the normal Sphinx search page or exclusively tackling the overlay?

Currently, I don't have any thoughts on improving the Sphinx search page, perhaps we may have to make changes to it if the suggested results and actual results don't match.
how customizable do we want to make this?

According to your comment above, we should atleast give the maintainers an option to opt-in/out to our search improvements. Also, I think maintainers should be able to change some UI parts like the colors. They might want to do this if the theme colors doesn't match with the search suggesstions colors.
Should any UI improvements be done to the search results page? This might be hard to do in a theme-agnostic way.

Does search results page need any improvements? Do you have anything in your mind?
Currently, our custom search doesn't integrate with MkDocs at all.

I am aware of this and I believe that we won't be able to support the search as you type feature for documentations built with MkDocs as of now. We can, once we have MkDocs integration with search.
Should we expose any facets in this search implementation (I'm thinking of how github issue search exposes facets like is:closed)?

How we will be using facets for our searches?
Projects should probably have a way to disable any modifications we make to the search page or even just have a way to just use the Sphinx defaults.

I agree with you here. We have to find a way to support this. I will research on this more. Do you anything in your mind?
How will the front-end work in a theme agnostic way?

I think, this is one of the biggest challenge for us. I faced this issue when making the short demo of this feature. I will research more on this.
Will project maintainers be able to opt-out of this or customize it in any way (either styles or something different)?

Yes, I think some customisations must be there for project maintainers as mentioned in Point 2 above.
How will we roll this out to users? For example, we probably want to turn it on for docs.readthedocs.io before anyone else. Maybe we make it opt-in for a beta period.

We can have something like Add a flag to disable docsearch #4570 for this.
I would like to start thinking about some example searches we could think of on specific projects and what the results should be. That way we can ensure that our search as you type implementation is returning desired results.

I did compared the suggested results (with the short demo) with the algolia's result. The results were pretty good. I wouldn't say that all the results were same but most of them were.

Linking some PRs on Algolia integration with the docs:

Thanks @davidfischer for the points. Those were some important ones which i didn't consider in the first place. I am updating the Design Docs to include those.

how this will fit into the codeline, how we'll roll it out, and how project maintainers will be able to customize things.

I don't have answers to these questions yet. I will do some research on these.

dojutsu-user · 2019-05-24T09:07:21Z

@davidfischer
I found this way to enable project maintainers to opt-in/out of our search-as-you-type feature.

For opt-in/out, we basically need a javascript bool type variable and from that we can wrap our code in if block to control it.
To enable search-as-you-type, project maintainers set it in their conf.py like

# conf.py
...
RTD_AUTOCOMPLETE = True
...

Then we can make changes in our readthedocs-sphinx-ext to check for this variable and if this is true, set a javasciript variable in the template to true, like

var RTD_AUTCOMPLETE = true;

This can be done in this way - https://github.com/thewtex/sphinx-contrib/blob/master/googleanalytics/sphinxcontrib/googleanalytics.py

and we got our variable which is globally available and we can use it to control this feature.
Thus, project maintainers have to do very little work to opt-in/out of this.

And for the styles, they can have a custom css file embed in their docs (which can be configured from conf.py) like this (tested with short demo) - https://github.com/rtfd/readthedocs.org/blob/f12f5fd4adae415959caf56323ed0eeb77103ae7/docs/conf.py#L107-L108

Also for rolling out this feature, after testing this with our docs, can we ask some other project maintainers who are currently using RTD to enable this in their project and provide some feedback on this?

davidfischer · 2019-05-24T18:50:42Z

Does search results page need any improvements? Do you have anything in your mind?

I think if we add the ability to disable our search (falling back to Sphinx's built-in search) or to search subprojects, we should probably add some messaging like "search provided by Read the Docs". We could even have a link to click to get the built-in search results. We also need a way in the UI to show if a search result came from a subproject or something like that.

How we will be using facets for our searches?

We don't currently. However, I could see some useful ones like excluding subprojects, looking for certain terms in titles only vs anywhere in the body, or something like that.

I agree with you here. We have to find a way to support this. I will research on this more. Do you anything in your mind?

One way that would involve a significant refactor would be to expose all the functionality we do on in-doc pages via a JavaScript global variable READTHEDOCS. This variable could have methods on it that handle things like in-doc search (eg. READTHEDOCS.handle_indoc_search). The defaults for these methods are in some file we include on the generated docs pages. The user could override these methods to change functionality.

This could be used for:

disabling overriding the built-in Sphinx search
disabling or modifying how the autocomplete/search overlay work

Thoughts?

dojutsu-user · 2019-05-24T20:52:49Z

@davidfischer

I think if we add the ability to disable our search (falling back to Sphinx's built-in search) or to search subprojects, we should probably add some messaging like "search provided by Read the Docs". We could even have a link to click to get the built-in search results. We also need a way in the UI to show if a search result came from a subproject or something like that.

I am +1 on the message. But, how will showing the built in results benefit?
Also, by UI, did you mean UI around search suggestions or actual results page? On the actual results page, we did have a UI to tell the users that a result is from a subproject. And we need to show something on the search suggestions for the results from subprojects.

We don't currently. However, I could see some useful ones like excluding subprojects, looking for certain terms in titles only vs anywhere in the body, or something like that.

It is great idea. We should consider this. This will increase the complexity though.

One way that would involve a significant refactor would be to expose all the functionality we do on in-doc pages via a JavaScript global variable READTHEDOCS.

We can do this. I don't know if it's going to increase the complexity or not. Also, I would like to know your thoughts on my comment above.

One thing that we can also provide is the search analytics to the project maintainers (who are using our search)? I don't know if it is required or not. But it will be a great feature. They can just go to the admin dashboard and see the analytics.

I think we are tracking many things here and we should prioritize them before starting the work. Like, search analytics, if agreed upon, might me the last on our to-do list.

humitos · 2019-05-29T14:30:34Z

To enable search-as-you-type, project maintainers set it in their conf.py like

I think we don't use the conf.py file to change behavior on Read the Docs. In case we need to add a config for this, it should probably go in our YAML file.

dojutsu-user · 2019-05-29T14:36:36Z

I think we don't use the conf.py file to change behavior on Read the Docs. In case we need to add a config for this, it should probably go in our YAML file.

Thanks. This is a valid point. YAML file makes more sense.

PS. Closed the PR by mistake. Opened it again though.

ericholscher · 2019-05-29T20:21:04Z

I think we are tracking many things here and we should prioritize them before starting the work. Like, search analytics, if agreed upon, might me the last on our to-do list.

Agreed. I want to focus this a bit more than the current discussion. My first question is:

Can we implement the front-end "search as you type" with the existing search API? Do we need to re-build the backend first, or can we simply update the existing search to make it better? The demo you showed looked like a great improvement to me.

Once we answer that question, we can break the work down into pieces. My goal is to focus mostly on the front-end here, at least to start, since that's the area where we historically have needed more work. Once we get a really good UX around search, then we can work on improving indexing and other backend related things to make the front-end function better.

ericholscher · 2019-05-29T22:33:05Z

(Sorry, didn't mean to close it, wrong button! :)

dojutsu-user · 2019-05-30T10:29:39Z

@ericholscher
I tried with the existing search backend. It is not effective as that of first demo but I think we can start working on frontend with it. 🎉

One problem I faced is how to include CSS file for this. I was trying to include it in the same way as we include readthedocs-doc-embed.js for all our docs but the grep wasn't able to tell me that how we are doing it.

We should define a bit more what we're going to do on the front-end in terms of technology.

What do you mean 'technology' here?

cc: @davidfischer

humitos · 2019-05-30T11:52:31Z

Wow, the demo is really good.

Just a comment here, since I suppose that the UI has not been discussed yet, but to consider in that case. If we are going to show these kind of results as you type, I'd suggest to open a modal once the user clicks on the "Search" box input, as Slack does.

It takes control of the screen over the page that you are seeing it suggest you things, but if you don't find what you are looking for from the suggested ones, you hit enter and you get more specific results in the same modal

So, this way we can differentiate our own search from the default search provided by Sphinx and both can live together.

dojutsu-user · 2019-05-30T12:04:20Z

@humitos
I think that's really a nice idea. Although the UI will be more complex though.
We have two options now. What do you think? @ericholscher @davidfischer

dojutsu-user · 2019-05-31T18:12:41Z

@humitos
It may look like this.

This is just a demo and it will much better than this. Like backdrop and other things.
Thoughts?

dojutsu-user · 2019-05-31T18:23:09Z

@ericholscher
@davidfischer

I am thinking to start the work. What are your thoughts?

davidfischer · 2019-05-31T22:39:37Z

Once we answer that question, we can break the work down into pieces. My goal is to focus mostly on the front-end here, at least to start, since that's the area where we historically have needed more work. Once we get a really good UX around search, then we can work on improving indexing and other backend related things to make the front-end function better.

I'm good focusing on in-doc modal search. That means that we are mostly ignoring the built-in Sphinx search page for the purpose of GSoC. I'm fine 👍 with that.

Basically, that limits the scope to:

JavaScript in-doc search as you type
Any API changes needed to enable in-doc search as you type

I think that's probably a good idea to scope this down for the purpose of GSoC. We have a clear, somewhat self-contained project. There's already enough complexity here.

What do you mean 'technology' here?

I think he means like any libraries we will need to do this. For example, we try to support all the way down to IE11 with our docs theme and Read the Docs generally and I think we want to support that here. For example, that means you can't use the JavaScript fetch API to make Ajax requests. Having a library might make this easier. Sphinx docs already rely on jquery which has an Ajax API but will that be sufficient for everything I'm not 100% sure.

One problem I faced is how to include CSS file for this.

Currently, the CSS included for in-doc search is here: https://github.com/rtfd/readthedocs.org/blob/master/media/css/readthedocs-doc-embed.css

Currently, we inject our CSS and JS into the Sphinx build process here: https://github.com/rtfd/readthedocs-sphinx-ext/blob/50e43731311006cb51a530bc34df89386de8bcf6/readthedocs_ext/readthedocs.py#L53

You have a few options.

You could add your CSS into readthedocs-doc-embed.css and your JS into readthedocs-doc-embed.js and it will get included.
We could package the in-doc search into it's own self-contained CSS and JS files and include them in a similar manner to readthedocs-doc-embed.*.
It might be possible to package up the in-doc CSS/JS as a sphinx extension. This might be nice because then it's easy to enable it on a per-project basis. When we are ready to roll it out to a wider audience, we can make a decision to just turn it on for everybody (put it in here) or we could enable it as an opt-in feature like the 404 extension.

With respect to @humitos comments around a modal window versus an overlay, I don't have a particularly strong opinion. I do hope that we can change the UI relatively easily and I don't want to have to make the decision between modal and overlay quite yet. It might be nice to roll it out to a subset of users and begin to get some real user feedback before committing to a specific UI/UX.

davidfischer · 2019-06-01T00:00:34Z

With respect to concrete feedback on the document in this PR, I think there's a few bits that need to be moved around and a few bits that need to be added.

A few things that we should probably add:

Intro - You have a small intro and that's good but I think there could be a bit more. I think it's worth taking one of the animated gifs in the comments of this PR and putting it in there. A picture is worth a 1000 words sometimes. It's ok if the final implementation won't exactly match the image. We can always update it. I think it's also worth mentioning that this was a GSoC project. This section only needs to be a few sentences, I think.
Goals and non-goals - This should probably be a short section near the top. I think that we should highlight all the goals required to make this a successful project. The goals should be really high level but it should include things like:
- Support a search-as-you-type/autocomplete interface backed by the existing Read the Docs search backend.
- Support across all (or virtually all) Sphinx themes. We should also mention that we are targeting Sphinx only for the initial release since we don't index MkDocs.
- Support for the JS user experience down to IE11 or graceful degradation where we can't support it
- Maintainers should have a way to opt-in/opt-out of this feature. Changing styles would be nice but isn't as important.
Milestones - This section should be near the bottom. You alluded to this in the "Shipping" section that you have but I think we should be more explicit. Most of these milestones will be dependent on the milestones before them so the order is important (although things can move around a bit). While you don't have to commit to dates, it might also help you coordinate the milestones. We do have specific evaluation dates for GSoC although they don't have to match exactly. I can envision some milestones like:
- A local implementation (this seems like what you have in the gifs)
- In-doc search on a test project hosted on Read the Docs using the RTD Search API
- In-doc search on docs.readthedocs.io
- Friendly user trial where users can add this on their own docs
- Additional UX testing on the top-10 Sphinx themes
- Finalize the UI
Open questions - It is ok that we don't have all the answers up-front. For example, should we rely on jquery, another library, or go 100% vanilla JS? I don't think we quite know the answer to this and we might not until we start on the project. However, we can discuss the trade-offs of these. For example, having as few dependencies as possible is good especially when we want to include this on lots of docs. However, if we have to reimplement the wheel of Ajax, it's probably worth it to use something. This should probably be near the bottom. I think there's a few other open questions like:
- Is our existing Search API is sufficient?
- Will we search subprojects?

Here's a short guide on writing design documents if that's helpful.

Headings

Looking at just the headings of the doc in this PR, I think we could have something like this:

Intro
Goals and non-goals
Existing search implementation
Proposed architecture for in-doc search
- Front-end - this could have subsections
- Back-end - you could highlight possibilities like the N-Gram Tokenizer here if you want but I'd keep it a little shorter than what's there now
Milestones
Open Questions
References (maybe)

dojutsu-user · 2019-06-01T15:23:28Z

@davidfischer
I really liked the idea of having separate sphinx extension for it. I think we should do it that way.

Is our existing Search API is sufficient?

I don't think so, I tried that with the demo #2 and the results were not very good. We get the suggestion after typing the whole word. Like.. It will not give any result for elasticse, but would give results for elasticsearch.
We have to make changes in our elasticsearch index to get the results similar to demo #1. That' why I have added backend improvements at the last of milestones. But I do think that it should be before 'friendly user trial', otherwise we won't be having much good reviews about it. What are your opinions on this?

Also, Thanks for a great advice on the design docs. I have restructured the whole docs according to it (I have copied few lines from it).

dojutsu-user · 2019-06-02T09:10:57Z

I was trying via sphinx-extension method (mentioned by @davidfischer in above comment). The main problem is that we can't use rtd-data.js and our custom js file (search_ext.js) gets included just below the readthedocs-embed.js

The problem it poses is that search_ext.js will be missing some parameters - project and version (and language) which are required for making query. This is because that readthedocs-data.js gets included a little below the search_ext.js and we won't be having object READTHEDOCS_DATA in search_ext.js. I think to solve this problem we might have to get a little hacky (like taking the current url, parsing the url to get project, version and language and then sending them via the query params to search_ext.js).

I think we might not want to that much hack around it (or we want?).
I don't know that the hacky way that I have mentioned will work or not. If we go in this direction, we need to find a way for this.

Or else we can use options 1 or 2 from the comment above.

Edit: We can use DOMContentLoaded event (I totallly forget about this) and can have access to READTHEDOCS_DATA global object. 🎉
I would like to go in this direction. What are your thoughts.. @ericholscher @davidfischer
If yes... Will there be a new repo (what should be the name) for that or will we be including in readthedocs-sphinx-ext?

davidfischer · 2019-06-03T20:27:55Z

We can use DOMContentLoaded event (I totallly forget about this) and can have access to READTHEDOCS_DATA global object.

This is what you want! Support is good for all the browsers we care about.

Will there be a new repo

I'd start with a new repo and we can change that later if necessary.

davidfischer

I think the document is very close to being ready to merge. I don't consider it a final document and we can always update it as we make more design decisions.

One thing I would add is a note at the top (like this)mentioning that this is not yet a live feature and it is in active development. That way if a user stumbles across it they know they can't use it yet.

docs/design/in-doc-search-ui.rst

davidfischer · 2019-06-03T20:58:38Z

docs/design/in-doc-search-ui.rst

+We plan to select the search bar, which is present in every documentation,
+using the `querySelector()`_ method of JavaScript.
+Then add an event listener to it to listen for the changes and
+fire a search query to our backend as soon as there is any change.


What I'm about to say doesn't need to be added to this document but may be helpful to you more generally. Sometimes events like this are "debounced". In this instance, that means that the search query will only be fired every X number of milliseconds and only when the user stops typing. While you could use vanilla JS for this, there is a handy version in underscore already and underscore is already included by Sphinx (under the variable $u instead of the usual).

Thanks @davidfischer.
But what will be the benefit of debouncing the event?
will it not make our suggestions loading time look slower?

Co-Authored-By: David Fischer <[email protected]>

dojutsu-user · 2019-06-04T04:36:49Z

@davidfischer
I have applied and pushed the changes you suggested.
I will make a new repo for the sphinx extension and start working on it from today.

humitos · 2019-06-06T10:45:57Z

docs/design/in-doc-search-ui.rst

+
+.. figure:: ../_static/images/design-docs/in-doc-search-ui/in-doc-search-ui-demo.gif
+    :align: center
+    :target: ../_static/images/design-docs/in-doc-search-ui/in-doc-search-ui-demo.gif


I think if you put just in-doc-search-ui-demo.gif in both places here, Sphinx will just do the magic. Not sure, though

I just tried it.
if we do this... then it is looking for the image in the same folder i.e., design/in-doc-search-ui.gif.

humitos · 2019-06-06T10:48:23Z

docs/design/in-doc-search-ui.rst

+* Support across all (or virtually all) Sphinx themes.
+* Support for the JavaScript user experience down to IE11 or graceful degradation where we can't support it.
+* Project maintainers should have a way to opt-in/opt-out of this feature.
+* (Optional) Project maintainers should have the flexibility to change some of the styles using custom CSS and JS files.


I'm not sure if this is something where we need to do something. If we start the project by adding a specific HTML class to all the new elements that we add in the UI, it should be possible to change the style by the user without us implementing something in particular. Maybe I'm missing something, though.

Yes I agree.
I think I should remove this point??
I am guessing that user won't be overwriting JS functions ... !!

We need to document how users can override it, so I'm 👍 on keeping it.

humitos · 2019-06-06T10:52:11Z

docs/design/in-doc-search-ui.rst

+  and it will get included.
+* Package the in-doc search into it's own self-contained CSS and JS files
+  and include them in a similar manner to `readthedocs-doc-embed.*`.
+* It might be possible to package up the in-doc CSS/JS as a sphinx extension.


If this would be a voting, I'd vote for this one :)

I would also vote for that one. We are currently in this direction only.

My only concern with the extension is that it doesn't work locally. You can build your docs locally, but even if we hit the prod API, it will only return results from prod docs. This feels like a weird thing to do with an extension when it's inherently linked to our RTD production builds.

@ericholscher
We have a global object READTHEDOCS_DATA and it contains api_host. In the extension, we are taking the API host from there.
So in local, it is - http://127.0.0.1:8000
And we can use the extension locally (given that the Elasticsearch is correctly set up).

Sure, but that requires a running instance of RTD -- I mean it isn't useful outside of RTD. If you already have a local RTD instance, then bundling it with RTD would have the same outcome :)

Yes... that is the limitation. 😕

Split of code is a win to me. Easier to contribute, easier to focus when debugging, easier to keep up to date, etc. The downside is deploying a bugfix immediately, where core & extension need to be deployed together.

I understand that could feel weird that the results come from production while you are seeing a different set of docs. The extension could disable itself if detecting that it's running outside RTD if we can avoid that weird UX. Not a solution, though.

I think it will increase complexity in development environment..!!

humitos · 2019-06-06T10:55:40Z

docs/design/in-doc-search-ui.rst

+++++++
+
+We have a few options to support `search as you type` feature,
+but we need to decide that which option would be best for our use-case.


Using one backend or the other does involve a lot of changes or it's just changing a setting and a class or similar? I want to know if this is something that we can test and see how it perform and decide after that or if testing this way is complicated since they are two different implementations.

If we want to have results like comment #1 then we have to do some changes in the backend and a new API endpoint. The changes may increase the size of elasticsearch index to x3 or x5. (In the demo, I did changes with my elasticsearch index locally to produce the results.)
Further changes can be.. like if we want to support section linking. Currently we just open the result page and user have to scroll down to the required section. For that... there will be considerable big changes. many parts of the elasticsearch will change -- how we index data/search data.
I don't think that we can test this without any changes.

Without any changes is okay too...the test-builds - https://readthedocs.org/projects/test-builds-dojutsu-user/versions/ are without any changes.

ericholscher

I believe this document is mostly ready. I'd be 👍 on merging it soon, and then we can update it with PR's as we go and make implementation decisions, but I think it's at a good place for now.

ericholscher · 2019-06-06T14:26:34Z

docs/design/in-doc-search-ui.rst

++++++++++
+
+* For the initial release, we are targeting only Sphinx documentations
+  as we don't index MkDocs documentations to our Elasticsearch index.


Of note, supporting all Sphinx themes will also make it much easier for us to support mkdocs. A large part of the issue with around the UI/UX and theme integration, so we should hopefully be able to work around that now :)

ericholscher · 2019-06-06T14:28:47Z

docs/design/in-doc-search-ui.rst

++++++++++++++
+
+* Should we rely on jQuery, any third party library or pure vanilla JavaScript?
+* Are the subprojects to be searched?


Yes, but the backend API should already be doing this.

Yes. It is a question when we are changing/updating the backend.

ericholscher · 2019-06-12T19:02:41Z

Merging this, we can make any updates to this doc in an additional PR.

study edge-ngrams

49016d7

dojutsu-user added the PR: work in progress Pull request is not ready for full review label May 17, 2019

dojutsu-user requested review from ericholscher, davidfischer and a team May 17, 2019 12:37

dojutsu-user added 8 commits May 17, 2019 18:59

lint fix

229d371

give analyzers definitions

072daa3

fix queries

8f41b3a

fuzziness

aec1e97

update cons of edge-ngrams

e68e5fa

update edge-ngram size

46c3218

Update queries

0c6cbdf

add details for completion suggester

0239240

little infor about frontend

34fc557

dojutsu-user removed the PR: work in progress Pull request is not ready for full review label May 20, 2019

dojutsu-user added 2 commits May 23, 2019 15:07

Merge branch 'master' into design-doc-search-ui

830a6bf

some challenges

d28c748

dojutsu-user closed this May 29, 2019

dojutsu-user reopened this May 29, 2019

ericholscher reopened this May 29, 2019

dojutsu-user added 2 commits June 1, 2019 01:13

Merge branch 'master' into design-doc-search-ui

aa1d828

improvements to the docs

097accb

dojutsu-user added 2 commits June 1, 2019 20:16

restructured doc

ac95f50

fix gif image

726dc45

davidfischer reviewed Jun 3, 2019

View reviewed changes

dojutsu-user and others added 5 commits June 4, 2019 09:56

add warning

4718cc0

Update docs/design/in-doc-search-ui.rst

0fa000a

Co-Authored-By: David Fischer <[email protected]>

Update docs/design/in-doc-search-ui.rst

d532119

Co-Authored-By: David Fischer <[email protected]>

Update docs/design/in-doc-search-ui.rst

608c2b8

Co-Authored-By: David Fischer <[email protected]>

Update docs/design/in-doc-search-ui.rst

efda71f

Co-Authored-By: David Fischer <[email protected]>

humitos reviewed Jun 6, 2019

View reviewed changes

ericholscher reviewed Jun 6, 2019

View reviewed changes

ericholscher approved these changes Jun 12, 2019

View reviewed changes

ericholscher merged commit 405f41a into readthedocs:master Jun 12, 2019

dojutsu-user deleted the design-doc-search-ui branch June 12, 2019 19:03

[Design Doc] In Doc Search UI #5707

[Design Doc] In Doc Search UI #5707

Conversation

dojutsu-user commented May 17, 2019 • edited Loading

dojutsu-user commented May 19, 2019 • edited Loading

safwanrahman commented May 19, 2019

dojutsu-user commented May 20, 2019

davidfischer commented May 23, 2019

dojutsu-user commented May 23, 2019 • edited Loading

dojutsu-user commented May 24, 2019

davidfischer commented May 24, 2019

dojutsu-user commented May 24, 2019

humitos commented May 29, 2019

dojutsu-user commented May 29, 2019 • edited Loading

ericholscher commented May 29, 2019

ericholscher commented May 29, 2019

dojutsu-user commented May 30, 2019

humitos commented May 30, 2019

dojutsu-user commented May 30, 2019

dojutsu-user commented May 31, 2019 • edited Loading

dojutsu-user commented May 31, 2019

davidfischer commented May 31, 2019 • edited Loading

davidfischer commented Jun 1, 2019

Headings

dojutsu-user commented Jun 1, 2019 • edited Loading

dojutsu-user commented Jun 2, 2019 • edited Loading

davidfischer commented Jun 3, 2019

davidfischer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dojutsu-user commented Jun 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dojutsu-user Jun 6, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dojutsu-user Jun 6, 2019 • edited Loading

Choose a reason for hiding this comment

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher commented Jun 12, 2019

dojutsu-user commented May 17, 2019 •

edited

Loading

dojutsu-user commented May 19, 2019 •

edited

Loading

dojutsu-user commented May 23, 2019 •

edited

Loading

dojutsu-user commented May 29, 2019 •

edited

Loading

dojutsu-user commented May 31, 2019 •

edited

Loading

davidfischer commented May 31, 2019 •

edited

Loading

dojutsu-user commented Jun 1, 2019 •

edited

Loading

dojutsu-user commented Jun 2, 2019 •

edited

Loading

dojutsu-user commented Jun 4, 2019 •

edited

Loading

dojutsu-user Jun 6, 2019 •

edited

Loading

dojutsu-user Jun 6, 2019 •

edited

Loading