Add "search as you type" functionality

[robhudson]

Relevant Elasticsearch docs:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-suggesters-completion.html

For project search we can autocomplete project name or slug.

For page search within a project we can autocomplete page titles and possibly headers? This may or may not be as useful as project autocompletion.

[ericholscher]

On the section index:

            'suggest': {
                'foo, bar'
                'payload': {
                    url
                    pk
                }

            }

[ronakkhunt]

I would like to give it a try.

[ericholscher]

Great -- we're using elastic search as the backend, which can be setup locally as documented here: http://docs.readthedocs.io/en/latest/rtfd.html#elastic-search-setup

The main search views are here: https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/search/views.py

I'm not 100% sure how to implement this on the ES side, so that would take some research.

[safwanrahman]

Really excited to see its coming next to my work.
@ericholscher Is it possible to provide any UI or something like that so I can try to implement them? Can anyone from the team help me do the frontend things? Can you design the API for the suggestions?

[safwanrahman]

@robhudson Thanks a lot for filling the issue. I will work on this in next couple of weeks.
Do you have any suggestion you would like to provide that will help me to implement this?

[ericholscher]

I'd really like to see us be able to index & autocomplete API objects as well. So that you could type Pro and it would autocomplete Project or similar. I don't know the best UI for it though.

This existing Sphinx extension has at least a basic take on this: https://github.com/jimfulton/sphinxautoindex/tree/master/findanything -- but could definitely use some UX improvements :)

[safwanrahman]

I have investigated about it and found that we need some UX decision about it.
From my thinking, we can implement one of the following

• Google style autocomplete suggestion for Document Search/Page Search. Also provide misspelled suggestion something like "Did you mean". It can be implemented by using Completion Suggester. As it keeps everything in RAM, for reducing the usage, we can only keep headers and titles for autocomplete query.

• Search as You type functionality can also be implemented in the Document Search. So while a person types anything into the search box, it will make the query and show the relavant result!

Example

We have search page where we can implement easily Search as you Type functionality easily. But we need to decide if we want that feature also in the sidebar.

Moreover, we need to learn about ngram thing of elasticsearch in order to implement a good suggestion and search functionality.

[humitos]

As it keeps everything in RAM, for reducing the usage, we can only keep headers and titles for autocomplete query.

This sounds to not scale in a good way and probably something that we can't support in the community site with thousands of projects.

Search as You type functionality can also be implemented in the Document Search

The example you showed here it's amazing to me and I'd like to follow that direction!

These are just opinions, though, I'm not really involved in the search world inside Read the Docs.

[dojutsu-user]

@ericholscher @davidfischer

I did some research on how algolia implemented its search on Requests docs...

When query is This part of the documentation, this is the response

POST data sent with each request -

{
	"requests": [
		{
			"indexName": "python-requests",
			"params": "query=This%20part%20of%20documentation&hitsPerPage=5"
		}
	]
}

GET data sent with each request -

x-algolia-agent: Algolia%20for%20vanilla%20JavaScript%20(lite)%203.30.0%3Bdocsearch.js%202.6.2
x-algolia-application-id: BH4xxxxxx
x-algolia-api-key: f177061e2354c50a97bfxxxxxxxxxxxx

Result (in json) -

Full response - here

{  
    "results":[  
        {  
            "hits":[  
                {  
                    "hierarchy":{  
                        "lvl2":null,
                        "lvl3":null,
                        "lvl0":"Installation of Requests",
                        "lvl1":null,
                        "lvl6":null,
                        "lvl4":null,
                        "lvl5":null
                    },
                    "url":"https://2.python-requests.org/en/master/user/install/#install",
                    "content":"This part of the documentation covers the installation of Requests.\nThe first step to using any software package is getting it properly installed.",
                    "anchor":"install",
                    "objectID":"3180085671",
                    "_snippetResult":{  
                        "content":{  
                            "value":"<span class=\"algolia-docsearch-suggestion--highlight\">This</span> <span class=\"algolia-docsearch-suggestion--highlight\">part</span> <span class=\"algolia-docsearch-suggestion--highlight\">of</span> the <span class=\"algolia-docsearch-suggestion--highlight\">documentation</span> covers the installation <span class=\"algolia-docsearch-suggestion--highlight\">of</span> Requests",
                            "matchLevel":"full"
                        }
                    },
                    "_highlightResult":{  
                        "hierarchy":{  
                            "lvl0":{  
                                "value":"Installation <span class=\"algolia-docsearch-suggestion--highlight\">of</span> Requests",
                                "matchLevel":"partial",
                                "fullyHighlighted":false,
                                "matchedWords":[
                                    "of"
                                ]
                            }
                        },
                        "content":{  
                            "value":"<span class=\"algolia-docsearch-suggestion--highlight\">This</span> <span class=\"algolia-docsearch-suggestion--highlight\">part</span> <span class=\"algolia-docsearch-suggestion--highlight\">of</span> the <span class=\"algolia-docsearch-suggestion--highlight\">documentation</span> covers the installation <span class=\"algolia-docsearch-suggestion--highlight\">of</span> Requests.\nThe first step to using any software package is getting it properly installed.",
                            "matchLevel":"full",
                            "fullyHighlighted":false,
                            "matchedWords":[  
                                "this",
                                "part",
                                "of",
                                "documentation"
                            ]
                        },
                        "hierarchy_camel":[  
                            {  
                                "lvl0":{  
                                    "value":"Installation <span class=\"algolia-docsearch-suggestion--highlight\">of</span> Requests",
                                    "matchLevel":"partial",
                                    "fullyHighlighted":false,
                                    "matchedWords":[  
                                        "of"
                                    ]
                                }
                            }
                        ]
                    }
                },
                {..},
                {..},
                {..},
                {..}
            ],
            "nbHits":7,
            "page":0,
            "nbPages":2,
            "hitsPerPage":5,
            "processingTimeMS":1,
            "exhaustiveNbHits":true,
            "query":"This part of documentation",
            "params":"query=This%20part%20of%20documentation&hitsPerPage=5",
            "index":"python-requests"
        }
    ]
}

Observations -

• Each result is divided into levels.
• levels are a way to categorise the content of the result, for example: lvl0 defines the main heading under which the result will appear. (https://github.com/algolia/docsearch-configs/blob/605ded4e961f232f8c5d45a6617b0c19121a7e16/configs/python-requests.json#L13-L20)
• Each result contains both its url and anchor.
• Result is already formatted with proper html tags and CSS classes.
• Everything else is more or less provided to make custom UI.

Conclusion -

For the UI part, I believe that sending the data pre-formatted with proper html tags is something we should do as it will reduce the load on client side. Also, I searched for a JS library which we can use but it was of no use. The UI shown in the above image is nice, but we want to have our own and better than this.

For the backend part, I believe that we should create a single API endpoint with parameters so that it can be used for all the projects.

Elasticsearch docs suggest to use Completion Suggester to implement search-as-you-type feature. But it comes with some disadvantage. As mentioned in their docs:

The suggester uses data structures that enable fast lookups, but are costly to build and are stored in-memory.

I am not sure that whether this will be a advantage/disadvantage for us.

And also for the Completion Suggester, we need to have some changes in our doc-types of the PageDocument.

Starting the work -

My suggestion is to start the backend work after some design decisions. And simultaneously start working on the frontend (after deciding on the UI). Frontend work can be started on Codepen.io.

Blog -

Also, I will be maintaining a blog for the GSoC period. Are there any suggestions for the name? I am thinking something like this - gsoc19-with-rtd.wordpress.com ??

[dojutsu-user]

Hi all
I am closing this issue as we have already worked on a Sphinx extension to provide "search as you type" functionality -- readthedocs-sphinx-search
More info on our blog: https://blog.readthedocs.com/improved-search-and-search-as-you-type/
Thanks for your patience.