Search: Highlight exact matches improvements

[humitos]

In #4292 exact match search was implemented, but I'm not sure it is working as expected or at least it could be improved in some ways.

Following this link, we search for "404 not found"

• I expect to get the whole phrase highlighted
• only get results where the whole text appears in that specific order
• there are results where not is highlighted alone, which makes me confuse

In this another example, I search for "results not found here"

• do not fallback on sphinx search when using exact match

Another search for "custom domains"

• clicking on the first result goes to https://docs.readthedocs.io/en/stable/custom_domains.html?highlight=%22custom%20domains%22 which contains ?highlight="custom domains" which makes the JS code to get confused and do not highlight anything

[ericholscher]

This is using the ES Simple String Query -- but it has almost no documentation on how it works. Quoting the search does result in a smaller set of results than not quoting, but I don't fully understand what is happening.

[ericholscher]

It seems that search is properly searching for only "custom domains" for example, but once it hits the highlighter, we're getting highlighted results that only contain part of the queried term.

[ericholscher]

You can see the results better on site search: https://readthedocs.org/search/?q=%22custom+domains%22&type=file&version=latest&project=docs

[ericholscher]

Looks like a known issue: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-highlighting.html

Highlighters don’t reflect the boolean logic of a query when extracting terms to highlight. Thus, for some complex boolean queries (e.g nested boolean queries, queries using minimum_should_match etc.), parts of documents may be highlighted that don’t correspond to query matches.

[dojutsu-user]

@humitos
We have removed the highlight url param, is this issue still valid?

[stsewd]

The highlight is back, but not sure if we can do much about this.

[stsewd]

From the docs https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-request-highlighting.html

fragmenter

    Specifies how text should be broken up in highlight snippets: simple or span. Only valid for the plain highlighter. Defaults to span.

    simple
        Breaks up text into same-sized fragments. 
    span
        Breaks up text into same-sized fragments, but tries to avoid breaking up text between highlighted terms. This is helpful when you’re querying for phrases. Default. 

Didn't test how it looks like, but maybe I'll give it a try later.

[stsewd]

I expect to get the whole phrase highlighted

I'll see if I can play around operators and see if there is any improvement, but this again falls to try to parse the query.

do not fallback on sphinx search when using exact match

We always need to fallback to the original search when there aren't results, this way we guarantee we don't miss anything. The search extension doesn't fallback.

which contains ?highlight="custom domains" which makes the JS code to get confused and do not highlight anything

That may be tricky to solve, we could analyze the original query and return the intended text without any operators (but this could result in dropping some original content from the query). And we would need to implement a mini-parser for the single simple query (probably not that hard).

[stsewd]

I'll see if I can play around operators and see if there is any improvement, but this again falls to try to parse the query.

Nope, using just the and operator didn't solve this.

[humitos]

This doesn't seem to be a problem with the highlight only. If you take a look at the example that I shared before, https://docs.readthedocs.io/en/stable/search.html?q=%22results+not+found+here%22&check_keywords=yes&area=default you will notice that "results not found here" does not appear in the resulting page.

[stsewd]

@humitos what you are looking there is the default Sphinx search, our search returns zero results for that query.

[humitos]

Closing in favor of readthedocs/addons#36