Changing the search engine

[Hetarth02]

Issue #1457, Issue #1749 This addresses the problem of getting insufficient or non-ideal search results after the search.

Issue #1832: This issue also proposes the idea of using a different search engine instead of LunrJs to reduce initial lag which happens during the building of the indexes.

Moreover, lunrjs repo has been stale for some time.

I mostly suggest minisearchjs.

• It is designed for memory-constrained cases.
• Is inspired by Lunrjs, so it will be easier to shift to it by not having to re-implement the entire search logic.
• Supports suggestions if you want to improve the search UI in future.

16 June, 2023
For anyone wanting to test it out, here is the link to compare both results from both engines.

Compare Searches

[Hetarth02]

For the cold start times of lunrjs, currently we can have something of an opt-in pre-building of indexes and just load up the serialised index on client-side.

A simple benchmark I did using julia's docs yielded these results,

The load times, went down from ~2900ms to ~800ms.

The serialised index size is around 7.6mb

For minisearchjs,

Serialised Index size is around 6.4mb (Around 16% less than lunrjs)

And Instantiation time with prebuild index is around ~500ms.

The only downside for prebuilding index is that we require NodeJs as an environment.

[mortenpi]

Looks good, I think we definitely want this on our roadmap. It would be good to have side-by-side demos with the same index for e.g. the Julia manual, to see how the search results differ. And also maybe have it print out the timings and index sizes in the consoles.

[Hetarth02]

For anyone wanting to test it out, here is the link to compare both results from both engines.

Compare Searches

[mortenpi]

Alright, I did a few small tweaks to the compare page (mortenpi/documenter-updated-search@04c766c; deployed here) and tried out a bunch of cases.

My take is that the results are usually slightly better (better priority for some things), but mostly the same. Only in some rare cases it seems that minisearch promotes some strange results. Generally, though, it often seems to filter out some of the less useful ones, which is good (e.g. base64).

Not super important, but for multi threading, minisearch seems to find a lot of duplicates though. Not sure what's up with that.

All in all though, considering the other benefits minisearch would bring that we've discussed, I'd say yeah, let's go for it!

[mortenpi]

Supports suggestions if you want to improve the search UI in future.

How do these work exactly by the way? If they're easy to implement, then we could add this right away, when we switch.

[Hetarth02]

@mortenpi It mostly uses all the options as the original search does. Here is the explanation in the docs.

[mortenpi]

Would be cool if we could maybe auto-suggest e.g. function names that way. I think often people will start searching for something, but don't necessarily remember the exact name.

[fredrikekre]

Partial exakt matches doesn't work great right now it looks like. For example, searching for cholesky doesn't give any good resault until cholesk.

[Hetarth02]

Partial exakt matches doesn't work great right now it looks like. For example, searching for cholesky doesn't give any good resault until cholesk.

That might have to do with the fuzzy search distance. Currently it only checks for one edit distance for the word. We can increase or decrease it as we please.

[fredrikekre]

Yea, that would be good. I think it is pretty common that you type correct thing, and it "feels nice" if you get the correct results before typing the whole thing.

[Hetarth02]

Would be cool if we could maybe auto-suggest e.g. function names that way. I think often people will start searching for something, but don't necessarily remember the exact name.

Yeah, we can do that. I will see what I can do UI wise to incorporate it.