Add segments of words with punctuation to index

[bglw]

Discussion in #215

Perhaps Pagefind should automatically index a word like color-accent as [color-accent, color, accent]

[bglw]

Addition from #267 — we should ideally apply this to a range of punctuation characters that make sense, so pagefind.toml indexes as [pagefind.toml, pagefind, toml].

(NB: We want to avoid indexing don't as [don't, don, t])

[lorenzolewis]

Firstly, amazing project! Found it through Astro's Starlight and love it!

Do you have any thoughts around similar situations with camelCase and snake_case variants? I imagine that snake_case could use a similar strategy but camelCase might have some added complexity to it to make sure you're not mistakenly over-splitting words (thinking proper names like "McDonalds", etc.).

[bglw]

Hey @lorenzolewis 👋

Ooh, I haven't yet given it thought but it seems fine!

I don't think I'm too worried about over-splitting. In that example it isn't a big deal if you can search for "donald" and get a result for "McDonalds" — perhaps having a minimum length on splitting words would help (3+ characters?).

I think the main thing I would like to do for segmented words is de-rank the partial matches. Currently results are ranked by how close your search word is, so if you're searching for con then cone will rank higher than conifer as it's closer in length. But with naive word splitting, myConFig would rank high, because con would be indexed as well as the other words, and it would look like an exact match.

I think this could tap into the new weighting feature, though, and make these partial matches weaker than they would otherwise be. In which case, over-indexing is a negligible problem.

I'll keep you posted — I'll actually look at this next 👀

[lorenzolewis]

RE: 3+ characters:

I know some Apple naming conventions use something like this: WKWebView (short for WebKitWebView. Would that be split into something like this: [WK, Web, View, WKWebView]? I think this is fairly common from languages that didn't respect namespaces very well, so might be a scenario to keep in mind if you're only wanting to treat 3+ characters as the "cutoff".

But this also opens up another question: Would this be picked up since it's WKWebView instead of WkWebView (notice the lower-case k)?

[bglw]

I'll likely use directly, or base this handling on, https://github.com/withoutboats/heck as it has pretty robust splitting:

t!(test8: "this-contains_ ALLKinds OfWord_Boundaries" => "This Contains All Kinds Of Word Boundaries")

Looking at their implementation, WKWebView would be Wk Web View 🎉
(Helpfully also in this example, it doesn't particularly matter whether the first word is indexed uniquely, as searching for wk will match both wk and wkwebview)

RE: RE: 3+ characters:
2+ is probably a safer bet anyway. OpenGL is a good example, where it would be nice to index Open GL.

[bglw]

Hey all ! 👋

Good news — this has landed in Pagefind v1.0.0! ✨

See the full release notes here: https://github.com/CloudCannon/pagefind/releases/tag/v1.0.0 💙

Ping me here if you have any questions about the implementation!