Experimental feature: Score details

[dureuill]

Feature name	How to enable	Description	Missing for stabilization	Expected stabilization date/version
score details	curl \ -X PATCH 'http://localhost:7700/experimental-features/' \ -H 'Content-Type: application/json' \ --data-binary '{ "scoreDetails": true }'	Enables computing detailed scores	Confidence on the computed details names	v1.7

Meilisearch v1.3 (released July 31) introduces score computations:

• Aggregate, normalized ranking score per document, available via the showRankingScore search query parameter
• Detailed scores per ranking rule for each document, available via the showRankingScoreDetails search query parameter. The detailed scores are an experimental feature in Meilisearch v1.3

Experimental feature abstract

Passing "showRankingScoreDetails" : true to a query from the /indexes/{:indexUid}/search or /multi-search routes injects a new _rankingScoreDetails field in each returned document.

This field has for value a JSON object representing the details of the score calculation for each of the ranking rules involved in the ranking of the document.

The various details of this object is given in the related specification.

How to use the feature?

1. Call the PATCH /experimental-features passing "scoreDetails": true.
2. Make search requests with "showRankingScoreDetails": true in the search query parameters.
3. The documents in the response will contain the _rankingScoreDetails field with the value described above.

What is an experimental feature

By enabling this feature via the /experimental-features route, you opt into the following:

• The API and the behavior of the score details can break between two minor versions of Meilisearch.
• In particular, the shape of the _rankingScoreDetails JSON object can change and contain different fields in the future.
• While unlikely at this point, the whole _rankingScoreDetails can be removed in the future.

You can use this feature in production, but be prepared to update your code from one version to the next.

Why is this feature not stable yet?

The _rankingScoreDetails object is expansive and adds a lot of surface to the API. We don't want to stabilize all the details of the various ranking rules instantly.

🗣️ You are welcome to give feedback about the score details or ask any question on its usage; we are eager to collect feedback on the feature

When will the feature potentially be stable?

The target version is Meilisearch v1.7 (released March 11th).
Stabilization PR: meilisearch/meilisearch#4389

[jr-cobweb]

Thanks for the new feature! As discussed separately with @macraig . I'd be interested in being able to filter against the scores (preferably on a per rule basis) too. Currently we're able to do this separately/after the fact, but as a result aren't able to use pagination.

[macraig]

Hi @jr-cobweb, We've published a prototype of the feature you requested, and we'd love your feedback. It would be especially helpful for us to know what you think about its current limitations and whether it would be enough for your use case. Thanks in advance!

[macraig]

Hi @jr-cobweb, I wanted to let you know that we've added filtering by score to our upcoming version v1.9. It's currently available in pre-release in case you want to test it.

[macraig]

Hello again @jr-cobweb , just so you know, we've just released v1.9 with the new rankingScoreThreshold feature. Let us know what you think!

[jmgibson1976]

I am using this feature, I personally like the aggregate score. For it to be useful to me, in production, I need to be able to set a threshold. For example I may want to filter out all results that have a score below .49. I can do it after the search but as @jr-cobweb mentioned, it means that total hits and facet distribution is incorrect.

[jmgibson1976]

@macraig thank you for the update. I have quite a busy week ahead but will try to test this out very soon, hopefully within a week or so and report back to you. Thanks again.

[jmgibson1976]

@macraig I was able to test on one of my indexes, specifically a categories index with 42,159 product categories. The search I performed prior to adding a rankingScoreThreshold returned 54 results, after adding a threshold of .5, returned 15 results that were more accurate, and most importantly had the correct facet distribution and counts. At first glance it looks like exactly what I was looking for, at least for my use case. With that being said, I do searches with a limit of 1000 results so may not run into the issues regarding incorrect hit count outlined here: meilisearch/meilisearch#4609 (comment)

I have another index to test that has many more records and I use cursor pagination with infinite scrolling. It will be a few days before I can test it and report back. Thanks again for working on this feature!

[macraig]

That's excellent news @jmgibson1976 ! Thank you for taking the time to test it. Let us know how the more intensive testing goes :)

[macraig]

Hi @jmgibson1976 , I wanted to let you know that we've added filtering by score to our upcoming version v1.9. It's currently available in pre-release in case you want to test it.

[macraig]

Hello again @jmgibson1976 , just so you know, we've just released v1.9 with the new rankingScoreThreshold feature. Let us know what you think!

[jmgibson1976]

@macraig thanks for the response. I wanted to follow up with an example if I could. I have either deleted all the data or replaced with asterisks to hide identifying or brand information but I left the relevant bits with the score. Now to preface... I have tweaked and tweaked and tweaked search and in a lot of cases I get what I want, the example Im about to show you however has been bothering me and when I turned on aggregate score it validated my assumptions. On my site, when I search for tshirt, ill get the 8 results I expect, when I search for t-shirt I get 1 result. Thats fine I guess as I ranked tags higher than title and one test item has the tag t-shirt. I can fix that with synonyms, but when I search for the plural 't-shirts' I get the 8 results Im expecting and an 24 more. The best i can figure is the '-' but why?

Anyhow, after weeks of tweaking and testing I couldnt figure it out and then I came across the scoring. And this is what I saw, the score drops from .65 to .08 after 8 results and everything is below .08. Why anything that poorly ranked be returned??? So, the scoring shows that search has the same opinion about the quality of the results as I do and yet they get returned. Why is this a big deal? When a user can sort or price or how new the product is or on the facets, an item that was at the bottom of the list (because it shouldnt be there at all) can now wind up at the top.

{
"hits": [
{
"title": "**** Embroided Unisex t-shirt",
"_rankingScore": 0.9941211052322163
},
{
"title": "**** Unisex t-shirt",
"_rankingScore": 0.6598079561042524
},
{
"title": "**** Unisex t-shirt",
"_rankingScore": 0.6598079561042524
},
{
"title": "**** Unisex t-shirt",
"_rankingScore": 0.6598079561042524
},
{
"title": "**** Unisex t-shirt",
"_rankingScore": 0.6598079561042524
},
{
"title": "**** Unisex t-shirt",
"_rankingScore": 0.6598079561042524
},
{
"title": "**** Unisex t-shirt",
"_rankingScore": 0.6598079561042524
},
{
"title": "**** Unisex t-shirt",
"_rankingScore": 0.6545169508132471
},
{
"title": "Egg Pan",
"_rankingScore": 0.08787878787878788 <-- huge drop off in the quality of results
},
{
"title": "**** Cover Print",
"_rankingScore": 0.0696969696969697
},
{
"title": "Test Product",
"_rankingScore": 0.051515151515151514
},
{
"title": "**** Pillowcase - ",
"_rankingScore": 0.04242424242424243
},
{
"title": "",
"_rankingScore": 0.03333333333333333
},
{
"title": "**** mouse pad",
"_rankingScore": 0.03333333333333333
},
{
"title": "**** Unisex Hoodie",
"_rankingScore": 0.03333333333333333
},
{
"title": "**** mouse pad - ",
"_rankingScore": 0.03333333333333333
},
{
"title": " mouse pad",
"_rankingScore": 0.03333333333333333
},
{
"title": "**** mouse pad - ",
"_rankingScore": 0.03333333333333333
},
{
"title": " mouse pad",
"_rankingScore": 0.03333333333333333
},
{
"title": "**** mouse pad",
"_rankingScore": 0.03333333333333333
}
],
"query": "t-shirts",
"processingTimeMs": 1,
"limit": 20,
"offset": 0,
"estimatedTotalHits": 32,
"facetDistribution": {
****,
****,
****
},
"facetStats": [],
"nbHits": 32
}

[macraig]

Thanks @jmgibson1976 for the detailed explanation, I get why it's important to your use case.

Just to be sure I understand correctly: for "t-shirt" and "tshirt" you are only getting the 8 results you'd expect by using synonyms. However when you search for the plural "t-shirts", you get the 8 results plus 24 irrelevant results that are effectively scored much lower. Is that correct? Have you tried adding "t-shirts" as a synonym as well?

[jmgibson1976]

I haven't used synonyms yet but I think it will help with the tshirt, t-shirt, and possibly t-shirts issue. T-shirts was just a recent example where unlike tshirt and t-shirt, I get back 3x as many results and 2/3rds of them are not relevant. Couldnt figure out why and then I turned on scoring and saw that even the scoring ranks the results really poorly.

As far as whether or not synonyms will help... probably, but I dont think it is a scalable solution for my application as I have potentially 10s of thousands of products that are changing constantly throughout the year. I am looking into the natural language stuff; however, the scoring seems like an excellent solution to my problems so I am looking forward to its evolution as well.

[macraig]

I get back 3x as many results and 2/3rds of them are not relevant

Is this due to having no other relevant results left to show? As in, are you getting the most relevant result on top (even if it's a low score) or are some relevant results not showing up and being replaced by less relevant ones?

[MustangZhong]

it's quite useful feature! thanks the team!

[dureuill]

Opened stabilization PR to stabilize this feature in v1.7: meilisearch/meilisearch#4389 🚀

[macraig]

Hi everyone 👋

🐇 v1.7 has been released and we're happy to announce that score details are now considered stable 🎉

Check out our documentation for usage instructions.