Batch integration result metrics are not in proper range

[adamgayoso]

https://openproblems.bio/benchmarks/batch_integration_embed/immune_batch/

It seems the silhouette is greater than 1? How is that possible if scib is rescaling to [0, 1]?

Also, is there a total score that is being computed to rank the methods? It would be great if that total score was included.

[LuckyMD]

Thanks for the issue @adamgayoso. We are currently rescaling the metrics to a range between a random and perfect baseline in open problems. We were discussing on Tuesday how to best document this on the website. One thought was to enable a toggle between raw and scaled values in the tables. Do you have any thoughts on this?

This rescaling is also used to compute a total score just using mean aggregation. Displaying that would indeed be a good idea.

[adamgayoso]

I like the idea of a toggle and including the mean score (as well as how it's computed; are you doing the 0.6/0.4 split)?

[LuckyMD]

We're not currently doing the 60-40 split to be consistent across tasks. I do think we should, but it's not currently something that is set task-specifically.

[adamgayoso]

To me it makes sense to maybe have a straight average, a bio-biased average, and a batch-biased average? (5050,6040,4060)

[LuckyMD]

hmm... i wonder if that will just over-complicate things. Either way it's not generalizeable across tasks and so would require some exception in the ranking. It should definitely be discussed though.

[adamgayoso]

It still seems like the metrics are in different ranges, yet a simple average of the metrics is being used.

For example, isolated label sil. is >1 for many method and seems to contribute a lot to the ranking right now.

[scottgigante-immunai]

See #685

[scottgigante-immunai]

Also, is there a total score that is being computed to rank the methods? It would be great if that total score was included.

This score is displayed as "Mean score". It's the first column.

[scottgigante-immunai]

Closing as duplicate