[v2] ci: run bm25 and ColBERT test in ci

[sam-hey]

cI: Enable skipped tests for ColBERT and BM25
fix: get model meta from CrossEncoder
ref: use tmp_path for tests
test: add tests for model_meta

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

start in: #1759 (comment)

[KennethEnevoldsen]

This only really checks if they can be installed. We should add an actual test as well?

[sam-hey]

Hi @KennethEnevoldsen,

I added two tests in this PR. However, they are not being run in the CI because the dependencies don't exist.

[KennethEnevoldsen]

Ahh thanks for the pointer. Any reason not to remove the skip? (I could later clean up the installs without knowing that I disabled a test).

Will you also add a time diff? (coverage report would also be nice, but not required)

[sam-hey]

The current issue with PyLate (referenced in GitHub Issue #79) is related to compatibility problems with older Python versions, which is causing the tests for Python 3.9 to fail.

Not skipping would require all developers to install additional dependencies: pylate, bm25s, and pystemmer. While this is feasible, it does introduce more requirements for everyone on the team. This decision is ultimately up to you, and I am fine with proceeding if you decide to include them.

[sam-hey]

Tests conducted on v2.0.0 with Python 3.11:

checks occasionally take up to 12 minutes.
This branch test runtime: 8-9 minutes.

Performance benchmarks:
ColBERT + BM25: Test execution on my machine completes in 15.17 seconds.

Test coverage summary:
Overall test coverage: 66%.
mteb/models/colbert_models.py: 63%
mteb/models/bm25.py: 94%

[sam-hey]

I believe I’ve added everything for now and would be happy to get your review of the changes!

• Skipping pylate: Skipped for Python 3.9.
• Updates:
  • Added extra tests.
  • Introduced ModelMeta for CrossEncoders passed as models to run().
• Pending Decision: Awaiting your input on skipping BM25s and ColBERT if the version is not available.

@KennethEnevoldsen

[sam-hey]

@Samoed can you give me a review if you have some time?

[Samoed]

Great!

[sam-hey]

Why was MaxSim renamed to max_sim? This change breaks ColBERT. @Samoed, I’ll fix it and add a test to prevent similar issues in the future.

[Samoed]

max_sim was used in main branch and I thought it was tested and worked with this name

[sam-hey]

max_sim was used in main branch and I thought it was tested and worked with this name

The max_sim function was updated to MaxSim to align with the upcoming pylate integration in a future release.

[sam-hey]

Awesome, thanks a lot!