fix(deps): relax tokenizers floor to >=0.22 so [rerank] install resolves (closes #333) - #334
Conversation
…ves (closes #333) A fresh `pip install mempalace[rerank]` fails with ResolutionImpossible. Root cause: the rerank extra pulls sentence-transformers → transformers, whose current releases pin `tokenizers>=0.22.0,<=0.23.0` (e.g. transformers 5.9.0). mempalace's core `tokenizers>=0.23` floor sits above the bottom of that window, leaving the resolver only the single point 0.23.0 to satisfy both constraints. pip can't hold that against the rest of the rerank tree and backtracks through dozens of transformers versions before failing. Relax the floor to `tokenizers>=0.22,<0.24`. 0.22 is the actual bottom of the transformers window and what the rerank stack resolves to (verified: tokenizers 0.22.2 + transformers 5.9.0 + sentence-transformers 5.5.1). mempalace's only direct use (embedding.py: `Tokenizer.from_file`) is stable well below 0.22, so lowering the floor is safe. `<0.24` keeps it bounded. Verified in a fresh venv: `pip install mempalace[rerank]` now resolves cleanly and the cross-encoder rerank path imports/loads. Full test suite green (3792 passed), ruff clean. uv.lock: the mempalace tokenizers specifier was already drifted to `>=0.15` (PR #207 bumped pyproject to >=0.23 without re-locking, and CI installs via `pip install -e ".[dev]"`, not `uv sync`, so the lock is a dev convenience, not a gate). Synced that one specifier to the new bound rather than regenerating the whole lock — a full `uv lock` is a separate ~3k-line churn best done in a dedicated housekeeping pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a dependency resolution failure encountered when installing the Highlights
Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the dependency constraint for tokenizers in pyproject.toml from >=0.23 to >=0.22,<0.24 to resolve a dependency resolution conflict with the [rerank] extra dependencies. No review comments were provided, and the changes appear correct and well-documented.
The bug
A fresh
pip install mempalace[rerank]fails:Root cause (verified, not guessed)
The
[rerank]extra pullssentence-transformers→transformers, whose current releases pintokenizers>=0.22.0,<=0.23.0(confirmed from the published wheel metadata oftransformers 5.9.0). mempalace core declarestokenizers>=0.23— a floor above the bottom of that window. The only version satisfying both is the single point0.23.0; pip can't hold that against the rest of the rerank tree (torch, scikit-learn, regex pins) and backtracks through dozens oftransformersversions before failing.Fix
Relax the core floor to
tokenizers>=0.22,<0.24:0.22is the actual bottom of the transformers window — and what the stack resolves to in practice:tokenizers 0.22.2+transformers 5.9.0+sentence-transformers 5.5.1.mempalace/embedding.py(from tokenizers import Tokenizer→Tokenizer.from_file), a stable API well below 0.22, so lowering the floor is safe.<0.24keeps it bounded.Verification
pip install mempalace[rerank]resolves cleanly;import mempalace, theembedding.pyTokenizerpath, andsentence_transformers.CrossEncoder/mempalace.cross_encoder_rerankall import.pytest tests/);ruff check+ruff format --checkclean.Note on
uv.lockThe mempalace
tokenizersspecifier inuv.lockwas already drifted (>=0.15) — PR #207 bumped pyproject to>=0.23without re-locking, and CI installs viapip install -e ".[dev]"(notuv sync --locked), so the lock is a developer convenience, not a CI gate. I synced just that one specifier line to the new bound rather than regenerating the whole lock; a fulluv lockis a separate ~3k-line churn (it would also pull the never-locked rerank/torch tree) best done in a dedicated housekeeping pass.Closes #333.
🤖 Generated with Claude Code