Stop testing for slow tokenizers as they will not exist soon#34235
Stop testing for slow tokenizers as they will not exist soon#34235DarkLight1337 merged 1 commit intovllm-project:mainfrom
Conversation
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
There was a problem hiding this comment.
Code Review
This pull request removes tests for slow tokenizers in anticipation of their deprecation. The change is straightforward, but it removes test coverage for code that still exists in the repository (e.g., handling for tokenizer_mode='slow'). This is risky. I've added a comment suggesting that the feature code and its tests should be removed together to avoid potential issues.
I am having trouble creating individual review comments. Click here to see my feedback.
tests/tokenizers_/test_basic.py (28-30)
While the goal of removing support for slow tokenizers is clear, this PR only removes the tests. The implementation logic for slow tokenizers (e.g., handling tokenizer_mode="slow" in vllm/tokenizers/registry.py) still exists in the codebase. Removing tests before the feature itself leaves this code untested, which can lead to regressions or dead code being left behind. It's recommended to remove the feature code and its tests in the same pull request.
…oject#34235) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
…oject#34235) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Slow tokenizders do not exist in Transformers v5. This PR pre-emptively removes the test that checks that slow tokenizers load slowly.