Skip to content

[codex] Add SA-Bench DeepSeek V4 tokenizer mode#100

Closed
alec-flowers wants to merge 1 commit into
mainfrom
codex/sa-bench-tokenizer-mode
Closed

[codex] Add SA-Bench DeepSeek V4 tokenizer mode#100
alec-flowers wants to merge 1 commit into
mainfrom
codex/sa-bench-tokenizer-mode

Conversation

@alec-flowers
Copy link
Copy Markdown
Collaborator

Summary

Adds a narrow SA-Bench tokenizer-mode pass-through for DeepSeek V4 without bringing in any recipe payload.

Current main already has the DSV4 custom tokenizer package and the fast chat-template failure with guidance from PR #76. This PR fills the remaining gap from aflowers/gb200-dsv4-recipes: recipes can now set benchmark.tokenizer_mode: deepseek_v4, and SA-Bench will pass that through to benchmark_serving.py.

What changed

  • Adds BenchmarkConfig.tokenizer_mode.
  • Passes tokenizer mode from SABenchRunner into bench.sh as a new trailing positional argument, preserving the existing dataset arguments.
  • Adds --tokenizer-mode deepseek_v4 support in benchmark_serving.py and backend_request_func.py.
  • Adds custom_tokenizer: deepseek_v4 as a short alias for the same vLLM DeepSeek V4 tokenizer.
  • Adds focused tests for command construction.

Validation

  • PYTHONPATH=src .venv/bin/python -m pytest tests/ -q -> 612 passed, 2 skipped, 6 deselected
  • PYTHONPATH=src .venv/bin/python -m pytest tests/test_benchmarks.py tests/test_configs.py -q -> 104 passed
  • UV_PROJECT_ENVIRONMENT=.venv uv run --frozen --no-sync ruff check src/srtctl tests/test_benchmarks.py --ignore SIM117 -> passed
  • UV_PROJECT_ENVIRONMENT=.venv uv run --frozen --no-sync ruff format --check src/srtctl tests/test_benchmarks.py -> passed
  • bash -n src/srtctl/benchmarks/scripts/sa-bench/bench.sh -> passed
  • PYTHONPATH=src .venv/bin/python -m py_compile src/srtctl/benchmarks/scripts/sa-bench/backend_request_func.py src/srtctl/benchmarks/scripts/sa-bench/benchmark_serving.py -> passed

Note: this intentionally does not port the side-branch auto-fallback behavior for missing chat templates; current main already fails fast with actionable DSV4 guidance instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant