chore: vLLM, kernels, and flashinfer upgrades

[mckornfield]

Summary

Pre-Review Checklist

Ensure that the following pass:

• mise run format && mise run check or via prek validation.
• mise run test passes locally
• mise run test:e2e passes locally
• mise run test:ci-container passes locally (recommended)
• GPU CI status check passes -- comment /sync on this PR to trigger a run (auto-triggers on ready-for-review)

Pre-Merge Checklist

• New or updated tests for any fix or new behavior
• Updated documentation for new features and behaviors, including docstrings for API docs.

Other Notes

• Closes #

Summary by CodeRabbit

• Bug Fixes
  • Improved vLLM engine initialization by aligning maximum sequence length with model metadata and applying RoPE context extension overrides when configured.
• New Features
  • Updated Linux Torch-related optional dependency pins for CPU and CUDA 12.9 (flashinfer and vLLM), including Transformers version constraints.
• Documentation
  • Refreshed CUDA 12.9 (cu129) vLLM wheel index URLs across the README and user/developer docs and tutorials.
• Chores
  • Updated install/source configuration for cu129 wheels in automation and PyPI-mode workflows.
• Tests
  • Expanded vLLM initialization tests to validate max_model_len and RoPE-related overrides.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Updates dependency pins and vLLM source entries, refreshes cu129 wheel index URLs in install instructions, and adds RoPE override handling in VllmBackend.initialize with matching tests.

Changes

vLLM dependency, install, and runtime configuration

Layer / File(s)	Summary
Dependency and source updates pyproject.toml	flashinfer, transformers, and vLLM pins are updated in the cpu and cu129 extras, and the uv vLLM source and index entries are switched to the new 0.24.0/cu129 location.
Install URL refresh script/slurm/slurm_nss_matrix.sh, README.md, docs/dev-notes/posts/introducing-nemo-safe-synthesizer.md, docs/user-guide/getting-started.md, docs/tutorials/*.ipynb	The cu129 wheels.vllm.ai index URL is updated in the Slurm install script and in the README, dev note, user guide, and tutorial notebooks, including notebook metadata changes in one tutorial.
RoPE overrides in vLLM initialization src/nemo_safe_synthesizer/generation/vllm_backend.py, tests/generation/test_vllm_backend.py	VllmBackend.initialize sets max_model_len from model_metadata.max_seq_length, builds optional RoPE-based hf_overrides, and the tests assert both the explicit length and override behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested labels: bug, test

Suggested reviewers: kendrickb-nvidia, binaryaaron

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main dependency upgrades to vLLM, kernels, and flashinfer.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch all-the-upgrades-06-30/mck

---

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Greptile Summary

This PR upgrades vLLM from 0.20.0 to 0.24.0 and FlashInfer from 0.6.8.post1 to 0.6.13, adapting the vLLM backend to the new API surface and adding proper RoPE context-extension support via hf_overrides.

• vLLM 0.24 API migration: prompt_token_ids= argument replaced with the TokensPrompt dict wrapper via prompts=; AttentionConfig now constructed as a typed object; max_model_len explicitly passed from ModelMetadata.max_seq_length; beam-search (beam_width) removed from the parameter mapping since vLLM 0.24 dropped it.
• RoPE context extension: New _build_rope_hf_overrides helper constructs the {"rope_parameters": {...}} dict passed as hf_overrides to the vLLM engine when factor > 1.0; RopeScaling gains a rope_parameters field that preserves the full native Transformers v5 config and is merged correctly from both the new rope_parameters attribute and the legacy rope_scaling dict.
• Reduced transformers coupling: PreTrainedTokenizerBase replaced by a local EncodeOnlyTokenizer Protocol exposing only the encode method used by processors.

Confidence Score: 5/5

Safe to merge — the vLLM 0.24 API migration is complete and internally consistent, the three issues raised in earlier review rounds are all resolved, and the new RoPE override path is covered by targeted tests.

The FlashInfer version mismatch flagged in a prior round is now fixed in both the cpu and cu129 extras. The vLLM index name is corrected. The test assertion for hf_overrides now matches the single-key return value of _build_rope_hf_overrides. The TokensPrompt migration, max_model_len alignment, and RopeScaling.rope_parameters round-trip are all covered by new unit tests.

No files require special attention beyond the minor transformers pin style difference in pyproject.toml.

Important Files Changed

Filename	Overview
src/nemo_safe_synthesizer/generation/vllm_backend.py	Adapts to vLLM 0.24 API: replaces prompt_token_ids with TokensPrompt, adds max_model_len and hf_overrides to LLM constructor, introduces _build_rope_hf_overrides for RoPE context extension, drops num_beams→beam_width mapping (logged at INFO when remapped)
src/nemo_safe_synthesizer/llm/metadata.py	Adds rope_parameters field to RopeScaling to preserve native Transformers v5 config; from_autoconfig now merges legacy rope_scaling dict into rope_parameters with new format taking precedence; to_hf_dict updated to propagate stored parameters while filtering rope_theta key
src/nemo_safe_synthesizer/generation/processors.py	Replaces PreTrainedTokenizerBase dependency with a local EncodeOnlyTokenizer Protocol, reducing coupling to the full transformers library while retaining structural typing
pyproject.toml	Bumps vLLM 0.20.0→0.24.0 and FlashInfer 0.6.8.post1→0.6.13 in both cpu and cu129 extras; transformers is pinned inconsistently (==5.12.0 exact in cpu vs >=5.12,<5.12.1 range in cu129)
tests/generation/test_vllm_backend.py	New tests cover max_model_len propagation, RoPE hf_overrides construction, and the updated TokensPrompt-based generation dispatch paths; test assertion for hf_overrides now correctly matches the single-key {"rope_parameters": ...} return value
tests/llm/test_metadata.py	Adds round-trip tests for Transformers v5 rope_parameters (Qwen2Config); imports Qwen2Config directly for realistic config fixtures

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant VB as VllmBackend.initialize()
    participant RH as _build_rope_hf_overrides()
    participant MM as ModelMetadata
    participant vLLM as vLLM(engine)

    VB->>MM: read max_seq_length, base_max_seq_length, rope_scaling
    VB->>RH: _build_rope_hf_overrides(model_metadata)
    alt "rope_scaling is None or factor <= 1.0"
        RH-->>VB: None
    else "factor > 1.0"
        RH->>RH: build rope_parameters dict (type, factor, original_max_pos_emb, rope_theta)
        RH-->>VB: "{"rope_parameters": {...}}"
    end
    VB->>vLLM: "LLM(model=..., max_model_len=max_seq_length, hf_overrides=rope_overrides, ...)"
    vLLM-->>VB: engine instance
    VB->>VB: cache engine runtime config
    VB->>vLLM: get_tokenizer() → EncodeOnlyTokenizer
    VB->>VB: create_processor(schema, metadata, config, tokenizer)

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant VB as VllmBackend.initialize()
    participant RH as _build_rope_hf_overrides()
    participant MM as ModelMetadata
    participant vLLM as vLLM(engine)

    VB->>MM: read max_seq_length, base_max_seq_length, rope_scaling
    VB->>RH: _build_rope_hf_overrides(model_metadata)
    alt "rope_scaling is None or factor <= 1.0"
        RH-->>VB: None
    else "factor > 1.0"
        RH->>RH: build rope_parameters dict (type, factor, original_max_pos_emb, rope_theta)
        RH-->>VB: "{"rope_parameters": {...}}"
    end
    VB->>vLLM: "LLM(model=..., max_model_len=max_seq_length, hf_overrides=rope_overrides, ...)"
    vLLM-->>VB: engine instance
    VB->>VB: cache engine runtime config
    VB->>vLLM: get_tokenizer() → EncodeOnlyTokenizer
    VB->>VB: create_processor(schema, metadata, config, tokenizer)

Loading

_{Reviews (9): Last reviewed commit: "chore: address Aaron CR" | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 94.33962% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...c/nemo_safe_synthesizer/generation/vllm_backend.py	78.57%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

[kendrickb-nvidia]

The upgrade hamster wheel never ends.

Is this motivated by a particular issue, or just general bump of our deps to keep up?

[coderabbitai]

🧹 Nitpick comments (2)

src/nemo_safe_synthesizer/generation/vllm_backend.py (1)

312-326: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Document why hf_overrides is needed here.

The rationale for conditionally injecting hf_overrides (vLLM 0.24 requiring explicit RoPE config overrides when max_model_len exceeds the model's native context) only appears in the test docstring, not as a source comment. Since this is a non-obvious, version-specific invariant, a brief inline comment here would help future maintainers understand why this branch exists without needing to trace back to the test file.

model_ref = ModelRef.parse(self.config.training.pretrained_model)
llm_kwargs: dict[str, Any] = {}
# vLLM 0.24 requires explicit rope_parameters hf_overrides when max_model_len
# extends beyond the model's native context via rope_scaling.
if hf_overrides := _build_rope_hf_overrides(self.model_metadata):
    llm_kwargs["hf_overrides"] = hf_overrides

tests/generation/test_vllm_backend.py (1)

398-441: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add boundary coverage for rope_scaling.factor <= 1.0 with rope_scaling set.

The two tests cover rope_scaling is None (no override) and factor=2.0 (override applied), but _build_rope_hf_overrides's explicit factor <= 1.0 branch is never exercised with a non-None rope_scaling (e.g., factor=1.0). This boundary is exactly the condition guarding whether hf_overrides gets sent to vLLM, so it's worth a focused test.

def test_initialize_omits_hf_overrides_when_rope_scaling_factor_not_extended(...):
    mock_model_metadata.rope_scaling = RopeScaling(rope_type="linear", factor=1.0, theta=10000.0)
    ...
    assert "hf_overrides" not in mock_vllm.call_args.kwargs

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1594b711-f9d7-4961-bfe9-a6b58c1e3f1d

📥 Commits

Reviewing files that changed from the base of the PR and between 9c4a585 and 9e28e7e.

📒 Files selected for processing (2)

• src/nemo_safe_synthesizer/generation/vllm_backend.py
• tests/generation/test_vllm_backend.py

📜 Review details

⏰ Context from checks skipped due to timeout. (2)

• GitHub Check: Greptile Review
• GitHub Check: Typecheck

🧰 Additional context used

📓 Path-based instructions (11)

**/*.{md,markdown,py}

📄 CodeRabbit inference engine (.cursor/rules/agent-markdown-style.mdc)

**/*.{md,markdown,py}: Avoid decorative bold (**text**) in list items, body text, and docstrings; use structural cues (headers, list markers, colons, backticks) for emphasis instead
Use backticks for code identifiers, paths, and CLI commands in markdown and docstrings

Files:

• src/nemo_safe_synthesizer/generation/vllm_backend.py
• tests/generation/test_vllm_backend.py

**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

**/*.py: Place durable implementation guidance in function and class docstrings for public contracts and source comments for local invariants
Target Python 3.11–3.13 with modern syntax (X | Y, list[str], Self). Python 3.14+ is not supported

**/*.py: Source code must remain Python 3.11 syntax-compatible; do not use Python 3.12-only syntax such as PEP 695 type statements or bracketed generic class/function parameters in shared package code
Use ruff for formatting and linting via mise run format and mise run check tasks; run formatting before committing
Use ty for type checking via mise run check task; ensure type hints are present and valid
New features must include tests; bug fixes must include regression tests

**/*.py: Use BaseSettings for env/CLI settings, prefer AliasChoices for per-field dual naming, and use env_prefix only for simple settings classes with a shared prefix.
Use Field(description=...) as the canonical field docstring for Pydantic models, and always include it.
Use assignment-style Field(default=..., description="...") as the default for model fields; prefer it over Annotated unless extra metadata is needed.
Use Annotated only when the field carries additional metadata beyond Field() (for example validators, reusable constrained aliases, nested-type constraints, or discriminated unions).
When Annotated is used, place defaults as bare assignments (= value), except default_factory, which should still use assignment-style Field(default_factory=...).
For immutable value objects and validators, prefer @dataclass(frozen=True); use mutable dataclasses only for builders, accumulators, and pipeline state.
Use field(default_factory=list) for mutable defaults; never use = [].
Use StrEnum for string-valued enums used in configs or serialization; use plain Enum for internal-only named constants.
Use observability.get_logger(__name__) for logging; do not call logging.getLogger() or structlog.get_logger() direc...

Files:

• src/nemo_safe_synthesizer/generation/vllm_backend.py
• tests/generation/test_vllm_backend.py

**/*.{py,sh,yaml,yml,md}

📄 CodeRabbit inference engine (CONTRIBUTING.md)

All source files (.py, .sh, .yaml, .yml, .md) require SPDX copyright headers; mise run format adds them automatically

Files:

• src/nemo_safe_synthesizer/generation/vllm_backend.py
• tests/generation/test_vllm_backend.py

src/nemo_safe_synthesizer/**/*.py

📄 CodeRabbit inference engine (CONTRIBUTING.md)

API reference pages are auto-generated from Python docstrings using Google-style format; write docstrings in src/nemo_safe_synthesizer/ and they will appear in the reference/

Files:

• src/nemo_safe_synthesizer/generation/vllm_backend.py

src/**/*.py

📄 CodeRabbit inference engine (STYLE_GUIDE.md)

src/**/*.py: Use relative imports inside src/, for example from ..observability import get_logger.
Every directory under src/ that contains Python files must include an __init__.py file.

Files:

• src/nemo_safe_synthesizer/generation/vllm_backend.py

---

⚙️ CodeRabbit configuration file

Review library code against STYLE_GUIDE.md. Focus on behavior, API contracts, error handling, resource cleanup, typing, logging, and user-facing failures. Public APIs and nontrivial functions need Google-style docstrings.

Files:

• src/nemo_safe_synthesizer/generation/vllm_backend.py

**/*

📄 CodeRabbit inference engine (STYLE_GUIDE.md)

**/*: Every source file must include the required SPDX copyright and license header; use HTML comments for Markdown, hash comments for .py, .sh, .yaml, and .yml, and hash-comment headers inside YAML frontmatter for Markdown files with frontmatter.
Ensure files end with a newline and have no trailing whitespace; use a single space between sentences.

Files:

• src/nemo_safe_synthesizer/generation/vllm_backend.py
• tests/generation/test_vllm_backend.py

---

⚙️ CodeRabbit configuration file

**/*: Review as a senior maintainer for NeMo Safe Synthesizer. Prioritize issues that can change behavior, break user workflows, weaken privacy guarantees, hide failures, make tests unreliable, or create maintenance risk. Avoid generic style commentary unless it points to a concrete project convention that automated tools will not catch.
Comment only when the finding is actionable and tied to changed code. For each finding, state the impact, the condition that triggers it, and the smallest practical fix. Prefer one precise comment over broad advice. Do not ask for refactors outside the PR scope unless the changed code creates the problem.
Review type guidance: - Potential issue: use for correctness bugs, data loss, privacy leaks,
security risks, broken public APIs, invalid config behavior, missing
validation, hidden failures, nondeterministic tests, or CI breakage.

• Refactor suggestion: use for local maintainability problems introduced
  by the diff when they have clear future cost, such as duplicated setup,
  unclear boundaries, over-mocking, avoidable complexity, or opaque test
  helpers.
• Nitpick: avoid in chill mode. Do not emit formatting, import-order,
  wording, or style-only comments unless automated tools cannot catch the
  issue and it affects maintainability.

Severity guidance: - Critical: security/privacy leaks, data loss, training/test/holdout
contamination, or broken release/package/core pipeline execution.

• Major: incorrect generation/training/evaluation behavior, broken
  CLI/SDK public API, invalid config defaults or validators, or GPU/vLLM
  cleanup and process-isolation bugs likely to fail CI or production
  runs.
• Minor: localized bugs, missing focused tests for changed behavior, or
  bad test patterns that weaken regression coverage.
• Trivial: small cleanup with no behavior impact. Usually suppress in
  chill mode.
• Info: context only. Avoid unless it helps reviewers understand risk.
  Safe-Synthesizer-specific review focus: - Data ...

Files:

• src/nemo_safe_synthesizer/generation/vllm_backend.py
• tests/generation/test_vllm_backend.py

**

⚙️ CodeRabbit configuration file

**:

AGENTS.md

Guide for AI agents (Cursor, Windsurf, Claude Code, etc.) working in the Safe-Synthesizer repo.

This project loads local developer preferences from @AGENTS.local.md. You MUST read this file if it exists and give its instructions top priority.

Skills

Repo-specific skills live in .agents/skills/; see .agents/README.md for the catalog. Read a skill when the task matches its scope instead of copying workflow details into this file.

Durable implementation guidance belongs with the code it describes: function and class docstrings for public contracts and source comments for local invariants. Test-suite guidance belongs in tests/TESTING.md.

Repo Conventions

See STYLE_GUIDE.md for detailed code style conventions (Python, markdown, Dockerfiles, shell scripts, testing, config files, docstrings).

Use uv for everything -- never pip or raw python. Python 3.11–3.13 with modern syntax (X | Y, list[str], Self). Python 3.14+ is not supported.

Common commands: mise run test (unit tests), mise run format (auto-fix formatting + lint + copyright), mise run check (read-only local quality checks), mise run validate (pre-PR quality, lock, and CI unit checks), mise run typecheck (ty only). Always use mise tasks or the wrapper scripts in tools/ instead of running ruff or ty directly. Use uv run for Python execution. When in doubt, inspect mise tasks and pytest --markers.

The canonical uv sync command for a full GPU/dev environment is:

uv sync --frozen --extra cu129 --extra engine --group dev

Bare uv sync --frozen (without extras) installs an incomplete environment -- ty, import checks, and GPU tests will fail.

Feature branches off main. Branch names often include an issue number prefix (e.g., <author>/123-short-name).

Do ...

Files:

• src/nemo_safe_synthesizer/generation/vllm_backend.py
• tests/generation/test_vllm_backend.py

src/nemo_safe_synthesizer/generation/**/*.py

⚙️ CodeRabbit configuration file

Review generation changes for retry loops, stopping conditions, invalid record handling, regex/structured output contracts, backend teardown, memory cleanup, and vLLM assumptions.

Files:

• src/nemo_safe_synthesizer/generation/vllm_backend.py

**/test_*.py

📄 CodeRabbit inference engine (AGENTS.md)

Use the unit marker instead of the deprecated unit_test marker for test identification

Files:

• tests/generation/test_vllm_backend.py

tests/**

📄 CodeRabbit inference engine (.cursor/rules/repo-navigation.mdc)

tests/**: Mirror src/ directory structure in tests/ directory for test organization
Auto-mark tests by directory: tests/e2e/ → e2e, tests/smoke/ → smoke, otherwise default to unit

Mirror source code directory structure in tests directory (e.g., tests/training/, tests/generation/ parallel to source structure)

Files:

• tests/generation/test_vllm_backend.py

---

⚙️ CodeRabbit configuration file

tests/**:

Testing Guide

Comprehensive testing reference for Safe-Synthesizer developers. Covers commands, markers, test data, fixtures, and gotchas.

Read First

1. tests/conftest.py -- auto-marking, load_test_dataset/load_test_dataframe, fixture_mock_processor pattern
2. pytest.ini -- markers, asyncio, timeout
3. tests/evaluation/conftest.py -- most complex: Faker-based make_df, nullable dtype conversion
4. tests/generation/conftest.py -- JSONL/schema fixtures, fixture_valid_iris_dataset_jsonl_and_schema

Running Tests

All mise test tasks, grouped by scope:

mise run test                              # Unit (excludes slow, e2e, and smoke)
mise run test:unit-slow                    # Unit tests including slow (excludes e2e and smoke)
mise run test:smoke                        # CPU smoke tests (~few min, no GPU required)
mise run test:smoke:gpu                    # All staged GPU smoke tests (requires CUDA)
mise run test:smoke:gpu:train-only
mise run test:smoke:gpu:generation
mise run test:smoke:gpu:resume
mise run test:smoke:gpu:structured-generation
mise run test:smoke:gpu:timeseries
mise run test:smoke:gpu:smollm2
mise run test:e2e                          # All e2e (requires CUDA) -- runs default + dp
mise run test:e2e:default                  # e2e default (no-DP) tests only
mise run test:e2e:dp                       # e2e DP tests only
mise run test:ci                           # CI unit tests with coverage (excludes slow, e2e, gpu, smoke)
mise run test:ci-slow                      # CI slow tests with coverage
mise run test:ci-container                 # CI tests in a Linux container (Docker/Podman)

Run a single test:

uv run --frozen pytest tests/path/test_file.py::test_name -vvs -n0

Test runner: uv run --frozen pytest -n auto --dist loadscope -vv...

Files:

• tests/generation/test_vllm_backend.py

tests/**/*.py

📄 CodeRabbit inference engine (tests/TESTING.md)

tests/**/*.py: Auto-mark tests based on file path: tests under /e2e/ get e2e marker, tests under /smoke/ get smoke marker, all others get unit marker (only if no category marker already present)
Every test should have exactly one category marker: unit, smoke, or e2e
Use pytest.mark.requires_gpu modifier on tests that need CUDA hardware
Use pytest.mark.vllm on tests using vLLM generation backend and ensure each vLLM test file runs in its own process for GPU memory isolation
Use pytest.mark.slow on long-running tests
Use pytest.mark.smollm2 for SmolLM2 Hub download tests to enable process isolation
Use pytest.mark.noautouse to skip autouse fixtures for specific tests
Use load_test_dataset(filename) helper to load test datasets from tests/stub_datasets/ as HuggingFace Dataset objects
Use load_test_dataframe(filename) helper to load test data files from tests/stub_datasets/ as pandas DataFrames
Convert pandas columns to nullable dtypes (pd.Int64Dtype(), pd.BooleanDtype()) before assigning np.nan values
Use fake.seed_instance(seed) and random.seed(seed) together for Faker-based test data reproducibility
When sharing methods across multiple test files, define them in conftest.py and import them using relative imports (e.g., from .conftest import train_with_sdk); note that importing from other test files like tests/cli/helpers.py does not work
Use fixture_mock_processor or fixture_mock_processor_without_valid_records for mocking ParsedResponse objects with valid_records, invalid_records, errors, and prompt_number fields
Use pytest.importorskip to gate tests on optional dependencies that require specific extras (e.g., sentence_transformers, vllm)
Run vLLM tests with separate pytest invocations (one per file) using -n 0 (single process) for GPU memory isolation, or use staged mise tasks for CI visibility
Print statements are allowed in tests (ruff T201 is suppressed for tests/ directory) and should...

Files:

• tests/generation/test_vllm_backend.py

---

⚙️ CodeRabbit configuration file

Review tests against tests/TESTING.md. Check marker usage, fixture naming, tmp_path usage, determinism, and GPU/vLLM process-isolation requirements. Flag slop tests that only check that code runs, assert result is not None when stronger invariants exist, over-mock internal implementation details, patch around the bug instead of reproducing it, or add broad snapshot/golden churn without a clear contract. Flag change detector tests that fail on harmless refactors, formatting, record ordering, incidental wording, or private implementation details without demonstrating a behavior regression. Prefer existing fixtures or focused new fixtures for repeated setup; keep tests DRY when reasonable without making the behavior under test opaque. print() is allowed in tests.

Files:

• tests/generation/test_vllm_backend.py

🧠 Learnings (2)

📚 Learning: 2026-05-27T22:20:37.354Z

Learnt from: kendrickb-nvidia
Repo: NVIDIA-NeMo/Safe-Synthesizer PR: 520
File: tests/generation/test_vllm_backend.py:556-587
Timestamp: 2026-05-27T22:20:37.354Z
Learning: In NVIDIA-NeMo/Safe-Synthesizer, `tests/conftest.py`’s `pytest_collection_modifyitems` hook applies pytest category markers automatically based on each test file’s path: tests under `/e2e/` get `pytest.mark.e2e`, tests under `/smoke/` get `pytest.mark.smoke`, and all other tests get `pytest.mark.unit`. Therefore, when reviewing pytest tests outside `tests/e2e/` and `tests/smoke/`, do not flag missing explicit `pytest.mark.unit` decorators on test classes/functions as an issue (the hook will add them during collection). If a new test directory/category is introduced, ensure the hook is updated so it’s categorized correctly.

Applied to files:

• tests/generation/test_vllm_backend.py

📚 Learning: 2026-06-04T16:14:09.868Z

Learnt from: binaryaaron
Repo: NVIDIA-NeMo/Safe-Synthesizer PR: 526
File: tests/generation/test_vllm_backend.py:399-509
Timestamp: 2026-06-04T16:14:09.868Z
Learning: In this repo, only apply `pytest.mark.vllm` to smoke tests under `tests/smoke/` that actually run real vLLM GPU generation and therefore require per-file process isolation (e.g., `test-smoke-gpu-*` Makefile targets). Do not apply `pytest.mark.vllm` to unit-style tests under `tests/generation/` that merely import `vllm_backend` but never instantiate a real vLLM engine and never call `.generate()` (GPU not required). Note that `tests/conftest.py` auto-marks these as `unit` via `pytest_collection_modifyitems`, and `vllm` is not among the auto-mark categories—so if a test in `tests/generation/` has `vllm`, it should be treated as a review issue unless it meets the real GPU generation criteria above.

Applied to files:

• tests/generation/test_vllm_backend.py

🔇 Additional comments (2)

tests/generation/test_vllm_backend.py (1)

23-23: LGTM!

Also applies to: 46-47

src/nemo_safe_synthesizer/generation/vllm_backend.py (1)

79-94: 🎯 Functional Correctness

No issue here: VllmBackend.initialize() is only reached with fully populated metadata, and ModelMetadata.stub() stays on the preflight path. The base_max_seq_length fallback mismatch does not reach model loading.

			> Likely an incorrect or invalid review comment.

[kendrickb-nvidia]

Slurm testing is successful? Fix up the test expectation and looks good.

[binaryaaron]

agent (review-pr): Request changes for vLLM 0.24 and Transformers 5.12 compatibility gaps

I found several migration gaps that should be addressed before merge.

Required changes

1. [P1] Preserve native Transformers 5 RoPE parameters

   RopeScaling.from_autoconfig() still reads the removed top-level config.rope_theta. Transformers 5.12 stores this value in config.rope_parameters["rope_theta"].

   For a standard Qwen configuration, NSS currently converts the native theta from 1_000_000 to the fallback 10_000. The new hf_overrides then replaces the correct vLLM value with the fallback. This is reachable through the normal train, save-artifact, resume, and generate flow.

   The same boundary silently drops scheme-specific fields required by dynamic, yarn, and llama3. Please preserve and validate the standardized native rope_parameters, then add a real Qwen2Config round-trip regression test.

2. [P2] Keep max_position_embeddings at the original context length

   vLLM treats max_position_embeddings as the pre-scaling context and multiplies it by the RoPE factor. The PR sets it to the already-extended length, so the rotary cache is scaled twice.

   For base=2048 and factor=2, the required cache length is 4096; the current override produces 8192. Keep the top-level LLM(max_model_len=...) argument and remove max_position_embeddings and the redundant nested max_model_len from hf_overrides.

3. [P2] Update the removed token-ID generation API

   vLLM 0.24 no longer accepts LLM.generate(prompt_token_ids=...). The _generate(input_ids=...) branches now raise TypeError. Pass token IDs through prompts= and add flat and batched token-ID coverage using the real 0.24 signature.

4. [P2] Update the dependency troubleshooting documentation

   docs/user-guide/troubleshooting.md still recommends Transformers >=5.6,<6 with vLLM 0.20.0. Please update this to the supported Transformers 5.12 and vLLM 0.24 pairing.

Follow-up cleanup

• Remove or replace the obsolete num_beams -> beam_width mapping. SamplingParams 0.24 no longer accepts beam_width.
• Pass hf_overrides= directly instead of routing it through dict[str, Any] and **llm_kwargs.
• Use the public vLLM AttentionConfig to validate the configured attention backend.
• Replace the TokenizerLike cast with an encode-only local Protocol.

Validation performed

• Focused generation tests passed.
• Type checks, lint, lock validation, and dependency resolution passed.
• Transformers and vLLM runtime probes reproduced the issues above.
• Existing GPU smoke coverage does not exercise scaled-context generation.

StructuredOutputsConfig, top-level max_model_len, and the current shutdown strategy remain compatible with vLLM 0.24.

[binaryaaron]

thanks! land away if we get clean slurm runs.