feat: vLLM benchmark harness

docs/developer-guide/vllm-benchmark.md

@@ -0,0 +1,102 @@
+<!-- SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+
+# vLLM Benchmark Harness
+
+The vLLM benchmark harness replays a captured generation trace against one or
+more benchmark candidates. It is a developer tool for comparing engine and
+sampling configurations. It is not part of the main Safe Synthesizer CLI
+workflow.
+
+Run it through `uv` with the full engine environment:
+
+```bash
+uv run --frozen --extra cu129 --extra engine --group dev \
+    python tools/vllm_benchmark.py list
+```
+
+## Corpus Format
+
+The input corpus is a JSONL file with one header record followed by prompt
+records:
+
+```json
+{"kind": "header", "run_id": "run-1", "pretrained_model": "model-ref", "dataset_schema": {}, "engine_parameters": {}}
+{"kind": "record", "row_index": 0, "prompt": "...", "sampling_params": {"temperature": 0.0}}
+```
+
+The header supplies the model, optional LoRA path, dataset schema, and captured
+engine parameters. Each record supplies the exact prompt and sampling parameters
+to replay.
+
+## Run a Matrix
+
+Use a preset matrix:
+
+```bash
+uv run --frozen --extra cu129 --extra engine --group dev \
+    python tools/vllm_benchmark.py run \
+    /path/to/trace.jsonl \
+    --output /path/to/benchmark.json \
+    --candidates default_matrix
+```
+
+Use `list` to see available presets. The `bracketed_ab_*` presets emit repeated
+baseline and candidate runs for noisier comparisons.
+
+Use a custom candidate file when a preset is too broad:
+
+```json
+{
+  "candidates": [
+    {
+      "name": "baseline",
+      "engine_config": {},
+      "sampling_overrides": {"seed": 42}
+    }
+  ]
+}
+```
+
+Then run:
+
+```bash
+uv run --frozen --extra cu129 --extra engine --group dev \
+    python tools/vllm_benchmark.py run \
+    /path/to/trace.jsonl \
+    --output /path/to/benchmark.json \
+    --candidates-file candidates.json
+```
+
+## Compare And Analyze
+
+Render one benchmark JSON:
+
+```bash
+uv run --frozen --extra cu129 --extra engine --group dev \
+    python tools/vllm_benchmark.py compare /path/to/benchmark.json
+```
+
+Analyze every `*.json` result in a directory:
+
+```bash
+uv run --frozen --extra cu129 --extra engine --group dev \
+    python tools/vllm_benchmark.py analyze /path/to/results-dir \
+    --cluster-signal auto \
+    --json-out /path/to/analysis.json
+```
+
+The analyzer reports candidate-run aggregates by condition. The report JSON
+uses `n_candidate_runs` for aggregate sample counts.
+
+Use `--min-runs-per-condition` to raise or lower the refusal threshold. The
+default threshold is 6 candidate runs per condition.
+
+## WandB Sink
+
+The harness can use WandB as a metrics sink. Each benchmark candidate run becomes
+one WandB run in a shared group. WandB failures do not fail the benchmark; the
+benchmark JSON is still written.
+
+WandB mode defaults to disabled. Use `WANDB_MODE`, `NSS_WANDB_PROJECT`, or
+`WANDB_PROJECT` consistently with the rest of Safe Synthesizer.

mkdocs.yml

@@ -194,6 +194,7 @@ nav:
       - Example Generation: developer-guide/example-generation.md
       - Observability: developer-guide/observability.md
       - Preflight Plugins: developer-guide/preflight-plugins.md
+      - vLLM Benchmark Harness: developer-guide/vllm-benchmark.md
       - API Reference: reference/
   - Dev Notes:
       - dev-notes/index.md

src/nemo_safe_synthesizer/config/generate.py

@@ -5,7 +5,7 @@
 
 import warnings
 from collections.abc import Mapping
-from typing import Annotated, Any, ClassVar, Literal, Self
+from typing import Annotated, Any, ClassVar, Literal, Self, cast, get_args
 
 from pydantic import (
     BaseModel,
@@ -27,15 +27,32 @@
 ResolvedStructuredGenerationSchemaMethod = Literal["regex", "json_schema", "structural_tag"]
 StructuredGenerationBackend = Literal["auto", "xgrammar", "guidance", "outlines", "lm-format-enforcer"]
 
+SUPPORTED_STRUCTURED_GENERATION_BACKENDS = cast(
+    tuple[StructuredGenerationBackend, ...],
+    get_args(StructuredGenerationBackend),
+)
+"""Structured-output backend values accepted by generation config."""
+
 STRUCTURAL_TAG_COMPATIBLE_BACKENDS = frozenset({"auto", "xgrammar"})
 
+COMMON_ATTENTION_BACKENDS: tuple[str, ...] = (
+    "FLASHINFER",
+    "FLASH_ATTN",
+    "TORCH_SDPA",
+    "TRITON_ATTN",
+    "FLEX_ATTENTION",
+)
+"""Common vLLM attention backend values accepted by ``generation.attention_backend``."""
+
 __all__ = [
+    "COMMON_ATTENTION_BACKENDS",
     "GenerateParameters",
     "ResolvedStructuredGenerationSchemaMethod",
     "StructuredGenerationParameters",
     "StructuredGenerationBackend",
     "StructuredGenerationSchemaMethod",
     "STRUCTURAL_TAG_COMPATIBLE_BACKENDS",
+    "SUPPORTED_STRUCTURED_GENERATION_BACKENDS",
     "ValidationParameters",
     "resolve_structured_generation_schema_method",
     "structural_tag_backend_error_message",
@@ -265,8 +282,8 @@ class GenerateParameters(Parameters, BaseModel):
         Field(
             title="attention_backend",
             description=(
-                "The attention backend for the vLLM engine. Common values: 'FLASHINFER', "
-                "'FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION'. "
+                "The attention backend for the vLLM engine. Common values: "
+                f"{', '.join(repr(backend) for backend in COMMON_ATTENTION_BACKENDS)}. "
                 "If ``None`` or 'auto', vLLM will auto-select the best available backend."
             ),
         ),