refactor(aime): drop structured runner, ship configs/aime/{run.sh,rescore.py}

[ishandhanani]

Summary

Drops the first-class type: aime runner and replaces it with two scripts under configs/aime/ that recipes invoke via type: custom. Cleaner, more reliable, easier to extend to other NeMo Skills benchmarks (GPQA / MMLU / etc. — see follow-up issue).

Why drop the runner

NeMo Skills' ns eval fans out parallel python -m nemo_skills.inference.generate subprocesses through nemo-run, which constructs unquoted bash command strings for those children. Backslash-bearing CLI overrides — including any custom ++eval_config.extract_regex=... — get stripped down to invalid regex by the time the inner shell parses them.

Three failed cluster runs walked through the consequences:

• 4836 — under-escaped regex → **Answer** collapsed to ** (invalid quantifier) → all 16 generate seeds crashed instantly → "Benchmark completed successfully" false positive.
• 4838 — doubled backslashes → still stripped one layer too few → same crash.
• 4840 — script approach (bash /configs/aime/run.sh) → real generations, no crash, real metrics.

Keeping the regex in Python source as a raw string sidesteps every shell/Hydra escape layer.

What ships

Removed

• src/srtctl/benchmarks/aime.py (91 LOC, structured runner)
• src/srtctl/benchmarks/scripts/aime/bench.sh (166 LOC, pip-install-into-container path)
• BenchmarkType.AIME enum
• BenchmarkConfig.aime_dataset schema field
• TestAIMERunner test class (4 tests)
• AIME entries in docs/config-reference.md benchmark-type table and section
• AIME mention in examples/example.yaml

Added

• configs/aime/run.sh — orchestrates ns prepare_data + ns eval against localhost:8000/v1 (the in-job dynamo frontend). Tuning knobs (MAX_TOKENS, REPEAT, NUM_THREADS, TEMPERATURE, TOP_P, SEED, DATASET, MODEL) are env-var overridable; defaults match the upstream reasoning-eval reference (max_tokens=400000, repeat=16, temperature=1.0).
• configs/aime/rescore.py — re-extracts answers from the cached per-seed output-rs<seed>.jsonl files using a broader regex (matches \boxed{}, **Answer**, final answer …, answer is/=/:). Regex is a Python raw string — no shell layers. Writes metrics-regex.json alongside NeMo Skills' default metrics.json for an easy A/B.

Updated

• docs/accuracy.md AIME section → script-based runbook with reasoning-mode env vars, container alias setup (nemo-skills), recipe shape, and a paragraph on why it's a script not a runner.
• src/srtctl/benchmarks/__init__.py — drop aime import + __all__ entry.

Recipe shape (from the new docs section)

backend:
  prefill_environment:
    SGLANG_ENABLE_THINKING: "1"
    SGLANG_REASONING_EFFORT: "max"
    # ...
  decode_environment:
    SGLANG_ENABLE_THINKING: "1"
    SGLANG_REASONING_EFFORT: "max"
    # ...

benchmark:
  type: custom
  container_image: nemo-skills    # alias from srtslurm.yaml `containers:`
  env:
    OPENAI_API_KEY: "EMPTY"
    HF_TOKEN: "${HF_TOKEN}"
    # MAX_TOKENS / REPEAT / TEMPERATURE / SEED / etc. all overridable
  command: |
    bash /configs/aime/run.sh

/configs is already mounted into the bench container by RuntimeContext, so the script and its companion are just there.

Backward compatibility

None. Recipes with type: aime will fail schema validation. Migration is 1:1: swap to the type: custom block above. The structured runner couldn't actually deliver the regex anyway, so anyone relying on it for reasoning models was getting silently-wrong (no_answer-inflated) metrics.

Test plan

• make check — 611 passed, 2 skipped. Drops 8 tests (the AIME runner suite).
• End-to-end on dsv4-pro 1P/1D (GB200), job 4840: workers healthy, 16 seeds spawn, real generation in progress with reasoning mode confirmed ('</think>' markers in output).
• (post-merge) Validate metrics-regex.json extraction recovers no_answer cases vs default metrics.json.

Follow-up

Issue to track porting GPQA / MMLU / longbenchv2 to the same configs/<bench>/ script pattern coming next.

🤖 Generated with Claude Code