chore(apertus): recipe registration, minimizer override fix, and CI repairs - #13
Merged
Conversation
test_all_recipe_yamls_accounted_for_in_test_suites has failed since the Apertus recipes landed (216 yamls vs 205 driver scripts), and with maxfail=1 that failure truncated the unit suite: everything after it had never run. Follows the upstream convention rather than scoping the test out: - Prunes two recipes nothing references, -gym-reasoning-alp-ngram and -gym-reasoning-overlong. - Renames the probe family to satisfy the algo-prefix rule the same module enforces (test_all_recipes_start_with_algo_hyphen): probe-grpo-apertus1p5-* becomes grpo-apertus1p5-8b-1n4g-megatron-probe[-*], with `probe` as a modifier. Recipe defaults chains, the three CSCS launchers and the docs follow the rename. - Adds a driver per remaining recipe under tests/test_suites/llm/, named to match its yaml because common.env derives the config path from the script name, and lists them in disabled.txt: their model, tokenizer and container paths are CSCS-specific, so nightly automation cannot run them. They are certified by the launchers in infra/slurm/cscs/ instead. common.env now skips the audio-dependency install under TEST_DRYRUN. The accounting test invokes every script that way as a cheap config check, but scripts whose filename matches (omni|audio|avqa|vlm) were triggering an apt install that needs root; ours are text-reasoning recipes on an omni checkpoint, so the name matched while the workload never needed it. Recipes and drivers are now 181 to 181, all 181 scripts dry-run clean, and the unit suite runs to completion for the first time: 1243 passed, 14 skipped. Signed-off-by: Alvorecer721 <yixuan.xu@ai.ethz.ch> (cherry picked from commit feb9234)
Apply the configured Ruff and recipe-minimization rules and retain the xIELU registration import explicitly. (cherry picked from commit 1500548, omitting the rollouts truncation-metric hunk that ships in PR #12 as a298c69 and the vllm0251 probe-script hunks whose file arrives with PR #12) Signed-off-by: Alvorecer721 <yixuan.xu@ai.ethz.ch>
Add the now-clean Math500 dataset module to Pyrefly's enforced project-includes list. Signed-off-by: Alvorecer721 <yixuan.xu@ai.ethz.ch> (cherry picked from commit ea1ae31)
Register the CI-confirmed clean RL prompt dataset and make the whitelist counter safe under bash -e so one run reports every missing entry. Signed-off-by: Alvorecer721 <yixuan.xu@ai.ethz.ch> (cherry picked from commit dd35ff1)
Add the final x86 CI-confirmed zero-error module to the Pyrefly project whitelist. Signed-off-by: Alvorecer721 <yixuan.xu@ai.ethz.ch> (cherry picked from commit b2b204d)
Keep replacement subtrees intact so minimize-check cannot strip required values from _override_ sections. Add direct and semantic round-trip coverage, then regenerate the affected Apertus omni recipe. Signed-off-by: Alvorecer721 <yixuan.xu@ai.ethz.ch> (cherry picked from commit 7b137e9)
Mark the two CSCS checkpoint paths as reviewed detect-secrets false positives so the changed-files CI scan passes. Signed-off-by: Alvorecer721 <yixuan.xu@ai.ethz.ch> (cherry picked from commit e3ea186)
The 3-step smoke inherits checkpointing.enabled from the 8-node recipe, and the mandatory end-of-training save OOMs the single node: the fp32 distributed-optimizer state of the 8B model cannot be host-staged without the 8-way sharding the full recipe provides (raylet killed the workers mid-save, job 3075160). Signed-off-by: Alvorecer721 <yixuan.xu@ai.ethz.ch> (cherry picked from commit 1bef52a)
github-actions
Bot
added
documentation
Signed-off-by: Alvorecer721 <yixuan.xu@ai.ethz.ch>
Alvorecer721
changed the title
The repo root now points CSCS users at the quickstart instead of only the generic cluster docs, and the quickstart advertises the shared vllm-0.25.1 certified image and its shipped EDF. Signed-off-by: Alvorecer721 <yixuan.xu@ai.ethz.ch>
Alvorecer721
marked this pull request as ready for review
August 14, 2026 05:38
Resolves the rename conflict on the async probe recipe: the old-named probe-grpo-apertus1p5-8b-1n4g-megatron-async.yaml (deleted by the recipe renaming in this branch) had gained vllm_cfg.tensor_parallel_size: 2 from 71ad3bc on main; that TP2-async certification override is ported into the renamed grpo-apertus1p5-8b-1n4g-megatron-probe-async.yaml. Signed-off-by: Alvorecer721 <yixuan.xu@ai.ethz.ch>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_override_: truesections:_prune_equal(tools/config_cli.py#L243-L258 at 150054886) pruned keys equal to the base even inside replace-don't-merge blocks, sominimize-check's--in-placeremediation silently deleted required keys from the omni ALP recipe — including the recipe's own warning comment about exactly this contract (recipe L35 at 103bc8896). Result: crash at launch (KeyError: 'max_input_seq_length') plus silent loss of the MATH-500 CoT prompt wrapper via the no-warning guard at nemo_rl/data/utils.py#L254. Fixed in d0f50c1 with direct + semantic round-trip regression tests, and the recipe regenerated by the fixed tool3075160); the full recipe shards optimizer state across 8 nodes# pragma: allowlist secretcomments (503d96e) stripped from dpo-apertus1p5-8b-maxmin-megatron.yaml#L24 and grpo-apertus1p5-8b-1n4g-megatron-probe.yaml#L14Upstream provenance
_override_minimizer bug is live on upstreammain:_prune_equalat 7cba25ac has no_override_handling while the merge contract pops override sections (nemo_rl/utils/config.py#L34-L41). The fix here is a clean single-mechanism commit held for a future upstream PRValidation
minimize-checkpasses on allexamples/configs/recipes/{llm,vlm}YAMLs on this branchconfig_cli.py expandat150054886^vs the regenerated file, verified 2026-08-13)3076512, COMPLETED 0:0, 28:44); the prior run reproduced the checkpoint OOM this fixes (job3075160)test_async_utils.py+test_config_cli.py, Slurm job3076335)Assembly notes
max_generation_failuresintroduced by the fail-fast work-xprovenance fromapertus/v0.7.0; the lint-gate commit omits itsnemo_rl/experience/rollouts.pytruncation-metric hunk (ships in build(vllm): upgrade to 0.25.1 and make CSCS images reproducible #12 as a298c69eb) and theprobe_nemo_rl_vllm0251_image.pyhunks (that file arrives with build(vllm): upgrade to 0.25.1 and make CSCS images reproducible #12)🤖 Generated with Claude Code