Skip to content

fix(ci): use ++ for token_capture override in SC functional tests - #4031

Closed
terrykong wants to merge 1 commit into
ci/bump-mbridge-5ed9799-20260904-070613from
fix/sc-token-capture-override
Closed

fix(ci): use ++ for token_capture override in SC functional tests#4031
terrykong wants to merge 1 commit into
ci/bump-mbridge-5ed9799-20260904-070613from
fix/sc-token-capture-override

Conversation

@terrykong

Copy link
Copy Markdown
Collaborator

What does this PR do ?

Unblocks L1_Functional_Tests_SingleController (and its GB200 twin) on #4022.

Both jobs fail before training starts:

omegaconf.errors.ConfigKeyError: Key 'token_capture' is not in struct
hydra.errors.ConfigCompositionException: Could not override 'token_capture.enabled'.
nemo_rl.utils.config.OverridesError: Failed to parse Hydra overrides: Could not override 'token_capture.enabled'.

L1_Functional_Tests_SingleController.sh passes token_capture.enabled=true as a plain Hydra
override, but the config that test runs — examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml
has no token_capture block. Hydra composes in struct mode, so overriding a key that is not in
the YAML is an error. MasterConfig.token_capture having a Pydantic default does not help,
because that default is applied after Hydra composition.

The fix is the ++ (add-or-override) prefix, which is how this same script already passes the
other SingleController-only knobs into this shared Gym config — ++data_plane.*,
++async_rl.*. One line.

Details

This is not a #4022 regression. It reproduces on main:

It came in with #3837 (7036e5d16), which added the test line and the token_capture feature but
did not add the key to the Gym config the test runs. Targeting #4022's branch because that is what
the failure is currently blocking; happy to retarget at main instead if you would rather it land
independently.

Why not add token_capture: to the YAML instead? That config is shared by eight other
functional scripts, most of them not SingleController, so declaring an SC-only block there has a
wider blast radius than the caller-side ++. Either fix works — say the word if you prefer the
config-side one and I will switch it.

The test still asserts what it is meant to. grpo_async_gym_single_controller.sh gates its
extra check on the argument string:

if [[ "$*" == *token_capture.enabled=true* ]]; then
    EXTRA_CHECKS+=('max(data["train/finalize/total_ms"]) > 0')
fi

++token_capture.enabled=true still matches that glob, so the finalizer assertion keeps running —
the test is not being quietly turned into a no-op.

Verification

Reproduced and fixed locally, CPU only, against the real config loader:

token_capture.enabled=true    -> FAILED: Could not override 'token_capture.enabled'.
++token_capture.enabled=true  -> OK, token_capture.enabled = True

(driving nemo_rl.utils.config.load_config + parse_hydra_overrides on
examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml, which is the exact path that raises in CI)

The glob check above was verified separately in bash. The GPU run itself is not something I can
execute — that needs CI.

Issues

Unblocks CI on #4022.

The SingleController L1 functional test passes token_capture.enabled=true
as a plain Hydra override, but examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml
does not declare a token_capture block. Hydra composes in struct mode, so the
run dies before training starts:

    omegaconf.errors.ConfigKeyError: Key 'token_capture' is not in struct
    hydra.errors.ConfigCompositionException: Could not override 'token_capture.enabled'.

Use the ++ (add-or-override) prefix, matching how the same script already
passes the other SingleController-only knobs to this shared Gym config
(++data_plane.*, ++async_rl.*). The downstream

    if [[ "$*" == *token_capture.enabled=true* ]]

check in grpo_async_gym_single_controller.sh still matches, so the
finalize/total_ms assertion continues to run.

Signed-off-by: Terry Kong <terryk@nvidia.com>
@terrykong
terrykong requested a review from a team as a code owner September 6, 2026 23:14
@copy-pr-bot

copy-pr-bot Bot commented Sep 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@terrykong

Copy link
Copy Markdown
Collaborator Author

Closing — @cspades landed the same fix independently in 28f4d08. No longer needed.

@terrykong terrykong closed this Sep 7, 2026
@terrykong
terrykong deleted the fix/sc-token-capture-override branch September 7, 2026 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant