[arun-elr]

Fixes #462

What does this PR do?

A malformed MSTAR_LAUNCH_WAIT_MS or MSTAR_MAX_CONSECUTIVE_SPEC_STEPS raised only after weight load, warmup, capture and SETUP_DONE, and then took the deployment down. The worker now reads both when it is built, before anything loads, and handles a bad value the way MSTAR_PY_SWITCH_INTERVAL_SEC and MSTAR_RECOMPILE_LIMIT already do:

• Malformed (not a number, or for the wait nan, inf or more than threading.TIMEOUT_MAX seconds): a warning, and the default is used.
• Below the minimum (0 ms for the wait, 0 for the cap): a warning, and the minimum is used.
• A cap of 0 still turns speculation off, now with a startup warning that says so, and the docs row says so too.
• Docs: MSTAR_LAUNCH_WAIT_MS gets the row in docs/environment_variables.rst it never had (default, units and the handling above).

Every value that worked before gives the same result.

Schematic, not to scale: both knobs are now read before anything loads, and a bad value warns and falls back instead of raising after SETUP_DONE.

How was it tested?

test/modular/test_worker_spec_knobs.py: 13 CPU tests, added to test/cpu-core.txt, running the real Worker.__init__; all 13 fail on main, which reads neither knob in __init__. ruff check . is clean and the file passes 13/13 on the head (rebased on main). @test/cpu-core.txt with the GPU hidden: 1307 passed, 46 skipped, before the rebase.

Not tested

@test/cpu-core.txt was not rerun after the rebase: the rebase only merged test/cpu-core.txt and the new commit is docs only.

Details

• _env_number(name, default, parse, minimum) is a small module helper. _wait_ms rejects nan, inf and anything past threading.TIMEOUT_MAX seconds, where Event.wait overflows.
• Both are read in Worker.__init__ right after worker_id, device and NVTX are set, before the communicator, the graph runtime and EngineManager.build. run() uses the stored values.
• Warn and fall back rather than raise, to match the two knobs above.
• 0 keeps its meaning rather than being rejected, since "cap on back-to-back speculative steps" of 0 literally means none, and it is a way to A/B speculation.

Tests

• test/modular/test_worker_spec_knobs.py: 13 CPU tests, added to test/cpu-core.txt. They run the real Worker.__init__ and stop it at the first call after the knobs, before anything is built. They cover:
  • defaults and valid values;
  • malformed values for each knob, including nan, inf, 1e20, 1e3 and 16.0;
  • negative values raised to the minimum;
  • the cap-of-0 warning;
  • Worker.run no longer reading either variable.
• All 13 fail on main, which reads neither knob in __init__.
• ruff check . is clean and test/modular/test_worker_spec_knobs.py passes 13/13 on the head (rebased on main).
• @test/cpu-core.txt with the GPU hidden: 1307 passed, 46 skipped, before the rebase. Not rerun after it: the rebase only merged test/cpu-core.txt and the new commit is docs only.

Checklist

• ruff check . passes
• Added or updated tests / docs where relevant

AI assistance: written with AI coding agent

[arun-elr]

/review

[github-actions]

Automated review

The worker now reads MSTAR_LAUNCH_WAIT_MS and MSTAR_MAX_CONSECUTIVE_SPEC_STEPS in Worker.__init__ instead of in run() after SETUP_DONE. Each knob goes through a new _env_number helper: a malformed value logs a warning and falls back to the default, and a value below the minimum logs a warning and is clamped to it. The PR also adds 13 CPU tests registered in test/cpu-core.txt. The code matches the description.

Findings: none.

I checked the claim that "every value that worked before gives the same result" against how the values are used:

• A negative cap such as -1 was accepted before. consecutive >= fair.max_consecutive is always true for both -1 and 0 (mstar/worker/worker.py:3716), so clamping to 0 changes nothing.
• A negative wait passed to Event.wait returns immediately, the same as 0.
• 1e20 used to raise OverflowError and 1e3 used to fail int(), so neither was a working value before.

The test count of 13 matches the file.

The description doesn't mention that MSTAR_LAUNCH_WAIT_MS still has no row in docs/environment_variables.rst (AGENTS.md invariant 9). The gap predates this PR. But the PR changes how that knob handles bad values and updates the row for its sibling knob, so adding a row for it here would be cheap.

---

_{Advisory, not a merge gate. Rules: AGENTS.md. Reply to discuss.}

[NSagan271]

🤖 Automated live test, AI-generated (Claude Code). Not checked line by line by a human. Numbers come from one allocation, so treat small deltas as provisional.

Live test: [Worker][Bug fix] Read the speculation knobs when the worker is built, not after warmup

Works. On base a malformed knob kills the deployment after weight load and CUDA-graph capture; on the PR it warns at startup, uses the default, and serves. No CPU-overhead regression outside noise.

1x H200, Qwen3.5 0.8B · PR head e98e2800 vs base 7fa8b8e7 (merge-base)

Claims

Claim	Result
A malformed MSTAR_LAUNCH_WAIT_MS no longer takes the deployment down after setup	✅ =5ms. Base: 76 CUDA graphs captured, then the worker dies in run() (ValueError: could not convert string to float: '5ms') and the API server's warmup requests fail. PR: "Ignoring invalid MSTAR_LAUNCH_WAIT_MS='5ms'; using 5.0", 16/16 served.
The same for MSTAR_MAX_CONSECUTIVE_SPEC_STEPS	✅ =1e3. Base: the same death after capture (invalid literal for int()). PR: warns, uses 1024, 16/16 served.
A cap of 0 still turns speculation off, with a warning	✅ PR: the warning is logged, 16/16 served
CPU tests	✅ test_worker_spec_knobs.py 13/13 fail on base, 13/13 pass on the PR

Performance

Qwen3.5 0.8B, closed loop, base, PR, PR, base, text tok/s, two jobs:

arm	c=8 (40 req)	c=32 (160 req)	c=128 (640 req)
base, job 1	2817 / 2769	6891 / 6767	10017 / 10632
PR, job 1	2723 / 2906	7335 / 6822	9752 / 8713
base, job 2	2901 / 2929	7210 / 7376	10325 / 10127
PR, job 2	3026 / 2522	7105 / 7347	10484 / 10252

• No regression. Job 1's low PR c=128 run (8713, −10.6% on the mean) didn't repeat: job 2 is +1.4% at c=128. Each job has one low PR run in a different cell (c=128 in job 1, c=8 in job 2), the noise pattern of this node. The diff only moves two env reads from run() into __init__, nothing per step.

Caveat: node load average 20–96.

Client command

python -m benchmark.runner --model qwen3.5 --model-id Qwen/Qwen3.5-0.8B --request-type text_to_text --ignore-eos --output-len-min 128 --output-len-max 384 --output-len-seed 0 --profiling-type closed_loop --num-requests <5*c> --num-warmup 1 --max-concurrency <c>

Other

• Not tested: nan/inf/huge values served (CPU tests cover them).

[merceod]

Ran main 84fd4a6 + e98e280 (463m, the only conflict is test/cpu-core.txt) against main on a klone H200 (clock-limited node), Qwen3.5-0.8B.

• MSTAR_LAUNCH_WAIT_MS=abc. Main captures 76 CUDA graphs, the conductor counts the worker ready, and 322 s after the first log line the worker dies in run() with "could not convert string to float: 'abc'", both warmup requests fail and the deployment shuts down. 463m logs "Ignoring invalid MSTAR_LAUNCH_WAIT_MS='abc'; using 5.0" before "Engine loaded" and serves 8/8.
• -5 is raised to 0.0 with a warning, MSTAR_MAX_CONSECUTIVE_SPEC_STEPS=0 warns that it turns speculation off and serves, both messages name the variable.
• Throughput. Cap 0 against the default at c=16 costs about the same on both trees (main 2156 to 1888 out tok/s, -12%, 463m 1920 to 1754, -9%, ranges overlap), as it should since the PR only moves where the value is read. MSTAR_LAUNCH_WAIT_MS=20 at c=32, 3037 vs 3096, within noise. 0 errors in every launch.
• 13/13 new tests, 13 fail on main. worker.py has no main commits since your base. cpu-core failing set equals main's, ruff clean.
• Reading agrees, both knobs are read in init right after worker_id, device and NVTX, before the communicator and the engine build, run() uses the stored values, and _wait_ms rejects nan, inf and anything past TIMEOUT_MAX.

One note, inline. MSTAR_LAUNCH_WAIT_MS still has no row in docs/environment_variables.rst. Pre-existing, but since this PR changes how it parses and touches the sibling row, this is the cheap moment to add it.

Approving. Needs the rebase for cpu-core.txt.

[merceod]

MSTAR_LAUNCH_WAIT_MS still has no row here. Pre-existing gap, but this PR changes how it parses bad values and already touches this sibling row, so adding it now would be cheap (default 5 ms, non-finite or past TIMEOUT_MAX falls back with a warning, below 0 is raised to 0).

[arun-elr]

Thanks for running this. Added the row (default 5 ms; a non-number, nan, inf or a value past threading.TIMEOUT_MAX falls back to 5 with a warning; a negative value is raised to 0), and rebased on main for test/cpu-core.txt.

[merceod]

Thanks. The row reads right, and 113df46 only adds it and rebases, the worker.py and test hunks are byte-identical to e98e280 against each base, so the approval stands.