[listar2000]

Merge with rebase, not squash: fix-016 and its patch page are two commits by ledger rule.

Outcome

Score centering stops slowing every model call: per-call overhead 20.4 s → 3.9 s on a 1-node GPU probe (3.2 s with no top-k at all). Each engine's HTTP process spent about a second of CPU per 2,000-token reply formatting the candidate tokens, and every call queued behind that. This PR asks SGLang for the same candidates in its compact binary format. It is on by default for every score-centering run through the session server, with no flag to set.

Terms: score centering is a loss that needs, for every generated token, the engine's top-k candidate tokens and their log-probabilities (top-k logprobs, k=32 here). The session server is Miles' OpenAI-compatible proxy that agents call during rollout. The tokenizer manager is SGLang's single Python HTTP process per engine.

1-node replay, 128 sessions, k=32	Before	This PR	No top-k
Fixed overhead per call	20.4 s	3.9 s	3.2 s
Calls finished in 16 min	3,154	4,637	4,940
Tokenizer manager CPU	~98% of a core	~30%

CPU replay of the engine's response code (sglang-miles 7b6993a, the engine in our image), k=32:

Reply length	Engine CPU per call, nested (before)	Flat (this PR)	No top-k	Reply size, before → PR
500 tokens	0.23 s	0.007 s	0.005 s	2.0 → 0.24 MB
2,000 tokens	1.02 s	0.019 s	0.019 s	8.2 → 1.0 MB
8,000 tokens	4.34 s	0.062 s	0.057 s	33 → 3.9 MB

Cause. With top_logprobs=32 on /v1/chat/completions, SGLang turns on return_text_in_logprobs. For every generated token, the tokenizer manager then calls tokenizer.batch_decode on 32 candidates. _build_token_logprobs_from_raw builds 32 TopLogprob objects with byte lists, and both copies (choices[].logprobs and meta_info.output_top_logprobs) get serialized.

Fix. SGLang's own option return_flat_raw_output_top_logprobs with return_flat_raw_top_logprobs_b64 (sgl-project/sglang#42890) returns the candidates once, as base64 int32 ids and float32 logprobs, built straight from the scheduler's arrays.

Patch page: fork/patches/fix-016-flat-topk-logprobs.md

Contract

• Requires SGLang with flat top-k logprobs ([Feature] Return output top logprobs as flat base64 arrays on request sgl-project/sglang#42890; in the image's sglang-miles) and a router that forwards the flag (fix(protocol): forward flat output top logprobs for chat completions radixark/sgl-router-for-miles#24). With an older one, score-centering runs fail at sample collection with "Requested flat output top logprobs but the response has none".
• Agents that ask the session server for logprobs during score-centering training now get empty top_logprobs lists (Harbor agents do not ask).
• Recipe fields, flags and defaults are unchanged.

Side-effects

• Support mode (--rollout-sampling-logprobs-mode support) and the native /generate path are unchanged. /generate never decodes candidate text.
• Upstream files touched, marked [fork fix-016]: miles/rollout/generate_utils/rollout_topk_logprobs.py (pure insertions plus the flat keyword on append_rollout_topk_logprobs), miles/rollout/session/samples/merge.py, and the two score-centering test files.
• [parked] ext(disagg): run the trainer in its own Modal app, joining late or per step [fork ext-021] #36 also appends a row to fork/patches/README.md; whichever PR lands second keeps both rows.

Done

• configure_rollout_topk_logprobs_request sets both flat flags on OpenAI (session) training requests in "selected" mode and drops client-sent copies elsewhere, because the rollout args own these fields.
• append_rollout_topk_logprobs(..., flat=True) decodes the base64 arrays, honoring SGLang's shape, null_prefix and dtype markers. The session producer passes flat from the backend request.
• Tests: the session producer test runs on both nested and flat replies and validates the same sample, and a flat request with no flat reply raises.

Verification

• python fork/tools/ledger.py check: ok
• pre-commit run --files <changed>: all pass
• pytest --noconftest tests/fast/rollout/session/test_score_centering.py tests/fast/rollout/generate_utils/test_score_centering_contracts.py: 70 passed. The wider tests/fast/rollout/session + generate_utils run gives 411 passed, against 410 on main, with the same environment failures.
• A CPU benchmark of SGLang's response code paths (not in the repo; it checks that flat and nested carry identical candidates) produced the second table.
• Router forwards the flag, checked in the current trainer image (main b021994): strings /usr/local/bin/sgl-model-gateway | grep -c return_flat_raw_output_top_logprobs → 1, and the same grep on sglang_router_rs.abi3.so (sglang-router 0.3.2) → 2.
• GPU probe on this head (1 node of 8 B300, 128 recorded score-centering sessions replayed, 16 min): numbers in the first table, and zero "Requested flat output top logprobs" errors.