rollout: consistent_hashing/manual routing for inference_rollout stack

[yueming-yuan]

Problem

The consistent_hashing router policy needs every generation request to carry an X-SMG-Routing-Key header; a request without the key silently degrades — the router's implicit-key fallback can hash a constant authorization header and pin all traffic to a single engine (the failure mode of #1657). The client half of this contract was only implemented in the legacy stack (#891): sglang_rollout.generate_and_rm_group assigns per-sample keys and generate() attaches the header. The inference_rollout stack has neither — which is why --sglang-router-policy was hard-blocked under MILES_EXPERIMENTAL_ROLLOUT_REFACTOR=1. The eval paths of both stacks also never assign keys, and #1690 is about to make manual a second routing-key-based policy with its own scattered == "consistent_hashing" checks.

Fix

• Rename Sample.session_id → Sample.routing_key (review feedback): the field is the per-sample routing key sent as X-SMG-Routing-Key, not a session — the new name disambiguates it from the session server's and the p2p transfer engine's session ids. Renamed repo-wide.
• One predicate for all sites: policy_uses_routing_key(args) (consistent_hashing or manual) in generate_endpoint_utils, used by every key site — both stacks' group assignment and eval, compute_routing_headers, prefill-recompute batch gating and per-sample headers. The legacy generate() and prefill per-sample path now reuse compute_routing_headers instead of hand-building the header. This closes the coordination gap with [router] set manual policy (sticky + min_load) as default agentic routing policy #1690: manual gets keys on all paths, not just the two legacy-stack sites, and a future key-based policy is a one-line change.
• inference_rollout_common.generate_and_rm_group assigns a per-sample routing_key (mirrors the legacy stack); generate_hub.single_turn / generate_hub.multi_turn attach the header.
• Both stacks' eval_rollout_single_dataset mint an independent uuid per eval sample, so eval traffic spreads across engines while each sample stays pinned.
• Drop the MILES_EXPERIMENTAL_ROLLOUT_REFACTOR=1 restriction on --sglang-router-policy.

Enforcement lives router-side: radixark/sgl-router-for-miles#8 rejects keyless requests with 400 missing_routing_key under consistent_hashing/manual policies (escape hatch: --allow-requests-without-routing-key). The router is the only chokepoint that sees every client — including proxy layers and raw-aiohttp callers that bypass miles' http utils — so this PR carries no client-side check.

Compat notes for the rename: pickled Samples in old partial-rollout buffers / debug dumps restore the stale session_id attr; the new field defaults to None and gets a fresh routing key on the next submit — affinity resets once across restart, harmless. Out-of-repo code reading sample.session_id must switch to sample.routing_key.

Validation

• Smoke tests: policy_uses_routing_key truth table (consistent_hashing/manual → true; None/cache_aware → false); compute_routing_headers attaches the header iff predicate + key set; batch prefill scoring disabled for both key policies; reset_for_retry preserves routing_key.
• pre-commit run --all-files passes. Generate paths are unchanged when the policy is off (headers=None as before).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request implements sticky routing for the consistent_hashing router policy by generating a unique session_id for each sample and passing it as an X-SMG-Routing-Key header in generation requests. It also adds a check in the HTTP utility to ensure this header is present for relevant endpoints when consistent hashing is enabled. The review feedback suggests replacing an assert statement with a ValueError in the routing key validation check to prevent it from being bypassed when Python is run with optimizations.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

According to the project guidelines, ValueError should be used instead of assert for validating function arguments. Using assert can be bypassed if Python is run with optimizations (-O), which would silently disable this critical routing check.

Suggested change

	assert headers is not None and any(k.lower() == _ROUTING_KEY_HEADER.lower() for k in headers), (
	f"consistent_hashing router policy requires an {_ROUTING_KEY_HEADER} header on {parts.path} requests, "
	f"but none was attached (url={url}). Set sample.session_id before generating so the generate "
	f"function can attach the routing key."
	)
	if headers is None or not any(k.lower() == _ROUTING_KEY_HEADER.lower() for k in headers):
	raise ValueError(
	f"consistent_hashing router policy requires an {_ROUTING_KEY_HEADER} header on {parts.path} requests, "
	f"but none was attached (url={url}). Set sample.session_id before generating so the generate "
	f"function can attach the routing key."
	)

References

1. Use ValueError instead of assert for validating function or constructor arguments (such as checking for positive or non-negative values).

[guapisolo]

To avoid confusion with session server's session id, could you also modify the sample.session_id to sample.sglang_router_session_id?

[guapisolo]

Good fix.
I think the session server should also generate a unique hash key, and append it into all request header. Then the harness has no need to care about the headers? What's your point.

[yueming-yuan]

@guapisolo yes if the user is using the session server then it's perfect. But in case some users might vibe their own session server, and they might not know adding this key..

[guapisolo]

LGTM

[yueming-yuan]

Update: dropped the client-side check from this PR — keyless-request enforcement moved to the router, where it covers every client (session server, custom proxy layers, raw aiohttp callers) instead of only requests going through miles' http_utils: radixark/sgl-router-for-miles#8. Both consistent_hashing and manual policies now reject keyless requests with 400 missing_routing_key by default; --allow-requests-without-routing-key disables the check.

[Zhichenzzz]

LGTM!