Repository navigation
feat: add Arena single- and multi-stream rollouts - #1702
Conversation
Run Arena Harness tasks through the existing rollout proxy, with weighted prompt mixtures, per-stream rewards, session-bound gateway routing, failure classification, and worker-scoped registration cleanup. Preserve main's processor cache and fail-fast grouped cancellation. Include portable two-node FSDP/SGLang examples and integration guidance. Keep PRM, RewardSystem, AWEX, and mean-only normalization out of scope. Port selected changes from swe-dev: - da1da65c7: initial Arena Stream integration - 1d0fc7ba0: configurable reward transforms - 714f733a0: episode and grouped metrics - 586d7cc06: separate task and proxy timeouts - 84f7bb341: recover typed context overflow - ddb7d7f3c: example-level continuous reward transform - 2b994c10c: preserve original rewards for debugging - 9ecbde97c: multi-stream routing and lifecycle support Validation: 227 targeted tests passed; full pre-commit stack run.
| messages = kwargs.get("messages") | ||
| if messages is not None: | ||
| try: | ||
| prepared_messages = _prepare_request_messages(messages) |
There was a problem hiding this comment.
When transitioning from anthropic to OpenAI, a processor is executed once. It looks like it will be executed again here; will this cause any issues when the prompt is processed repeatedly, such as some content being inserted twice?
There was a problem hiding this comment.
You are right: the Anthropic path ran the configured message preprocessors during translation and again in _call_client_create, so a non-idempotent preprocessor could insert content twice. Fixed in fceab48: translation no longer receives the preprocessors; the shared request path applies them once for both APIs.
Added an endpoint regression using a preprocessor that appends an instruction, covering OpenAI and Anthropic requests in both streaming and non-streaming modes. The two Anthropic cases failed before the fix with two calls; all four pass after it. Related proxy/streaming/translation tests: 41 passed without Arena credentials and with external socket connections blocked. Full pre-commit passes.
GitHub Actions
Failure
GitHub Actions
Active
GitHub Actions
Error
GitHub Actions
Active
GitHub Actions
Active
Description
Allow Arena-hosted Harnesses to generate RL trajectories through AReaL's rollout proxy. The examples support a single Stream or a deterministic weighted mixture, with per-Stream Harness/reward settings and terminal-result audit records.
Port Arena client, dataset pagination, mixture configuration, preflight validation, reward transforms, and portable two-node FSDP/SGLang examples.
Add session-bound gateway routing and worker-scoped registration cleanup, stream heartbeats and completion-ID reward mapping, and model/system failure classification.
Preserve main's processor caching, sample identities, and fail-fast grouped cancellation. PRM/RewardSystem, AWEX, internal Astra launchers, and mean-only reward normalization are outside this port.
Include
examples/swe/submit_arena.sh: environment-configured CPU controller submission, container preflight, and optional check-only mode. Site paths, mounts, model, partition, and credentials file are supplied by environment. GPU topology stays in the selected YAML.Related Issue
No linked issue. Selected port from the Arena integration and multi-Stream work on
swe-dev; source commits and scope are documented inexamples/swe/README_arena.md.Type of Change
Checklist
main/review-prcommand/create-prValidation
After removing launcher mocks, YAML profile assertions, and preflight import smoke tests (
1453cb29c), the retained changed test files pass without Arena credentials and with external Python socket connections blocked: 216 passed, 3 skipped. The skips are pre-existing IPv6 route-dependent cases; those tests remain at the user's request. Arena requests use mock transports or test doubles, and require no live Arena service or token. Full pre-commit passes.The retained tests cover Arena routing, reward mapping, retries, cancellation, failure classification, configuration redaction, cleanup, port randomness, and uneven microbatch execution. The two-process CPU regression uses local Gloo/DDP collectives and requires no GPUs or Arena installation.
Distributed debugging and validation
This integration also exercises two existing core assumptions that fail with shared-node jobs and variable-turn Arena episodes:
f7e06cab0uses SystemRandom so equal training seeds no longer select the same port sequence or have their RNG advanced by network setup. Network tests: 11 passed, 2 skipped because the host lacks the corresponding IPv6 routes.e888007bapreserves rank-local token packing and pads execution with zero-contribution forwards/backwards. It reuses valid inputs, skips dummy result/loss/stat callbacks, and keeps real output metadata and loss normalization unchanged. The shared split helper keeps synchronized behavior by default for other engines; tree packing retains its existing path. A two-process CPU regression with real DDP collectives verifies unequal counts, real output order, and accumulated gradients; that test plus two existing multimodal tests pass. SP/TP hardware validation and empty ranks remain outside this smoke test.Full local pre-commit and GitHub pre-commit pass on
e888007ba: https://github.com/areal-project/AReaL/actions/runs/34577115502/job/103191990515 .Other diagnostic findings were corrected in the example or private site configuration: the old model-gateway credential returned HTTP 401, while the control-plane bearer passed gateway authentication; SGLang rejects total tokens equal to its context length, so
034b24265reserves one token of headroom; private NCCL interface settings are explicit; DP8 requires eight accepted prompt groups, not four groups multiplied by two episodes. No credentials or site paths are committed.Standard preflight currently blocks both Streams because each has 100/100 recent failures. Diagnostic retries explicitly enabled the existing allow_unhealthy option in a private launcher while retaining other validations. The committed launcher remains strict. An earlier batch8 run accepted all eight groups (12 Harness OK, four SWE failures recovered with a locally confirmed context-overflow marker), but exposed the FSDP packing failure above.
The FSDP retry on
e888007bacollected eight accepted groups after one rejected group was resampled. Real DP8 logprob recomputation and advantage calculation completed; backward was still running when the user requested switching acceptance to Megatron. The jobs were stopped; no optimizer step was confirmed.A two-node Megatron experiment was submitted on
1453cb29c: controller963023, trialarena-megatron-2n-20260911-163123, Qwen3-4B-Instruct-2507, actormegatron:d1p1t8, rolloutsglang:d4p1t2, two prompt groups across two Streams and two samples per group, one training-step target. The controller is RUNNING; training success is not yet established. Megatron, mbridge, and TransformerEngine imports passed in the existing runtime image. DP1 intentionally avoids the shared multi-DP packing limitation while validating Arena on Megatron; this is not multi-DP Megatron validation.Review focus: generation-only gateway credentials cannot mutate sessions; reward mapping stays aligned across streaming IDs; context-overflow recovery does not accept classified system failures; worker cleanup and grouped cancellation preserve resource lifetimes; dummy FSDP execution must not change loss normalization or returned samples. Launcher, algorithm, and FSDP experts reviewed the relevant boundaries.