Random fully async agent example

[jthomy]

Random fully async agent example as a minimal example for agentic RL(hicache, disaggregated setup, EP, etc).

EDIT: The patches are temporary necessary fixes, I believe they should be commited to Miles' SGLang branch instead.

Expected throughput:

[gemini-code-assist]

Code Review

This pull request introduces a new 'random fully-async' example designed to stress-test agent infrastructure by simulating multi-turn rollouts with random tokens and rewards, bypassing the need for real datasets. It includes a suite of temporary SGLang patches to resolve issues with attention backends, synchronization, and memory management, alongside a multi-node runner script and configuration. Feedback focuses on improving the efficiency of the asynchronous worker loop by addressing blocking queue operations and slow concurrency ramp-up. Additionally, it is recommended to avoid hardcoding model dimensions like vocabulary size and absolute file paths to improve the portability and flexibility of the example.

[lawrence-harmonic]

Purpose of patches:

No tbo support (patch 00)

Fixed in sgl-project/sglang#20114

Weight sync with disagg wasn't pausing the engines (patch 01)

Should be fixed in sgl-project/sglang#24462 (haven't verified)

Mamba indices need to be on Cuda (patch 4)
Race bug where data is registered only on froward path but used in load/write streams resulting in corrupted pointers (patch 13)

Fixed in:
sgl-project/sglang#22940
sgl-project/sglang#23427

OOM on startup although the host has enough RAM, removing strict NUMA policy (patch 15)
At least in our cluster, SGLang crashes with a certain probability on Inconsistency detected by ld.so: ../elf/dl-tls.c: 1044: _dl_add_to_slotinfo: Assertion `idx == 0' failed!  - added support to preload it which appears to fix it.
Weight sync with multi tokenizer pool (this is necessary for good speed if you need tokenization) appears to be broken with weight sync. In our case we can simply disable tokenizer init as we are using token in token out for our RL (more on policy anyway)

Needs upstream fix?

[lawrence-harmonic]

At least in our cluster, SGLang crashes with a certain probability on Inconsistency detected by ld.so: ../elf/dl-tls.c: 1044: _dl_add_to_slotinfo: Assertion `idx == 0' failed! - added support to preload it which appears to fix it.

This turns out to be a rare bug in glibc, which we have produced a repro using only dlopen: https://gist.github.com/lawrence-harmonic/995d2ef964e2f40af1d81205d7285355

LD_PRELOAD is not the real fix, it only avoids the issue since the issue is sensitive to load order (dlopen must fail at multiple of 62). We will report the issue to glibc mailing list.

Remaining issues with this test script:

OOM on startup although the host has enough RAM, removing strict NUMA policy (patch 15)

Weight sync with multi tokenizer pool (this is necessary for good speed if you need tokenization) appears to be broken with weight sync. In our case we can simply disable tokenizer init as we are using token in token out for our RL (more on policy anyway)

[jthomy]

I removed the patches. All patches are solved in sgl-project/sglang#25201 and additionally for R3 in sgl-project/sglang#24908

[Zhichenzzz]

Most LGTM! Just a small nit here! Thanks @jthomy!