[Perf] Reuse opaque metadata during multi-worker output fanout

[oguzserdar]

Motivation

I was reading the multi-worker output path for CPU work that could be removed without changing what the tokenizer workers receive. I ended up in the row-splitting loops and found the same batch metadata being decoded again for every request.

With MessagePack IPC, time_stats and customized_info arrive wrapped. _extract_field_by_index unwraps the whole field before picking one row. Both fanout loops call the slicer once per request, so a 32-request batch with both fields decodes the wrappers 64 times. The payload has not changed between those calls.

This patch sends the first row through the existing path, then materializes the wrapped fields once for the remaining rows. That brings the count down to four: two for the first row, two for the rest. The existing slicer still builds and wraps each outgoing row.

Where this helps is a multi-worker generation batch carrying timing or custom metadata under MessagePack IPC. Default pickle IPC does not get this benefit. Neither do embedding outputs or singleton batches.

Modifications

• Reuse wrapped metadata in the multi-detokenizer router and the detokenizer-to-HTTP-worker fanout loops.
• Keep the existing per-row slicer and wire wrappers. Leave the received batch untouched.
• Return the original object for default pickle IPC, embedding outputs and batches with at most one request.
• Add regressions to the registered CPU unit suite and a standalone benchmark using the actual detokenizer loop, a tiny local tokenizer and real ZeroMQ IPC.

Related: #42238 groups router-to-detokenizer sends. This patch removes repeated metadata decoding during row splitting and also covers downstream output fanout. It does not require that PR or #29952, #30005 or #41206. If metadata becomes natively serialized upstream, this optimization can become unnecessary.

Accuracy Tests

I reproduced the decoding count with your own row slicer and both fanout loops. Same 32-request batch, both fields present:

A. unmodified main   64 wrapper decodes per loop
B. this patch         4 wrapper decodes per loop
C. default pickle     0 wrapper decodes; no metadata materialization

The new regression fails on A and passes on B. C is the transport control: there are no opaque wire wrappers to eliminate there.

14 registered tests pass with each transport. They check full row payloads, opaque custom metadata, native scheduler timing objects, input immutability, ordering, first-send sequencing, and send-failure behavior.

PYTHONPATH=python SGLANG_USE_PICKLE_IPC=0 python test/registered/unit/managers/test_multi_tokenizer_mixin.py
PYTHONPATH=python SGLANG_USE_PICKLE_IPC=1 python test/registered/unit/managers/test_multi_tokenizer_mixin.py

The real-IPC benchmark also validates all output fields, final decoded text, per-worker order and duplicate-free FIFO fences. Each transport verified 123,480 output rows plus 4,320 fence rows, including warmup trials.

No model forward code or kernels changed. I have not run GPU/model accuracy tests.

Speed Tests and Profiling

CPU results on an Apple M1 Pro, macOS 15.0, Python 3.12.14. Reference: unmodified main at 482b062d282b2ef4e449bd108df5228d81da2112. The relevant manager and codec files also match main at af1bef3eaf723babb83df1f0e25f8cca2c5b9e58.

Nine paired rounds, three streams per arm per round, one warmup round excluded, randomized arm order. Each stream has three incremental token batches. Completion time runs from input send through decoded consumer metadata reads; validation is outside the timed region.

MessagePack IPC (SGLANG_USE_PICKLE_IPC=0):

Requests per batch	Payload	Reference median ms	Patch median ms	Median paired time reduction	Faster rounds
1	no opaque metadata	4.03	3.23	+29.4%	5/9
1	timing + custom metadata	4.40	4.12	+17.2%	6/9
32	no opaque metadata	11.18	11.74	+19.0%	5/9
32	timing + custom metadata	15.15	9.07	+37.3%	9/9
310	no opaque metadata	52.93	45.07	+6.8%	5/9
310	timing + custom metadata	612.90	51.27	+91.6%	9/9

The paired reduction is the median of per-round ratios, not the ratio of the two independently computed medians. The clear result is on metadata-bearing batches: 32 requests improved in 9/9 rounds; 310 requests improved in 9/9 rounds. A paired bootstrap of these nine rounds gives 95% intervals of 17.0–42.9% and 89.7–92.1%, respectively. These intervals describe this local experiment only.

What I still cannot tell from this CPU run is how much of a real server's end-to-end time this saves. These are synthetic output-pipeline results; no model or GPU is involved. First-output latency did not consistently improve: the 310-request metadata case had a -5.0% median paired reduction (3/9 faster; bootstrap interval -30.6–24.8%). Default pickle IPC has no metadata-reuse benefit; the 310-request metadata case measured -4.6% (4/9 faster; interval -9.8–8.7%). Those controls are noisy and do not establish the absence of a regression. The first send is structurally unchanged, but consumer-visible latency still needs a real serving check.

Reproduce from this branch with the normal SGLang Python dependencies installed; no model download is needed:

git worktree add --detach /tmp/sglang-fanout-reference 482b062d282b2ef4e449bd108df5228d81da2112
python benchmark/io/bench_multi_worker_fanout.py \
  --reference /tmp/sglang-fanout-reference/python --candidate "$PWD/python" \
  --transport msgpack --rounds 9 --repetitions 3 --output /tmp/sglang-fanout-msgpack
python benchmark/io/bench_multi_worker_fanout.py \
  --reference /tmp/sglang-fanout-reference/python --candidate "$PWD/python" \
  --transport pickle --rounds 9 --repetitions 3 --output /tmp/sglang-fanout-pickle

The benchmark saves raw paired trials, source hashes and verified output counts. Local versions included msgspec 0.22.0, PyTorch 2.14.1 and Transformers 5.17.0; Linux CI with upstream-pinned dependencies is still needed.

Checklist

• Run pinned pre-commit hooks on all five changed files. Full-repository checks remain for upstream CI.
• Add registered CPU unit tests.
• Include benchmark scope and reproduction commands here and in its CLI help.
• Run upstream serving/GPU checks; CPU output parity and CPU benchmark results are provided above.
• Follow existing row slicing, transport and test conventions.

Review and Merge Process

Ready for review of the CPU evidence. The separate upstream lint workflow passed. Upstream serving validation remains pending; an authorized maintainer needs to enable the appropriate CI. Could a scheduler merge oncall advise which multi-worker serving test is appropriate and authorize it?

---

CI States

Latest PR Test (Base): ❌ Run #37094999291
Latest PR Test (Extra): ❌ Run #37094999141
Latest PR Test (AMD ROCm 10): ❌ Run #37094999278

[oguzserdar]

@hnyls2002 I have marked this ready for review. This removes repeated decoding of timing/custom metadata in the multi-worker MessagePack output path; it does not depend on the other open metadata PRs. Registered CPU regressions pass with both transports, and the real-IPC measurements and their limits are in the description. Could you review or route this and enable the appropriate multi-worker serving CI? The earlier gate failure was because the PR was a draft; serving validation is still pending.