[Perf] Exploit out-of-band buffers in shm_broadcast

[njhill]

Avoid serialization + copy overhead when pickled payload contains large buffers by exploiting https://peps.python.org/pep-0574.

This is beneficial for example when broadcasting bitmask arrays with the multiproc executor.

Benchmark using structured outputs + multiproc executor:

canhazgpu run --gpus 1 -- vllm serve Qwen/Qwen3-1.7B --uvicorn-log-level=error  --no-enable-prefix-caching --distributed-executed-backend mp

python3 benchmarks/benchmark_serving_structured_output.py --backend vllm --model Qwen/Qwen3-1.7B --structured-output-ratio 0.8 --request-rate 120 --max-concurrency 800 --num-prompts 5000 --json-schema-path ./test3.json  --output-len 128

Before:

============ Serving Benchmark Result ============
Successful requests:                     5000      
Maximum request concurrency:             800       
Request rate configured (RPS):           120.00    
Benchmark duration (s):                  83.96     
Total input tokens:                      2405000   
Total generated tokens:                  639936    
Request throughput (req/s):              59.55     
Output token throughput (tok/s):         7621.48   
Total Token throughput (tok/s):          36264.42  
---------------Time to First Token----------------
Mean TTFT (ms):                          327.90    
Median TTFT (ms):                        296.12    
P99 TTFT (ms):                           674.63    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          96.30     
Median TPOT (ms):                        102.40    
P99 TPOT (ms):                           106.44    
---------------Inter-token Latency----------------
Mean ITL (ms):                           96.54     
Median ITL (ms):                         92.78     
P99 ITL (ms):                            251.71    
==================================================
correct_rate(%) 99.6 

After:

============ Serving Benchmark Result ============
Successful requests:                     5000      
Maximum request concurrency:             800       
Request rate configured (RPS):           120.00    
Benchmark duration (s):                  71.28     
Total input tokens:                      2405000   
Total generated tokens:                  639937    
Request throughput (req/s):              70.15     
Output token throughput (tok/s):         8977.91   
Total Token throughput (tok/s):          42718.52  
---------------Time to First Token----------------
Mean TTFT (ms):                          251.35    
Median TTFT (ms):                        236.60    
P99 TTFT (ms):                           586.14    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          80.32     
Median TPOT (ms):                        86.29     
P99 TPOT (ms):                           90.48     
---------------Inter-token Latency----------------
Mean ITL (ms):                           80.32     
Median ITL (ms):                         78.74     
P99 ITL (ms):                            198.15    
==================================================
correct_rate(%) 99.58 

[gemini-code-assist]

Code Review

This pull request introduces a performance optimization for shared memory broadcasting by leveraging pickle's out-of-band (OOB) buffer support. This avoids serialization and copy overhead for large buffers, which is a great improvement. The implementation correctly adapts both shared memory and ZMQ communication paths to handle multipart data. However, I've identified a critical bug in the size calculation for shared memory writes that could lead to a buffer overflow and crash the writer process. A fix is suggested to address this.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR.

• @codex fix this CI failure
• @codex address that feedback

[russellb]

This is really nice!

One minor question and one minor suggestion, otherwise lgtm!

[russellb]

just curious, what led to this change? What's special about 24 MB?

[njhill]

It's just to ensure that the largest bitmasks are covered, I observed that they could be up to ~20MiB.

[russellb]

got it. It would be helpful to leave a comment in the code to explain where the number came from, and that it's "big enough based on observation" versus some observed technical limitation of the shared memory pathway.

[russellb]

I think an expanded docstring here that gives an overview of the encoding format would be helpful here.