perf(embedding): build embedding base64 directly from the tensor (numpy.tobytes)

[tzulingk]

Important

Recommended embedding fix. Captures the full −30–35% latency win cross-node with a one-line, behavior-preserving change. Benchmarks (below) show it matches the shared-memory path (#10220) without that PR's same-node constraint or extra failure modes, so it — not SHM — is the one to ship for text embeddings. (Complements the already-merged base64 wire-format change in #10139: that made shipping the payload cheap; this makes producing it cheap.)

Note: this branch is currently stacked on #10220; to merge standalone it should be rebased onto main (the numpy.tobytes change is independent of the shared-memory code).

Overview:

Speeds up embedding-response serialization on the worker by building the base64 payload directly from the pooling tensor via numpy.tobytes, instead of going through a Python float list + struct.pack varargs expansion.

The current path is tensor.detach().cpu().flatten().tolist() → struct.pack("<{N}f", *floats) → base64. For a batch-15 × 3072-dim response that's 46,080 Python float objects materialized and then unpacked as 46,080 positional args into struct.pack — an O(N) interpreter-level pass over every element. This PR replaces it with tensor → numpy.tobytes() → base64, a single C-level copy. The emitted base64 bytes are identical on little-endian hosts.

Measured on GB200 (Qwen3-Embedding-0.6B, dim 3072): −30% at batch 15 (124 → 87 ms) and −35% at batch 64 (470 → 307 ms) per-request latency. The win grows with batch size, since it scales with the number of floats serialized. It also works cross-node (no shared memory required) and complements the base64-wire-format change in #10139 (this PR makes producing that base64 cheaper; #10139 made shipping it cheaper).

Details:

• components/src/dynamo/vllm/handlers.py: new _pooling_output_to_base64() builds base64 straight from the tensor. Shared tensor-prep (detach().cpu().flatten().to(float32)) is factored into a small helper reused by _pooling_output_to_list, so there's no duplicated tensor handling. A list/tuple fallback preserves behavior for non-tensor pooling outputs.
• .to(torch.float32) makes bf16/fp16 pooling outputs match the struct.pack("<f") width, keeping the emitted bytes identical.

Benchmark (GB200, Qwen3-Embedding-0.6B, dim 3072, ISL 80):

Per-request latency (ms, avg). Small batches: rate-limited (no queueing), 200/100 reqs. Large batches: closed-loop --concurrency 1, 40/25 reqs.

batch	response (raw f32)	baseline (tolist+struct.pack)	numpy.tobytes (this PR)	SHM (#10220)	numpy.tobytes + SHM
15	~184 KB	124.25	86.94	85.49	85.31
64	~786 KB	469.54	306.57	290.69	294.72
128	~1.5 MB	n/m	584.31	573.98	566.64
256	~3 MB	n/m	1182.62	1128.59	1126.00
512	~6 MB	n/m	2364.84	2301.73	2200.67
1024	~12 MB	n/m	4694.72	4615.99	4447.78

n/m = not measured — the baseline's struct.pack("<{N}f", *floats) becomes pathological at these batch sizes (millions of positional args). This PR gives −30% (batch 15) to −35% (batch 64) vs baseline and tracks SHM within noise across all sizes; SHM's ~2–5% nominal edge is partly cross-node measurement noise and shrinks against the client-facing float-JSON response, which neither change touches.

Where should the reviewer start?

• components/src/dynamo/vllm/handlers.py — _pooling_output_to_base64 and its use in the embedding response loop.

🤖 Generated with Claude Code

[tzulingk]

Superseded by #10231 — the same numpy.tobytes change rebased cleanly onto main (independent of the shared-memory PR #10220), so it can merge standalone as the recommended embedding fix.