[mglyn]

Summary

This draft adds an opt-in transport path that removes three redundant host-side copies after video generation:

Existing path                              Experimental path
-------------                              -----------------
GPU RGB tensor                             GPU RGB tensor
      | mandatory D2H                            | mandatory D2H
      v                                          | directly into registered SHM
pinned CPU buffer                                v
      | host memcpy [removed]              CUDA-registered POSIX SHM
      v                                          | borrowed view
POSIX SHM                                        v
      | host memcpy [removed]              NumPy / Torch view
      v                                          | borrowed RGB AVFrame
private CPU buffer                               v
      | RGB plane copy [removed]           PyAV encoder -> MP4
      v
PyAV encoder -> MP4

The implementation writes CUDA output directly into registered POSIX shared memory, preserves the mapping for the lifetime of the receiver's borrowed view, and uses VideoFrame.from_numpy_buffer for eligible RGB uint8 frames (including row-padded layouts). The mandatory device-to-host transfer remains; "zero copy" refers to the eliminated host-side copies around IPC and encoding.

The existing path remains the default. Enable the experiment with:

--video-output-transport '{"enable_registered_shm": true, "enable_borrowed_frames": true}'

Video-only, opt-in: both options default to false. Registered SHM is limited to typed video media (currently WAN2.2) and explicitly marked MiniMax-H3 video tensors; audio, images, trajectories and unmarked legacy outputs retain their existing transport. Borrowed PyAV frames apply only to eligible RGB input in the Videos API encoder. Pre-encoded MP4 bypasses both raw-frame transport and API-side encoding. CPU, non-CUDA and synchronous/step-mode transfers retain the existing path.

The direct GPU-to-SHM direction was proposed earlier in upstream PR #2197, which removed the NumPy staging copy. This draft extends that early work by registering POSIX SHM for D2H, borrowing the receiver mapping, and borrowing RGB buffers in PyAV. It remains a small copy-elision baseline that can compose with the chunkwise producer/consumer design in RFC #6872.

Evaluation: cumulative RGB-to-MP4 ablation

One NVIDIA H200, CPU/memory bound to NUMA node 0. Real MiniMax-H3 RGB decoded beforehand; smaller shapes are temporal prefixes/spatial subsamples. Actual pack/unpack functions run in separate producer/consumer processes, with descriptors sent through a multiprocessing Pipe. This measures GPU-ready RGB → CPU MP4, not generation or VAE-to-HTTP: VAE, DiT, production ZMQ/HTTP and audio are excluded.

A = existing cached pinned D2H + SHM copies + planar PyAV input. Each following column adds one optimization: B direct D2H into registered SHM; C borrowed receiver view; D borrowed RGB PyAV input. These are cumulative alternatives, not additive phases. B's receiver-copy isolation exists only in the external harness.

Mean latency in milliseconds; positive D/A means regression:

RGB shape (W×H×frames)	Payload MiB	A baseline	B +direct D2H	C +borrow receiver	D +borrow PyAV	D/A
448×256×4	1.31	10.40	15.38	16.65	14.84	+42.6%
672×384×4	2.95	14.32	27.38	20.22	21.18	+47.9%
1344×768×1	2.95	42.77	86.12	84.43	63.75	+49.0%
448×256×16	5.25	19.65	22.44	19.39	29.20	+48.6%
1344×768×122	360.28	691.77	577.89	378.00	380.35	−45.0%
1344×768×362	1069.03	1914.25	1630.10	1092.68	1085.78	−43.3%

Per shape/variant: one first-use sample, two warmups, 20 measured samples. Balanced orders ABDC/BCAD/CDBA/DACB repeat five times; no outliers removed. H.264 ultrafast, CRF 18, 24 FPS, encoder threads=0; baseline planar converter uses eight workers. Hash/decode validation is outside timing.

Small shapes regress; this is not a general speedup. Baseline measured calls reuse PyTorch's pinned allocations (zero new pinned allocations). The experimental path allocates/registers/unregisters SHM per transfer, without pooling. Registration overhead and variability can outweigh saved copies; CPU encoding also fluctuates, so not every regression is attributable to registration. Shared-host CPU workloads were not isolated. These samples establish neither a universal crossover threshold nor a tail-latency guarantee; the existing >1,000,000-byte SHM routing threshold is unchanged, not a performance threshold.

Most large-payload benefit comes from IPC copy elimination. For 362 frames, A→B/B→C/C→D total mean reductions are 284.15/537.42/6.90 ms; PyAV encoding alone improves 9.65 ms from C to D. At 122 frames, D is slightly slower overall than C despite removing another copy. These marginal differences are not direct memcpy timings.

All 552 RGB and MP4 hashes match their respective baseline; 24 saved MP4s were fully decoded. These performance measurements predate the latest main rebase; post-rebase checks below verify correctness, not a new performance run. This table supersedes the previous replay table for this revision and uses a different measurement boundary.

Validation

Rebased onto main 5a93ec1b4, squashed into one signed-off commit. 864 existing CPU tests pass; external GPU checks pass for video-only scope, BF16 bit preservation, NumPy/Torch/PyAV mapping lifetimes, and registration/copy/unregistration failure cleanup. Upstream H3 pre-encoded MP4 handling is preserved. No test files were added or changed.

Applicable lint/format hooks pass. Local mypy retains 60 existing diagnostics versus 61 on unmodified main, with no new diagnostics; upstream CI skips that hook. Remote CI must validate the pushed revision.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/profiling.md, docs/design/module/entrypoints.md, docs/design/module/observability.md, docs/design/module/diffusion/diffusion_runtime.md.

Module owners: @alex-jw-brooks @gcanlin @linyueqian

Routing: @alex-jw-brooks via module of the changed files, CODEOWNERS; @gcanlin via module of the changed files; @linyueqian via module of the changed files

@mglyn, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[linyueqian]

The shape of this is right: opt-in, both switches default off, the existing path untouched as the default, and a description that is careful to say "zero copy" means the host-side copies rather than the mandatory D2H. The lease design is also sound. __array_finalize__ hands every view the same _SharedMemoryLease object, so the mapping outlives the last view rather than the first, and unlinking the segment immediately while keeping it mapped is the correct POSIX idiom here.

One blocking issue is simply what is keeping main CI red, and it is small. The rest is two things in the CUDA path that only bite under VLLM_OMNI_ZERO_COPY_SHM=1.

The one worth reading carefully is the bfloat16 widening, because the invariant it breaks is written down a hundred lines below it in the same file. The existing path widens on the host after the D2H specifically so a payload that already fell back from an fp32 OOM does not allocate fp32 on the device a second time. The new path widens before the copy, on the accelerator. That is a real tension with the zero-copy design rather than a simple slip, since the destination buffer has to have a numpy dtype and numpy has no bfloat16, so if you keep the device-side widening it is worth saying so where the old comment is, rather than leaving the two paths silently disagreeing.

Validation: static review of the two-file author diff at 653c1d61 against merge base 3d952d13, the pack and unpack round trip including failure atomicity, and the Buildkite log for the failing step. Fork head, so the branch was not executed. A panel reviewed the same head: codex timed out, and deepseek refuted two hypotheses of mine that are consequently not raised here. I had suspected the strides[1:] == (3, 1) gate was an incomplete contiguity check, but PyAV's own guard in av/video/frame.py is byte-for-byte the same comparison and it uses strides[0] as the plane linesize, so a row-padded view is consumed correctly rather than skewed; the gate is exactly right. I had also suspected the suppressed BufferError in the lease could strand a mapping, but since every view shares one lease object, __del__ runs only once the last view is gone and the buffer is no longer exported.

[linyueqian]

[blocking] This is what is keeping main CI red, and it is the only thing.

Build 15098 fails one test, tests/config/test_environment_variables.py::test_statically_resolvable_environment_accesses_are_classified, with {'VLLM_OMNI_VIDEO_BORROWED_FRAMES', 'VLLM_OMNI_ZERO_COPY_SHM'} == set(). 2973 tests pass in the same step, so this is not inherited. Both new switches are read through os.getenv without being classified in the typed environment configuration that guard enforces, the one added under the #6232 migration.

Registering both, here and for VLLM_OMNI_VIDEO_BORROWED_FRAMES in video_api_utils.py, is the whole fix and turns the lane green.

[mglyn]

Fixed in 14cd96f. Removed both ad-hoc environment switches; the options now use the existing typed video_output_transport configuration, both default off. The environment inventory test passes locally. Registered SHM is scoped to video outputs.

[linyueqian]

[important] Widening bfloat16 here happens on the accelerator, which is exactly what the existing path documents that it must not do.

source is still a CUDA tensor at this point, so .to(torch.float32) allocates a second, full-size fp32 tensor on the device before any copy has happened. The non-zero-copy branch of _tensor_to_shm deliberately does the opposite, and says why in a comment: numpy has no bfloat16, so widen on the host after D2H, and never widen on the accelerator before the copy, because a payload that already fell back after an fp32 OOM would OOM again on a second device-side fp32 allocation.

That is the case this reintroduces. A bf16 video tensor large enough to have taken the fallback is precisely the one that cannot afford a device-side fp32 copy, so the new path can OOM on exactly the payload the old path was built to survive.

The tension is real rather than accidental, since the destination is a numpy-typed shared segment and numpy has no bfloat16, so the widening cannot simply move after the copy the way it does above. Registering the fp32 host buffer and having the device copy convert into it, or falling back to the existing path for bf16 sources, both avoid the second device allocation. If neither is practical, the constraint should at least be written down next to the existing comment so the two paths do not silently contradict each other.

[mglyn]

Fixed in 14cd96f. BF16 now travels as original bits in uint16-backed SHM and is restored via a dtype view; no FP32 accelerator expansion. GPU checks pass for contiguous/strided BF16 and special-value bit preservation.

[linyueqian]

[important] registered is cleared before the error it guards is checked, so the finally retry can never fire on the one failure it exists for.

The sequence is error = cudart.cudaHostUnregister(pointer), then registered = False, then if int(error) != 0: raise. On a failed unregister the raise happens with the flag already false, so the finally block's if registered: cudart.cudaHostUnregister(pointer) is skipped. The same finally then runs shm.close() and, because succeeded is still false, shm.unlink().

So the failure mode is the worst available one: the driver keeps a host registration on a segment that has just been closed and unlinked, and that memory goes back to the OS to be reused while CUDA still believes it is pinned. That is the bookkeeping corruption the flag and the finally retry were added to prevent.

Raising first and clearing the flag only after a successful unregister fixes it, as does leaving registered true across the check so the finally retries.

[mglyn]

Fixed in 14cd96f. The flag is cleared only after successful unregister. Cleanup retries; if it still fails, the mapping is retained and further registrations require a worker restart. Fault-injection checks pass for both retry success and persistent failure.

[lishunyang12]

Keep the mapping alive through plain NumPy views; np.asarray currently drops the lease.

[mglyn]

Fixed in 14cd96f. Replaced the ndarray-subclass lease attribute with an array_interface owner retained through the array base chain. Lifetime checks pass for np.asarray, slices, torch.from_numpy, and borrowed PyAV frames.

[mglyn]

Follow-up: small-payload transport regressions and variability

Further testing exposed small-payload regressions and latency spikes in the registered-SHM path. The large-video results should not be interpreted as a general speedup.

Setup: one NVIDIA H200, CPU and memory bound to NUMA node 0, PyTorch 2.13.0+cu130. A CUDA producer and a separate CPU consumer exercise the IPC pack/unpack functions, passing handles through a multiprocessing pipe. Timing ends when the consumer has a usable view; VAE, MP4 encoding, HTTP, and payload validation are excluded.

A is the existing transport; B is this PR’s per-transfer registered-SHM path, without pooling. Each sweep includes two warmups, eight measured A samples and four B samples per size. B was sampled after baseline/reuse-prototype trials, not in an interleaved A/B comparison.

All timings below are milliseconds; no outliers were removed.

Sweep	Payload	A mean / median	B mean / median	B min–max
1	1.1 MB	1.79 / 1.79	1.80 / 1.66	1.54–2.34
1	4 MiB	5.04 / 5.05	36.05 / 34.82	8.45–66.13
1	16 MiB	17.25 / 17.15	9.25 / 8.59	7.84–11.99
2	1.1 MB	2.10 / 1.94	60.43 / 8.79	3.03–221.10
2	4 MiB	5.36 / 5.21	67.25 / 4.06	3.07–257.81
2	16 MiB	18.68 / 18.67	35.83 / 14.77	8.91–104.85

This is not a uniform slowdown: some medians improve while means regress due to spikes. These exploratory samples do not establish a reliable crossover threshold or tail-latency estimate.

The baseline reuses PyTorch’s cached pinned allocations—the host-allocation counter did not increase during measured A calls. This PR instead creates fresh SHM and registers/unregisters it for every transfer. Our working explanation is that repeated allocation/registration overhead can outweigh the copies saved on smaller payloads. The spikes occur in producer-side packing; we have not isolated the cause of every spike.

For MP4-only requests, worker-side CPU encoding may be a simpler alternative: keep raw frames local, retain ordinary pinned-buffer reuse, and transfer only the compressed result.

[hsliuustc0106]

@specture724 PTAL

[hsliuustc0106]

fix CI

[mglyn]

@hsliuustc0106 Pushed 14cd96f, rebased onto main and squashed with DCO. The reported environment-classification failure is fixed; 864 existing CPU tests, external GPU correctness checks, and applicable lint/format hooks pass locally. Remote CI is pending. The description now includes the cumulative ablation, explicitly reports small-shape regressions, and documents video-only, default-off opt-in scope.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit e04324eb6aeb produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[linyueqian]

Re-review at 14cd96ff against the last reviewed head 653c1d61. The three items from the first round are fixed in the delta: the inventory test that kept main red is handled, the bfloat16 widening no longer happens on the accelerator (the transfer carries a dtype view instead), and the registered flag is now cleared only after the cleanup it guards has run, so the retry path can fire. The revised transport stays opt-in with both options off by default. One new problem in the delta has to be fixed before merge, inline: the frontend reads video_output_transport from the diffusion config it can see, and for out-of-process diffusion stages that view is the metadata-only fallback, so enable_borrowed_frames=true is silently ignored in exactly the multi-replica deployments the option is for. Static read of the 653c1d61..14cd96ff delta and the full merge-base diff against 5a93ec1b; fork head, no PR code executed. The ready label predates the push, so no general lane has run at this head.

[linyueqian]

[important] _resolve_diffusion_od_config goes through AsyncOmniBase.get_diffusion_od_config, which returns a stage client's od_config when one exposes it and otherwise falls back to OmniEngineBase.get_diffusion_od_config, a SimpleNamespace built from model metadata that has no video_output_transport. StageDiffusionClient (the out-of-process, multi-replica diffusion path) exposes neither od_config nor an inner engine, so in that deployment this lookup yields {} and both Videos API response paths keep the old encoding even with enable_borrowed_frames=true. Carry the effective transport configuration into the frontend config view, or resolve it from the stage configuration, and add a regression test for the out-of-process case.

[mglyn]

Fixed in 39ea9be. The Videos API now reads the effective video transport option from the resolved diffusion stage config when the frontend only has the metadata-only config view. Both typed and legacy stage configs are covered, and regression tests exercise raw MP4 and base64 responses for an out-of-process-style client. The five focused tests and applicable pre-commit hooks pass locally. The broader video-server file has seven unrelated Cosmos3 failures in this local environment because its installed vLLM lacks ModelOptLinearMethod; remote CI will validate the pushed head.