Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions docs/user_guide/diffusion/device_side_video_postprocess.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,42 @@ as safety checks or audio/video packaging.

See [RFC #6541](https://github.com/vllm-project/vllm-omni/issues/6541) for the
contract, lifecycle, batching rules, and migration plan.

## Experimental registered-SHM video transport

`video_output_transport.enable_registered_shm` is a separate, default-off option
for CUDA request-mode workers. It copies eligible video tensors directly into
CUDA-registered shared memory and returns borrowed CPU views, removing the
intermediate pinned-to-SHM and SHM-to-consumer host copies. The D2H transfer still
occurs.

`video_output_transport.enable_borrowed_frames` independently enables borrowed
RGB AVFrames in the Videos API MP4 response encoder, removing the third host
copy into PyAV frame buffers. It also defaults to false. Eligible uint8 RGB
frames are wrapped with `VideoFrame.from_numpy_buffer`; other dtypes/layouts use
the existing encoder path. Color conversion, compression and muxing still occur.

Enable it through the same CLI, Python or deploy-stage configuration:

```bash
--video-output-transport '{"enable_registered_shm": true, "enable_borrowed_frames": true}'
```

Registered SHM applies only to prepared typed video media (currently WAN2.2) and
explicitly marked legacy video outputs (currently MiniMax-H3). H3 marks the video
entry of its `(video, audio)` output; audio, trajectory data, images and unmarked
legacy outputs keep their existing transport. Device-side postprocessing is
independent and does not need to be enabled for registered SHM.
Pre-encoded MP4 outputs bypass raw-frame transport and API-side encoding.

The existing SHM routing threshold is unchanged: the tensor or its backing
storage must exceed 1,000,000 bytes. This is not a measured performance crossover.
SHM is allocated and registered on every transfer, without pooling; small outputs
can regress and registration latency can vary. Benchmark representative output
sizes before opting in. CPU outputs, non-CUDA backends, and synchronous/step-mode
transfers keep their existing path.

Borrowed views keep the unlinked mapping alive until their last user releases
it. CUDA registration/copy failures fail the request; if CUDA cleanup cannot be
confirmed, the worker retains the mapping for safety and must be restarted before
further registered-SHM transfers.
13 changes: 12 additions & 1 deletion vllm_omni/diffusion/data.py
Original file line number Diff line number Diff line change
Expand Up @@ -804,10 +804,16 @@ def uses_diffusers_adapter(od_config: object) -> bool:
@dataclass
class VideoOutputTransportConfig:
enable_device_postprocess: bool = False
enable_registered_shm: bool = False
enable_borrowed_frames: bool = False

def __post_init__(self) -> None:
if not isinstance(self.enable_device_postprocess, bool):
raise TypeError("enable_device_postprocess must be a bool")
if not isinstance(self.enable_registered_shm, bool):
raise TypeError("enable_registered_shm must be a bool")
if not isinstance(self.enable_borrowed_frames, bool):
raise TypeError("enable_borrowed_frames must be a bool")


@dataclass
Expand Down Expand Up @@ -1756,11 +1762,16 @@ class DiffusionOutput:
# Internal control-plane event emitted on first scheduler admission.
request_started: bool = False

# Typed video-media contract. Declared last so the pre-existing positional
# Typed video-media contract. Appended so the pre-existing positional
# constructor order (output, trajectory_timesteps, ...) that out-of-tree
# pipelines rely on is preserved. Mutually exclusive with ``output``.
media: DiffusionMediaOutput | None = None

# Compatibility adapter for joint (video, audio) outputs that have not
# migrated to typed media. Only this tuple entry is eligible for video
# transport optimizations; unmarked legacy outputs retain their old path.
video_output_index: int | None = None

def __post_init__(self) -> None:
if self.media is not None and not isinstance(self.media, DiffusionMediaOutput):
raise TypeError(f"media must be DiffusionMediaOutput, got {type(self.media).__name__}")
Expand Down
Loading
Loading