[Frontend] Preserve encoded video in ComfyUI audio handoff

[Dong1017]

Scope: CUI-03 single-clip audio/video handoff

This PR adds a concrete media-processing helper plus a reusable ComfyUI template for an existing CFR VIDEO and supplied AUDIO.

Included	Behavior
Audio to Video Duration node	Derive target audio samples from VIDEO frame count/rational FPS; trim long audio; loop, silence-pad, or retain short audio. Preserve supplied audio channels/sample rate.
Media handoff template	Separate original, replacement, mean-mix and loop outputs; soundtrack branches pass the original VIDEO into single-input Concatenate Video with complete_audio. Encoded file inputs retain their video packets and frame timing.
Tests and usage guide	Tensor/graph regression tests, policy semantics and instructions for connecting independently configured remote producers.

The scope is a partial CUI-03 contribution for single-clip, constant-frame-rate media. It does not add H3 shot generation, SeedVR2/YuE2 serving APIs, multi-clip orchestration, VFR timestamp preservation or seamless audio crossfades. Fresh remote-producer integration remains pending; this PR stays draft.

Contributor intent (required human answers)

1. What outcome do you want to achieve?

Deliver concrete CUI-03 media handoff behavior requested in #8251: connect H3 video and supplied audio, make trim/loop policies explicit, and preserve original media separately.

2. What is the highest-leverage variable for that outcome?

Reduce graph wiring and manual timing settings for the explicit trim/loop policies requested in CUI-03.

3. How will this work move that variable?

Provide the template plus an Audio to Video Duration node that derives audio length from the video's frame count/rational frame rate. Standard nodes handle extraction, mixing, soundtrack override and saving; the helper fills the short-audio looping/channel-preservation gap requested in #8251.

4. Whose customer or user needs does this address, and what are those needs?

The H3/MV users described in #8251 need original/replacement/mix policies, explicit trim/loop behavior, source timing and independently configured remote services.

Coordination boundary

The video-response decoder is left to #6782/#5905, and SeedVR2 integration to #8505 (CUI-01). This contribution consumes the existing VIDEO/AUDIO contracts. H3 shot generation and YuE2 APIs remain with their assigned tasks.

Purpose

Refs #8251 (CUI-03).

• Add Audio to Video Duration with VIDEO/AUDIO inputs and loop, pad, trim policies. All policies trim long audio. Short audio loops to fill the video, receives a silent tail, or remains short.
• Compute target samples using the video's rational frame rate, rounding up by less than one sample. Preserve audio batch/channels, sample rate, dtype and sample values; do not resample or modify the VIDEO.
• Replace the repeat-twice branch with a duration-aware loop branch. Standard AudioConcat duplicates mono into stereo and does not determine how many repeats are needed; TrimAudioDuration only trims. This is why the small helper is needed.
• Keep four outputs: original, replace, mean mix, loop. The three soundtrack branches use single-input Concatenate Video with complete_audio and codec auto, avoiding an IMAGE/FLOAT reconstruction path. The original VIDEO goes directly to Save Video.
• Add CPU regression tests and document service wiring and exact policy semantics. No new dependencies.

The helper targets constant-frame-rate video such as H3 output. Loop joins are hard, without a crossfade. Compatible encoded file inputs copy video packets into the MP4 output; tensor-backed producer outputs still require ComfyUI encoding. This graph cannot recover timing already discarded by a producer. This PR does not implement the assigned H3-03 shot-generation workflow.

Test Plan

vLLM Version: not used by this media-processing path.

vLLM-Omni Commit: base 744d6dae9d7e9b718ea2ff93a62dccc240f93cd6; current head 53b1a31b9334a73a2cc4e07e952480f8e63c5dd8.

PYTHONPATH=apps/ComfyUI-vLLM-Omni python -m pytest --noconftest tests/e2e/features/comfyui/test_audio_timing.py tests/e2e/features/comfyui/test_media_handoff_workflow.py -q
# CPU-only CI selection: add -m "core_model and cpu" to the pytest command above.
pre-commit run --files apps/ComfyUI-vLLM-Omni/README.md apps/ComfyUI-vLLM-Omni/__init__.py apps/ComfyUI-vLLM-Omni/comfyui_vllm_omni/nodes.py apps/ComfyUI-vLLM-Omni/comfyui_vllm_omni/utils/audio_timing.py apps/ComfyUI-vLLM-Omni/docs/media-handoff.md "apps/ComfyUI-vLLM-Omni/example_workflows/vLLM-Omni Media Handoff.json" tests/e2e/features/comfyui/test_audio_timing.py tests/e2e/features/comfyui/test_media_handoff_workflow.py

Test Result

• 16 tests passed locally and on the server. Tests cover per-channel/per-batch sample order, trimming, silent padding, unchanged short audio in trim mode, empty/invalid policy errors and graph wiring. They use actual torch tensors; these commands disable the existing ComfyUI test conftest. Existing core_model and cpu CI selection includes them.
• Actual custom node: 9 MP4 exports passed on the H200 host's CPU, with ComfyUI 06c67806bcb36789a1e8e1278c6a764b389ff858, Python 3.12.13, torch 2.13.0, PyAV 19.0.1. Audio inputs: 0.6 s/mono/44.1 kHz, 1.5 s/stereo/32 kHz, 6 s/stereo/48 kHz; all three policies tested. Assertions check exact PCM repetition/padding/trimming, preserved channels/rates and exported audio duration. Each MP4 retains 107 video frames and 24 FPS.
• Fractional rate: a real ComfyUI VideoFromComponents with 3 frames at 30000/1001 FPS and 44.1 kHz audio produces exactly 4,415 samples.
• ComfyUI browser queue smoke: imported this exact JSON into a temporary, self-hosted ComfyUI instance and queued it through the web UI without signing in. The job completed and saved preserve/replace/mix/loop MP4s. ffprobe confirms all four have 107 video frames at 24 FPS; replacement audio is trimmed to video duration, loop audio fills it, and preserve/mix keep source audio. CPU only, no model service or GPU inference.
• Current integrated template: 16 actual MP4 exports passed. Existing H3 video (107 frames at 24 FPS) and an actually encoded 9-frame 30000/1001 FPS clip, each paired with 0.1 s mono/44.1 kHz and 6 s stereo/32 kHz WAV inputs, across all four output policies. Video packet payloads, PTS, DTS, packet duration, time base, rate, pixel format and color-space fields match the input in every export. Audio rate/channel and replace/loop duration checks passed. The real ComfyUI dynamic-input parser accepts the single videos.video0 input and complete_audio wiring.
• Pre-submit quality/simplification checks and local pre-commit passed. The helper has no remote calls, model loading, new shared state or dependencies.

Source video: an existing MiniMax-H3 Ref2VA output generated on 2026-09-23, checkpoint 428986c4aa5280855b9ac26eff8d1a46b5999cd0, 512×512, 107 frames, 24 FPS, 32 kHz stereo. Its historical generation experiment reported a mismatch against its reference baseline; only the valid, decodable media is reused here. The historical source copy has no available Git revision.

Validation covers actual node execution and exported media on CPU. Fresh remote H3/SeedVR2/YuE2 inference remains outside the completed validation, so this PR remains draft.

[Dong1017]

Encoded-video handoff update — 53b1a31b9334

The replacement, mean-mix and duration-aware loop branches now use the standard Concatenate Video node with one VIDEO and complete_audio, rather than reconstructing VIDEO from IMAGE/FLOAT outputs. For compatible encoded file inputs, codec auto retains the encoded video stream and frame timing while replacing the soundtrack. Tensor-backed producer outputs still require encoding; this graph cannot restore timestamps discarded earlier.

16 actual exports passed on CPU with the pinned ComfyUI revision: H3 107-frame/24 FPS and an encoded 9-frame/30000/1001 FPS clip × short mono/44.1 kHz and long stereo/32 kHz audio × four outputs. Packet payloads, PTS/DTS, durations, time base and relevant video-format fields match each input. The actual dynamic-input parser accepts the graph wiring. The 16 tensor/graph tests and normal hooks also pass. I also imported this workflow in a temporary self-hosted ComfyUI web UI (no login) and queued it on CPU; the queue completed and saved all four MP4 outputs, which retain 107 frames at 24 FPS.

To avoid overlap, this update leaves response decoding to #6782/#5905 and SeedVR2 integration to #8505. It adds no model-service implementation. This remains a CFR single-clip CUI-03 contribution; fresh remote-producer integration is pending, so the PR stays draft.