[krishung5]

Overview

Add audio URL/data-URI passthrough to the TensorRT-LLM sidecar, stacked on the updated video wiring PR #16870. The image foundation #16613 is merged, and both branches include current main (3057891245). Support is gated by the loaded engine's advertised media and input capabilities. Text-only processors such as Whisper receive the complete formatted prompt, preserving language/task prefixes and empty default prompts.

Key Changes

116 added production lines relative to the video branch (excluding tests, CI, comments, and blank lines).

• Retain the startup remote-media policy inherited from the image/video foundation. Disabling DYN_MM_ALLOW_REMOTE_URLS rejects image, video, and audio HTTP(S) sources before engine dispatch while inline data URIs remain supported.
• Map audio through the existing ordered conversion and capability gates; preserve full text prompts for text-only processors. Configure shared testkit ModelInfo responses and cover task-prefix/empty-prompt forwarding with empty token IDs.
• Extend the official upstream nightly matrix to image, video, and audio. The video case explicitly installs the repository’s existing TRT-LLM decoder dependencies. The audio case uses pinned Whisper-tiny weights and a checksum-verified speech sample; require the expected transcript, completion usage, and exactly one executed Rust integration test.
• Keep the shared-runtime HTTP audio test in a separate local follow-up, following the image/video split. No personal-fork source overlay is used.

Validation

Merged the video branch's terminal-usage fix: hold a trailing hidden EOS until terminal engine usage arrives, then remove the confirmed hidden stop. This prevents the frontend from ending the stream before receiving authoritative prompt/cached-prompt usage.

• 381 sidecar/testkit unit tests passed across the five sidecar/testkit crates; six pre-existing live-only TRT-LLM cases remain ignored in that unit invocation.
• All 11 shared mock-engine integration tests passed, with no skips. Clippy with warnings denied, formatting, whitespace and targeted pre-commit hooks passed.
• Four full HTTP audio cases passed locally: HTTP URL and data URI, each with JSON and SSE responses, through Dynamo frontend → native sidecar → OpenEngine → real Whisper-tiny inference. The expected transcript phrase and complete stream are checked. Prompt usage is now 4, matching the engine, rather than the previous 0.
• The same stack also passed all four full HTTP video cases.

The local GPU run used the unmodified official nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc3.dev202610300406 image (source ac6fb72d9fa1caf2b2d730aa4797727e6b3f6d23), on one RTX 5880 Ada GPU with one aggregated worker. An isolated container used matching userspace driver libraries to work around the host library mismatch; host configuration was unchanged. No live disaggregated or multiworker routing validation is claimed here.

Completion accounting follows the shared frontend's delivered-token convention: Whisper reports 16 engine tokens including hidden EOS; HTTP reports 15 after that EOS is removed, and total 19 (=4+15). Checks explicitly verify the removed EOS and prompt/completion/total consistency.

The official-nightly audio GPU job passed at the prior published audio head 3b11847574. The new fix has local GPU validation; CI for the new head is separate.

Issues and Ongoing Work

refs DIS-2971. Stacked on #16870. Upstream audio support (NVIDIA/TensorRT-LLM#20655) is merged. An official nightly confirmed to contain audio support is 1.4.0rc3.dev202610300406, source ac6fb72d9fa1caf2b2d730aa4797727e6b3f6d23; its ancestry and handler table include audio support. The resolver selects the latest published regular nightly automatically. The audio GPU lane passed in the linked official-nightly run.

The GPU lane covers the Rust sidecar adapter through native OpenEngine and real inference. Full Dynamo frontend HTTP CI remains in the shared-runtime follow-up, pending its runtime upgrade; local full-path GPU validation is recorded above.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill for the review fixes and CI update.

Summary by CodeRabbit

• New Features

  • Added audio input support for TensorRT-LLM multimodal requests.
  • Audio data can now be passed using audio_url and mapped to the audio modality.
  • Added support for Whisper-based aggregated audio serving configurations.
  • Added configurable engine arguments and an option to disable guided decoding.

• Bug Fixes

  • Improved engine configuration merging for custom settings and generated overrides.
  • Model capability detection now recognizes audio support and refreshes limits accordingly.