feat(llm): add extra_body passthrough to media request structs

[merceod]

Overview:

Adds passthrough of unknown top level fields on the media request structs so backend specific knobs can reach the backend without a typed field for each one. Follows the recent discussion where we agreed typed fields stay preferred for anything generic and a passthrough covers the rest.

Details:

The media request structs are closed today, so any field the frontend doesn't know about gets silently dropped. That blocks things like audio generation toggles on video models, image size overrides and TTS emotion controls.

Two layers, matching how /inference/v1/generate already separates its public shape from its worker contract.

At the public edge, NvCreateImageRequest, NvCreateVideoRequest and NvCreateAudioSpeechRequest keep unknown top level fields, matching how the OpenAI client's extra_body option merges into the top level of the body. Video and audio use a flattened passthrough map like the existing GenerateRequest. The image request already flattens its inner OpenAI struct, and a second flattened map would capture the typed keys too, so it gets hand rolled serde that keeps the same external contract.

At the worker boundary, the handlers call nest_passthrough() before dispatch, which moves the captured fields under an explicit extra_args["media_passthrough"] entry. So the worker sees the normal typed fields plus one nested, namespaced bucket instead of loose top level unknowns, and backend extensions can't collide with canonical fields. The Python media protocol models declare extra_args as an explicit field rather than relying on pydantic extra field storage, and the Omni handlers consume the namespace: knobs matching a diffusion sampling params attribute apply directly, the rest ride OmniDiffusionSamplingParams.extra_args (TTS merges them into its engine params). Knobs that prove out can be promoted to typed fields over time.

Tests pin the SDK wire shape at the edge, the nesting at the boundary, and that a passthrough knob reaches the actual sampling params in the Omni handler.

Where should the reviewer start?

lib/llm/src/protocols/openai.rs has the nesting helper and the namespace key. videos.rs shows the plain flattened form plus nesting tests, images.rs the hand rolled serde. On the worker side, components/src/dynamo/vllm/omni/omni_handler.py consumes the namespace and the boundary test lives in test_omni_handler.py.

Related Issues

🚫 This PR is NOT linked to an issue:

• Confirmed — no related issue

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi merceod! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bbb17286-10b5-4315-9fc9-e90d6b5399a8

📥 Commits

Reviewing files that changed from the base of the PR and between 27f09d5 and a5dc5d9.

📒 Files selected for processing (3)

• lib/llm/src/protocols/openai/audios.rs
• lib/llm/src/protocols/openai/images.rs
• lib/llm/src/protocols/openai/videos.rs

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The OpenAI audio, image, and video request types now accept optional extra_body JSON maps. Serialization omits the maps when absent. Tests cover round trips and related unknown-field behavior.

Changes

OpenAI extra body support

Layer / File(s)	Summary
Request extra_body contracts lib/llm/src/protocols/openai/{audios,images,videos}.rs	The audio, image, and video request types add optional JSON maps for backend-specific parameters.
Serialization and deserialization validation lib/llm/src/protocols/openai/{audios,images,videos}.rs	Tests verify round-trip preservation, omission when unset, coexistence with flattened image fields, and unrelated unknown-field handling.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to a5dc5

This localized change adds optional passthrough fields to media requests without any identified merge-blocking risk; it is merge-ready after normal checks and review.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 3 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly describes the main change: adding extra_body passthrough support to media request structs.
Description check	✅ Passed	The description includes all required sections, explains the passthrough behavior, identifies review starting points, and confirms that the PR has no related issue.

---

_{Comment @coderabbitai help to get the list of available commands.}

[furionw]

Thanks for working on this, @merceod . how about streamlining on passthrough which captures top-level fields?

---

[P2: Real correctness/reliability issue under scoped configurations or conditions; normally fix before merge]
align server passthrough with OpenAI extra_body wire semantics

I think a literal extra_body request field has different semantics from the OpenAI client option whose name it borrows. The Python SDK stores the client-side extra_body option as extra_json and then merges that mapping into the normal JSON body (option construction, request construction). Therefore:

extra_body={"generate_sound": True}

arrives at the server as a top-level "generate_sound": true, not as "extra_body": {"generate_sound": true}. The tests in this PR validate only the latter shape, and the video test explicitly pins unknown top-level fields as dropped, so the normal OpenAI client path would not populate these maps.

Could media requests follow the existing /inference/v1/generate pattern instead?

• GenerateRequest keeps an internally named #[serde(flatten)] passthrough: Map<String, Value>, so unknown wire fields stay top-level.
• VllmTitoEnvelope::new borrows that map and flattens it again into the backend-owned envelope.
• The frontend then stores the envelope in internal PreprocessedRequest.extra_args["vllm_tito"]. Neither passthrough nor extra_args becomes part of the public wire contract.

For audio and video, that would be approximately:

#[serde(default, flatten)]
pub passthrough: serde_json::Map<String, serde_json::Value>,

Image already has a flattened inner, so if a second flattened map is ambiguous for that type, a targeted/custom deserializer can preserve the same external top-level contract rather than introducing a different nested contract for all three media APIs.

The Python media request models also need to preserve and forward these unknown fields; otherwise the Rust frontend can capture them but the worker boundary will still discard them.

[merceod]

you're right, thanks for catching it. The sdk merges extra_body into the top level so the nested field would never see traffic from a normal openai client. Ireworked it to follow the GenerateRequest pattern, video and audio now carry a flattened passthrough map, and since the image request already flattens its inner openai struct (a second flattened map would capture the typed keys too) it got hand rolled serdee that keeps the same external contract, typed fields at the top level, unknowns retained, typed fields winning on conflict.

[furionw]

Thanks for reworking the Rust side so the public wire shape now matches OpenAI extra_body semantics. I think the latest revision still stops one layer too early at the Python boundary: ConfigDict(extra="allow") preserves unknown fields in Pydantic model_extra, but the Omni handlers do not read or forward model_extra to the engine.

Could we make this an explicit extra_args contract between the Rust frontend and Python worker instead?

Ask

Concretely, I am asking for:

1. Keep the flattened Rust passthrough on the public media request types. That correctly captures the top-level fields produced by the OpenAI SDK.

2. In the Rust frontend, before engine.generate(...), convert the public request to an internal worker request. Keep the normal typed media fields available to the worker, but place the captured backend-owned fields under a namespaced entry, for example:

   {
     "prompt": "a cat",
     "model": "wan",
     "size": "832x480",
     "extra_args": {
       "vllm_omni": {
         "generate_sound": true
       }
     }
   }

   If this needs to remain backend-neutral, a neutral namespace such as media_passthrough is fine; the important part is that the internal extension contract is explicit and nested.

3. Make extra_args an explicit field in the Python worker request contract, rather than depending on BaseModel.model_extra.

4. Have OmniHandler consume that namespace and forward the supported values into OmniDiffusionSamplingParams.extra_args and/or the appropriate engine kwargs. Stable fields can still be promoted to typed fields over time.

5. Add a boundary test showing that a top-level SDK extra_body field reaches the actual Omni sampling/engine input, not only that Pydantic retains it.

Why

Why I prefer this:

• We should not require every Python receiver to inherit from and configure Pydantic BaseModel. For example, the common backend GenerateRequest is only a TypedDict, and it explicitly exposes extra_args.
• A nested namespace makes ownership clear and prevents backend-specific extensions from colliding with Dynamo's canonical fields or extensions from another backend.
• It gives us one recognizable Rust↔Python pattern instead of one path using explicit extra_args and media paths relying on Pydantic-specific implicit storage.
• extra="allow" by itself only retains the value; it does not forward it to vLLM-Omni.

Example

The Rust frontend side of /inference/v1/generate is the precedent I have in mind:

• Public GenerateRequest captures future top-level fields with #[serde(flatten)] passthrough.
• VllmTitoEnvelope retains the backend-owned request shape.
• preprocessed_from_generate runs in the Rust HTTP frontend and stores that envelope in PreprocessedRequest.extra_args["vllm_tito"] before dispatch.
• The handler performs that conversion before constructing Context<PreprocessedRequest> and calling the engine: handler_generate.

So the requested separation is: public top-level compatibility at the HTTP edge, then an explicit namespaced extra_args contract across the Rust-to-Python worker boundary.

[merceod]

Done! Went with the full separation you described.

@biswapanda let me know what you think of the PR and if its ready to merge.

[biswapanda]

/ok to test 62b88f9

[biswapanda]

/ok to test 7350323

[merceod]

@biswapanda Addressed all your concerns! Lmk if there are any more.

[biswapanda]

/ok to test b9dabff

[biswapanda]

lgtm

[rmccorm4]

/ok to test 399a66e

[rmccorm4]

Fixed the pre-commit failures and triggered CI