[TRTLLM-13143][feat] BREAKING: VisualGen serving API: response_format support path and modify async job status

docs/source/commands/trtllm-serve/trtllm-serve.rst

@@ -236,7 +236,7 @@ Visual Generation Serving
    trtllm-serve nvidia/Cosmos3-Nano \
        --enable_visual_gen
 
-For checkpoints that support both LLM and Visual Generation, such as Cosmos3, pass ``--enable_visual_gen`` to select the VisualGen runtime when ``--visual_gen_args`` is not specified. The ``--visual_gen_args`` flag accepts a YAML file that configures quantization, parallelism, and TeaCache. Available visual generation endpoints include ``/v1/images/generations``, ``/v1/videos``, ``/v1/videos/generations``, and video management APIs.
+For checkpoints that support both LLM and Visual Generation, such as Cosmos3, pass ``--enable_visual_gen`` to select the VisualGen runtime when ``--visual_gen_args`` is not specified. The ``--visual_gen_args`` flag accepts a YAML file that configures quantization, parallelism, and TeaCache. Available visual generation endpoints include ``/v1/images/generations``, ``/v1/videos``, ``/v1/videos/sync`` (with ``/v1/videos/generations`` kept as a deprecated alias), and video management APIs.
 
 For full details, see the :doc:`../../models/visual-generation.md` feature documentation. Example client scripts are available in the `examples/visual_gen/serve/ <https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/visual_gen/serve>`_ directory.
 

docs/source/models/visual-generation.md

@@ -107,12 +107,17 @@ When served via `trtllm-serve`, the following OpenAI-compatible endpoints are av
 | `/v1/images/generations` | POST | Synchronous image generation |
 | `/v1/images/edits` | POST | Image editing |
 | `/v1/videos` | POST | Asynchronous video generation |
-| `/v1/videos/generations` | POST | Synchronous video generation |
+| `/v1/videos/sync` | POST | Synchronous video generation |
+| `/v1/videos/generations` | POST | Deprecated alias of `/v1/videos/sync` (kept for back-compat) |
 | `/v1/videos/{id}` | GET | Video status / metadata |
 | `/v1/videos/{id}/content` | GET | Download generated video |
 | `/v1/videos/{id}` | DELETE | Delete generated video |
 | `/v1/videos` | GET | List all videos |
 
+The asynchronous `/v1/videos` job advances through `GET /v1/videos/{id}`: `queued` → `generating` (model inference) → `postprocessing` (encode the media and/or write the output file) → `completed`. The `generating` → `postprocessing` transition marks the end of inference; the video is downloadable via `/content` once `completed`.
+
+`response_format="path"` returns the generated file's server-side path (under `TRTLLM_MEDIA_STORAGE_PATH`) for co-located clients, enabled by default. Set `TRTLLM_DISALLOW_LOCAL_MEDIA_PATH=1` to reject such requests with HTTP 400. See the [serve examples](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/visual_gen/serve) for the full `response_format` reference.
+
 ## Optimizations
 
 ### Quantization

docs/source/release-notes.md

@@ -34,6 +34,8 @@ All published functionality in the Release Notes has been fully tested and verif
 
 - <span style="color: red">**[BREAKING CHANGE] KV Cache Manager V2 reports cold-tier `secondary*` statistics in `kvCacheIterationStatsByColdPoolGroup`. These keys are no longer present in the hot pool-group or window-size views.**</span>
 
+- <span style="color: red">**[BREAKING CHANGE] VisualGen video `response_format`.**</span> Video generation (`POST /v1/videos/sync`, `POST /v1/videos`) narrows `response_format` to `{file, path}` (default `file`): the old `url` value — which returned raw bytes, not a URL — is renamed to `file`, and `b64_json` is removed. Requests still sending `url` or `b64_json` now get an error that names the replacement. The internal `output_path` is no longer emitted by `GET /v1/videos/{id}` or `GET /v1/videos` (status only); a co-located client obtains the on-disk path via `response_format="path"`. Image `response_format` is unchanged and additively gains `path`. The synchronous route is now `POST /v1/videos/sync`, with `POST /v1/videos/generations` kept as a deprecated alias. `response_format="path"` returns absolute server-side file paths and can be disabled server-side with `TRTLLM_DISALLOW_LOCAL_MEDIA_PATH=1` (enabled by default; paths stay under the media-storage directory).
+
 ### Fixed Issues
 
 ### Known Issues

examples/visual_gen/serve/README.md

@@ -146,7 +146,7 @@ python sync_video_gen.py --mode ti2v \
 - `--size` - Video resolution in WxH format (default: 256x256)
 - `--output` - Output video file path (default: output_sync.mp4)
 
-**API Endpoint:** `POST /v1/videos/generations`
+**API Endpoint:** `POST /v1/videos/sync`
 
 **API Details:**
 - T2V uses JSON `Content-Type: application/json`
@@ -275,7 +275,7 @@ You can customize these by:
 - `seed`: Random seed; `null` / omitted means the engine draws a fresh seed
 - `num_inference_steps`, `guidance_scale`, `max_sequence_length`, `negative_prompt`: per-request denoise controls (override pipeline defaults when sent)
 - `extra_params`: model-specific overflow as a JSON object (see "Model-Specific `extra_params`" below). Unknown keys are rejected by the executor.
-- `response_format`: `"b64_json"` or `"url"`
+- `response_format`: `"url"` (default; HTTP URL to `/content`), `"b64_json"` (inline base64), or `"path"` (server-side on-disk path, for co-located clients)
 - `format`: Generation content encoding. Image encoders: `"png"`, `"webp"`, `"jpeg"`. Tensor formats: `"safetensors"`, `"pt"`.
 - Accept-and-warn OpenAI-shape fields (no engine semantic): `model`, `quality`, `style`, `user`. Sending `quality`/`style` logs a server-side WARNING; sending `model` warns on mismatch. None of these change generation behavior.
 
@@ -289,9 +289,11 @@ You can customize these by:
 - `input_reference`: Reference image (I2V/TI2V) or video (V2V), accepted as a base64-encoded string in JSON or as a file in multipart form-data
   - **Supported formats**: PNG and JPEG images; MP4 and AVI video, with H.264 the tested codec and others best-effort. HEIF/AVIF are not supported.
 - `extra_params`: model-specific overflow (see below)
-- `response_format`: `"b64_json"` or `"url"`
+- `response_format`: `"file"` (default; `FileResponse` byte download) or `"path"` (server-side output path JSON, for co-located clients)
 - `format`: Generation content encoding. Video encoders: `"mp4"`, `"avi"`, `"auto"`. Tensor formats: `"safetensors"`, `"pt"` (carries video + audio + scalar metadata in one payload for LTX-2).
 
+> **`response_format="path"`** (image and video) returns absolute server-side file paths under the server's media-storage directory (`TRTLLM_MEDIA_STORAGE_PATH`), for clients co-located with the server (shared filesystem). Enabled by default; set `TRTLLM_DISALLOW_LOCAL_MEDIA_PATH=1` to reject `path` requests with HTTP 400.
+
 #### Tensor-format consumer contract
 
 When `format="safetensors"` or `format="pt"`, the payload bundles every populated media tensor (`image` / `video` / `audio`) and the scalar metadata (`frame_rate`, `audio_sample_rate`) into one file.
@@ -406,6 +408,8 @@ curl -X POST "http://localhost:8000/v1/videos" \
 curl -X GET "http://localhost:8000/v1/videos/{video_id}"
 ```
 
+The async job's `status` advances `queued` → `generating` (model inference) → `postprocessing` (encode the media and/or write the output file) → `completed`. The `generating` → `postprocessing` transition marks the end of inference; poll for `completed` to download via `/content`.
+
 ### Download Video
 ```bash
 # The server returns either MP4 (with ffmpeg) or AVI (without ffmpeg)
@@ -426,14 +430,14 @@ curl -X DELETE "http://localhost:8000/v1/videos/{video_id}"
 | Endpoint | Method | Mode | Content-Type | Purpose |
 |----------|--------|------|--------------|---------|
 | `/v1/videos` | POST | Async | JSON or Multipart | Create video job (T2V/TI2V) |
-| `/v1/videos/generations` | POST | Sync | JSON or Multipart | Generate video sync (T2V/TI2V) |
+| `/v1/videos/sync` | POST | Sync | JSON or Multipart | Generate video sync (T2V/TI2V) |
 | `/v1/videos/{id}` | GET | - | - | Get video status/metadata |
 | `/v1/videos/{id}/content` | GET | - | - | Download video file |
 | `/v1/videos/{id}` | DELETE | - | - | Delete video |
 | `/v1/videos` | GET | - | - | List all videos |
 | `/v1/images/generations` | POST | - | JSON | Generate images (T2I) |
 
-**Note:** Both `/v1/videos` (async) and `/v1/videos/generations` (sync) support:
+**Note:** Both `/v1/videos` (async) and `/v1/videos/sync` (sync) support:
 - **JSON**: Standard text-to-video (T2V)
 - **Multipart/Form-Data**: Text+image-to-video (TI2V) with file upload
 

examples/visual_gen/serve/sync_video_gen.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 """Test script for synchronous video generation endpoint.
 
-Tests POST /v1/videos/generations endpoint which waits for completion and returns video data.
+Tests POST /v1/videos/sync endpoint which waits for completion and returns video data.
 The video is generated synchronously and the response contains the video file.
 
 Supports two modes:
@@ -65,7 +65,7 @@ def test_sync_video_generation(
     print(f"   Size: {size}")
 
     try:
-        endpoint = f"{base_url}/videos/generations"
+        endpoint = f"{base_url}/videos/sync"
 
         if input_reference:
             # TI2V mode - Use multipart/form-data with file upload

tensorrt_llm/_torch/visual_gen/models/ltx2/pipeline_ltx2_two_stages.py

@@ -56,10 +56,11 @@ class _TwoStagePhaseTimer(CudaPhaseTimer):
     """CudaPhaseTimer + the two-stage extras: the stage-2 refinement loop and
     the decode section.
 
-    Inherited marks keep their contract (``denoise`` = the whole stage-1
-    forward; stage 2 folds into ``post_denoise`` on ``PipelineOutput``).
+    ``mark_post_start`` is placed at the decode boundary, so ``denoise``
+    spans stage 1 plus the stage-2 upsample / LoRA bind / refinement loop
+    and ``post_denoise`` is the decode only (matching single-stage models).
     The extra event pair brackets the stage-2 refinement step loop only
-    (upsample / LoRA bind / text-cache prep stay outside).
+    (upsample / LoRA bind / text-cache prep stay outside), for logging.
 
     Event deltas are GPU-stream distances: they include GPU work plus any
     CPU time exposed to the stream, and stay correct under CUDA graphs and
@@ -1356,8 +1357,6 @@ def forward(
         audio_latents = out.audio  # (B, C, F_aud, M) or None
         assert video_latents is not None, "stage-1 latents missing on this rank"
 
-        timer.mark_post_start()
-
         # ================================================================
         # Stage 2: spatial upsample + refinement denoise — all ranks, collectively
         # ================================================================
@@ -1466,6 +1465,9 @@ def forward(
             else:
                 self._lora_cuda_graph_state = "original"
 
+        # Denoise (stage 1 + stage 2) complete; decode is the post phase.
+        timer.mark_post_start()
+
         # ================================================================
         # Decode
         # ================================================================

tensorrt_llm/serve/openai_protocol.py

@@ -1628,7 +1628,7 @@ class ImageGenerationRequest(OpenAIBaseModel):
 
     # Prompt + transport (OpenAI-standard, always honored)
     prompt: str
-    response_format: Literal["url", "b64_json"] = "url"
+    response_format: Literal["url", "b64_json", "path"] = "url"
     format: Literal["png", "webp", "jpeg", "safetensors", "pt"] = Field(
         default="png",
         description=(
@@ -1762,6 +1762,7 @@ class ImageObject(OpenAIBaseModel):
     """Generated image object in the response."""
     b64_json: Optional[str] = None
     url: Optional[str] = None
+    path: Optional[str] = None
     revised_prompt: Optional[str] = None
 
 
@@ -1792,7 +1793,7 @@ class VideoGenerationRequest(OpenAIBaseModel):
 
     # Prompt + transport
     prompt: str
-    response_format: Literal["url", "b64_json"] = "url"
+    response_format: Literal["file", "path"] = "file"
     format: Literal["mp4", "avi", "auto", "safetensors", "pt"] = Field(
         default="auto",
         description=(
@@ -1865,6 +1866,26 @@ def _check_paired_dimensions(self):
                 f"{self.width!r}, height={self.height!r}")
         return self
 
+    @field_validator("response_format", mode="before")
+    @classmethod
+    def _reject_removed_response_format(cls, value):
+        """Give migrating callers an actionable error for removed values.
+
+        ``url``/``b64_json`` were valid before the transport rewrite; run
+        before the ``Literal`` check so the error names the replacement
+        instead of the generic "Input should be 'file' or 'path'".
+        """
+        removed = {
+            "url":
+            ("'url' was removed for video; use 'file' (raw bytes -- the "
+             "old 'url' behavior, renamed) or 'path' (server-side path)."),
+            "b64_json": ("'b64_json' was removed for video; use 'file' (raw "
+                         "bytes) or 'path' (server-side path)."),
+        }
+        if isinstance(value, str) and value in removed:
+            raise ValueError(removed[value])
+        return value
+
 
 class VideoJob(OpenAIBaseModel):
     """Metadata for an asynchronous video generation job.
@@ -1886,8 +1907,13 @@ class VideoJob(OpenAIBaseModel):
         default=None,
         description="Progress of the video generation job (0-100)")
     prompt: str = Field(description="The prompt used to generate the video")
-    status: Literal["queued", "in_progress", "completed", "failed"] = Field(
-        description="Current status of the video generation job")
+    status: Literal["queued", "generating", "postprocessing", "completed",
+                    "failed"] = Field(description=(
+                        "Current status of the video generation job. "
+                        "``generating`` (model inference) becomes "
+                        "``postprocessing`` (encode and/or write the output "
+                        "file) when inference finishes, then ``completed`` "
+                        "once downloadable via ``/content``."))
 
     # Video properties
     duration: Optional[float] = Field(default=None,
@@ -1900,17 +1926,35 @@ class VideoJob(OpenAIBaseModel):
     )
     size: Optional[str] = Field(default=None,
                                 description="Video dimensions in 'WxH' format")
+    # exclude=True: internal file-location for /content resolution + delete;
+    # never on the wire (the path payload is the hand-built {id, output_path}
+    # envelope in /content), so status/list model_dump() stays status-only.
     output_path: Optional[str] = Field(
-        default=None, description="Actual path where the video file was saved")
+        default=None,
+        exclude=True,
+        description="Server-side saved path (internal; excluded from the wire)."
+    )
     output_paths: Optional[List[str]] = Field(
-        default=None, description="Paths for all generated videos when n > 1")
-    response_format: Optional[Literal["url", "b64_json"]] = Field(
+        default=None,
+        exclude=True,
+        description=
+        "Server-side paths for n>1 (internal; excluded from the wire).")
+    # exclude=True internal timings, never on the wire (status/list
+    # model_dump() stays status-only). ``request_started`` is a
+    # ``perf_counter()`` stamped at the POST handler; the background task
+    # computes ``total`` from it and stores the header timings
+    # (``generation``/``denoise``/``total``) in ``timing_metrics`` so
+    # ``/content`` emits the same Server-Timing header as the sync route.
+    request_started: Optional[float] = Field(default=None, exclude=True)
+    timing_metrics: Optional[Dict[str, float]] = Field(default=None,
+                                                       exclude=True)
+    response_format: Optional[Literal["file", "path"]] = Field(
         default=None,
         description=(
             "Transport the client requested. ``GET /v1/videos/{id}/content`` "
-            "honors this: ``b64_json`` returns the encoded payload as a "
-            "base64 string inside a JSON envelope; ``url`` (or unset) "
-            "returns the file as a ``FileResponse`` download."),
+            "honors this: ``path`` returns the server-side output path(s) in a "
+            "JSON envelope; ``file`` (or unset) returns the file as a "
+            "``FileResponse`` download."),
     )