Skip to content

docs(inference): add TensorRT-LLM static FP8 cookbook - #229

Closed
ishovkun wants to merge 1 commit into
NVIDIA:mainfrom
ishovkun:trt-llm-nano-super-fp8
Closed

docs(inference): add TensorRT-LLM static FP8 cookbook#229
ishovkun wants to merge 1 commit into
NVIDIA:mainfrom
ishovkun:trt-llm-nano-super-fp8

Conversation

@ishovkun

@ishovkun ishovkun commented Sep 1, 2026

Copy link
Copy Markdown

What this PR adds

This is a documentation-only PR. It adds a single-GPU cookbook for running
local, ModelOpt-calibrated FP8 versions of the Cosmos3 Nano and Super generators
with TensorRT-LLM.

The guide includes complete commands for:

  • text to image
  • text to video
  • image to video
  • video to video
  • decoding and checking every generated artifact

It is linked from the main README and inference guide.

About the checkpoints

These are FP8 variants of the Cosmos3 Nano and Super generator checkpoints.
Their metadata identifies static FP8 and contains the calibrated weight and
activation scales, so TensorRT-LLM selects the correct execution path without a
separate quantization flag.

The checkpoints are supplied as local directories; there are no directly
loadable FP8 Hub IDs. The current TensorRT-LLM implementation supports these
FP8 checkpoints on one GPU only. The guide does not validate audio generation
or claim that FP8 output quality matches BF16.

TensorRT-LLM dependency

This cookbook depends on TensorRT-LLM #17476
and will remain a draft until that change is merged. The instructions are written
for TensorRT-LLM main after the merge; validation used the exact PR head
fdd26874d32b3feaaf13be7ccc11f3c949c78281.

Validation

  • Built the exact TensorRT-LLM PR head from source and ran its focused static-FP8
    tests on one H200: 59 passed for step-precision behavior and 15 passed for
    real Nano/Super checkpoint loading, topology, weights, and calibration scales.
  • Ran all eight documented Nano/Super commands on one H100.
  • Decoded both 1024x1024 PNGs and all six 1280x720, 24 fps, 189-frame H.264 MP4s.
  • Kept guardrails enabled. All cookbook prompts passed the text checks, and an
    unsafe control prompt was blocked. In cosmos_guardrail==0.3.0, upstream does
    not run the generated-frame content classifier because it produced too many
    false positives; RetinaFace face blurring remains active.
  • All nine GitHub checks passed.

@lfengad

lfengad commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

We already have the TensorRT cookbook in the https://github.com/NVIDIA/cosmos repo. That is the repo to hold the cookbook for non our framework cases. Thanks!

@lfengad lfengad closed this Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants