docs(inference): add TensorRT-LLM static FP8 cookbook - #229
Closed
ishovkun wants to merge 1 commit into
Closed
Conversation
Collaborator
|
We already have the TensorRT cookbook in the https://github.com/NVIDIA/cosmos repo. That is the repo to hold the cookbook for non our framework cases. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR adds
This is a documentation-only PR. It adds a single-GPU cookbook for running
local, ModelOpt-calibrated FP8 versions of the Cosmos3 Nano and Super generators
with TensorRT-LLM.
The guide includes complete commands for:
It is linked from the main README and inference guide.
About the checkpoints
These are FP8 variants of the Cosmos3 Nano and Super generator checkpoints.
Their metadata identifies static FP8 and contains the calibrated weight and
activation scales, so TensorRT-LLM selects the correct execution path without a
separate quantization flag.
The checkpoints are supplied as local directories; there are no directly
loadable FP8 Hub IDs. The current TensorRT-LLM implementation supports these
FP8 checkpoints on one GPU only. The guide does not validate audio generation
or claim that FP8 output quality matches BF16.
TensorRT-LLM dependency
This cookbook depends on TensorRT-LLM #17476
and will remain a draft until that change is merged. The instructions are written
for TensorRT-LLM
mainafter the merge; validation used the exact PR headfdd26874d32b3feaaf13be7ccc11f3c949c78281.Validation
tests on one H200:
59 passedfor step-precision behavior and15 passedforreal Nano/Super checkpoint loading, topology, weights, and calibration scales.
unsafe control prompt was blocked. In
cosmos_guardrail==0.3.0, upstream doesnot run the generated-frame content classifier because it produced too many
false positives; RetinaFace face blurring remains active.