Skip to content

Update quantization/README.md #2862

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
juney-nvidia merged 1 commit into from
Mar 24, 2025
Merged

Update quantization/README.md #2862

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions examples/quantization/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Checkpoint saved in `output_dir` can be directly passed to `trtllm-build`.

- model_dir: Hugging Face model path.
- qformat: Specify the quantization algorithm applied to the checkpoint.
- nvfp4: Weights are quantized to NVFP4 block-wise with size 16. Actiavation global scale are calibrated.
- nvfp4: Weights are quantized to NVFP4 block-wise with size 16. Activation global scale are calibrated.
- fp8: Weights are quantized to FP8 tensor wise. Activation ranges are calibrated tensor wise.
- int8_sq: Weights are smoothed and quantized to INT8 channel wise. Activation ranges are calibrated tensor wise.
- int4_awq: Weights are re-scaled and block-wise quantized to INT4. Block size is specified by `awq_block_size`.
Expand Down Expand Up @@ -99,7 +99,7 @@ Checkpoint saved in `output_dir` can be directly passed to `trtllm-build`.

### Building Arguments:

There are several arguments for building stage which related to quantizaion.
There are several arguments for the building stage which relate to quantization.
- use_fp8_context_fmha: This is Hopper-only feature. Use FP8 Gemm to calculate the attention operation.

```python
Expand Down Expand Up @@ -170,7 +170,7 @@ There will be another file about per-layer quantization information named `quant
}
```

TensorRT-LLM will automatically read `quant_cfg.json` after recogniziong the `MIXED_PRECISION` quantization method in `config.json`. All the specific algorithm keeps the same as what in `quantization` filed before. If some layers are not listed, they'll be treated as no quantization.
TensorRT-LLM will automatically read `quant_cfg.json` after recogniziong the `MIXED_PRECISION` quantization method in `config.json`. All the specific algorithm keeps the same as what in `quantization` field before. If some layers are not listed, they'll be treated as no quantization.

## APIs

Expand Down