Conversation
|
|
||
| FP8-kernel modules were never wrapped (their weights loaded directly in FP8) and hold no | ||
| compressed-tensors scheme, so neither the call below nor that hook touches them.""" | ||
| if self.quantization_config.dequantize and self.quantization_config.is_quantization_compressed: |
There was a problem hiding this comment.
we bring this back, should help with dequantizing without using too much memory cc @kylesayrs
| # `use_optimized_inference` has been resolved against the checkpoint by | ||
| # `CompressedTensorsConfig.post_init`; what is left to check is whether the hardware can run | ||
| # the kernels. When it cannot, the model goes through the regular compressed-tensors route. | ||
| self.use_fp8_kernel = self.quantization_config.use_optimized_inference |
There was a problem hiding this comment.
we have use_optimized_inference arg now that the user must set if they want to hit non CT path
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| dequantize: bool = False, | ||
| use_optimized_inference: bool = False, |
There was a problem hiding this comment.
I think it's fine to have this default to True
There was a problem hiding this comment.
Whatever you prefer, this was a bit breaking on your side no ? We can wait a bit until you start using this arg and always set it to False on your side.
|
|
||
| ## FP8 kernel acceleration | ||
|
|
||
| Pass `use_optimized_inference=True` to keep an FP8 compressed-tensors model in FP8 and run its matmuls through hardware-accelerated FP8 kernels (`torch._scaled_mm`), instead of dequantizing the weights back to BF16. Keeping weights in FP8 throughout inference lowers memory usage and speeds up computation. This is inference only, so leave it off to fine-tune. |
There was a problem hiding this comment.
maybe we should primarily mention the public api, the reason being that the public one dispatches to torch._scaled_mm_v2 and torch._scaled_mm
|
[For maintainers] Suggested jobs to run (before merge) run-slow: compressed_tensors_integration |
CI recapDashboard: View test results in Grafana |
* fix compressed tensors * update
4 participants
What does this PR do?
This PR updates CT implementation. One of the PR were a bit too breaking so we are requiring to set an attributed to unlock the optimized inference path vs before where it was the default one. Also, we don't decompress the model by default as this was creating issues.
cc @kylesayrs