[Feature] W8A8 support for turbomind engine

[binhtranmcs]

Motivation

For now, W8A8 quantization is only supported with pytorch engine. But the turbomind engine is much more performant. Do you have any plan to support W8A8 for turbomind engine.

Also, only SmoothQuant is supported as in here. I think FP8 quantization is also a feature to consider.

Thanks!

Related resources

No response

Additional context

No response

[lvhan028]

FP8 is being supported in pytorch engine.
Due to shorthand, we haven't started w8a8 inference including fp8 in turbomind engine.
We probably start this job after Chinese New Year.