How to deploy a quantized FP8 multimodel llm ?

[zhishao]

Do both the visual encoder and the language model need to be quantized to fp8?
How to modify the config.pbtxt?

[zhishao]

When I launch an Internvl2 multimodal model, it can perform inference normally. However, after I quantize its language model component, it loads successfully but encounters errors during inference :

[TensorRT-LLM][ERROR] Request embedding table data type doesn't match model weight data type.
[TensorRT-LLM][ERROR] Encountered an error in forwardAsync function: Request embedding table data type doesn't match model weight data type.