Update TensorRT-LLM

[kaiyux]

• Model Support
  • Support VILA (see “LLaVA and VILA” section in examples/multimodal/README.md)
• Features
  • Support loading Gemma from Hugging Face
  • Add support to context chunking to work with KV cache reuse
  • Support auto parallelism planner for high-level API and unified builder workflow
  • Enable multi-LoRA for BART LoRA
• API
  • [BREAKING CHANGE] Remove model parameter from gptManagerBenchmark and gptSessionBenchmark
• Bug fixes
  • Fix ChatGLM2-6B building failure on INT8 chatglm2-6b int8+kv8 build failed on 0.8.0 branch #1239
  • Fix wrong relative path in Baichuan documentation Incorrect documentation in examples /baichuan/ #1242
• Performance
  • Remove router tensor parallelism to improve performance for MoE models, thanks to the contribution from @megha95 in moe router tp removed #1091
• Infra
  • TensorRT dependency is updated to 9.3.
  • Base Docker image for TensorRT-LLM is updated to nvcr.io/nvidia/pytorch:24.01-py3
  • Base Docker image for TensorRT-LLM backend is updated to nvcr.io/nvidia/tritonserver:24.01-py3

[yiakwy-xpu-ml-framework-team]

Hi, previously, almost 1 year ago, we also wrote a python/cpp separately to find pipeline partitions upon ONNX IR and cpp IR (where we can have more accurate memory estimation).

Since Tensor-LLM now uses its own memory-pool to mange memory allocation. I guess the esitmation will be better now.

The method we used (I also wrote a solver with dyanmical programming) is multi-level graph based.

The problem of performaning solver upon ONNX IR graph is that you cannot always garantee the same topological order in frontend and backend. I guess you will have the same problem.

Obviously one could start with 2-device partition, the answer is naive. (that was initial reason why I used dynamic programming), then extend the solution to 4, 8 or many devices mesh.

However, iterating the nodes is time consuming, so we have to do it in cpp backend with heuristic search.

I wonder how did you managed to run into these problems ? And what is the compile time you have , for example 1000 nodes of graph ?

[yiakwy-xpu-ml-framework-team]

@Shixiaowei02 @kaiyux