Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update TensorRT-LLM #2333

Merged
merged 1 commit into from
Oct 15, 2024
Merged

Update TensorRT-LLM #2333

merged 1 commit into from
Oct 15, 2024

Conversation

kaiyux
Copy link
Member

@kaiyux kaiyux commented Oct 15, 2024

  • Model Support
  • Features
    • Supported the Python LLM API for Mamba2.
    • Supported INT8 GPTQ quantization.
    • Supported FP8 for Deepseek-v1, see the “FP8 Quantization” section in examples/deepseek_v1/README.md.
    • Added TensorRT OOTB support for INT8 Smooth Quantization.
    • Supported quantization for Exaone model, see examples/exaone/README.md.
    • Enable Medusa for Qwen2 models, see “Medusa with Qwen2” section in examples/medusa/README.md.
    • Added request queueing time to gptManagerBenchmark tool.
    • Optimized pipeline parallelism with ReduceScatter and AllGather for Mixtral models.
  • API
    • [BREAKING CHANGE] Moved the flag builder_force_num_profiles in trtllm-build command to env var.
  • Bug fixes
  • Infra
    • The dependent ModelOpt version is updated to 0.17.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants