Skip to content

Add TensorRT-LLM support as another backend #205

@kerthcet

Description

@kerthcet

What would you like to be added:

Add support for https://github.com/NVIDIA/TensorRT-LLM?tab=readme-ov-file

see blog like https://developer.nvidia.com/blog/optimizing-inference-on-llms-with-tensorrt-llm-now-publicly-available/

Why is this needed:

Completion requirements:

This enhancement requires the following artifacts:

  • Design doc
  • API change
  • Docs update

The artifacts should be linked in subsequent comments.

Metadata

Metadata

Assignees

Labels

needs-kindIndicates a PR lacks a label and requires one.needs-priorityIndicates a PR lacks a label and requires one.needs-triageIndicates an issue or PR lacks a label and requires one.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions