Skip to content

fix: Change tensorrt_llm api from tensorrt_llm._torch to tensorrt_llm - #446

Closed
sarattha wants to merge 389 commits into
ai-dynamo:dynamo/trtllm_llmapi_v1from
sarattha:f/trtllm-add_engine_support
Closed

fix: Change tensorrt_llm api from tensorrt_llm._torch to tensorrt_llm#446
sarattha wants to merge 389 commits into
ai-dynamo:dynamo/trtllm_llmapi_v1from
sarattha:f/trtllm-add_engine_support

Conversation

@sarattha

@sarattha sarattha commented Mar 31, 2025

Copy link
Copy Markdown
  • Add support for TensorRT-LLM on both pytorch and pre-built (engine) model

rmccorm4 and others added 30 commits February 26, 2025 13:29
@rmccorm4
refactor: Move arg parsing into app for cleaner signatures (ai-dynamo…
e04325e 
…#281)
@paulhendricks @grahamking
refactor: using async_openai
ca72b8c 
Co-authored-by: Graham King <grahamk@nvidia.com>
@rmccorm4
chore: Remove 'Roff' section from repo language stats (ai-dynamo#286)
0b45baa
@sshchoi @alec-flowers
feat: Add HTTP completions endpoint to kv router example (ai-dynamo#258)
a6f6bc2 
Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com>
@tanmayv25 @NVShreyas
feat: LLM API example integration (ai-dynamo#182)
b4d54bd 
Co-authored-by: NVShreyas <158103197+NVShreyas@users.noreply.github.com>
@paulhendricks
refactor: rename ChatCompletionRequest to NvCreateChatCompletionReque…
1384839 
…st (ai-dynamo#284)
@paulhendricks
refactor: rename ChatCompletionResponse to NvCreateChatCompletionResp…
23d7e60 
…onse (ai-dynamo#291)
@paulhendricks
refactor: rename ChatCompletionResponseDelta to NvCreateChatCompletio…
58b661b 
…nStreamResponse (ai-dynamo#292)
@nv-anants
ci: build wheel from root directory (ai-dynamo#274)
cadad17
@ptarasiewiczNV Piotr Tarasiewicz Nvidia
@nnshah1 @alec-flowers
feat: vLLM + NIXL example
33f2349 
Co-authored-by: Piotr Tarasiewicz Nvidia <ptarasiewicznv@Piotrs-MacBook-Pro.local>
Co-authored-by: nnshah1 <neelays@nvidia.com>
Co-authored-by: alec-flowers <aflowers@nvidia.com>
@ptarasiewiczNV
Update wheel build in Dockerfile.vllm_nixl (ai-dynamo#295)
f163b41
@ryanolson
refactor: service/endpoint stats_handler (ai-dynamo#282)
162c599
@paulhendricks
refactor: removes wrapper for ChatCompletionContent and adds document…
7de3174 
…ation (ai-dynamo#296)
@paulhendricks
fix: add skip_serializing if none (ai-dynamo#297)
4db09ad
@grahamking
feat: llama.cpp engine for tio (ai-dynamo#298)
fe946cb 
Docs in README
@NVShreyas
Updates to support DS R1 in TRTLLM example (ai-dynamo#301)
21f71bc
@piotrm-nvidia
Add missing inits (ai-dynamo#307)
5158c5a
@saturley-hall @nvda-mesharma
ci: initial public ci for PRs (ai-dynamo#241)
06041ac 
Signed-off-by: Harrison Saturley-Hall <454891+saturley-hall@users.noreply.github.com>
Signed-off-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>
Co-authored-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>
@grahamking
feat: vllm engine (ai-dynamo#308)
a49c781 
triton-distributed-llm component and support in tio
@rmccorm4
feat: Add initial prometheus/grafana support for count (ai-dynamo#303)
497f5e1
@alec-flowers @rmccorm4
[fix] KV Router Example fixes (ai-dynamo#314)
8ed9f22 
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
@grahamking
feat: TensorRT-LLM engine (ai-dynamo#317)
14b9f06 
Engine, `tio` support and docs.

Proof of concept / experimental.
@paulhendricks
refactor: use async-openai CompletionRequest (ai-dynamo#310)
7405264
@piotrm-nvidia
Add lazy import to nixl.py
17aa573
@alec-flowers
[fix] OpenAI API object: completion to text_completion (ai-dynamo#318)
3923236
@nnshah1
chore: Update README.md
c627f4f 
Signed-off-by: Neelay Shah <neelays@nvidia.com>
@rmccorm4
chore: Tweak prometheus/grafana config defaults (ai-dynamo#319)
2671498
@piotrm-nvidia
fix: Copy nixl example for vllm_nixl container (ai-dynamo#323)
a789154
@grahamking
fix: Install specific toolchain (ai-dynamo#329)
a4b78de 
`cargo build --locked` won't let you use "1.85.0" if you only have "stable" installed, even if those are the same thing right now.
@bispnv
feat: Add compound AI python SDK
20364c0
@copy-pr-bot

copy-pr-bot Bot commented Mar 31, 2025

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Mar 31, 2025

Copy link
Copy Markdown
Contributor

👋 Hi sarattha! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@sarattha sarattha closed this Mar 31, 2025
@sarattha sarattha changed the title Change tensorrt_llm api from tensorrt_llm._torch to tensorrt_llm fix: Change tensorrt_llm api from tensorrt_llm._torch to tensorrt_llm Mar 31, 2025
@sarattha sarattha reopened this Mar 31, 2025
@dmitry-tokarev-nv dmitry-tokarev-nv added the external-contribution Pull request is from an external contributor label Apr 10, 2025
@saturley-hall
saturley-hall force-pushed the dynamo/trtllm_llmapi_v1 branch from db35630 to 304cd37 Compare April 11, 2025 03:28

This branch had an error being deployed

1 failed deployment
GITLAB — 97ac9344 Deployed Mar 31, 2025 by sarattha
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nnshah1 nnshah1 Awaiting requested review from nnshah1 nnshah1 is a code owner
@tanmayv25 tanmayv25 Awaiting requested review from tanmayv25 tanmayv25 is a code owner
@piotrm-nvidia piotrm-nvidia Awaiting requested review from piotrm-nvidia piotrm-nvidia is a code owner
@ptarasiewiczNV ptarasiewiczNV Awaiting requested review from ptarasiewiczNV ptarasiewiczNV is a code owner
@ryanolson ryanolson Awaiting requested review from ryanolson ryanolson is a code owner
@grahamking grahamking Awaiting requested review from grahamking grahamking is a code owner
@paulhendricks paulhendricks Awaiting requested review from paulhendricks paulhendricks is a code owner
@biswapanda biswapanda Awaiting requested review from biswapanda biswapanda is a code owner
@tmonty12 tmonty12 Awaiting requested review from tmonty12 tmonty12 is a code owner
@GuanLuo GuanLuo Awaiting requested review from GuanLuo GuanLuo is a code owner
@rmccorm4 rmccorm4 Awaiting requested review from rmccorm4 rmccorm4 is a code owner
@tedzhouhk tedzhouhk Awaiting requested review from tedzhouhk tedzhouhk is a code owner
@alec-flowers alec-flowers Awaiting requested review from alec-flowers alec-flowers is a code owner

Assignees

No one assigned

Labels

contact-us external-contribution Pull request is from an external contributor

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.