Skip to content

feat: LLMAPI PoC with dynamo-run launcher - #114

Merged
tanmayv25 merged 12 commits into
mainfrom
tanmayv-llmapi-launch
Mar 14, 2025
Merged

feat: LLMAPI PoC with dynamo-run launcher#114
tanmayv25 merged 12 commits into
mainfrom
tanmayv-llmapi-launch

Conversation

@tanmayv25

@tanmayv25 tanmayv25 commented Mar 12, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overview:

A simple proof of concept on using llmapi to start serving single-GPU monolithic model using pystr in dynamo-run.

  • Monolithic with duplicate code
  • Monolithic with simple code

Following steps can be used to run llmapi with dynamo-run:

  1. cargo build --release --features python
  2. /workspace/target/release/dynamo-run out=pystr:./monolith/dynamo_engine.py -- --engine_args llm_api_config.yaml

Will add documentation once this PR is merged.

@tanmayv25 tanmayv25 changed the title PoC: LLMAPI with dynamo-run launcher feat: LLMAPI PoC with dynamo-run launcher Mar 12, 2025
@github-actions

github-actions Bot commented Mar 12, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Test Results

104 tests   104 ✅  52s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit af574b5.

♻️ This comment has been updated with latest results.

rmccorm4
rmccorm4 reviewed Mar 12, 2025
View reviewed changes
Comment thread launch/dynamo-run/engines/llmapi/__init__.py Resolved
@tanmayv25
tanmayv25 force-pushed the tanmayv-llmapi-launch branch from 446770e to a201c23 Compare March 13, 2025 10:05
@tanmayv25 tanmayv25 changed the title feat: LLMAPI PoC with dynamo-run launcher WIP: LLMAPI PoC with dynamo-run launcher Mar 13, 2025
@tanmayv25
tanmayv25 force-pushed the tanmayv-llmapi-launch branch from a80523e to 9aef228 Compare March 13, 2025 23:37
@tanmayv25
tanmayv25 requested a review from rmccorm4 March 14, 2025 00:04
@tanmayv25 tanmayv25 changed the title WIP: LLMAPI PoC with dynamo-run launcher feat: LLMAPI PoC with dynamo-run launcher Mar 14, 2025
rmccorm4
rmccorm4 reviewed Mar 14, 2025
View reviewed changes
Comment thread examples/python_rs/llm/trtllm/common/parser.py Resolved
rmccorm4
rmccorm4 reviewed Mar 14, 2025
View reviewed changes
Comment thread lib/llm/src/engines/python.rs Resolved
@tanmayv25
tanmayv25 requested a review from rmccorm4 March 14, 2025 07:13
@tanmayv25
tanmayv25 requested a review from rmccorm4 March 14, 2025 15:46
NVShreyas
NVShreyas reviewed Mar 14, 2025
View reviewed changes
Comment thread examples/python_rs/llm/trtllm/common/parser.py Resolved
@tanmayv25
tanmayv25 enabled auto-merge (squash) March 14, 2025 17:22
@tanmayv25
tanmayv25 merged commit 4cb5a2a into main Mar 14, 2025
@tanmayv25
tanmayv25 deleted the tanmayv-llmapi-launch branch March 14, 2025 17:41
NVShreyas
NVShreyas reviewed Mar 14, 2025
View reviewed changes
Comment thread examples/python_rs/llm/trtllm/monolith/launch.py
@@ -0,0 +1,34 @@
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

@NVShreyas NVShreyas Mar 14, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this launch script?

kylehh pushed a commit to kylehh/dynamo that referenced this pull request Apr 11, 2025
@tanmayv25
feat: LLMAPI PoC with dynamo-run launcher (ai-dynamo#114)
e0bb5bd
@ziqifan617 ziqifan617 mentioned this pull request Oct 28, 2026
Merged
@callanjfox callanjfox mentioned this pull request Oct 28, 2026
Draft

This branch was previously deployed

1 inactive deployment
GITLAB — 7a60c006 Deployed Mar 14, 2025 by tanmayv25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@NVShreyas NVShreyas NVShreyas left review comments
@grahamking grahamking grahamking approved these changes
@GuanLuo GuanLuo GuanLuo approved these changes
@nnshah1 nnshah1 Awaiting requested review from nnshah1
@piotrm-nvidia piotrm-nvidia Awaiting requested review from piotrm-nvidia
@ptarasiewiczNV ptarasiewiczNV Awaiting requested review from ptarasiewiczNV
@ryanolson ryanolson Awaiting requested review from ryanolson
@paulhendricks paulhendricks Awaiting requested review from paulhendricks
@biswapanda biswapanda Awaiting requested review from biswapanda
@tmonty12 tmonty12 Awaiting requested review from tmonty12
@tedzhouhk tedzhouhk Awaiting requested review from tedzhouhk
@rmccorm4 rmccorm4 Awaiting requested review from rmccorm4

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@tanmayv25 @grahamking @rmccorm4 @GuanLuo @NVShreyas