Repository navigation

fix: Change tensorrt_llm api from tensorrt_llm._torch to tensorrt_llm - #446

Closed

sarattha wants to merge 389 commits into

ai-dynamo:dynamo/trtllm_llmapi_v1from

sarattha:f/trtllm-add_engine_support

Closed

fix: Change tensorrt_llm api from tensorrt_llm._torch to tensorrt_llm#446
sarattha wants to merge 389 commits into
ai-dynamo:dynamo/trtllm_llmapi_v1from
sarattha:f/trtllm-add_engine_support

This pull request is big! We're only showing the most recent 250 commits

Commits on Feb 26, 2025
- refactor: Move arg parsing into app for cleaner signatures (#281)
  rmccorm4
  authored
  e04325e
- refactor: using async_openai
  
  paulhendricks
  and
  grahamking
  authored
  ca72b8c
- chore: Remove 'Roff' section from repo language stats (#286)
  rmccorm4
  authored
  0b45baa
Commits on Feb 27, 2025
Commits on Feb 28, 2025
- Updates to support DS R1 in TRTLLM example (#301)
  NVShreyas
  authored
  21f71bc
- Add missing inits (#307)
  piotrm-nvidia
  authored
  5158c5a
- ci: initial public ci for PRs (#241)
  
  saturley-hall
  and
  nvda-mesharma
  authored
  06041ac
- feat: vllm engine (#308)
  grahamking
  authored
  a49c781
- feat: Add initial prometheus/grafana support for count (#303)
  rmccorm4
  authored
  497f5e1
- [fix] KV Router Example fixes (#314)
  
  alec-flowers
  and
  rmccorm4
  authored
  8ed9f22
- feat: TensorRT-LLM engine (#317)
  grahamking
  authored
  14b9f06
- refactor: use async-openai CompletionRequest (#310)
  paulhendricks
  authored
  7405264
Commits on Mar 1, 2025
- Add lazy import to nixl.py
  piotrm-nvidia
  authored
  17aa573
Commits on Mar 2, 2025
- [fix] OpenAI API object: completion to text_completion (#318)
  alec-flowers
  authored
  3923236
- chore: Update README.md
  nnshah1
  authored
  c627f4f
- chore: Tweak prometheus/grafana config defaults (#319)
  rmccorm4
  authored
  2671498
- fix: Copy nixl example for vllm_nixl container (#323)
  piotrm-nvidia
  authored
  a789154
Commits on Mar 3, 2025
- fix: Install specific toolchain (#329)
  grahamking
  authored
  a4b78de
- feat: Add compound AI python SDK
  bispnv
  committed
  20364c0
- Merge branch 'mkhadkevich/publish-compound-ai-sdk-gl' into 'main'
  bispnv
  committed
  1483d87
Commits on Mar 4, 2025
- Update README.md
  nvda-mesharma
  authored
  752d181
- ci: update container registry location
  saturley-hall
  committed
  fea0239
- ci: fix copyright checks container image
  saturley-hall
  committed
  583efbf
- ci: split out jobs into separate workflows
  saturley-hall
  committed
  4d09196
- Merge branch 'main' into harrison/ci-test
  saturley-hall
  authored
  a8e4bea
- ci[fix]: indentation
  saturley-hall
  committed
  6484762
- ci: use inline --cache-to to be able to pull image
  saturley-hall
  committed
  7a60b58
- remove timeout for cargo deny check
  nv-anants
  committed
  11ff1ce
- fix
  nv-anants
  committed
  c9a3b10
- ci: follow naming conventions
  saturley-hall
  committed
  6cccc48
- codeowners: update to dynemo org
  saturley-hall
  committed
  7c24ae1
- ci: fix copyright checks
  saturley-hall
  committed
  7a482c4
- rerun: check timing
  nv-anants
  committed
  d750624
- ci: revert to working cache location
  saturley-hall
  committed
  23e0ab9
- ci: fix copyright again
  saturley-hall
  committed
  02c7176
- Merge branch 'anants/fix-rust-ci' into harrison/ci-test
  saturley-hall
  committed
  c1b81af
- nit: update name of project in CODEOWNERS
  saturley-hall
  committed
  c7118b9
- Merge pull request #3 from dynemo-ai/harrison/ci-test
  saturley-hall
  authored
  c008780
- Update README.md (#1)
  nvda-mesharma
  authored
  5033457
- Update NIXL Dockerfile + vLLM patch with variable TP (#9)
  
  ptarasiewiczNV
  and
  Piotr Tarasiewicz Nvidia
  authored
  6d6a551
- CI: setup caching for any merge to main (#8)
  saturley-hall
  authored
  0e673dc
- feat: nixl metadata store and retrieved from etcd (#6)
  
  authored
  57d19b5
- init: helm charts (#5)
  whoisj
  authored
  c909567
- chore(github): improve PR process and rename workspace (#12)
  ishandhanani
  authored
  4d31cc6
- feat: add python binding for rust llm modules (#13)
  biswapanda
  authored
  2da0921
- feat: vllm engine tensor parallel and pipeline parallel (#16)
  grahamking
  authored
  e64935b
Commits on Mar 5, 2025
- feat: OAI compatible endpoints for TRTLLM (#14)
  
  authored
  b7cca38
- refactor: Rename 'tio' to 'dynemo-run' (#18)
  grahamking
  authored
  7dc246d
- feat: moved compoundAI operator, APIserver, and examples (#10)
  hutm
  authored
  a014b97
- ci: Add python "pre-merge" tests when merging to main (#19)
  saturley-hall
  authored
  968ca0b
- refactor: rename triton_distributed to dynemo (#22)
  
  nnshah1
  and
  grahamking
  authored
  13ab5de
- refactor: updated command line arg (#26)
  nnshah1
  authored
  af23c75
- fix: mistralrs use auto device map (#31)
  grahamking
  authored
  0ea0d09
- chore: regenerate patch (#29)
  nnshah1
  authored
  8559e0c
Commits on Mar 6, 2025
Commits on Mar 7, 2025
Commits on Mar 8, 2025
- test: add tests for kv bindings (#35)
  GuanLuo
  authored
  bfb0997
- ci: skip test redundancy in Gitlab CI (#36)
  pvijayakrish
  authored
  cc6c510
- chore: rename dynamo (#44)
  
  nnshah1
  and
  biswapanda
  authored
  2b28487
- chore: remove debug statements (#57)
  nnshah1
  authored
  f31211c
- chore: Renamed Triton Distributed to Dynamo (#56)
  dmitry-tokarev-nv
  authored
  e12685f
- ci: rename project to dynamo (#60)
  saturley-hall
  authored
  9f823fe
- Update README.md
  nvda-mesharma
  authored
  a6c8d3b
Commits on Mar 9, 2025
- fix: vLLM disagg fix incorrect block ids order (#63)
  ptarasiewiczNV
  authored
  f5db9c1
- feat: kv aware router + disagg router + prefill queue (#11)
  
  authored
  039f9a5
- integrate kv router into trtllm ex
  
  NVShreyas
  authored and
  tanmayv25
  committed
  9fb2a94
- Publishing metrics framework
  tanmayv25
  committed
  3db4668
- WIP
  tanmayv25
  committed
  d19ece9
- Some active tests
  tanmayv25
  committed
  f36eee2
- Get stats from llm api
  tanmayv25
  committed
  3777426
- Format
  tanmayv25
  committed
  8cd54b7
- Successfully publish the stats from workers
  tanmayv25
  committed
  cbcf7a7
- fix imports
  tanmayv25
  committed
  ceed883
- Fix e2e scheduler
  tanmayv25
  committed
  d640731
- Fix the worker
  tanmayv25
  committed
  14487e0
- Format fix
  tanmayv25
  committed
  fcd3dcf
- Complete the rename
  tanmayv25
  committed
  4808d90
- Rebase fixes
  tanmayv25
  committed
  0e5f8f6
- clean-up
  tanmayv25
  committed
  b826891
- Format fix
  tanmayv25
  committed
  d38c4b1
- fix
  tanmayv25
  committed
  7a6fc97
- Fix config
  tanmayv25
  committed
  8a79380
- Format fix
  tanmayv25
  committed
  cda3476
- Restore vllm
  tanmayv25
  committed
  76cd233
- doc fix
  tanmayv25
  committed
  c16f293
- Add documentation with instructions
  tanmayv25
  committed
  3a869d6
- chore: address comments for #35 (#53)
  GuanLuo
  authored
  342c6f9
- chore: left over renaming (#67)
  
  nnshah1
  and
  saturley-hall
  authored
  884ce42
- ci: remove caching of docker layers on PR builds (#61)
  saturley-hall
  authored
  a53e237
- feat: make vllm baseline support both chat and completions (#70)
  alec-flowers
  authored
  4e8b97c
- chore: stragglers rename (#69)
  
  nnshah1
  and
  saturley-hall
  authored
  53e28ab
- feat: make block_size input for indexer, router, publisher (#66)
  alec-flowers
  authored
  381bb02
Commits on Mar 10, 2025
- Format fix
  tanmayv25
  committed
  90eb0d4
- build(deps): bump transformers from 4.45.2 to 4.48.3 (#58)
  dmitry-tokarev-nv
  authored
  b4a6a5d
- Fix format
  tanmayv25
  committed
  7bf0e58
- Fix mypy test issues
  tanmayv25
  committed
  469ad42
- Merge branch 'main' into tanmayv-exp-router
  tanmayv25
  authored
  1bf9f49
- Address review comments
  tanmayv25
  committed
  7e159a2
- format fix
  tanmayv25
  committed
  c5bf392
- init kv aware agg
  NVShreyas
  committed
  fd8751e
- chore: Add dynamo-run to workspace file (#76)
  rmccorm4
  authored
  61e13a8
- working multi instance
  NVShreyas
  committed
  6d93838
- style: fix formatting for .go file (#62)
  nv-anants
  authored
  65f98b2
- add kv publishing
  NVShreyas
  committed
  d036aea
- ci: start using ECR for container caches (#77)
  saturley-hall
  authored
  36210d5
- fix(dynamo-run): Text input doesn't need a name (#80)
  grahamking
  authored
  ee93f80
- feat: LLM API integration with smart routing bits (#55)
  
  tanmayv25
  and
  NVShreyas
  authored
  4482f8a
- feat: Add configurable DYN_TOKEN_ECHO_DELAY_MS for echo engine testing (#81)
  rmccorm4
  authored
  0954c94
- add script to launch
  NVShreyas
  committed
  e697131
- update scripts
  NVShreyas
  committed
  8e5c288
- Merge branch 'main' into shreyasm/kv-aware-agg
  NVShreyas
  committed
  56f621b
- only run router when n workers>1
  NVShreyas
  committed
  c2ca4f2
- update scrip
  NVShreyas
  committed
  a39840a
- chore: update wheel name and reset versions (#73)
  nv-anants
  authored
  a7ee871
Commits on Mar 11, 2025
- feat: add openai http service (#82)
  biswapanda
  authored
  5f3882f
- log level
  NVShreyas
  committed
  ebb12dc
- add args for routing strategy
  NVShreyas
  committed
  ff21a18
- style: fix go linting errors (#86)
  nv-anants
  authored
  e2e913a
- feat: unified entry point for vllm-nixl (#83)
  tedzhouhk
  authored
  56b0d4d
- feat(sdk): pass in CLI args when running `serve` (#78)
  ishandhanani
  authored
  ba806a0
- fix: Include GAP hot fix in VLLM NIXL container (#90)
  piotrm-nvidia
  authored
  426c810
- chore: Simplify the container build instructions for LLMAPI example (#87)
  tanmayv25
  authored
  ebec5c1
- fixes
  NVShreyas
  committed
  275d940
- fix: add multi-node deployment instruction for vllm-nixl (#93)
  tedzhouhk
  authored
  b347fe1
- fix: update vLLM patch to 0aa204 (#92)
  ptarasiewiczNV
  authored
  cd56ad0
- fix: inconsistent router args (#94)
  tedzhouhk
  authored
  e8092e6
- feat: add new metrics and simple router cost fn (#88)
  alec-flowers
  authored
  188f21a
- feat: add script to deploy a Dynemo pipeline in k8s using helm (#42)
  julienmancuso
  authored
  c3af89f
- fix: Support vllm_nixl (custom vllm patch) from dynamo-run (#84)
  rmccorm4
  authored
  4559c16
- refactor: Move rust binaries out of examples, update nixl dockerfile (#89)
  
  nnshah1
  and
  nvda-mesharma
  authored
  012ac9b
- feat(dynamo-run): Upgrade mistral.rs (#97)
  grahamking
  authored
  518bf85
- feat: kv aware disagg router (#98)
  
  tedzhouhk
  and
  alec-flowers
  authored
  52892ba
- fix(pystr): Output python errors (#99)
  grahamking
  authored
  818da69
- docs(dynamo-run): Fix for workspace (#102)
  grahamking
  authored
  65caf24
- fix: Add missing arg to echo_full example and source cargo env in setup steps (#101)
  rmccorm4
  authored
  e4d821c
- refactor: rename vllm_nixl to vllm and make default (#100)
  nnshah1
  authored
  aa4e78b
- fix: Fix build regression in TRTLLM container with th rename. (#103)
  tanmayv25
  authored
  20963b0
- feat: remove Dynamo API Server's dependency on NDS (#54)
  hhzhang16
  authored
  72c7301
Commits on Mar 12, 2025
- fix: Add missing util files to vllm example (#105)
  nnshah1
  authored
  7b420b2
- feat: rename dynamo-sdk to dynamo; add dynamo run to call dynamo-run under the hood for unification (#104)
  ziqifan617
  authored
  e2f0581
- build: Changes to support building trtllm image in CI (#108)
  tanmayv25
  authored
  8be417b
- fix: bump nixl version (#110)
  tedzhouhk
  authored
  ea7f489
- feat: added back smart routing and basic vllm examples (#111)
  hutm
  authored
  5532212
- revert: "feat: added back smart routing and basic vllm examples (#111)" (#115)
  nv-anants
  authored
  1c09ef3
- test: add basic support for pytest codeblocks (#117)
  nnshah1
  authored
  0a67e77
- Create SECURITY.md (#95)
  dmitry-tokarev-nv
  authored
  3a8ed4d
- Merge branch 'main' into shreyasm/kv-aware-agg
  NVShreyas
  committed
  7b653e4
- add custom cost fn
  NVShreyas
  committed
  3527628
- some fixes
  NVShreyas
  committed
  c391d08
- fix: Update trigger_ci.yml to enable CI pipeline trigger for dynamo deploy (#119)
  hutm
  authored
  1e04641
- chore: Reduce conditional prefill logs (#121)
  ptarasiewiczNV
  authored
  95b86dd
- chore: add codeowner (#122)
  
  tedzhouhk
  and
  nv-anants
  authored
  7a3c283
- feat: Support prometheus push gateway for use cases behind a firewall (#64)
  rmccorm4
  authored
  d9cf9d0
- feat(pystr): Pass command line arguments (#123)
  grahamking
  authored
  63adf11
- worker scores always 0
  NVShreyas
  committed
  23b191b
- test: enable unit tests for Dynamo API server (#107)
  hhzhang16
  authored
  be1f88d
- publish each block event individually
  NVShreyas
  committed
  2a49019
Commits on Mar 13, 2025
- feat(dynamo-run): Download models from HF, smart model defaults (#126)
  
  grahamking
  and
  rmccorm4
  authored
  6b016ff
- fix: Fix TRTLLM chat to work with latest ToT (#127)
  
  tanmayv25
  and
  rmccorm4
  authored
  4c3fdfb
- feat: onboard nixl based vllm example to dynamo serve (#120)
  biswapanda
  authored
  b89a3ed
- docs: Updated macOS build instructions for dynamo-run. (#131)
  dmitry-tokarev-nv
  authored
  ba57201
- build: add CARGO_BUILD_JOBS argument to Dockerfiles (#132)
  pziecina-nv
  authored
  e46a511
- fix(dynamo-run): Network interface detection is Linux only (#133)
  grahamking
  authored
  f0759bd
- build: remove hardcoded cargo jobs args (#136)
  nv-anants
  authored
  6071c15
- feat(mistralrs): Let the engine enforce max tokens (#134)
  grahamking
  authored
  453e017
- Merge branch 'shreyasm/kv-aware-agg' of https://github.com/ai-dynamo/dynamo into shreyasm/kv-aware-agg
  NVShreyas
  committed
  6f5f261
- still debugging
  NVShreyas
  committed
  fa03ed1
- build: add top level rust workspace (#137)
  nv-anants
  authored
  e2aa009
- fixed
  NVShreyas
  committed
  4226ee6
- refactor: update dynamo run cmd to hint user to install dynamo-run when missing (#143)
  ziqifan617
  authored
  65d65ba
- LLMAPI PoC with dynamo-run launcher
  tanmayv25
  committed
  a6d4afe
- Clean-up extra code to avoid confusion
  tanmayv25
  committed
  1d50d1b
- Update to use new CLI options
  tanmayv25
  committed
  718757c
- Refactor Monolith worker to integrate with dynamo-run
  tanmayv25
  committed
  9aef228
- build: Remove nats and etcd sources from final build (#144)
  rmccorm4
  authored
  0fec087
Commits on Mar 14, 2025
- Format fix
  tanmayv25
  committed
  61c03e5
- Merge branch 'main' into tanmayv-llmapi-launch
  tanmayv25
  authored
  82693a9
- feat: global kv block manager (#45)
  ryanolson
  authored
  44fedb6
- Merge branch 'main' into tanmayv-llmapi-launch
  tanmayv25
  authored
  0b4bfaa
- Address review comments
  tanmayv25
  committed
  c1501b7
- Remove extra artifacts
  tanmayv25
  committed
  8fed2c1
- refactor: Remove STANDARD and VLLM_NIXL choices from build/run (#148)
  rmccorm4
  authored
  86761f0
- fix: Add missing binaries back into container build (#152)
  rmccorm4
  authored
  4b7b53e
- fix: Fix cargo doc warnings for lib/llm (#151)
  rmccorm4
  authored
  26f8a22
- fix: Fix cargo doc warnings for lib/runtime (#150)
  rmccorm4
  authored
  da35236
- Address review comments
  tanmayv25
  committed
  71a14b6
- Merge branch 'main' into tanmayv-llmapi-launch
  tanmayv25
  authored
  af574b5
- feat: add helm charts for deployment (#145)
  
  hhzhang16
  and
  julienmancuso
  authored
  8d78e33
- fix: Various for MacOS (#155)
  grahamking
  authored
  4eb7b10
- new metrics
  NVShreyas
  committed
  c557c8b
- Merge branch 'tanmayv-llmapi-launch' into shreyasm/kv-aware-agg
  NVShreyas
  committed
  375a6fd
- feat: LLMAPI PoC with dynamo-run launcher (#114)
  tanmayv25
  authored
  4cb5a2a
- build: reorganize python packaging to build new wheels (#118)
  nv-anants
  authored
  5a0aca7
- refactor: Update default log level to INFO and promote/demote a few log messages (#159)
  rmccorm4
  authored
  4402982
- combine disagg and agg chat processors
  NVShreyas
  committed
  1d7d7a9
- more fixes
  NVShreyas
  committed
  44dca9c
- fix: Improve error handling for failed HF download (#160)
  rmccorm4
  authored
  0f505d2
- some refactoring
  NVShreyas
  committed
  d13d3c7
- add known issues
  NVShreyas
  committed
  0c5c1d4
- reset bindings path
  NVShreyas
  committed
  7227d19
- Merge branch 'main' into shreyasm/kv-aware-agg
  NVShreyas
  committed
  26e6968
- rename to kv router
  NVShreyas
  committed
  a1013f7
- build: use wheel for vllm install (#163)
  nv-anants
  authored
  dc1caff
- fix: wrong indent -> only one worker metric (#161)
  tedzhouhk
  authored
  49e8ce9
- chore: add alec to .py and .rs code owner (#162)
  tedzhouhk
  authored
  a2f241d
- cleanup
  NVShreyas
  committed
  728f52a
- fix(mac): Fix for virtual env (#164)
  grahamking
  authored
  c8942c7
- feat(sdk): add initial graph structure for prebuilt components (#130)
  
  ishandhanani
  and
  biswapanda
  authored
  46d0b92
- feat: Support caching nixl build stage (#147)
  rmccorm4
  authored
  bebb85a
- fix: indent issue (duplicated source code) (#165)
  tedzhouhk
  authored
  1e39832
- ci: Improve summarizing the test report (#153)
  pvijayakrish
  authored
  27afbb9
- Update examples/python_rs/llm/trtllm/common/base_engine.py
  
  NVShreyas
  and
  tanmayv25
  authored
  b03c778
- Merge branch 'main' into shreyasm/kv-aware-agg
  NVShreyas
  committed
  f114e43
- address comments
  NVShreyas
  committed
  49829d7
Commits on Mar 15, 2025
- fix
  NVShreyas
  committed
  56848dd
- minor fixes
  NVShreyas
  committed
  17621a9
- update readme
  NVShreyas
  committed
  e883288
- another fix cause trtllm keeps changing
  NVShreyas
  committed
  507d217
Commits on Mar 16, 2025
- build: Remove internal reference from TensorRT-LLM build (#189)
  tanmayv25
  authored
  4d99f64
Commits on Mar 17, 2025
- remove request.stream
  NVShreyas
  committed
  be8bf32
- Move the example to outer directory
  tanmayv25
  committed
  07c3739
- Fix docs in kv routing
  tanmayv25
  committed
  26ebe20
Commits on Mar 25, 2025
- fix: Heavy logging by TensorRT-LLM monolith worker (#347)
  kthui
  authored
  db35630
Commits on Mar 31, 2025
- Change tensorrt_llm api from tensorrt_llm._torch to tensorrt_llm
  sarattha
  committed
  97ac934

Commits

Commits on Feb 26, 2025

Commits on Feb 27, 2025

Commits on Feb 28, 2025

Commits on Mar 1, 2025

Commits on Mar 2, 2025

Commits on Mar 3, 2025

Commits on Mar 4, 2025

Commits on Mar 5, 2025

Commits on Mar 6, 2025

Commits on Mar 7, 2025

Commits on Mar 8, 2025

Commits on Mar 9, 2025

Commits on Mar 10, 2025