Simple and stable Inference APIs - #4697

Merged

YangFei1990 merged 49 commits into

NVIDIA:mainfrom

YangFei1990:inference_apis

May 29, 2026

Commits on May 7, 2026

feat(inference): add ServeConfig and _EventLoopManager

YangFei1990
and
claude
committed
8debc5e
feat(inference): add coordinator runtime and _MegatronLLMBase

YangFei1990
and
claude
committed
4ca3c95
feat(inference): add MegatronAsyncLLM, slim base class

YangFei1990
and
claude
committed
d7e68f1
feat(inference): add MegatronLLM

YangFei1990
and
claude
committed
c03ab48
refactor(inference): drop model_name fields from ServeConfig

YangFei1990
and
claude
committed
5c38044
feat(inference): add MegatronAsyncLLM.serve(), drop ServeConfig.role

YangFei1990
and
claude
committed
6331cc0
feat(inference): add offline_inference example, fix high-level API bugs

YangFei1990
and
claude
committed
b6448e8

Commits on May 8, 2026

feat(inference): add launch_inference_server example, fix daemon-thread CUDA device

YangFei1990
and
claude
committed
44ec5a9
fix(tests): repoint inference recipes and cuda_graphs.sh to examples/inference/legacy/

YangFei1990
and
claude
committed
0d8ae8b
test(inference): add unit tests for the high-level inference API

YangFei1990
and
claude
committed
589626d
test(inference): add functional tests for offline_inference 4 modes with reused legacy goldens

YangFei1990
and
claude
committed
2eb2e81
test(inference): add HTTP smoke test for launch_inference_server with bespoke driver

YangFei1990
and
claude
committed
739f3b0
docs(inference): add README for the high-level inference API

YangFei1990
and
claude
committed
8ff9a26
docs(inference): rewrite examples README and remove stale llama_mistral inference sections

YangFei1990
and
claude
committed
9e7fae3
Merge branch 'main' into inference_apis
YangFei1990
authored
5f651b7
ci(inference): satisfy linting, copyright-check, and build-docs

YangFei1990
and
claude
committed
34494c5
Merge branch 'inference_apis' of https://github.com/YangFei1990/Megatron-LM into inference_apis
YangFei1990
committed
8caa84d
Merge branch 'main' into inference_apis
YangFei1990
authored
963f663

Commits on May 18, 2026

handle coordinator subprocess during partial shutdown
YangFei1990
committed
67a9122
do not hardcode return_log_probs
YangFei1990
committed
b754bd2
raise when running sync inside a eventloop
YangFei1990
committed
e1c6353
add future roadmap and knwn limitations
YangFei1990
committed
dd70796
always use Optional instead of |; remove unneeded doc
YangFei1990
committed
db9f475
raise error when ep > 1 and coordinator is not enabled
YangFei1990
committed
30f1020
use event to wait the loop ready
YangFei1990
committed
93b30a0
docs(inference): rename examples/inference/legacy to advanced and explain purpose

YangFei1990
and
claude
committed
2759cdd
fix bcast and import path
YangFei1990
committed
856c851
forbid running async + direct
YangFei1990
committed
c38eb66
refactor(inference): move high-level API to megatron/core/inference/apis

YangFei1990
and
claude
committed
34ca470
Merge branch 'main' into inference_apis
YangFei1990
committed
dac4fc2
update doc
YangFei1990
committed
4be95ff
Merge branch 'main' into inference_apis
YangFei1990
authored
a3aade9

Commits on May 20, 2026

fix lint
YangFei1990
committed
dd41c05
simplify tests
YangFei1990
committed
4cff952
add meaningful tests
YangFei1990
committed
450380d
Merge branch 'main' into inference_apis
YangFei1990
authored
1bbdada
fix lint
YangFei1990
committed
ec780e5

Commits on May 21, 2026

Merge branch 'main' into inference_apis
YangFei1990
authored
7f53bd1
modify inaccurate doc and make torch import default
YangFei1990
committed
7b513a0

Commits on May 28, 2026

Commits on May 29, 2026