Skip to content

[TRTLLM-12721][feat] Add disagg transfer state consensus - #2

Closed
chienchunhung wants to merge 1129 commits into
docs-and-plansfrom
codex/disagg-inflight-cancel-impl
Closed

[TRTLLM-12721][feat] Add disagg transfer state consensus#2
chienchunhung wants to merge 1129 commits into
docs-and-plansfrom
codex/disagg-inflight-cancel-impl

Conversation

@chienchunhung

@chienchunhung chienchunhung commented Jun 9, 2026

Copy link
Copy Markdown
Owner

Description

Adds a V1 CacheTransceiver request-state consensus step for disaggregated KV transfer terminal transitions.

The implementation records local completed/failed transfer outcomes, packs {request_id, state} entries, gathers them through the existing communication backend, and applies a deterministic consensus result before mutating request state. Context-side consensus uses the existing TP/CP sync group and then the PP group; generation-side consensus uses the existing generation sync group.

This keeps PR#1 scoped to the consensus foundation for safe in-flight cancellation. It does not yet add deadline enforcement, deferred cleanup/quarantine, richer cancellation states, or hard mid-transfer abort semantics; those remain for the follow-up cancellation/lifetime PR.

Notes

  • Failed terminal state wins over completed once all participating ranks report a terminal state.
  • Generation transfer no longer overwrites a receiver-side kDISAGG_TRANS_ERROR with kDISAGG_GENERATION_TRANS_COMPLETE.
  • The existing ready-request-ID collectives are intentionally left in place for this first slice.

Testing

  • pre-commit run --files cpp/include/tensorrt_llm/batch_manager/cacheTransceiver.h cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp
  • git diff --check -- cpp/include/tensorrt_llm/batch_manager/cacheTransceiver.h cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp

sara4dev and others added 30 commits May 27, 2026 21:26
@sara4dev @chang-l
[https://nvbugs/6115560][fix] catch OSError in config_file_lock for N…
59d4369 
…FS compatibility (NVIDIA#11960)

Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
Co-authored-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
@bmarimuthu-nv
[None][fix] Fix OSRB source header and provenance issues in AutoDeplo…
89c90eb 
…y modeling code (NVIDIA#14670)

Signed-off-by: Balamurugan Marimuthu <246387390+bmarimuthu-nv@users.noreply.github.com>
@tensorrt-cicd @nvchenghaoz
[https://nvbugs/6185173][fix] Set mamba ssm cache to fp32 for Nemotro…
fc3b69e 
…nV2 (NVIDIA#14448)

Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com>
Co-authored-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com>
@byshiue @tensorrt-cicd
[https://nvbugs/5800725][fix] Restore Mistral Large 3 text-only proce…
83ec591 
…ssor (NVIDIA#14248)

Signed-off-by: bhsueh <11360707+byshiue@users.noreply.github.com>
Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
Co-authored-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
@sunnyqgg
[None][test] Unwaive fp8 blockscale baseline mtp1 (NVIDIA#14666)
9feaa76 
Signed-off-by: qgai <qgai@nvidia.com>
@fuergaosi233
[None][docs] fix incorrect auto sampler behavior description for beam…
82f69a0 
… search (NVIDIA#14487)
@eopXD
[None][feat] Expose host/GPU per-iter time and clarify iter labeling …
fe07957 
…in /metrics (NVIDIA#14127)

Signed-off-by: Yueh-Ting Chen <yuehtingc@nvidia.com>
Signed-off-by: Yueh-Ting Chen <yueh.ting.chen@gmail.com>
@pengbowang-nv
[None][chore] Add test lists with multi-gpu test to CI multi-gpu test…
f391155 
… trigger files (NVIDIA#14087)

Signed-off-by: Pengbo Wang <221450789+pengbowang-nv@users.noreply.github.com>
@yuxianq
[None][refactor] Add derived properties for the thop.attention call s…
09bad73 
…ite (NVIDIA#14279)

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>
@tensorrt-cicd
[https://nvbugs/6160085][fix] At `tensorrt_llm/tokenizer/tokenizer.py…
e73d068 
…` import time, re-export `bytes_to_unicod (NVIDIA#14116)

Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
@ZhanruiSunCh
[None][infra] Waive 1 failed cases for main in post-merge 2740 (NVIDI…
89bcd80 
…A#14688)

Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>
@greg-kwasniewski1 @galagam
[https://nvbugs/6221483][fix] Revert auto_deploy _mamba_ssm_prepare_m…
6484b71 
…etadata to pre-NVIDIA#13566 state (NVIDIA#14640)

Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com>
Signed-off-by: Gal Hubara-Agam <96368689+galagam@users.noreply.github.com>
Co-authored-by: Gal Hubara-Agam <96368689+galagam@users.noreply.github.com>
@pcicotti
[TRTLLM-12762][fix] Enable multi-node TP for MiniMax-M2 (NVIDIA#14314)
82679b5 
Signed-off-by: Pietro Cicotti <5833013+pcicotti@users.noreply.github.com>
@yibinl-nvidia
[https://nvbugs/6043248][fix] Validate tensor payload size on deseria…
0432a81 
…lization (NVIDIA#14648)

Signed-off-by: Yibin-Li <109242046+yibinl-nvidia@users.noreply.github.com>
@tcherckez-nvidia
[None][perf] Add AutoDeploy NVFP4 RMSNorm quant fusion (NVIDIA#14361)
56be625 
Signed-off-by: Tal Cherckez <127761168+tcherckez-nvidia@users.noreply.github.com>
Signed-off-by: Tal Cherckez <tcherckez@nvidia.com>
@eopXD
[None][feat] Support Gemma4 multi-head_dim pools and host-side slicin…
13ca44a 
…g to provide local view to Triton kernels for SWA (NVIDIA#13745)

Signed-off-by: Yueh-Ting Chen <yuehtingc@nvidia.com>
@greg-kwasniewski1
[TRTLLM-13960][test] Offline equivalence test for sharding IR (NVIDIA…
0715e15 
…#13963)

Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com>
@NVShreyas
[TRTLLM-11410][feat] MoT World Model Support (NVIDIA#14012)
5c896cd 
Signed-off-by: Shreyas Misra <shreyasm@nvidia.com>
Signed-off-by: Shreyas Mista <shreyasm@nvidia.com>
@hyukn @claude
[https://nvbugs/6115036][fix] Fix NVFP4 engine size estimation and at…
7443f1d 
…tention DP batch size in trtllm-bench (NVIDIA#13498)

Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@yibinl-nvidia
[https://nvbugs/5972776][fix] Pass IPC HMAC key through file descript…
50ca49f 
…or (NVIDIA#14378)

Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>
@yibinl-nvidia
[https://nvbugs/5911594][fix] Restrict HTTP cluster storage to loopba…
e8a42a1 
…ck (NVIDIA#14161)

Signed-off-by: Yibin Li <109242046+yibinl-nvidia@users.noreply.github.com>
@achartier
[None][fix] Exclude post-merge stages from CBTS force-keep filters (N…
e173446 
…VIDIA#14594)

Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>
@tburt-nv
[TRTLLMINF-67][infra] use pre-configured idle GPU exemption (NVIDIA#1…
776bebc 
…4587)

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>
@tensorrt-cicd
[https://nvbugs/6207749][fix] Replace the spec with onnx>=1.21.0 in…
e96b710 
… `requirements.txt`; mirror in `security_ (NVIDIA#14577)

Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
@nvchenghaoz
[https://nvbugs/6185480][fix] Autodeploy skip the GLM accuracy test f…
22c7956 
…or pre-hopper (NVIDIA#14656)

Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com>
@taylor-yb-lee
[https://nvbugs/6165866][infra] Waive 1 failed cases for main in pre-…
da9db3a 
…merge 40081 (NVIDIA#14653)

Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>
@tensorrt-cicd @nvchenghaoz
[https://nvbugs/6187185][fix] Apply the existing `low_memory_override…
56df200 
…s()` helper in `TestNemotronV2.test_auto_ (NVIDIA#14584)

Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com>
Co-authored-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com>
@MrGeva
[https://nvbugs/6192201][fix] AutoDeploy: unwaive llama perf test and…
e6784d8 
… increase its concurrency to 256 (NVIDIA#14691)

Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>
@ixlmar
[TRTLLM-12982][feat] improve attention backend selection (NVIDIA#14635)
f6ba936 
Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com>
@lowsfer
[None][test] Enable test for kv_cache_manager_v2 for A10 (NVIDIA#12885)
e35f0f4 
Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>
greg-kwasniewski1 and others added 25 commits June 8, 2026 09:55
@greg-kwasniewski1
[None][feat] AutoDeploy: propagate layer_type hint across pattern-mat…
02f6b2f 
…cher rewrites (NVIDIA#14835)

Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com>
@tensorrt-cicd @crazydemo
[None][test] Waive 15 failed cases for main in QA CI (NVIDIA#15056)
28dc25e 
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Co-authored-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
@ZhanruiSunCh
[None][infra] Waive 1 failed cases for main in pre-merge 41894 (NVIDI…
2febb37 
…A#15089)

Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com>
@QiJune
[TRTLLM-13262][ci] Move non-default-feature tests to post merge (NVID…
09c21b6 
…IA#15038)

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
@reasonsolo
[None][feat] Enable disk cache config for KV cache v2 (NVIDIA#14845)
c93c63d 
Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com>
@pengbowang-nv
[https://nvbugs/6185446][fix] Add warmup for trtllm-gen fmha JIT kern…
6dee167 
…els (NVIDIA#14851)

Revert c37992c to remove padding.
Added warmup with a grid that should be dense enough to catch almost every possible JIT kernels. Note that this PR only enables warmup and have no shape binning at runtime. I'd like to see how it performs and if it can warmup most kernels then we may not need further works.
Added a message when a possible JIT happened, to give a clear clue why performance dropped.

Signed-off-by: Pengbo Wang <221450789+pengbowang-nv@users.noreply.github.com>
@longlee0622
[https://nvbugs/6162940][chore] Unwaive fixed test (NVIDIA#15078)
b14794c 
Signed-off-by: Jonas Li <6110159+longlee0622@users.noreply.github.com>
@nv-guomingz
[None][perf] Support Gemma RMSNorm + interleaved mRoPE in fused_qk_no… (
2bf4d3d 
NVIDIA#14898)

Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
@chenfeiz0326
[None][test] Half K25 Agg Multi Round to Solve Timeout Issue (NVIDIA#…
9eaa468 
…15083)

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
@tburt-nv @janbernloehr
[None][infra] Reduce Docker image layer count in release stage (NVIDI…
9af8a16 
…A#14972)

Signed-off-by: jbernloehr <jbernloehr@nvidia.com>
Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>
Co-authored-by: jbernloehr <jbernloehr@nvidia.com>
@MrGeva
[NVIDIA#14828][feat] AutoDeploy: support multi KV cache memory pool i…
cb01607 
…n trtllm attention (NVIDIA#14911)

Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com>
Signed-off-by: egeva <19514940+MrGeva@users.noreply.github.com>
@nv-guomingz
[None][doc] Refine Nemotron Ultra doc (NVIDIA#15113)
15d06c0 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
@mzweilz
[None][infra] Waive TestQwen3NextInstruct nvfp4 cases (NVIDIA#15086)
8036cde 
Signed-off-by: Abby Wei <18545893+mzweilz@users.noreply.github.com>
@tensorrt-cicd
[https://nvbugs/6248757][fix] Avoid running all reduce in aux stream (N…
1998324 
…VIDIA#14917)

Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
@govind-ramnarayan
[https://nvbugs/6221483][fix] AutoDeploy: Fix Eagle metadata host syn…
900d069 
…cs (NVIDIA#14714)

Signed-off-by: Govind Ramnarayan <105831528+govind-ramnarayan@users.noreply.github.com>
@karljang
[None][feat] add FLUX visual generation examples (NVIDIA#14987)
9827c21 
Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
@tensorrt-cicd @govind-ramnarayan
[https://nvbugs/6261164][fix] In the kvcache insert transform (`_Inse…
b222246 
…rtCachedOperator._apply`), when… (NVIDIA#15020)

Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
Signed-off-by: Govind Ramnarayan <105831528+govind-ramnarayan@users.noreply.github.com>
Co-authored-by: Govind Ramnarayan <105831528+govind-ramnarayan@users.noreply.github.com>
@tensorrt-cicd
[https://nvbugs/6211189][fix] Lower the reference to 46.5 (matching c…
c1e9b00 
…ross-GPU empirical mean) and remove the t (NVIDIA#14799)

Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
@bobboli
[None][refactor] split VisualGen pipeline and model configs (NVIDIA#1…
bfb4537 
…4956)

Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
@luyiyun1021
[TRTLLM-11457][feat] Async Ulysses pipeline (Enabled for LTX-2 + WAN) (…
5e3af40 
…NVIDIA#13978)

Signed-off-by: Yiyun Lu <55233584+luyiyun1021@users.noreply.github.com>
@nv-guomingz
[TRTLLM-11548][doc] Add Qwen3.5 deployment guide doc (NVIDIA#15111)
09ebc59 
Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com>
@tensorrt-cicd
[https://nvbugs/6181383][fix] Build inner text/vision/audio sub-confi…
a33dec7 
…gs as empty PretrainedConfig() then setat (NVIDIA#14399)

Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
@tensorrt-cicd
[None][infra] Check in most recent lock file from nightly pipeline
041ed83 
Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com>
@tburt-nv
[https://nvbugs/6273850][chore] waive TestQwen3_5_4B::test_bf16 for a…
2490441 
…ll GPUs (NVIDIA#15112)

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>
@bmarimuthu-nv
[None][doc] Add docs for AutoDeploy transforms (NVIDIA#15122)
64497e2 
Signed-off-by: Balamurugan Marimuthu <246387390+bmarimuthu-nv@users.noreply.github.com>
@chienchunhung
[TRTLLM-12721][feat] Add disagg transfer state consensus
5115366 
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung
chienchunhung force-pushed the codex/disagg-inflight-cancel-impl branch from 30d689f to 5115366 Compare June 9, 2026 05:05
@chienchunhung

chienchunhung commented Jun 9, 2026

Copy link
Copy Markdown
Owner Author

Closing this accidental draft because it targeted docs-and-plans. The implementation branch has been rebased onto upstream/main and will be opened against NVIDIA/TensorRT-LLM:main instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@chienchunhung chienchunhung

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.