feat(trtllm): plumb conversation_id to engine for conversation-aware ADP routing - #10517
Conversation
…ADP routing In disaggregated serving, TRT-LLM's ConversationAwareADPRouter pins each conversation to an attention-DP rank (KV locality) by reading disaggregated_params.conversation_id. Native trtllm-serve populates this from the client's conversation routing; the Dynamo TRT-LLM workers did not (gap #4), so the engine's conversation-affinity router never engaged and CTX prefill load scattered across ranks -> prefill queue -> high TTFT tail. This plumbs conversation_id end-to-end so the engine router engages: - RoutingHints gains conversation_id, seeded by the OpenAI preprocessor from nvext.session_control.session_id (lib/llm). - Both the unified worker (llm_engine.py) and the legacy handler (request_handlers/handler_base.py) set disaggregated_params.conversation_id on the engine request, on prefill and decode. - Gated by DYN_ENGINE_CONV_AFFINITY=1: when set, the worker stops force-feeding attention_dp_rank (an explicit rank is honored before conversation affinity in the engine and would bypass ConversationAwareADPRouter). - dp_rank.conversation_id_from_request helper + unit test; one-shot INFO log to confirm the runtime invariants without enabling debug. Validated on Lyris GB300 (DSV4-Pro, 3xCTX-DEP4 + 1xGEN-DEP8, c460, perf-off) with engine config kv_cache_routing_conversation_affinity=true: TTFT p95 8.39 -> 4.84s (~42%), matching native trtllm-serve (4.81); p99 7.12 vs 7.00; 46 -> 50 req/s. Runtime-confirmed: conversation_id set + attention_dp_rank not forced, on both PREFILL and DECODE. Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
|
👋 Hi nv-yna! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
github-actions
Bot
added
feat
external-contribution
| # disaggregated_params.conversation_id (round-robin balanced + sticky), matching | ||
| # native trtllm-serve. Otherwise honour the frontend KV router's DP rank decision | ||
| # (without it TRT-LLM picks its own rank and KV events land on the wrong publisher). | ||
| if self._engine_conv_affinity: |
There was a problem hiding this comment.
With DYN_ENGINE_CONV_AFFINITY=1, this suppresses the router-forced attention_dp_rank even when _set_on_disagg is false, so requests without a propagated conversation id can run on a different ADP rank than the router selected. Fix: only bypass SchedulingParams when the conversation id was actually attached to disaggregated_params.
🤖 AI Fix
In components/src/dynamo/trtllm/llm_engine.py, inside TrtllmLLMEngine._generate_started, change the scheduling branch to if self._engine_conv_affinity and _set_on_disagg: scheduling_params = None and keep the existing validate_global_dp_rank(forced_dp_rank(...)) forced-rank construction in the else branch.
| engine_conv_affinity = os.environ.get("DYN_ENGINE_CONV_AFFINITY") == "1" | ||
| scheduling_params = None | ||
| if dp_rank is not None: | ||
| if dp_rank is not None and not engine_conv_affinity: |
There was a problem hiding this comment.
With DYN_ENGINE_CONV_AFFINITY=1, this skips SchedulingParams for every request with routing.dp_rank, even when disaggregated_params has no conversation_id, so non-session or aggregated requests lose strict Dynamo DP routing. Fix: suppress attention_dp_rank only when a non-empty engine conversation id is present.
🤖 AI Fix
In components/src/dynamo/trtllm/request_handlers/handler_base.py, inside HandlerBase._generate_locally_impl, compute has_engine_conversation_id = bool(getattr(disaggregated_params, "conversation_id", None)) after engine_conv_affinity, and change the condition to if dp_rank is not None and not (engine_conv_affinity and has_engine_conversation_id):.
krishung5
left a comment
krishung5
left a comment
There was a problem hiding this comment.
Could you take a look at dynamo-ops comments and see if we want to address it?
3 participants
Summary
In disaggregated serving, TRT-LLM's
ConversationAwareADPRouterpins each conversation to an attention-DP rank (for KV locality) by readingdisaggregated_params.conversation_id. Nativetrtllm-servepopulates this; the Dynamo TRT-LLM workers did not (gap #4), so the engine's conversation-affinity router never engaged and CTX prefill load scattered across ranks → prefill queue → high TTFT tail.This plumbs
conversation_idend-to-end so the engine router engages.Changes
RoutingHintsgainsconversation_id, seeded by the OpenAI preprocessor fromnvext.session_control.session_id(lib/llm).llm_engine.py) and the legacy handler (request_handlers/handler_base.py) setdisaggregated_params.conversation_idon the engine request, on prefill and decode.DYN_ENGINE_CONV_AFFINITY=1: when set, the worker stops force-feedingattention_dp_rank— an explicit rank is honored before conversation affinity in the engine and would bypassConversationAwareADPRouter.dp_rank.conversation_id_from_requesthelper + unit test; one-shot INFO log to confirm the runtime invariants without enabling debug.Validation — Lyris GB300, DSV4-Pro, 3×CTX-DEP4 + 1×GEN-DEP8, c460, perf-off
With the engine config
kv_cache_routing_conversation_affinity: true(25-min run; 55-min full in progress):TTFT p95 dropped ~42%, matching native. Runtime-confirmed (one-shot INFO):
conversation_idset +attention_dp_ranknot forced, on both PREFILL and DECODE.Notes / dependencies
DisaggregatedParams.conversation_id(TRT-LLM PR #14983 / equivalent) and the client to emitnvext.session_control(use_dynamo_conv_aware_routing/ artificial-analysis MR 129).DYN_ENGINE_CONV_AFFINITY); could be promoted to a config knob.