feat: add EncoderCacheManager to TRT-LLM PrefillHandler

[furionw]

Just plug in as member variable.

Not use it yet.

Test Plan

running a few unit tests inside container --

pytest components/src/dynamo/trtllm/tests/request_handlers -v

Summary by CodeRabbit

Release Notes

• New Features

  • Added encoder cache configuration support via new --encoder-cache-capacity-gb command-line argument.
  • Introduced intelligent request coalescing to prevent duplicate encoding computations for concurrent requests with the same input.
  • Added cache statistics tracking and monitoring capabilities.

• Tests

  • Expanded test coverage for encoder cache functionality and request handler configurations.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

This PR introduces AsyncEncoderCache, an async wrapper around EncoderCacheManager that implements request coalescing to prevent duplicate concurrent computations for the same key. Configuration changes propagate encoder_cache_capacity_gb through RequestHandlerConfig, CLI arguments, and utility layers. PrefillHandler is updated to accept and utilize an encoder_cache parameter during initialization. Validation for negative capacity in EncoderCacheManager is removed.

Changes

Cohort / File(s)	Summary
Core encoder cache validation removal components/src/dynamo/common/memory/encoder_cache_manager.py	Removed positive capacity validation in __init__, allowing zero or negative capacity values
New AsyncEncoderCache implementation components/src/dynamo/common/multimodal/async_encoder_cache.py, components/src/dynamo/common/multimodal/__init__.py	New class with get_or_compute() async method for request coalescing; in-flight future tracking per key; stats property augmenting cache stats; exported via __all__
Configuration field propagation components/src/dynamo/trtllm/request_handlers/handler_base.py, components/src/dynamo/trtllm/utils/trtllm_utils.py, components/src/dynamo/trtllm/main.py	Added encoder_cache_capacity_gb (float, default 0) to RequestHandlerConfig and Config; CLI argument --encoder-cache-capacity-gb wired through; passed to RequestHandlerConfig in main.py
Handler integration components/src/dynamo/trtllm/request_handlers/handlers.py	PrefillHandler now accepts encoder_cache: AsyncEncoderCache parameter; factory initializes EncoderCacheManager and AsyncEncoderCache wrapper when disaggregation_mode is "prefill"
Test suite updates components/src/dynamo/common/tests/memory/test_encoder_cache_manager.py	Removed initialization validation tests for capacity constraints
New test coverage components/src/dynamo/common/tests/multimodal/test_async_encoder_cache.py, components/src/dynamo/trtllm/tests/request_handlers/test_trtllm_prefill_handler.py, components/src/dynamo/trtllm/tests/request_handlers/test_trtllm_request_handler_factory.py, components/src/dynamo/trtllm/tests/utils.py	AsyncEncoderCache tests covering retrieval, caching, coalescing, exception handling; PrefillHandler initialization tests; RequestHandlerFactory mode tests; mock config utility function
Build and logging .gitignore, test_concurrent_determinism_under_load[llm_server0]/test.log.txt	Added "core" to ignore rules; augmented test log with server startup entries

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 An async cache hops into view,
With coalescing futures, clever and true,
No duplicate work when requests collide,
One computation serves them all with pride,
The configuration spreads through every layer,
Prefill handlers now have async care! ✨

🚥 Pre-merge checks | ❌ 3

❌ Failed checks (2 warnings, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	⚠️ Warning	The PR title mentions EncoderCacheManager but the actual implementation adds AsyncEncoderCache, which is the key new public entity. The primary change is introducing AsyncEncoderCache to PrefillHandler, not EncoderCacheManager.	Update the title to 'feat: add AsyncEncoderCache to TRT-LLM PrefillHandler' to accurately reflect the main component being added and match the PR objectives.
Description check	❓ Inconclusive	The PR description is minimal and lacks key sections required by the template (Overview, detailed explanation of changes, and specific files to review).	Expand the description to include an Overview section explaining the purpose of AsyncEncoderCache, a Details section covering all modified files and the rationale, and a 'Where should the reviewer start?' section highlighting critical files.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[milesial]

Do we prefer to expose the absolute amount of bytes, or rather the VRAM fraction? Since this will interact with free_gpu_memory_fraction for example, which is relative

[furionw]

Do we prefer to expose the absolute amount of bytes, or rather the VRAM fraction? Since this will interact with free_gpu_memory_fraction for example, which is relative

Thanks for asking! we're following convention in this PR --

• G1 (GPU): Configured via percentage/fraction (in LLM runtime) or absolute blocks
• G2/G3/G4 (KVBM): Configured via absolute GB values or exact block counts
  • Dynamo KVBM does this
  • vLLM mm-processor-cache-gb too

e.g.

  # G2: CPU cache only (GPU -> CPU offloading)
  export DYN_KVBM_CPU_CACHE_GB=4
  # G3: Add disk tier (GPU -> CPU -> Disk tiered offloading)
  export DYN_KVBM_DISK_CACHE_GB=8

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@components/src/dynamo/common/multimodal/async_encoder_cache.py`:
- Around line 110-123: The in-flight Future created for coalescing can be left
unresolved if the compute_fn coroutine is cancelled; update the try/except to
explicitly handle asyncio.CancelledError: upon catching CancelledError, cancel
or set_exception on the future (e.g., call future.cancel() so waiters receive a
cancellation) and then re-raise the CancelledError. Modify the block around
compute_fn/ future (the future variable, self._in_flight[key], and the
try/except/finally) to include an "except asyncio.CancelledError:" clause that
cancels the future and re-raises, leaving the existing finally that deletes
self._in_flight[key] intact; keep the _suppress_unhandled_future_exception
callback behavior.

In `@components/src/dynamo/trtllm/utils/trtllm_utils.py`:
- Around line 289-294: Add a non-negative validation for the
"--encoder-cache-capacity-gb" CLI flag by either supplying a custom argparse
type (e.g., non_negative_float) to the parser.add_argument call for that flag or
by adding a post-parse check that calls parser.error when
args.encoder_cache_capacity_gb < 0; apply the same validation pattern to the
other cache/size flags in the 352-387 block so all capacity/size arguments (the
parser.add_argument entries for those flags) reject negative values with a clear
error message.

🧹 Nitpick comments (1)

components/src/dynamo/trtllm/tests/request_handlers/test_trtllm_prefill_handler.py (1)

40-44: Cover encoder_cache wiring in the init test.
Line 42 only asserts the engine; consider also asserting the encoder cache is stored to exercise the new constructor parameter.

♻️ Suggested test enhancement

 def test_init_with_encoder_cache(self, mock_config, mock_encoder_cache):
     """Test PrefillHandler can be initialized with encoder_cache."""
     handler = PrefillHandler(mock_config, encoder_cache=mock_encoder_cache)

     assert handler.engine == mock_config.engine
+    assert handler._encoder_cache is mock_encoder_cache