[yechank-nvidia]

Description

Problem. trtllm-serve decodes the media of a multimodal chat request in the frontend process. MultimodalDataTracker.retrieve_all_async() gathers every item of a request at once. The media worker pool (--num_media_load_workers) limits how many decodes run at the same time, not how much decoded data a request or the process holds: every finished item stays referenced until the whole request is decoded. The decoded data and the processed tensors stay referenced until the request reaches the engine, and for a non-streaming request openai_chat keeps them in local variables until the response is complete. There is no per-request or per-process byte limit for this. Inline base64 media is not covered by the remote-download size limit, and FastAPI buffers and parses the whole request body before any handler runs.

The effect on memory, measured against 95cb50c358 (method in the first table below): one request with 16 / 64 / 128 inline 1024x1024 JPEGs grows frontend RSS by +468 / +1060 / +1829 MiB.

Mechanism. This PR adds two opt-in options to trtllm-serve, both prototype and disabled by default (items 1 and 2), plus one small change that always applies (item 3).

1. --max_multimodal_cpu_bytes_per_request caps one request:
   • A small ASGI middleware on POST /v1/chat/completions, /v1/messages, and /v1/responses checks the request body before FastAPI parses it.
     • If Content-Length is above the cap, it returns 413 without reading the body.
     • For a chunked body, it counts bytes as they arrive and returns 413 once the count passes the cap.
     • It stores the count in the request state.
   • The media tracker starts from that raw-body count. It adds each item's CPU storage as the item finishes decoding, counting each storage only once:
     • torch storage, the base buffer of a NumPy array, or the pixel size of a PIL image;
     • views of the same buffer count once.
   • When the total passes the cap, the tracker cancels the loads that are left and the request returns 413. Queued executor jobs and downloads are cancelled; decodes already running finish and are dropped.
   • After LLM.preprocess, the decoded media plus the processed multimodal tensors are checked once more, before generate_async. Shared CPU handles count by their storage size; CUDA handles are not counted.
   • LLM.preprocess has already exported the processed CPU tensors to shared memory by then, and only a consumer that rebuilds the tensors frees that memory. So when this check rejects a request, the server rebuilds and drops those CPU handles before it returns 413. Without this step, each rejected request would leave its processed tensors in /dev/shm until the frontend process exits.
2. --max_multimodal_cpu_bytes caps one frontend process:
   • The server creates max_multimodal_cpu_bytes // max_multimodal_cpu_bytes_per_request request slots and logs the count at startup.
   • A chat request with at least one media item takes one slot before any of its media loads start. It gives the slot back as soon as generate_async accepts the request, or on any error.
   • Text-only requests and generation-only disaggregated requests don't take a slot, because they load no media.
   • If only this option is set, the per-request cap defaults to the same value, so only one request with media is decoded and preprocessed at a time. A per-request cap larger than the total is rejected at startup.
3. Once generate_async has accepted a chat request, openai_chat drops its local references to the decoded media and the processed inputs. Without this, a non-streaming request would keep them until generation finished, after its slot had already been returned, so any number of admitted requests could each hold up to a per-request cap of decoded media in the frontend. Streaming requests already dropped them. This runs with or without the options; its memory effect was not measured separately.

On the OpenAI routes, every 413 has error type RequestTooLargeError (the middleware's body uses the same ErrorResponse envelope as the chat path). On the server's /v1/messages, every 413 uses the Anthropic request_too_large error type. The middleware answers in that envelope, and the route's translation of chat-path errors maps status 413 to it; other 4xx statuses stay invalid_request_error as before. With --grpc, the options raise an error instead of being silently ignored.

Why this is safe.

• Without the options, almost nothing new runs:
  • no middleware is installed;
  • the tracker takes the existing single-asyncio.gather path (one None check added);
  • openai_chat only does a few attribute checks per request, plus dropping its local references to the inputs after generate_async (item 3 above), which does not change what is sent to the engine.
• /v1/completions, embeddings, the multimodal encoder server (mm_embedding_serve), and VisualGen are unchanged.
• Accepted requests pass the same decoded objects to the processor as before. In the end-to-end run below, greedy outputs are identical with the options on and off.
• Slots are released in a finally on every exit path. The end-to-end run shows that requests after several 413s, and six concurrent requests through two slots, all succeed.
• A request rejected after preprocessing leaves no shared memory behind. In the end-to-end run below, the frontend process has no /dev/shm/torch_<pid>_* files after any of these rejections.

Measured impact. This PR bounds memory; it does not make anything faster, so there is no throughput claim.

1. RSS of one many-image request (CPU only). All runs used PR revision ee506fbe54 against main baseline 95cb50c358, in the TRT-LLM dev container (PostMerge build of that commit), each row in a fresh process:

   • random 1024x1024 JPEG data URLs;
   • ImageMediaIO(format="pt"), so each decoded item is a 12 MiB float32 tensor;
   • an 8-thread media executor;
   • the production MultimodalDataTracker.retrieve_all_async();
   • RSS sampled after collection;
   • "off" runs the unchanged upstream gather path. The host was shared (load average about 25 on 56 cores); that affects time, not RSS.

   Per-request cap	Items	Finished decodes	Outcome	RSS delta
   off	16	16	192 MiB returned	+467.9 MiB
   off	64	64	768 MiB returned	+1060.4 MiB
   off	128	128	1536 MiB returned	+1829.4 MiB
   256 MiB	16	16	192 MiB returned	+483.8 MiB
   256 MiB	64	23	413 at 264.0 MiB	+739.4 MiB
   256 MiB	128	22	413 at 264.0 MiB	+801.1 MiB
   256 MiB, raw body charged	16	16	19.1 MiB raw + 192 MiB returned	+475.0 MiB
   256 MiB, raw body charged	64	15	413 at 256.5 MiB	+650.5 MiB
   256 MiB, raw body charged	128	9	413 at 261.1 MiB	+645.9 MiB

   With the cap, decoded output stops at the cap: the 64- and 128-item requests keep only 22–23 decoded items.

   • Without the raw-body charge, RSS still rises from 64 to 128 items (+739 to +801 MiB), because the request's base64 strings were all resident before decoding started.
   • With the raw body charged, the two RSS deltas are within 5 MiB of each other (+650.5 and +645.9 MiB).

   The same experiment on an earlier development branch (2026-09-01, older main) gave matching results:

   • off: +482 / +1071 / +1839 MiB;
   • 256 MiB cap: stops after 22 items, +723 / +818 MiB;
   • raw body charged: 15 and 9 items, +657 / +659 MiB.

   RSS stays well above the counted bytes because decoder temporaries, allocator high-water, and decodes still running are not counted. The cap bounds counted storage; it is not an RSS limiter.

   Overhead on an accepted request. The 16-item case (192 MiB, under a 256 MiB cap) ran 4 times each in alternating fresh processes. The median was 124.3 ms off versus 125.2 ms on, with a spread of 122–134 ms, so the difference is within noise.

2. End-to-end check. Setup:

   • Qwen3-VL-2B-Instruct on one H200;
   • trtllm-serve from this branch with --max_batch_size 8 --max_num_tokens 32768 --max_seq_len 32768, KV-cache memory fraction 0.2, CUDA graphs off;
   • three server runs:
     • no new options;
     • --max_multimodal_cpu_bytes 128MiB --max_multimodal_cpu_bytes_per_request 64MiB (two slots);
     • --max_multimodal_cpu_bytes 80MiB --max_multimodal_cpu_bytes_per_request 40MiB (two slots). With these limits, one 2048x2048 image passes the decode check but fails the processed-tensor check.

   Results:

   • A text request, a 512x512 image, a 1920x1080 photo, and a 2048x2048 image return 200 in the first two runs, with identical greedy outputs.
   • Six 2048x2048 images (about 0.5 MiB body, 72 MiB decoded as uint8):
     • without the options, the frontend decodes and preprocesses all six, and then the engine rejects the request with 400 (encoder output budget);
     • with the 64 MiB cap, the request gets 413 (type RequestTooLargeError) during decode at 76,027,401 bytes, before preprocessing.
   • A 70 MiB body gets 413 (type RequestTooLargeError) before parsing, both with Content-Length and chunked. On /v1/messages, the 413 uses the Anthropic request_too_large envelope.
   • With the 40 MiB cap, a request with one 2048x2048 image gets 413 after preprocessing ("materialized at least 63,134,620 CPU bytes"). Results:
     • five such chat requests in a row all get this 413;
     • the same image sent to /v1/messages gets 413 with type request_too_large;
     • the frontend process has 0 /dev/shm/torch_<pid>_* files after each of these requests, after 5 s idle, and after the successful requests that follow;
     • an earlier revision of this branch, without the release step, left the processed tensors' shared-memory files in /dev/shm after such rejections; that pre-fix run was not saved, so no counts are given here.
   • After the 413s in both limited runs, a normal image request still succeeds. Six concurrent image requests through the two slots all return 200 with identical text.
   • The server log has no tracebacks for any of the 413s. It reports the two slots at startup.

   Not measured: serving throughput or latency under load with the options on. When the total cap is reached, requests with media wait for a slot.

Scope and limitations.

• The limits count logical CPU storage: tensors, NumPy buffers, and PIL pixels. They do not count RSS, so leave headroom for decoder temporaries and allocator retention. Processed tensors can be larger than the decoded media. In the run above (Qwen3-VL-2B), one 2048x2048 image is 12 MiB decoded, but the request counts about 60 MiB once its processed tensors are added.
• --max_multimodal_cpu_bytes uses uniform worst-case slots, not a weighted byte ledger, so a small request reserves a full per-request share.
• Waiting for a slot has no timeout (no 429). A client that disconnects while waiting is noticed only after its request has been decoded, preprocessed, and submitted to the engine.
• The slot is returned when the engine accepts the request. Storage the engine keeps after that (for example shared CPU handles) is not charged.
• The parsed request (including inline base64 media in request.messages) stays resident in the frontend until the response is complete. Its raw size is charged to the per-request cap during admission, but it is no longer covered by a slot after the slot is returned.
• The processed-tensor check runs after preprocessing. It rejects the request, but it does not lower the processor's own peak memory; processor concurrency is limited only by the slot count.
• The release step after that check frees CPU shared-memory handles only. On an E/P disaggregated prefill worker with these options set, a processed-tensor 413 does not release the encoder's CUDA embedding handles; this case was not tested.
• /v1/responses gets only the raw-body cap: no decoded or processed accounting, no slot.
• Message Batches items go through the chat path, but their raw body is not charged.
• Message Batches results and the disaggregated orchestrator's /v1/messages keep their existing error mapping, so a 413 from the chat path shows up there as invalid_request_error.
• mm_embedding_serve does not expose the options.
• The body cap applies to every request on the three routes, including text-only ones.
• The limits apply per frontend process: with --num_serve_frontends N, the total is N times the cap.
• Remote URL downloads still use only the existing per-response download limit and are not added up across a request before they finish.

Test Coverage

Tests

• New: tests/unittest/llmapi/apps/test_openai_server_multimodal_admission.py, 12 CPU-only cases, registered in tests/integration/test_lists/test-db/l0_cpu.yml. It covers:
  • storage counting with deduplication, and charging the raw body;
  • cancelling the remaining loads at the cap;
  • checking the two limits against each other;
  • shared CPU handles made by MultimodalParams.to_handle: counted once per storage, and their shared-memory file removed by the release step;
  • the body middleware: Content-Length rejection on both the chat and /v1/messages paths, chunked rejection, staying 413 with type RequestTooLargeError through a real FastAPI app, recording the byte count, and leaving non-multimodal routes alone;
  • slot serialization.
• Updated: tests/unittest/api_stability/references/trtllm_serve_cli.yaml lists the two new prototype options.

Commands run (all in the TRT-LLM dev container, PostMerge build of 95cb50c358):

• pytest -v tests/unittest/llmapi/apps/test_openai_server_multimodal_admission.py tests/unittest/api_stability/test_serve_cli.py: 16 passed (12 admission cases, 4 CLI stability cases).

• pytest -m cpu_only on all 22 unittest/llmapi/apps entries in l0_cpu.yml, plus:

  • tests/unittest/disaggregated/test_openai_disagg_service.py, test_disagg_internal_auth.py, test_openai_server_info.py, and test_openai_server_stream_errors.py;
  • tests/unittest/llmapi/apps/test_openai_server_iteration_stats.py and tests/unittest/llmapi/test_rl_control_auth.py;
  • tests/unittest/api_stability;
  • tests/unittest/inputs.

  Result: 1275 passed, 11 deselected (not cpu_only).

• pytest without a marker filter on the Anthropic route and batch tests, test_openai_chat_disagg_multimodal.py, test_serving_extensions.py, test_chat_utils.py, test_openai_server_iteration_stats.py, the new admission tests, test_rl_control_auth.py, and the four disaggregated server tests above: 334 passed.

• The end-to-end trtllm-serve check described above (Qwen3-VL-2B-Instruct, one H200), run on the final head of this branch: all cases as listed.

• The fresh-process RSS runs described above.

• pre-commit run --files <changed files>: passed.