[jthomson04]

The container inherits Python protobuf, SMG copies the complete token history on each streaming update, and shared result queues schedule a coroutine for each notification. This PR makes native upb the container default, removes the unnecessary SMG history copies, and schedules direct event-loop callbacks for single and batched queue notifications.

Both gRPC launchers warn before model startup when Python protobuf is active and continue startup. Native upb and cpp do not produce a warning. Backend detection is optional: if the private protobuf implementation API cannot be imported or has no Type attribute, the launchers skip the warning and continue startup. The notification change keeps queue order, existing batch deduplication, event-loop ownership, and shutdown handling.

Streaming boundaries, token order, text deltas, logprob alignment, terminal metadata, cancellation, and slow-consumer isolation are preserved. The benchmark engine emits one token per result. No streaming-interval change, response merging, extra token buffering, public API change, or protocol change is included. OpenEngine text decoding remains enabled.

Current main versus this PR

Baseline: main at campaign start, 269365738865803f637cf6120a1d2098cc791004. Measured candidate: 88f331b90f043315a5689fe2d3e2ac33a8a6e5f5. The candidate contains main plus this PR's changes; no other unmerged PR is included. The follow-up commit 2b34042bd88b4baa21d9de368b048729f1cc4022 makes backend detection optional and changes startup diagnostics and tests only. The campaign was not repeated; the results below remain measurements of 88f331b90f043315a5689fe2d3e2ac33a8a6e5f5.

One unprofiled run per adapter and revision at trace concurrency 32:

Adapter	Main output tokens/s	PR output tokens/s	PR / main	Main requests/s	PR requests/s
smg	5,254.9	13,620.5	2.59x	6.35	13.12
openengine	5,067.0	10,553.7	2.08x	6.08	11.25

Adapter / revision	TTFT p50 / p99 ms	Request latency p50 / p99 ms	Response bytes/token	Frontend us/token	Worker + mock us/token	Client us/token
smg / main	79.68 / 283.65	2445.31 / 38436.86	23.33	206.38	34.36	19.52
smg / pr	20.06 / 58.05	1258.49 / 17956.86	23.61	88.44	31.73	17.36
openengine / main	77.95 / 305.92	2523.14 / 41844.74	23.23	213.22	33.96	19.87
openengine / pr	22.69 / 55.26	1385.47 / 22757.38	23.46	109.85	32.59	17.70

Response bytes are serialized protobuf payloads, including terminal messages; network framing is excluded. SMG's final repeated token sequence is checked without counting those tokens twice. Latencies include drained completions for requests admitted during measurement. Full latency, response-gap, request-length, and client-dispatch distributions are in the report.

Producer capacity and profiles

Four direct-LLM checks retain production sampling, request processing, worker dispatch, IPC, queues, notifications, result processing, and adapter decoding. They omit gRPC response construction, serialization, and transport. Their concurrency is 32 requests, while the full-server concurrency is 32 AgentX traces with dependencies and fan-out.

Adapter / revision	Direct tokens/s	Direct requests/s	Token headroom	Request headroom	Capacity input / output mean tokens	Full-server input / output mean tokens
smg / main	32,074.4	28.93	6.10x	4.56x	165,993 / 1,179	71,894 / 832
smg / pr	48,978.2	44.00	3.60x	3.35x	151,038 / 1,105	84,425 / 1,051
openengine / main	25,471.2	28.97	5.03x	4.77x	137,203 / 919	69,973 / 855
openengine / pr	36,882.1	32.57	3.49x	2.89x	152,854 / 1,138	79,926 / 951

The longer capacity prompts provide conservative headroom evidence for the retained CPU path. Both request and token rates are shown to avoid a claim based only on different output lengths. Four additional profile runs use separate 30-second perf and ordinary Python py-spy windows at 99 Hz. Their throughput does not replace the unprofiled measurements.

At C32, the observed output-throughput gain is 2.59x for SMG and 2.08x for OpenEngine. Frontend CPU per token falls by 57.1% and 48.5%, respectively. These results measure the three changes together against main. They do not isolate the contribution of each change.

All four capacity checks exceed 2x in both token and request throughput. The PR has 3.60x token / 3.35x request headroom for SMG and 3.49x / 2.89x for OpenEngine, with longer prompts in the capacity checks. The retained producer path therefore has substantial headroom over these full-server measurements. The busiest client thread averages less than 0.01 cores in every timing run. The largest p99 dispatch delay is 0.142 ms; isolated maxima reach 9.135 ms. These records do not indicate a sustained client CPU or dispatch limit.

Remaining PR frontend samples include adapter request and response handling (SMG 29.8%, OpenEngine 40.7%), shared result processing (16.1%, 25.4%), and IPC (19.0%, 13.5%). OpenEngine tokenizer work appears in 5.4% of frontend stacks. Worker stacks contain more IPC work (59.5% for SMG, 51.8% for OpenEngine) than mock generation (8.9%, 12.5%). These are overlapping inclusive Python sample fractions, not exact CPU-time shares or predicted optimization gains. Native perf reports also show Python evaluation, allocation, and kernel work. Python protobuf appears in 35.9% / 37.8% of main frontend stacks; its absence from PR Python stacks does not mean native protobuf is free.

All workload, placement, CPU-record, drain, and shutdown checks pass. The main SMG frontend py-spy capture has one sampling error among 3,169 recorded stacks; the other seven Python captures have no sampling errors. All eight perf captures report zero lost samples. This profile qualification is retained separately and does not reject an unprofiled timing result.

These are directional measurements, with one observation per point. C32 alone does not establish a serving ceiling or a causal GIL limit. The results cover the production serving path, this AgentX workload, frozen runtime, and CPU allocation with a CPU engine mock. They exclude GPU inference. The adapters perform different protocol work.

Validation

• Optional protobuf detection follow-up: all 16 optional-dependency and launcher tests passed in the saved CPU runtime, including fresh Python/upb processes, simulated cpp, missing private API import/Type, and both launchers reaching model initialization. All 11 registered CPU cases collect successfully. Formatting and lint hooks passed. No benchmark or container build was repeated for this follow-up.
• Both adapters and revisions pass production-launcher checks for token order, terminal metadata, cancellation, slow-consumer isolation, CPU placement, drain, and shutdown. Complete protobuf message sequences match main for 1-, 17-, and 257-token requests, including text and response boundaries.
• Queue tests: main 1 passed; PR 9 passed. The eight new cases cover real cross-thread wakeups, repeat delivery after draining, cancellation, repeated/batched notifications, and stopped/closed loops. The existing test file remains in CPU CI.
• Shared result tests: 15 passed on each revision. Affected executor tests: 83 passed and the same 9 GPU-startup import failures on both revisions in this CPU runtime.
• Affected adapter suite: main 102 passed / 8 failed; PR 112 passed / 6 failed; 51 deselected on each. All remaining PR failures also occur on main (sampling float comparisons, a LoRA field, and protobuf-message assertions). An existing PR correction to the reflection import mock makes two startup-cleanup tests pass. No new test failure was found.
• Metadata: 58 passed per revision. The adapter tests use upb; serving measurements retain main's Python protobuf and the PR's upb. Existing subprocess tests check the real Python and native backends.
• Pre-commit and source checks passed. Both new production commits are signed and have DCO sign-off. Loaded production source hashes match the measured Git revisions.
• Earlier validation of the protobuf change at 96e282482740ee2f345c9d49a6ce29674b8641d8 included a successful fresh x86 development-image build with upb, plus PyTorch, ONNX, ModelOpt, and ONNX protobuf serialization checks (image sha256:12e517048a913c11e869d336dce2c6ce49096b33bf337daf7f8fefd04043c981). That image build was not repeated for this notification update. GPU inference, full release/Triton image builds, and ModelOpt GPU plugins remain untested.

Method and provenance

The harness calls the production server launchers. It replaces only model/GPU initialization and execution: the model factory, worker engine setup, empty GPU model registry, and unused startup CUDA stream. Production LLM processing, executor IPC, worker dispatch, result queues, response construction, serialization, and transport remain. The deterministic CPU mock is unchanged across revisions.

• Exclusive Grace allocation 2936939: server ptyche0282, client ptyche0288. Frontend CPUs 0-35; worker/mock 36-71; client 0-71; memory NUMA 0. Client: 32 workers and 32 connections. Allocation released after collection.
• Timing: 32 warmups, 30-second ramp, 120-second measurement, up to 300-second drain. Capacity: 32 warmups, 10-second ramp, 30-second measurement. Four timing runs, four capacity runs, four profile runs, and eight correctness runs; no automatic repetitions.
• Python 3.12.3; PyTorch 2.14.0+cu132; grpcio 1.81.1; protobuf 6.33.6; py-spy 0.4.2; perf 6.14.11. Tokenizer: Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca. Trace loader: AIPerf edfbf94f004a610471c3454371e637f4969cc5a7.
• Frozen native source: affd82461714d182d54d222fe28384cedd60a8bb; wheel SHA256 df67a40e85c46e0792436040976623aa59530e4608812165b96160bb3b387945. Executor/bindings C++ source is unchanged through the pinned main. These are frozen native binaries with each revision's Python source, not fresh main/release image builds.
• Runtime image SHA256: 1b5f90fd2a61b60da12ba08c4dc062267a78bbb507651ff97ec6cfa86244629e.
• Mock source SHA256: 5c95942deddd48aa2e62d61338e57ab4666c4b08cbd1f9440133bad2a8efc72d; client SHA256: d1a020cb72f3c8446cc0100178cb22cf3f30d0cde62474b64185517899f46186.
• Prepared dataset SHA256: ae47f2e5c8263fe71a539c23104660baf18716ebf7ff236e52f42e809538059d; frozen prompt tokens SHA256: 64e176ca86a3609a3a6e6f3a2e24b2f3d6d7bf52bdde13aa1e389f8852db6939.

Harness changes and raw artifacts are separate from this production PR. The report, exact source archives, loaded-source hashes, runtime replacements, dependencies, request distributions, CPU records, raw profiles, validation logs, and release record are saved on Tyche at /lustre/fsw/coreai_comparch_trtllm/jothomson/trtllm-grpc-bench-20260930/main-notify-20261001.

Evidence archive: main-notify-20261001-evidence.tar.gz, 1,275,796,416 bytes, SHA256 5f71919fa00d92141a2e7e875f97adb59af0da76d37a9c393561d8fa58f8640f. The downloaded archive and loaded source hashes were verified locally.

Dev Engineer Review

The base and release container stages now default to native upb. Both gRPC launchers warn when protobuf uses the Python implementation and continue startup. If the private detection API is unavailable or lacks Type, the warning is skipped. Native upb and cpp do not warn.

The SMG response path reuses token histories instead of copying them. _SyncQueue.put and notify_many schedule synchronous callbacks with loop.call_soon_threadsafe and retain shutdown error handling. These changes target lower copy and notification overhead without changing the public API or protocol.

No current review findings were supplied, so review severity counts are unavailable. Reported benchmarks use a CPU engine mock and combine all three performance changes. They do not establish GPU-serving performance or a causal GIL limit. GPU inference, full release/Triton image builds, and ModelOpt GPU plugins remain untested.

QA Engineer Review

Changed tests cover SMG streaming response construction, protobuf warning and launcher startup paths, queue notifications and shutdown, and the remote-dictionary test’s independence from the Python protobuf environment variable. The l0_cpu test list adds two SMG streaming-delta tests and five optional-gRPC tests. The supplied evidence does not include test IDs for the added entries.

Reported validation includes 9 queue test passes on the PR, 15 shared-result test passes per revision, and 58 metadata test passes per revision. Affected executor tests had 83 passes and 9 GPU-startup import failures on both revisions. The affected adapter suite had 112 passes and 6 failures on the PR, compared with 102 passes and 8 failures on main; the author reports that all remaining PR failures also occur on main. CI results included failures and unstable pipelines. Coverage verdict: needs follow-up. GPU inference and the listed builds remain untested, and CI was not uniformly successful.

Per-File QA Perspective

• docker/Dockerfile.multi: Verify that PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb is active in both base and release images.
• docs/source/installation/containers.md: Verify that the protobuf setup guidance matches image defaults and launcher warning behavior.
• tensorrt_llm/grpc/_protobuf.py: Verify warnings for Python protobuf and continued startup when detection raises ImportError or AttributeError.
• tensorrt_llm/grpc/openengine/README.md: Verify that documented protobuf setup matches image defaults and launcher behavior.
• tensorrt_llm/grpc/openengine/server.py: Verify that the warning runs before the server lifecycle and does not block startup.
• tensorrt_llm/grpc/smg/server.py: Verify that the warning runs before reflection loading and does not block startup.
• tensorrt_llm/grpc/smg/servicer.py: Verify token order, counts, and logprobs in streaming and completion responses when token histories are reused.
• tests/integration/test_lists/test-db/l0_cpu.yml: Adds two SMG streaming-delta tests and five optional-gRPC tests to the CPU CI list.
• tests/unittest/disaggregated/test_remoteDictionary.py: Verify that the test remains independent of the Python protobuf environment setting. No test-list change for this unit test is identified.
• tests/unittest/grpc/smg/test_smg.py: Covers streaming deltas, completion responses, and empty token histories. The supplied evidence does not establish whether this unit test is listed in a CI or manual-QA list.
• tests/unittest/grpc/test_grpc_optional.py: Covers protobuf warnings, unavailable detection APIs, launcher startup, and missing reflection imports. The supplied evidence does not establish whether this unit test is listed in a CI or manual-QA list.
• tensorrt_llm/llmapi/utils.py: Verify ordered queue delivery, batched notifications, and shutdown errors with synchronous event-loop callbacks.
• tests/unittest/llmapi/test_llm_utils.py: Covers queue notifications across queues, cancellation, late notifications, and stopped or closed loops. The supplied evidence does not establish whether this unit test is listed in a CI or manual-QA list.
• jenkins/current_image_tags.properties: Verify that the updated staging image tags point to the intended x86_64, SBSA, Rocky Linux, Python, and wheel configurations used by CI.

[jthomson04]

/bot run --disable-fail-fast

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

• Last reviewed commit: 7d1f3a8
• Latest commit, not reviewed: f09845e

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review.

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 048e6b29-dea1-445f-b38a-6321665482e0

📥 Commits

Reviewing files that changed from the base of the PR and between b299e8b and 7d1f3a8.

📒 Files selected for processing (1)

• jenkins/current_image_tags.properties

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The change selects upb in container stages and adds protobuf implementation warnings to both gRPC launchers. It also updates SMG streaming token handling, changes async queue notification scheduling, and points Jenkins image variables to staging images.

Changes

Protobuf Runtime Selection

Layer / File(s)	Summary
Container runtime setting and guidance docker/Dockerfile.multi, docs/source/installation/containers.md, tensorrt_llm/grpc/openengine/README.md	The base and release stages set the protobuf implementation to upb. Documentation describes runtime selection, wheel requirements, and launcher warning behavior.
Startup warning and validation tensorrt_llm/grpc/_protobuf.py, tensorrt_llm/grpc/{openengine,smg}/server.py, tests/unittest/grpc/test_grpc_optional.py, tests/unittest/disaggregated/test_remoteDictionary.py, tests/integration/test_lists/test-db/l0_cpu.yml	A shared helper warns when Python protobuf is active. Both launchers call it before startup work proceeds. Tests cover runtime detection, startup behavior, and test registration. The remote dictionary test no longer sets or removes the protobuf environment variable.

SMG Streaming Responses

Layer / File(s)	Summary
Streaming response token handling tensorrt_llm/grpc/smg/servicer.py, tests/unittest/grpc/smg/test_smg.py	Chunked and final responses use truthy completion token histories directly. CPU-only tests cover token and logprob deltas, counts, final responses, and empty token histories.

Async Queue Notifications

Layer / File(s)	Summary
Notification scheduling and validation tensorrt_llm/llmapi/utils.py, tests/unittest/llmapi/test_llm_utils.py	Regular and batched notifications use thread-safe callback scheduling instead of coroutine submission. Scheduling errors become EventLoopShutdownError when the loop is closed. Tests cover ordered delivery and shutdown states.

Jenkins Image Tags

Layer / File(s)	Summary
Staging image tag update jenkins/current_image_tags.properties	Five image variables now reference staging Triton development images with torch_skip tags.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant Launcher as gRPC launcher
  participant Helper as Protobuf warning helper
  participant Detector as Protobuf implementation detector
  participant Logger
  Launcher->>Helper: Check protobuf implementation
  Helper->>Detector: Detect active implementation
  Detector-->>Helper: Return implementation
  Helper->>Logger: Warn if implementation is python

Loading

Suggested reviewers: tanmayv25

Merge Risk: Minimal · up to 7d1f3

No confirmed issue blocks merging. The image owner should confirm that Jenkins can pull all five staging tags.

Pre-merge checks | 4 | 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 32.35% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 34 functions across 9 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check		The title clearly identifies the main performance changes: native protobuf usage and streamlined gRPC result delivery. It is concise and follows the required ticket and type format.
Description check		The description clearly explains the problem, solution, scope, performance measurements, validation, limitations, and untested areas. It does not use the template headings or include the explicit chec…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 32.35% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 34 functions across 9 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #76002 [ run ] triggered by Bot. Commit: 96e2824 Link to invocation

[tensorrt-cicd]

PR_Github #76002 [ run ] completed with state SUCCESS. Commit: 96e2824
/LLM/main/L0_MergeRequest_PR pipeline #62666 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[jthomson04]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #76086 [ run ] triggered by Bot. Commit: 88f331b Link to invocation

[tensorrt-cicd]

PR_Github #76086 [ run ] completed with state FAILURE. Commit: 88f331b
/LLM/main/L0_MergeRequest_PR pipeline #62736 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation