[https://nvbugs/6388787][fix] Revert Pass IPC HMAC key through file descriptor (#15654) by chenfeiz0326 · Pull Request #15961 · NVIDIA/TensorRT-LLM

chenfeiz0326 · 2026-07-06T06:28:26Z

Summary

Reverts #15654 (48fc753).

PR #15654 passes the IPC HMAC key from the parent process to the LLM-API proxy child via a file descriptor. In the QA-pytest Slurm/MPI launch path used by trtllm-bench and trtllm-serve, the fd handshake between Rank0's forked subshell (spawning internal mpirun) and the mgmn_leader_node RemoteMpiCommSessionServer deadlocks silently.

Symptoms

trtllm-bench (nvbugs/6388787 — bisected there): after [llmapi] start MpiSession with N workers the bench process emits no further stdout for ~30 min, workers stay alive at 0% GPU / 4 MiB VRAM, no exception, no MPI abort. Perf harness SIGKILLs at _STALL_TIMEOUT=1800s.
trtllm-serve (nvbugs/6405747): the /health endpoint never binds and pytest fails with Server http://localhost:<port>/health did not become ready within 3600s on both baseline and candidate wheels. Reproduced on nemotron_3_ultra_550b_nvfp4 serve mode, B200 EP4, on 2026-07-05.

Prior history

This is the third time the "pass IPC HMAC key via fd" idea has broken bench/serve:

PR	Merged	Symptom	Reverted by
#14378	first attempt	`BlockingIOError: [Errno 11]` in `_read_spawn_proxy_process_ipc_hmac_key_fd`	#14782
#15654	2026-07-03	silent handshake deadlock in bench + serve	this PR

Both prior attempts had the same root cause: the writer's fd does not reliably survive the fork+exec into the internal mpirun child spawned by trtllm-llmapi-launch, and the reader (mgmn_leader_node) can't complete os.read(fd, ...) before its ZMQ PAIR accept() times out. Attempt #2 (this revert) fails silently instead of raising because os.set_blocking(fd, True) was added — same deadlock, no exception.

Test plan

Run trtllm-bench perf on any large-model + EP config (deepseek_v3.2_fp4-bench-pytorch-float4 ep:8) — should proceed past start MpiSession with N workers and produce a throughput number instead of stalling to the perf-harness SIGKILL.
Run trtllm-serve perf on nemotron_3_ultra_550b_nvfp4-serve (nvbugs/6405747 case) — /health should bind before pytest's 3600s server-ready timeout.
Confirm existing HMAC-encrypted IPC still functions (the reverted code path still generates and consumes the HMAC key via env var, matching pre-[https://nvbugs/6208457][fix] Pass IPC HMAC key through file descriptor #15654 behavior).

Next steps

Reverting until the fd-inheritance across fork+exec into the mpirun child is verified end-to-end for both the bench and serve launch paths (not just the unit-test-level test_launcher_envs.py coverage that #15654 added).

Signed-off-by: Chenfei Zhang chenfeiz@nvidia.com

Summary by CodeRabbit

New Features
- Updated the launcher to configure secure IPC credentials through environment variables, simplifying startup behavior.
Bug Fixes
- Improved consistency in process launch and worker setup when security credentials are required.
- Removed an unsupported configuration path related to disabling message encryption.

chenfeiz0326 · 2026-07-06T06:32:00Z

/bot run --disable-fail-fast --stage-list "*PerfSanity*"

coderabbitai · 2026-07-06T06:35:05Z

📝 Walkthrough

Walkthrough

This PR replaces the file-descriptor-based mechanism for passing the spawn-proxy IPC HMAC key with a direct environment variable (TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY). It updates the launcher script, serve command, executor utilities, and validation logic (ValueError → assert), and removes tests covering the removed FD-based approach.

Changes

IPC HMAC key handling migration

Layer / File(s)	Summary
HMAC key retrieval helper `tensorrt_llm/executor/utils.py`	`get_spawn_proxy_process_ipc_hmac_key_env()` now reads the key directly from `TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY`, asserting presence and decoding hex to bytes; removes FD-based caching/normalization helpers and `LlmLauncherEnvs` gains the new constant.
Launch script env export `tensorrt_llm/llmapi/trtllm-llmapi-launch`	Generates HMAC key via `openssl rand -hex 32` and exports it directly; removes `run_with_ipc_hmac_key` wrapper, running task and MGMN leader commands directly.
Disaggregated leader launch `tensorrt_llm/commands/serve.py`	Sets the HMAC key via `secrets.token_hex(32)` directly in `os.environ`, updates non-MPI env assertions, starts child process without pipe/`pass_fds`, and removes the prior `finally` cleanup block for signal handlers and child termination.
Assertion changes and test cleanup `tensorrt_llm/executor/ipc.py`, `tensorrt_llm/executor/worker.py`, `tests/unittest/executor/test_ipc.py`, `tests/unittest/executor/test_launcher_envs.py`	Replaces `ValueError` raises with `assert` for HMAC requirements in `ZeroMqQueue` and `GenerationExecutorWorker`; removes tests validating the removed ValueError/FD-based behaviors.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant LaunchScript as trtllm-llmapi-launch
  participant Env as Environment
  participant ServeCmd as serve.py (leader)
  participant ChildProc as Child process
  participant Worker as GenerationExecutorWorker

  LaunchScript->>Env: export TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY
  ServeCmd->>Env: set TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY (secrets.token_hex)
  ServeCmd->>ChildProc: start subprocess with non_mpi_env
  ChildProc->>Env: read TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY
  ChildProc->>Worker: initialize with hmac_key
  Worker->>Worker: assert hmac_key is set

Possibly related PRs

NVIDIA/TensorRT-LLM#15654: Both PRs modify the same IPC HMAC-key plumbing in serve.py's _launch_disaggregated_leader and executor/utils.py, switching between env-based vs FD-based TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY* handling.

Suggested reviewers: JunyiXu-nv

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the main change: reverting the IPC HMAC key file-descriptor approach.
Description check	✅ Passed	The description explains the issue, symptoms, history, and test plan, though it doesn't follow the template headings exactly.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tensorrt_llm/executor/ipc.py`:
- Around line 43-49: The IPC docstring is stale because it still describes
disabling HMAC even though the constructor now always requires it. Update the
docstring in the Queue/IPC initializer to remove the unsupported
`use_hmac_encryption is False` case and clearly state that HMAC encryption is
always required, keeping the parameter description consistent with the assertion
in the initialization logic.
- Line 51: Replace the mandatory HMAC guard in the IPC path with a runtime
validation instead of an assert, since the check in the executor/ipc.py logic
can be removed under optimized Python runs. Update the relevant check around
use_hmac_encryption to explicitly raise ValueError when it is false, preserving
the security enforcement in all deployments.

In `@tensorrt_llm/executor/utils.py`:
- Around line 39-43: The HMAC key validation in the utility that reads
TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY must reject both missing and empty values
instead of relying on assert. Update the key-loading logic in the function that
calls os.getenv and bytes.fromhex so it uses the
LlmLauncherEnvs.TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY constant directly for the
environment lookup and performs an explicit empty-string check before converting
the key.

In `@tensorrt_llm/llmapi/trtllm-llmapi-launch`:
- Line 43: The launch script currently generates
TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY with openssl, which can silently yield an
empty value if openssl is unavailable. Update the trtllm-llmapi-launch logic to
avoid relying on openssl by generating the key with a safer source such as
secrets.token_hex(32), or explicitly check the result and fail fast before
export. Keep the change focused around the TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY
setup so get_spawn_proxy_process_ipc_hmac_key_env() never receives an empty key.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cd7fa32b-ace1-4786-ad4e-f819d25f2ea2

📥 Commits

Reviewing files that changed from the base of the PR and between 1728261 and 41f4ada.

📒 Files selected for processing (7)

tensorrt_llm/commands/serve.py
tensorrt_llm/executor/ipc.py
tensorrt_llm/executor/utils.py
tensorrt_llm/executor/worker.py
tensorrt_llm/llmapi/trtllm-llmapi-launch
tests/unittest/executor/test_ipc.py
tests/unittest/executor/test_launcher_envs.py

💤 Files with no reviewable changes (2)

tests/unittest/executor/test_launcher_envs.py
tests/unittest/executor/test_ipc.py

tensorrt-cicd · 2026-07-06T06:37:01Z

PR_Github #57708 [ run ] triggered by Bot. Commit: 41f4ada Link to invocation

tensorrt-cicd · 2026-07-06T06:37:21Z

PR_Github #57709 [ run ] triggered by Bot. Commit: 41f4ada Link to invocation

tensorrt-cicd · 2026-07-06T06:37:26Z

PR_Github #57708 [ run ] completed with state ABORTED. Commit: 41f4ada

Link to invocation

chenfeiz0326 · 2026-07-06T06:50:13Z

/bot run --disable-fail-fast --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-1,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-3,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-4"

tensorrt-cicd · 2026-07-06T06:55:43Z

PR_Github #57712 [ run ] triggered by Bot. Commit: 41f4ada Link to invocation

tensorrt-cicd · 2026-07-06T06:59:20Z

PR_Github #57709 [ run ] completed with state ABORTED. Commit: 41f4ada

Link to invocation

tensorrt-cicd · 2026-07-06T07:46:45Z

PR_Github #57712 [ run ] completed with state FAILURE. Commit: 41f4ada
/LLM/main/L0_MergeRequest_PR pipeline #46424 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-07-06T08:15:38Z

/bot run --disable-fail-fast --stage-list "*PerfSanity*"

tensorrt-cicd · 2026-07-06T08:22:30Z

PR_Github #57732 [ run ] triggered by Bot. Commit: 2e3ba4d Link to invocation

tensorrt-cicd · 2026-07-06T12:38:16Z

PR_Github #57732 [ run ] completed with state FAILURE. Commit: 2e3ba4d
/LLM/main/L0_MergeRequest_PR pipeline #46441 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…escriptor (NVIDIA#15654) Reverts NVIDIA#15654 (48fc753). PR NVIDIA#15654 passes the IPC HMAC key from the parent process to the LLM-API proxy child via a file descriptor. In the QA-pytest Slurm/MPI launch path used by trtllm-bench and trtllm-serve, the fd handshake between Rank0's forked subshell (spawning internal mpirun) and the mgmn_leader_node RemoteMpiCommSessionServer deadlocks silently: - trtllm-bench (nvbugs/6388787): after "start MpiSession with N workers" the bench process emits no further stdout for ~30 min, workers stay alive at 0% GPU / 4 MiB VRAM, no exception, no MPI abort. Perf harness SIGKILLs at _STALL_TIMEOUT=1800s. - trtllm-serve (nvbugs/6405747): serve mode's /health endpoint never binds, and pytest fails with "Server http://localhost:<port>/health did not become ready within 3600s" on both baseline and candidate wheels. This is the third time the "Pass IPC HMAC key via fd" idea has broken bench/serve (previously PR NVIDIA#14378 was reverted by PR NVIDIA#14782 for the same class of failure: BlockingIOError in _read_spawn_proxy_process_ ipc_hmac_key_fd). Reverting until the fd inheritance across fork+exec into the mpirun child is verified end-to-end for both bench and serve launch paths. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

…pre-merge job 46441 Removed 70 perf sanity waives that passed in pre-merge job 46441 (OpenSearch-verified). Kept 12 waives whose test cases did not appear in the job 46441 OpenSearch results (pytest failed before uploading data). Breakdown: - 70 removed (passed in OpenSearch job 46441) - 12 kept (not present in OpenSearch job 46441) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-07-06T14:09:20Z

/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline"

tensorrt-cicd · 2026-07-06T14:15:52Z

PR_Github #57775 [ skip ] triggered by Bot. Commit: c514f81 Link to invocation

tensorrt-cicd · 2026-07-06T14:22:07Z

PR_Github #57775 [ skip ] completed with state SUCCESS. Commit: c514f81
Skipping testing for commit c514f81

Link to invocation

…escriptor (NVIDIA#15654) (NVIDIA#15961) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 requested a review from a team as a code owner July 6, 2026 06:28

chenfeiz0326 requested a review from hchings July 6, 2026 06:28

github-actions Bot assigned chenfeiz0326 Jul 6, 2026

chenfeiz0326 requested review from JunyiXu-nv and yibinl-nvidia July 6, 2026 06:30

coderabbitai Bot reviewed Jul 6, 2026

View reviewed changes

Comment thread tensorrt_llm/executor/ipc.py

Comment thread tensorrt_llm/executor/ipc.py

Comment thread tensorrt_llm/executor/utils.py

Comment thread tensorrt_llm/llmapi/trtllm-llmapi-launch

chenfeiz0326 force-pushed the chenfei/revert-15654-ipc-hmac-fd branch 2 times, most recently from 7d18710 to 2e3ba4d Compare July 6, 2026 08:13

JunyiXu-nv approved these changes Jul 6, 2026

View reviewed changes

chenfeiz0326 force-pushed the chenfei/revert-15654-ipc-hmac-fd branch from 2e3ba4d to 0745fba Compare July 6, 2026 13:48

chenfeiz0326 force-pushed the chenfei/revert-15654-ipc-hmac-fd branch from 0745fba to c514f81 Compare July 6, 2026 13:51

chenfeiz0326 enabled auto-merge (squash) July 6, 2026 14:09

chenfeiz0326 merged commit 0d97e9c into NVIDIA:main Jul 6, 2026
8 checks passed

chienchunhung mentioned this pull request Jul 6, 2026

[https://nvbugs/6342844][fix] Prevent disaggregated KV transfer stalls #15737

Merged

BrianLi23 pushed a commit to BrianLi23/TensorRT-LLM that referenced this pull request Jul 9, 2026

[https://nvbugs/6388787][fix] Revert Pass IPC HMAC key through file d…

962f5f4

…escriptor (NVIDIA#15654) (NVIDIA#15961) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 mentioned this pull request Jul 10, 2026

[https://nvbugs/6422339][test] Unwaive 3 disagg-perf-sanity GB200 tests after #15961 revert #16220

Open

2 tasks

Uh oh!

Conversation

chenfeiz0326 commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Symptoms

Prior history

Test plan

Next steps

Summary by CodeRabbit

Uh oh!

chenfeiz0326 commented Jul 6, 2026

Uh oh!

coderabbitai Bot commented Jul 6, 2026

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

chenfeiz0326 commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

chenfeiz0326 commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

chenfeiz0326 commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

tensorrt-cicd commented Jul 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chenfeiz0326 commented Jul 6, 2026 •

edited

Loading