[TRTLLM-12670][feat] add /start_profile and /stop_profile endpoints to trtllm… by JunyiXu-nv · Pull Request #13872 · NVIDIA/TensorRT-LLM

JunyiXu-nv · 2026-05-08T00:03:35Z

…-serve

Expose iteration-scoped runtime profiling over HTTP on the OpenAI-compatible trtllm-serve endpoint, mirroring SGLang's developer-profiling interface.

POST /start_profile accepts output_dir, num_steps, start_step, activities (any of "CPU", "GPU", "CUDA_PROFILER"). With "CUDA_PROFILER" only, the server skips torch.profiler entirely so it composes cleanly with nsys profile -c cudaProfilerApi.
POST /stop_profile terminates the active window and flushes the trace.
PyExecutor.start_profile/stop_profile reuse the existing profile_start_iters / profile_stop_iters / cudaProfilerStart-Stop plumbing so env-var-based profiling behavior is preserved.
_profiler() now builds torch.profiler lazily on the first start iteration so runtime configuration takes precedence over the TLLM_TORCH_PROFILE_TRACE env var and consecutive start/stop windows work.
GenerationExecutor, BaseWorker, and GenerationExecutorRpcProxy proxy the new calls into the worker process; non-PyTorch backends fall through to a warning.
LLM.start_profile / stop_profile added as @set_api_status("prototype").
Smoke test covers default values, explicit overrides, and the stop path via mocked generator (no GPU required).

Dev Engineer Review

Added POST /start_profile and POST /stop_profile to trtllm-serve.
Added runtime profiling across LLMs, executors, workers, proxies, and PyExecutor.
Added iteration-scoped scheduling, lazy profiler creation, multi-rank traces, activity selection, and consecutive profiling windows.
Added input validation for CPU, GPU, and CUDA_PROFILER.
Added proxy-worker acknowledgements, timeout handling, and shutdown-state protection.
Added unsupported-backend warnings and profiling documentation.
Test-list entries use valid paths and cover endpoint smoke tests, executor profiling, and proxy synchronization.
CI results remain mixed. PR_Github #49393 succeeded, but multiple main pipelines failed.

QA Engineer Review

Added tests cover:

OpenAI server defaults, custom arguments, backend errors, and event-loop responsiveness.
PyExecutor scheduling, cancellation, validation, and clean failure state.
Proxy acknowledgements, error propagation, timeouts, serialization, thread ownership, and shutdown races.
test_trtllm_serve_profile_example.

The endpoint smoke test, test_profile_endpoints.py, and test_proxy_profile_sync.py are listed in tests/integration/test_lists/test-db/l0_a10.yml.

Verdict: sufficient.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

JunyiXu-nv · 2026-05-11T09:06:54Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-11T09:13:31Z

PR_Github #47712 [ run ] triggered by Bot. Commit: 86b3237 Link to invocation

tensorrt-cicd · 2026-05-11T19:55:51Z

PR_Github #47712 [ run ] completed with state SUCCESS. Commit: 86b3237
/LLM/main/L0_MergeRequest_PR pipeline #37608 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

…ints Two follow-up fixes for the CI failures introduced by the profile endpoints PR (NVIDIA#13872): 1. API stability reference: add start_profile / stop_profile entries to tests/unittest/api_stability/references/llm.yaml so TestLLM::test_{signature,docstring,modified_init,...} stops failing with "LLM has different methods: {'start_profile', 'stop_profile'}". 2. Ray WorkerExtension collision: allow a user-supplied WorkerExtension to override BaseWorker.start_profile / stop_profile. These names are explicitly designated extension points (the RLHF WorkerExtension in tensorrt_llm/llmapi/rlhf_utils.py uses torch.cuda.profiler directly), so _inject_worker_extension now copies overridable extension methods into the derived class dict, which makes the extension's impl win regardless of MRO order and keeps the conflict check strict for every other name. Fixes the cascade of "RayGPUWorker died during initialization / Worker class RayGPUWorker already defines 'start_profile'" errors across the Ray test suites (H100, H100_PCIe, B200). Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-05-13T14:00:17Z

/bot run

tensorrt-cicd · 2026-05-13T14:06:41Z

PR_Github #48187 [ run ] triggered by Bot. Commit: c5bc255 Link to invocation

tensorrt-cicd · 2026-05-13T15:23:53Z

PR_Github #48187 [ run ] completed with state SUCCESS. Commit: c5bc255
/LLM/main/L0_MergeRequest_PR pipeline #38007 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

The Pre-commit Check on PR NVIDIA#13872 failed with two issue families: * ruff-format (PEP-8 + 80-col yapf-equivalent) reformatted multiline asserts and f-strings in the new test files. * ruff-legacy reported 8 D205 regressions ('1 blank line required between summary line and description') vs the baseline. Some are in docstrings I added (proxy._wait_profile_ack, proxy.start_profile, proxy.stop_profile, test_e2e.test_trtllm_serve_profile_example, 4 docstrings in tests/unittest/executor/test_proxy_profile_sync.py) and a few are pre-existing violations whose line numbers shifted because of my additions (proxy.submit, test_e2e bench tests). Fix all 8 by inserting the required blank line after each docstring summary line and tightening summary lines so they are real sentences (D415 fix on the test_proxy_profile_sync.py module docstring). Also accept the ruff-format reflow on the two test files. Verified locally with: pre-commit run --files <touched files> — all hooks pass (yapf, ruff, ruff-format, ruff-legacy, autoflake, codespell, etc.). Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>

…ints Two follow-up fixes for the CI failures introduced by the profile endpoints PR (NVIDIA#13872): 1. API stability reference: add start_profile / stop_profile entries to tests/unittest/api_stability/references/llm.yaml so TestLLM::test_{signature,docstring,modified_init,...} stops failing with "LLM has different methods: {'start_profile', 'stop_profile'}". 2. Ray WorkerExtension collision: allow a user-supplied WorkerExtension to override BaseWorker.start_profile / stop_profile. These names are explicitly designated extension points (the RLHF WorkerExtension in tensorrt_llm/llmapi/rlhf_utils.py uses torch.cuda.profiler directly), so _inject_worker_extension now copies overridable extension methods into the derived class dict, which makes the extension's impl win regardless of MRO order and keeps the conflict check strict for every other name. Fixes the cascade of "RayGPUWorker died during initialization / Worker class RayGPUWorker already defines 'start_profile'" errors across the Ray test suites (H100, H100_PCIe, B200). Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

The Pre-commit Check on PR NVIDIA#13872 failed with two issue families: * ruff-format (PEP-8 + 80-col yapf-equivalent) reformatted multiline asserts and f-strings in the new test files. * ruff-legacy reported 8 D205 regressions ('1 blank line required between summary line and description') vs the baseline. Some are in docstrings I added (proxy._wait_profile_ack, proxy.start_profile, proxy.stop_profile, test_e2e.test_trtllm_serve_profile_example, 4 docstrings in tests/unittest/executor/test_proxy_profile_sync.py) and a few are pre-existing violations whose line numbers shifted because of my additions (proxy.submit, test_e2e bench tests). Fix all 8 by inserting the required blank line after each docstring summary line and tightening summary lines so they are real sentences (D415 fix on the test_proxy_profile_sync.py module docstring). Also accept the ruff-format reflow on the two test files. Verified locally with: pre-commit run --files <touched files> — all hooks pass (yapf, ruff, ruff-format, ruff-legacy, autoflake, codespell, etc.). Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-05-14T03:38:49Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-14T03:44:34Z

PR_Github #48285 [ run ] triggered by Bot. Commit: 41eca47 Link to invocation

tensorrt-cicd · 2026-05-14T16:22:17Z

PR_Github #48285 [ run ] completed with state SUCCESS. Commit: 41eca47
/LLM/main/L0_MergeRequest_PR pipeline #38097 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…ints Two follow-up fixes for the CI failures introduced by the profile endpoints PR (NVIDIA#13872): 1. API stability reference: add start_profile / stop_profile entries to tests/unittest/api_stability/references/llm.yaml so TestLLM::test_{signature,docstring,modified_init,...} stops failing with "LLM has different methods: {'start_profile', 'stop_profile'}". 2. Ray WorkerExtension collision: allow a user-supplied WorkerExtension to override BaseWorker.start_profile / stop_profile. These names are explicitly designated extension points (the RLHF WorkerExtension in tensorrt_llm/llmapi/rlhf_utils.py uses torch.cuda.profiler directly), so _inject_worker_extension now copies overridable extension methods into the derived class dict, which makes the extension's impl win regardless of MRO order and keeps the conflict check strict for every other name. Fixes the cascade of "RayGPUWorker died during initialization / Worker class RayGPUWorker already defines 'start_profile'" errors across the Ray test suites (H100, H100_PCIe, B200). Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

The Pre-commit Check on PR NVIDIA#13872 failed with two issue families: * ruff-format (PEP-8 + 80-col yapf-equivalent) reformatted multiline asserts and f-strings in the new test files. * ruff-legacy reported 8 D205 regressions ('1 blank line required between summary line and description') vs the baseline. Some are in docstrings I added (proxy._wait_profile_ack, proxy.start_profile, proxy.stop_profile, test_e2e.test_trtllm_serve_profile_example, 4 docstrings in tests/unittest/executor/test_proxy_profile_sync.py) and a few are pre-existing violations whose line numbers shifted because of my additions (proxy.submit, test_e2e bench tests). Fix all 8 by inserting the required blank line after each docstring summary line and tightening summary lines so they are real sentences (D415 fix on the test_proxy_profile_sync.py module docstring). Also accept the ruff-format reflow on the two test files. Verified locally with: pre-commit run --files <touched files> — all hooks pass (yapf, ruff, ruff-format, ruff-legacy, autoflake, codespell, etc.). Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>

The Pre-commit Check on PR NVIDIA#13872 failed with two issue families: * ruff-format (PEP-8 + 80-col yapf-equivalent) reformatted multiline asserts and f-strings in the new test files. * ruff-legacy reported 8 D205 regressions ('1 blank line required between summary line and description') vs the baseline. Some are in docstrings I added (proxy._wait_profile_ack, proxy.start_profile, proxy.stop_profile, test_e2e.test_trtllm_serve_profile_example, 4 docstrings in tests/unittest/executor/test_proxy_profile_sync.py) and a few are pre-existing violations whose line numbers shifted because of my additions (proxy.submit, test_e2e bench tests). Fix all 8 by inserting the required blank line after each docstring summary line and tightening summary lines so they are real sentences (D415 fix on the test_proxy_profile_sync.py module docstring). Also accept the ruff-format reflow on the two test files. Verified locally with: pre-commit run --files <touched files> — all hooks pass (yapf, ruff, ruff-format, ruff-legacy, autoflake, codespell, etc.). Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-05-15T03:12:35Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-15T03:18:57Z

PR_Github #48498 [ run ] triggered by Bot. Commit: 941d851 Link to invocation

tensorrt-cicd · 2026-05-15T23:23:31Z

PR_Github #48498 [ run ] completed with state FAILURE. Commit: 941d851
/LLM/main/L0_MergeRequest_PR pipeline #38294 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tensorrt-cicd · 2026-08-28T04:52:14Z

PR_Github #69881 [ run ] triggered by Bot. Commit: c29f879 Link to invocation

tensorrt-cicd · 2026-08-28T04:53:38Z

PR_Github #69883 [ run ] triggered by Bot. Commit: c29f879 Link to invocation

tensorrt-cicd · 2026-08-28T04:59:52Z

PR_Github/13872-c29f879 #69881 was force-killed by a newer pipeline run.
L0 job information not available (job may not have been triggered yet).

Link to superseding invocation

tensorrt-cicd · 2026-08-28T06:34:25Z

PR_Github #69883 [ run ] completed with state SUCCESS. Commit: c29f879
/LLM/main/L0_MergeRequest_PR pipeline #57168 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-08-28T06:37:01Z

/bot run

JunyiXu-nv · 2026-08-28T06:37:34Z

Pausing re-runs: the RDMA/UCX fabric is unhealthy cluster-wide

Four runs, four different failures, none of them this PR — and the last two are the same class:

run	pipeline	artifacts	failure
1	#57090	4632 (complete)	`batch_manager` C++ ctest
2	#57123	268	VAE B200 TCPStore rendezvous timeout
3	#57150	188	`ucx send failed: Endpoint timeout` → `MPI_ABORT`
4	#57168	117	`ibv_create_ah ... Connection timed out` → `NIXL_ERR_NOT_FOUND`

Runs 3 and 4 are both UCX/RDMA transport failures on DGX H100 multi-GPU disagg stages, so I checked whether that is specific to this PR. It is not.

test_disaggregated_serving has failed 40 times in the last 24 hours across 11 different PRs plus post-merge main, concentrated on exactly those stages:

DGX_H100-4_GPUs-PyTorch-Others-2     10
DGX_H100-4_GPUs-PyTorch-DeepSeek-1    8
DGX_H100-4_GPUs-PyTorch-Others-1      8
DGX_H100-2_GPUs-PyTorch-Others-1      6
DGX_H100-2_GPUs-PyTorch-Others-2      5

This PR accounts for 5 of those 40. Re-running will not converge until the fabric is healthy, so I have stopped re-triggering rather than keep burning CI.

Runs 2-4 also carry no evidence about this PR either way: fail-fast aborted them before the single-GPU stages ran — 117-268 artifacts against 4632, and zero test_disaggregated_perf_metrics artifacts in any of them.

Run #57090 remains the only complete exercise of this PR, and everything it touches passed:

check	result
disagg handshake delay	1.23 / 1.47 ms (was 36/47 ms with the eager queue, same node class)
applied clock offset	0.15 / 0.26 ms (was ~74 ms of pure error)
HMAC keys generated	6 (was 8 — confirms the lazy path is taken)
`test_disaggregated_perf_metrics`	passed, first attempt, 0 assertions
executor unit tests (x86 / SBSA)	1066 / 1066 passed, 0 FAILED
`test_proxy_profile_sync`	12 passed
`test_profile_endpoints`	12 passed
`test_trtllm_serve_profile_example`	17 passed

Suggest waiting for the RDMA issue to clear before re-running.

tensorrt-cicd · 2026-08-28T06:42:34Z

PR_Github #69910 [ run ] triggered by Bot. Commit: c29f879 Link to invocation

tensorrt-cicd · 2026-08-28T08:26:01Z

PR_Github #69910 [ run ] completed with state FAILURE. Commit: c29f879
/LLM/main/L0_MergeRequest_PR pipeline #57196 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-08-31T02:59:21Z

Rebased onto latest main (64 commits) and resolved four conflicts. Head is now 366d795e37, MERGEABLE, full pre-commit clean.

The conflicts were semantic, not textual — two of them would have silently reverted main's work if resolved by taking this branch's side:

conflict	what main did	resolution
`l0_a10.yml`	moved the MoE test tree `_torch/modules/moe/` → `_torch/moe/`	took main's path, merged in our entry
`openai_server.py` imports	added `ChatCompletionMessageParam`, `ChatCompletionToolsParam`, `StreamOptions`	union of 35 symbols, each verified to exist in `openai_protocol.py`
`openai_server.py` routes	refactored the RL routes into `_register_rl_control_routes()`	kept our profile routes + main's helper call, after checking the helper registers exactly the three routes our inline copy did
`l0_a10.yml` (clockfix)	replaced `- unittest/tools` with an explicit per-file list to keep `test_layer_wise_benchmarks.py` off A10 (nvbugs/6337228)	kept main's list and its comment, merged in our one entry

Re-triggering CI.

JunyiXu-nv · 2026-08-31T02:59:22Z

/bot run

tensorrt-cicd · 2026-08-31T03:07:11Z

PR_Github #70231 [ run ] triggered by Bot. Commit: 366d795 Link to invocation

JunyiXu-nv · 2026-08-31T06:00:12Z

/bot run

tensorrt-cicd · 2026-08-31T06:06:19Z

PR_Github #70266 [ run ] triggered by Bot. Commit: 366d795 Link to invocation

tensorrt-cicd · 2026-08-31T06:06:23Z

PR_Github #70231 [ run ] completed with state ABORTED. Commit: 366d795
/LLM/main/L0_MergeRequest_PR pipeline #57484 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tensorrt-cicd · 2026-08-31T07:18:26Z

PR_Github #70266 [ run ] completed with state SUCCESS. Commit: 366d795
/LLM/main/L0_MergeRequest_PR pipeline #57512 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-08-31T07:18:30Z

/bot run

JunyiXu-nv · 2026-08-31T07:19:49Z

/bot run

tensorrt-cicd · 2026-08-31T07:26:01Z

PR_Github #70289 [ run ] triggered by Bot. Commit: 366d795 Link to invocation

tensorrt-cicd · 2026-08-31T07:27:08Z

PR_Github #70290 [ run ] triggered by Bot. Commit: 366d795 Link to invocation

tensorrt-cicd · 2026-08-31T07:27:11Z

PR_Github #70289 [ run ] completed with state ABORTED. Commit: 366d795

Link to invocation

tensorrt-cicd · 2026-08-31T08:25:11Z

PR_Github #70290 [ run ] completed with state SUCCESS. Commit: 366d795
/LLM/main/L0_MergeRequest_PR pipeline #57533 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-09-01T08:05:22Z

/bot run

tensorrt-cicd · 2026-09-01T08:11:17Z

PR_Github #70639 [ run ] triggered by Bot. Commit: 366d795 Link to invocation

tensorrt-cicd · 2026-09-01T09:16:44Z

PR_Github #70639 [ run ] completed with state FAILURE. Commit: 366d795
/LLM/main/L0_MergeRequest_PR pipeline #57844 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…o trtllm-serve Expose iteration-scoped runtime profiling over HTTP on the OpenAI-compatible trtllm-serve endpoint. - POST /start_profile accepts output_dir, num_steps, start_step, activities (any of "CPU", "GPU", "CUDA_PROFILER"). With "CUDA_PROFILER" only, the server skips torch.profiler entirely so it composes with `nsys profile -c cudaProfilerApi`. - POST /stop_profile terminates the active window and flushes the trace. - PyExecutor.start_profile/stop_profile reuse the existing profile_start_iters / profile_stop_iters / cudaProfilerStart-Stop plumbing so env-var-based profiling behavior is preserved. - GenerationExecutor, BaseWorker and GenerationExecutorRpcProxy proxy the calls into the worker process; non-PyTorch backends fall through to a warning. - LLM.start_profile / stop_profile added as @set_api_status("prototype"). Rebased onto main; the branch's 15 development commits are collapsed into one because several predate the tensorrt_llm/executor/ray_gpu_worker.py -> ray/gpu_worker.py move and could not be replayed onto current main without hand-porting each intermediate state. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

… before applying it `_sync_server_clock` estimates each ctx/gen server's clock offset with the NTP algorithm over a single HTTP round trip, then adds it to every perf-metric timestamp that server reports. That estimate's error is bounded by half the round-trip delay, because an asymmetric round trip is indistinguishable from a real clock offset -- and the handshake runs while the servers are still finishing startup, so a single sample regularly lands on a stalled event loop. On one host the true offset is exactly 0 (everything reads CLOCK_MONOTONIC), so whatever it computes is pure injected error. Across CI, 2394 of 2407 measured offsets are positive: the bias is systematic and lands entirely on the outbound leg, where DNS resolution, connection setup and the server's first-ever request are paid. `test_disaggregated_perf_metrics` asserts a 5 ms tolerance against that offset and has failed 4/4 runs here with 33-69 ms of injected skew, versus a 0.87 ms median across 128 baseline runs on main. - Warm up the connection first so DNS and connect costs stay out of the samples. - Probe up to `_CLOCK_SYNC_PROBES` times and keep the least-delayed round trip (NTP's clock filter), stopping early once a sample is conclusive so a healthy server still costs one probe. - Discard the offset when even the best delay exceeds `_CLOCK_SYNC_MAX_DELAY_SECONDS`: past that the estimate is worth less than the zero it would replace, since co-located servers share CLOCK_MONOTONIC and NTP-synced hosts agree to well under a millisecond. - Bound the cost: a dedicated per-request timeout instead of inheriting `_req_timeout_secs`, a wall-clock budget for the probe loop, and no retries against a server that errors -- servers are prepared sequentially, so an unresponsive one must not stall the queue behind it. Adds CPU-only unit tests driving the handshake against a stub endpoint with an injectable one-sided stall. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-09-01T09:32:44Z

Rebased again — this time to pick up a fix, not just to clear a conflict.

Seven of the last eight runs died on things this PR does not touch, and the recurring one was test_combined_fmha.py::test_flashinfer_context_fallback_scope on DGX_B200. Rather than keep re-running, I checked where that test actually stands on main:

f152eb29af  2026-09-01  [https://nvbugs/6697142][fix] Wire the stub the way
                        `TrtllmAttention.forward` does and parametrize over both… (#18504)

It has a tracked bug (nvbugs/6697142) and a landed fix, and it is not in waives.txt — so it was genuinely fixed, not waived around. The branch was based on 73ee6b0b19 (08-31 10:25), a day older than that fix, which is why every run kept hitting it. The 24 failures visible in the last 24 h all belong to a single post-merge main build, i.e. also from before the fix.

Head is now 0e462d0f79, based directly on f152eb29af, zero commits behind main, MERGEABLE, full pre-commit clean. The rebase produced no conflicts — rerere reused the four resolutions from the previous rebase (MoE test-tree move, the import union, the RL-route refactor, and the per-file unittest/tools list).

Worth recording so it does not get repeated: I had earlier concluded from OpenSearch that this test was "0 failures in the last 12 h" and treated that as a green light. That was wrong — the index runs more than six hours behind, so the recent windows read as "0 runs", not "0 failures". Commit history is the reliable signal here; failure statistics are not.

Re-triggering.

JunyiXu-nv · 2026-09-01T09:32:45Z

/bot run

tensorrt-cicd · 2026-09-01T09:38:39Z

PR_Github #70663 [ run ] triggered by Bot. Commit: 0e462d0 Link to invocation

tensorrt-cicd · 2026-09-01T11:37:34Z

PR_Github #70663 [ run ] completed with state SUCCESS. Commit: 0e462d0
/LLM/main/L0_MergeRequest_PR pipeline #57864 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned JunyiXu-nv May 8, 2026

JunyiXu-nv force-pushed the feat/trtllm-serve-profile-endpoints branch 2 times, most recently from 69b3d68 to 86b3237 Compare May 11, 2026 09:05

JunyiXu-nv force-pushed the feat/trtllm-serve-profile-endpoints branch 2 times, most recently from 0b70c55 to 029a94b Compare May 11, 2026 09:20

JunyiXu-nv changed the title ~~[None][feat] add /start_profile and /stop_profile endpoints to trtllm…~~ [TRTLLM-12670][feat] add /start_profile and /stop_profile endpoints to trtllm… May 14, 2026

JunyiXu-nv added the api-compatible Accepted LLM API contract change that is backwards-compatible label May 14, 2026

JunyiXu-nv force-pushed the feat/trtllm-serve-profile-endpoints branch from a07a3da to 41eca47 Compare May 14, 2026 03:34

JunyiXu-nv force-pushed the feat/trtllm-serve-profile-endpoints branch from 41eca47 to fcb8426 Compare May 15, 2026 03:03

JunyiXu-nv force-pushed the feat/trtllm-serve-profile-endpoints branch 2 times, most recently from 1e41133 to 941d851 Compare May 15, 2026 03:10

JunyiXu-nv added 2 commits September 1, 2026 09:31

Uh oh!

Conversation

JunyiXu-nv commented May 8, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

JunyiXu-nv commented May 11, 2026

Uh oh!

tensorrt-cicd commented May 11, 2026

Uh oh!

tensorrt-cicd commented May 11, 2026

Uh oh!

JunyiXu-nv commented May 13, 2026

Uh oh!

tensorrt-cicd commented May 13, 2026

Uh oh!

tensorrt-cicd commented May 13, 2026

Uh oh!

JunyiXu-nv commented May 14, 2026

Uh oh!

tensorrt-cicd commented May 14, 2026

Uh oh!

tensorrt-cicd commented May 14, 2026

Uh oh!

JunyiXu-nv commented May 15, 2026

Uh oh!

tensorrt-cicd commented May 15, 2026

Uh oh!

tensorrt-cicd commented May 15, 2026

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

JunyiXu-nv commented Aug 28, 2026

Uh oh!

JunyiXu-nv commented Aug 28, 2026

Pausing re-runs: the RDMA/UCX fabric is unhealthy cluster-wide

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

JunyiXu-nv commented Aug 31, 2026

Uh oh!

JunyiXu-nv commented Aug 31, 2026

Uh oh!

tensorrt-cicd commented Aug 31, 2026

Uh oh!

JunyiXu-nv commented Aug 31, 2026

Uh oh!

tensorrt-cicd commented Aug 31, 2026

Uh oh!

tensorrt-cicd commented Aug 31, 2026

Uh oh!

tensorrt-cicd commented Aug 31, 2026

Uh oh!

JunyiXu-nv commented Aug 31, 2026

Uh oh!

JunyiXu-nv commented Aug 31, 2026

Uh oh!

tensorrt-cicd commented Aug 31, 2026

Uh oh!

tensorrt-cicd commented Aug 31, 2026

Uh oh!

tensorrt-cicd commented Aug 31, 2026

Uh oh!

tensorrt-cicd commented Aug 31, 2026

Uh oh!

JunyiXu-nv commented Sep 1, 2026

Uh oh!

JunyiXu-nv commented May 8, 2026 •

edited by coderabbitai Bot

Loading