Skip to content

[NVBUG-6448152][test] TEST ONLY retry pre-admission native 85665f5f - #16960

Closed
chienchunhung wants to merge 10 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-native-pre-admission-85665f5f-compat
Closed

[NVBUG-6448152][test] TEST ONLY retry pre-admission native 85665f5f#16960
chienchunhung wants to merge 10 commits into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6448152-native-pre-admission-85665f5f-compat

Conversation

@chienchunhung

Copy link
Copy Markdown
Collaborator

TEST ONLY

Retry of the exact pre-admission historical-source/current-runtime discriminator after the prior attempt was censored in prerequisite compilation. This draft is diagnostic only and is not intended for merge.

  • Native source checkpoint: 85665f5fd3, exact native tree 8ada9734721bcd127722b367152eeb5bb22fa4ca, parent eaf5693b3aee9acea0d322dc8be997c4c0975280.
  • This is the exact first-parent immediately before the admission and bounded-polling change, and 100 first-parent commits before the known-slow native checkpoint 9ff00c869a36055fcf2ebf4de4950a0471a3039f.
  • Admission-specific controller, requester timestamping, bounded polling, scheduler/configuration, and serialization changes are absent from the checkout tree. Current main appears only through no-tree CI-ancestry merges; it contributes no product content to the tested tree.
  • Runtime overlay, Jenkins harness, no-home-mount behavior, asynchronous-consensus factor, protocol mode 2 with cancellation disabled, CMake normalization/restoration, and the compile-only Marlin stub repair are unchanged from the prior attempt and completed slow discriminators.
  • The only new delta is the exact relevant three-file compile/toolchain portion of the later DLFW dependency compatibility change, with stable patch ID 2128f376c81a96a070037f6d139abaf6c9b37092. It pins DeepEP compilation to the vendored NVSHMEM headers, removes a Clang-only flag after the sub-build switches to GCC, and normalizes CUDA virtual-memory aggregate initialization. It changes no admission, scheduling, transceiver, workload, or benchmark behavior.
  • The exact workload YAML, test list, selector, performance baseline, performance test, and Slurm workload files remain byte-identical to the native checkpoint and the completed slow discriminators. No DSpark, workload, selector, timeout, or product compatibility adaptation is included.
  • The target remains the exact GB300 3-node/12-GPU disaggregated performance selector. A valid result at or above 1402 output tokens/s places the regression at the admission change or within the following 99 commits; a valid result near 800 proves the slowdown predates admission and moves the source bound earlier. Any build/import failure, timeout, protocol mismatch, cancellation, or failed request is non-interpretable.

The merged asynchronous-consensus change is not under re-review here; this draft only localizes the separate source-throughput regression.

chienchunhung and others added 9 commits July 27, 2026 20:24
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 6fc7f33)
…sensus factor

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 9c8bf59)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
(cherry picked from commit 4b182f1)
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62236 [ run ] triggered by Bot. Commit: fd6fb71 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #62236 [ run ] completed with state FAILURE. Commit: fd6fb71
/LLM/main/L0_MergeRequest_PR pipeline #50405 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Copy link
Copy Markdown
Collaborator Author

Terminal outcome: censored before pytest

The single pre-admission discriminator run reached the exact requested GB300 stage at head fd6fb71e35ea2dd2113294596c065ce0afee312d. The prerequisite SBSA build succeeded, the exact three-node/twelve-GPU topology and selector were established, and artifact installation completed.

Before pytest collection or workload startup, the disaggregated-server process failed while importing tensorrt_llm/_utils.py:

ModuleNotFoundError: No module named 'tensorrt'

This is a runtime/package compatibility censor, not a performance result. There is no JUnit result, request count, protocol-mode evidence, shutdown evidence, or output-token metric. Consequently:

  • no fast/slow classification is valid;
  • the source regression bound does not move;
  • this run must not be compared with completed throughput checkpoints;
  • no manual rerun was issued.

This TEST ONLY draft is being closed unmerged. Its diagnostic branch is preserved.

@chienchunhung

Copy link
Copy Markdown
Collaborator Author

The import blocker was traced to the historical TensorRT 10.15/PyTorch 26.02 dependency input being installed inside the frozen TensorRT 10.16.1 runtime. The dependency-compatible replacement discriminator ports the exact upstream DLFW 26.04 requirements update used by the completed comparator, without changing product source or workload behavior. Its single targeted GB300 launch is active; this closed branch remains preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants