[NVBUG-6448152][test] isolate gen status caller guard

[chienchunhung]

Caution

TEST ONLY — never merge. This draft is a single-factor performance diagnostic for NVBUG 6448152.

Question

Does the unconditional per-loop generation-transfer status call introduced by PR #15356 cause the localized output-throughput regression by issuing empty/idle status work?

The adjacent comparison already localizes the regression to PR #15356 as a commit unit: parent 1515.84 output tok/s versus child 799.73. Prior admission diagnostics did not recover throughput, and the complete first-gate replay did not change an admission decision on its measured trace: terminal first-gate evidence.

Frozen causal identity

• Exact slow baseline: dd4a7ac2992f2b02dce381ce4e630c4ba4199436
• Signed treatment commit: 3835a8758ca0a20a69a6c13275ce767d2bfc2e06
• CI ancestry head: 7f36854604a2f9aee4bd8826b9f30788b0124ea6
• Treatment tree: bed490ecfc567777ebbdbc8b3721d6b60fa9b39f
• Current-main ancestry-only second parent: 73efcee5922de49560ca39ba7938a944e5d13bd3
• The CI head tree is byte-identical to the signed treatment tree.
• Stable treatment patch-id: f71a36a24d3a718087eb20b23cb45d4e88031abf
• Baseline-to-treatment binary diff SHA256: 1e2161b613f9190ea1689c476c4221c717dc850b0d73e4de167bd9d52f51ae62

The pre-main treatment is preserved at codex/backup-nvbug-6448152-gen-status-caller-guard-pre-main-20260803-3835a87.

Single treatment

With the TEST ONLY environment switch enabled, _check_disagg_gen_transfer_status restores only the pre-PR #15356 entry predicate outside KV-capacity warmup:

• no generation transfer active: skip that loop-entry status call;
• at least one generation transfer active: retain the existing nonblocking atLeastRequestNum=0 call.

It deliberately does not restore the old atLeastRequestNum=1 selection. The immediate receive-side poll and admission-progress poll are unchanged. The bounded C++ callee, packed terminal consensus, ready-ID gathers, both admission gates, scheduler, runtime, image, model, dataset, and topology remain unchanged.

Warmup preserves the baseline unconditional call and is excluded from diagnostic counters.

Exercised evidence

ID-free startup, first-decision, and shutdown markers report topology and per-loop counts on every worker:

• total post-warmup checks;
• retained nonblocking loop status calls;
• loop calls skipped because no transfer was active;
• maximum active transfers;
• cumulative and maximum retained-call wall time.

The exact topology makes the diagnostic safe and distinguishes the expected effects:

• CTX: PP4, Attention-DP off. All CTX ranks have no generation-transfer requests and skip symmetrically. Each skipped baseline empty call would otherwise execute the ready-ID count/payload and packed-terminal count/payload collectives.
• GEN: PP1, Attention-DP on with data groups of size one. The local predicate cannot mismatch a multi-rank generation-status collective.

Frozen workload and acceptance

Run only the existing GB300 three-node DeepSeek-R1 128K/8K context-first Python/NIXL selector: CTX PP4, GEN DEP8, 256 concurrency, two rounds (512 total requests), multiplier 1, and the current rendered image.

Required before interpreting throughput:

• exact CI head/tree, selector, image, three nodes, and 12 GPUs;
• 512/512 requests, zero failures, b_is_valid=true;
• mode 2 / cancellation off on CTX ranks 0–3;
• clean CTX, GEN, disaggregated-service, and benchmark teardown;
• effective-config and shutdown summary on CTX ranks 0–3 and GEN ranks 0–7;
• skipped_loop_calls > 0 on every CTX rank;
• active-transfer call evidence on the relevant GEN ranks.

Interpretation against the exact 799.73 tok/s slow baseline:

• exercised markers plus recovery to at least about 1402 tok/s implicates unconditional caller cadence as a dominant cause;
• exercised markers plus throughput near 800 tok/s rules this factor down as sufficient;
• missing markers, topology mismatch, invalid metric, or workload failure censors the result.

Local validation

• Python compilation and diff checks passed.
• Configured formatting, lint, YAML, secret, and static checks passed.
• The two test-list checks were rerun successfully under Python 3.13 (2683 unique entries validated) because the configured older hook runner cannot parse str | None.
• Focused test collection is locally blocked only by the absent transformers package.
• DCO and independent no-confound review passed with no P0–P3 findings.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #63548 [ run ] triggered by Bot. Commit: 7f36854 Link to invocation

[tensorrt-cicd]

PR_Github #63548 [ run ] completed with state FAILURE. Commit: 7f36854
/LLM/main/L0_MergeRequest_PR pipeline #51515 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

Terminal TEST ONLY result: the exact three-node GB300 workload completed 512/512 requests with zero client failures, b_is_valid=true, clean CTX/GEN/disaggregated shutdown, and 800.66 output tok/s (13,611.26 total tok/s). The exact slow baseline was 799.73 output tok/s, so the treatment changed throughput by only +0.12%. The caller guard was fully exercised: every CTX rank skipped all 643 empty loop-entry checks; every GEN rank emitted both skip and active-call evidence, skipping 194,534–241,284 of 244,601 checks while retaining active-transfer calls. Therefore restoring the pre-PR #15356 PyExecutor caller guard does not recover throughput and rules empty/idle status-poll cadence down as a sufficient or dominant cause for this workload. The CI failure is the expected performance-regression threshold, not a functional failure. Closing unmerged; the exact diagnostic branch is preserved and must not be rerun unchanged.