Skip to content

fix(agent): make the functional-test report say what it actually tested - #3524

Merged
shanmugamr1992 merged 2 commits into
nemorl-testing-agentfrom
nrlta-carry-bridge-fixes
Aug 6, 2026
Merged

fix(agent): make the functional-test report say what it actually tested#3524
shanmugamr1992 merged 2 commits into
nemorl-testing-agentfrom
nrlta-carry-bridge-fixes

Conversation

@shanmugamr1992

@shanmugamr1992 shanmugamr1992 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #3522, from continuing to pull on the same sweep. Two commits, each
fixing a way the functional-test report could mislead the person reading it.

1. Known breaks were reported as PR-caused failures

known_issues.py annotate labelled a matched failure with its diagnosis and fix
link but left status: fail, which the report vocabulary defines as "genuine
failure caused by the PR"
. Nothing looked wrong while apply_baseline.py
happened to cover for it: it downgrades any failure the cached baseline also
hit, and the baseline was red for the same reason.

That cover disappears exactly when a fix is raised. The baseline gets re-taken on
a stack carrying the fix, goes green, stops matching — so the next sweep reports
the same known break as fail against whichever PRs are under test, with a
link to the fix printed directly underneath.

An entry only ever exists for a break the PR did not cause, so annotate now
downgrades to fail (pre-existing) on match. It deliberately does not read
first_seen_megatron_pr as a culprit: that field records which run first
surfaced the break, usually neither the cause nor even the same repository — the
Bridge break in the registry was first seen on Megatron-LM#5382 and caused by
#5865, merged weeks earlier.

Carrying a fix the integration branch cannot reach. sync_integration.sh
cherry-picks open NeMo-RL fixes onto the branch under test, so those reach later
runs by themselves. A Megatron-Bridge or Megatron-LM fix is checked out inside
the container instead, so it had to be passed as --bridge-ref on every single
submit until it merged; forgetting it produces a confident wrong answer rather
than a missing one. run_suite.sh now asks the registry through a new
pending-fix-ref query, and errors rather than guessing when two fixes are
pending in one repo. This makes --fix-branch load-bearing at record time,
which the triage and fix-delivery skills now say.

2. The report did not say which revisions it ran

The header carried four bare shas, retyped by hand into --meta on every post.
Nothing checked them against the run, so a stale copy-paste read exactly like a
result, and the refs that give a sha its meaning were dropped on the way.

That got worse once runs stopped being uniform. Megatron-Bridge is now either the
sha NeMo-RL pins or a fix branch carried in its place, and those are different
claims about what a green table proves. NeMo-RL is normally a fork branch, so its
sha links nowhere a reader can check. megatron-core main and a PR head are
indistinguishable once written as hex.

The report now builds this from the prep block the run already recorded:

Component Ref Commit
megatron-core #5382 head eb01b689
Megatron-Bridge mcore-5382-fix — an override, not the Bridge NeMo-RL pins 2940a635
NeMo-RL nrlta/integration 86774472

Clone URLs are recorded in prep so those links can be built at all. The
watchdog issue imports the same renderer rather than keeping its own, which had
begun asserting a bare Bridge sha that is no longer necessarily the pin.

Test plan

  • test_known_issues.sh — 18 checks, extended with the downgrade, the
    suspect-pass and unmatched-failure cases that must not move, the
    stale-entry case that keeps its fail, and pending-fix-ref on none /
    one / two pending fixes
  • test_report_stack.sh — new; 17 checks covering all three Bridge ref
    shapes, the fork link, PR-head vs main, header de-duplication, results
    predating the URL record, --state running, and the watchdog heading
  • Remaining four agent test scripts pass unchanged
  • run_suite.sh --dry-run picks up mcore-5382-fix from the live registry
  • Re-annotating the real PR 5382 L1 results downgrades only topp_topk
  • Rendered against the real PR 5382 and watchdog baseline results
  • ruff check / ruff format

A registry entry only ever describes a break the revision under test did not
cause, but annotate added a comment and left the status at `fail`, which the
report reads as "this PR broke it". apply_baseline was covering for that by
accident: it downgrades a failure the baseline also hit, and the baseline was
red for the same reason. The moment a fix existed and the baseline went green,
three innocent PRs were reported as breaking the suite with a link to the fix
printed directly underneath. Downgrade on match instead.

Carry unmerged out-of-repo fixes automatically while we are here. A NeMo-RL fix
reaches later runs on the integration branch; a Megatron-Bridge one cannot ride
a NeMo-RL branch, so it had to be remembered as --bridge-ref on every submit for
as long as review took. run_suite.sh now asks the registry, via a new
pending-fix-ref query, and refuses to guess when two fixes are pending at once.

Signed-off-by: shanmugamr1992 <shanmugamr1992@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

The header carried four bare shas, retyped by hand into --meta on every post.
Nothing checked them against the run, so a stale copy-paste read exactly like a
result, and the refs that give a sha its meaning were dropped on the way.

Which mattered more once runs stopped being uniform. Megatron-Bridge is either
the sha NeMo-RL pins or a fix branch carried in its place, and those are
different claims about what a green table proves. NeMo-RL is normally a fork
branch, so its sha links nowhere the reader can check. megatron-core `main` and
a PR head are indistinguishable once written as hex.

Render the table from the prep block the run already recorded instead, naming
ref and commit together and linking each to the repo it was fetched from. The
clone URLs are now recorded so those links can be built at all. The watchdog
issue imports the same renderer rather than keeping its own, which had begun
asserting a bare Bridge sha that is no longer necessarily the pin.

Signed-off-by: shanmugamr1992 <shanmugamr1992@gmail.com>
@shanmugamr1992 shanmugamr1992 changed the title fix(agent): stop reporting known breaks as PR-caused failures fix(agent): make the functional-test report say what it actually tested Aug 6, 2026
@shanmugamr1992
shanmugamr1992 marked this pull request as ready for review August 6, 2026 20:12
@shanmugamr1992
shanmugamr1992 requested a review from a team as a code owner August 6, 2026 20:12
@shanmugamr1992
shanmugamr1992 merged commit 7c32c52 into nemorl-testing-agent Aug 6, 2026
11 of 14 checks passed
@shanmugamr1992
shanmugamr1992 deleted the nrlta-carry-bridge-fixes branch August 6, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant