Skip to content

fix(ci): unblock nightly startup after Brev reusable workflow merge - #3376

Closed
jyaunches wants to merge 1 commit into
NVIDIA:mainfrom
jyaunches:fix/nightly-startup-after-brev-3350
Closed

fix(ci): unblock nightly startup after Brev reusable workflow merge#3376
jyaunches wants to merge 1 commit into
NVIDIA:mainfrom
jyaunches:fix/nightly-startup-after-brev-3350

Conversation

@jyaunches

@jyaunches jyaunches commented May 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Nightly E2E started failing with startup_failure after #3350 merged. These runs have zero jobs and no logs, which means GitHub failed to instantiate the workflow graph before any test job could start.

Root cause: #3350 added a new brev-e2e reusable workflow call whose callee declares required BREV_API_KEY and BREV_ORG_ID secrets. Those secrets were validated in the fork, but they are not yet installed in the upstream NVIDIA/NemoClaw repo (only legacy BREV_API_TOKEN and NVIDIA_API_KEY are present there). A reusable workflow with missing required secrets fails during workflow startup, which aborts the entire nightly workflow graph.

This PR restores nightly startup immediately by disabling the brev-e2e job until the upstream secrets are installed. It also keeps the caller conservative:

  • if: false on brev-e2e until upstream BREV_API_KEY / BREV_ORG_ID exist.
  • branch: main for nightly runs. Branch-specific Brev validation should use e2e-branch-validation.yaml directly.
  • secrets: inherit for the same-repo reusable workflow call once it is re-enabled.

Validation

  • YAML parses locally for .github/workflows/nightly-e2e.yaml.
  • The failed nightly runs are startup failures with zero jobs/logs, matching missing-required-secret/reusable-workflow graph instantiation failure rather than product test failures.
  • Confirmed upstream secrets currently lack BREV_API_KEY and BREV_ORG_ID; fork has them.

Follow-up

After adding upstream repo/org secrets:

  1. BREV_API_KEY
  2. BREV_ORG_ID

remove the temporary if: false and re-enable the nightly brev-e2e matrix.

Related

Summary by CodeRabbit

  • Chores
    • Updated nightly end-to-end CI workflow: temporarily disabled the reusable job to avoid startup failures.
    • Standardized the workflow branch source to "main".
    • Changed secrets handling to inherit secrets from the calling workflow.

Note: Internal infrastructure updates only; no user-facing changes.

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

The nightly E2E workflow's brev-e2e job is updated to always target the main branch instead of dynamically using the current ref, with added inline documentation. The job also switches from explicit secret mappings to secrets: inherit for cleaner secret propagation to the called reusable workflow.

Changes

Nightly E2E Workflow

Layer / File(s) Summary
Brev E2E Job Configuration
.github/workflows/nightly-e2e.yaml
The brev-e2e job hardcodes branch: main with inline documentation clarifying scheduled vs manual behavior. Secret passing is simplified from explicit BREV_API_KEY, BREV_ORG_ID, and NVIDIA_API_KEY mappings to secrets: inherit.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

🐰 A workflow takes shape, so nimble and bright,
With main branch secured, and secrets held tight,
No more mapping dances, inheritance flows,
The nightly E2E test suite now goes!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title accurately describes the main change: a fix to unblock nightly CI startup that was broken after a Brev reusable workflow merge, directly addressing the core issue.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
⚔️ Resolve merge conflicts
  • Resolve merge conflict in branch fix/nightly-startup-after-brev-3350

Comment @coderabbitai help to get the list of available commands and usage tips.

Nightly E2E started failing at workflow startup after NVIDIA#3350 merged: the
runs have conclusion=startup_failure and zero jobs/logs, which means GitHub
could not instantiate the workflow graph rather than tests failing.

The only workflow-graph change from NVIDIA#3350 was the new brev-e2e reusable
workflow call. Make that call more conservative:

- pass branch: main again for nightly-e2e. Scheduled nightly runs validate
  main; branch-specific Brev validation should use e2e-branch-validation.yaml
  directly.
- use secrets: inherit for the same-repository reusable workflow call instead
  of mapping individual secrets in the caller. The callee still declares the
  required BREV_API_KEY, BREV_ORG_ID, and NVIDIA_API_KEY secrets.

This keeps the Brev job wired into nightly while removing the likely
startup-time expression/secret mapping issue that prevented any nightly jobs
from being created.
@jyaunches
jyaunches force-pushed the fix/nightly-startup-after-brev-3350 branch from 986bd52 to 924a61f Compare May 12, 2026 00:36

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
.github/workflows/nightly-e2e.yaml (1)

1800-1813: ⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

Merge duplicate if conditions in brev-e2e job to fix workflow parse error.

Line 1813 introduces a second if key that duplicates the condition at line 1800, which blocks workflow instantiation. Linting tools (actionlint, yamllint) report key "if" is duplicated in mapping.

Merge the two conditions by prepending the disabling logic to the existing condition:

Suggested fix
   brev-e2e:
     if: >-
+      false &&
       github.repository == 'NVIDIA/NemoClaw' &&
       (github.event_name != 'workflow_dispatch' ||
        inputs.jobs == '' ||
        contains(format(',{0},', inputs.jobs), ',brev-e2e,'))
     strategy:
       fail-fast: false
       matrix:
         test_suite: [all, messaging-providers, full]
     # Re-enable once BREV_API_KEY/BREV_ORG_ID are installed in the upstream
     # NVIDIA/NemoClaw repo. Referencing a reusable workflow that requires
     # missing secrets causes GitHub to fail the entire workflow at startup
     # (zero jobs/logs), which is what broke nightly after `#3350` merged.
-    if: false
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/nightly-e2e.yaml around lines 1800 - 1813, The job-level
duplicate "if" in the brev-e2e job causes a YAML parse error; remove the second
"if: false" and merge its disabling logic into the existing "if" condition for
brev-e2e by combining them into a single boolean expression (i.e., incorporate
the false-disabling clause with an AND into the original complex condition).
Edit the brev-e2e job's existing "if" key so it contains the full, merged
condition and delete the redundant "if: false" entry to resolve the duplicated
key error.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In @.github/workflows/nightly-e2e.yaml:
- Around line 1800-1813: The job-level duplicate "if" in the brev-e2e job causes
a YAML parse error; remove the second "if: false" and merge its disabling logic
into the existing "if" condition for brev-e2e by combining them into a single
boolean expression (i.e., incorporate the false-disabling clause with an AND
into the original complex condition). Edit the brev-e2e job's existing "if" key
so it contains the full, merged condition and delete the redundant "if: false"
entry to resolve the duplicated key error.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a0ceb970-a47a-4a32-a424-0537837939f6

📥 Commits

Reviewing files that changed from the base of the PR and between 986bd52 and 924a61f.

📒 Files selected for processing (1)
  • .github/workflows/nightly-e2e.yaml

@cv cv added v0.0.40 and removed v0.0.39 labels May 12, 2026

@ericksoa ericksoa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this PR should be closed as superseded rather than repaired.

Current main no longer has the brev-e2e block this PR is trying to patch; #3373 already reverted the #3350 nightly wiring that caused the startup failure. When I tested the branch against current main, the conflict was exactly that obsolete block: this branch wants to keep/disable brev-e2e, while main has removed it.

There is also still a real issue on this head: the proposed nightly-e2e.yaml has duplicate job-level if keys in the brev-e2e job, which risks workflow parse/startup failure if merged. Since the underlying broken job is already gone from main, I would not spend time fixing this branch. Please close this PR as superseded by #3373/current main.

@ericksoa

Copy link
Copy Markdown
Contributor

Closing as superseded by #3373/current main. The broken brev-e2e wiring this PR disables has already been removed from nightly-e2e.yaml, and this branch now conflicts with current main on that obsolete block.

@ericksoa ericksoa closed this May 13, 2026
@wscurran wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure bug-fix PR fixes a bug or regression chore Build, CI, dependency, or tooling maintenance and removed CI/CD chore Build, CI, dependency, or tooling maintenance labels Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure bug-fix PR fixes a bug or regression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants