Skip to content

ci(e2e): restore Docker Hub authentication coverage - #6100

Merged
cv merged 3 commits into
mainfrom
codex/fix-e2e-dockerhub-auth
Jul 1, 2026
Merged

ci(e2e): restore Docker Hub authentication coverage#6100
cv merged 3 commits into
mainfrom
codex/fix-e2e-dockerhub-auth

Conversation

@cv

@cv cv commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Restores Docker Hub authentication across every image-consuming scheduled/manual-main E2E job after the legacy nightly workflow retirement left 27 jobs anonymous. Trusted main runs now use one guarded login contract, while untrusted refs retain isolated anonymous Docker configs.

Related Issue

Fixes #4036

Changes

  • Apply the canonical guarded login to all 67 image-consuming E2E jobs, with six no-image jobs kept as explicit exemptions.
  • Add hardened isolated cleanup, a trusted-login marker, bounded retries, and fail-closed behavior for trusted runs.
  • Centralize regression validation and behavior tests so future E2E jobs cannot silently omit authentication.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: Internal CI workflow/auth boundary only; no user-facing behavior or documentation surface changes.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Credential scope and cleanup boundaries were reviewed locally; guards restrict secrets to NVIDIA/NemoClaw main schedule/manual runs, and focused adversarial tests cover traversal and symlink attacks.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Git hooks passed during commit and push, or npx prek run --from-ref main --to-ref HEAD passes
  • Targeted tests pass for changed behavior
  • Full npm test passes (broad runtime changes only)
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

  • New Features
    • Centralized Docker Hub authentication and cleanup across E2E workflows for more consistent CI behavior.
    • Standardized guarded login and cleanup so image jobs follow the same audited process.
  • Bug Fixes
    • Reduced the risk of leftover Docker credentials by cleaning up auth state reliably after runs.
    • Improved handling when authentication is skipped or fails, preventing unsafe credential persistence.
  • Tests
    • Expanded workflow boundary coverage to enforce the canonical auth/cleanup steps, ordering, and secret-safety rules.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv cv self-assigned this Jul 1, 2026
@github-code-quality

github-code-quality Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the codex/fix-e2e-docker... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main codex/fix-e2e-docker... 4ede2f0 +/-
nemoclaw/src/se...cret-scanner.ts 100%
nemoclaw/src/commands/slash.ts 100%
nemoclaw/src/li...bprocess-env.ts 100%
nemoclaw/src/bl...eprint/state.ts 98%
nemoclaw/src/onboard/config.ts 98%
nemoclaw/src/bl...int/snapshot.ts 97%
nemoclaw/src/bl...print/runner.ts 95%
nemoclaw/src/co...ration-state.ts 94%
nemoclaw/src/bl...ate-networks.ts 94%
nemoclaw/src/index.ts 94%

TypeScript / code-coverage/cli

The overall coverage in the codex/fix-e2e-docker... branch is 68%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main codex/fix-e2e-docker... 4ede2f0 +/-
src/lib/shields...nsition-lock.ts 86%
src/lib/actions...dbox/rebuild.ts 80%
src/lib/actions...all/run-plan.ts 80%
src/lib/state/o...oard-session.ts 80%
src/lib/state/sandbox.ts 72%
src/lib/onboard/preflight.ts 69%
src/lib/shields/index.ts 67%
src/lib/onboard...er-gpu-patch.ts 59%
src/lib/actions...licy-channel.ts 58%
src/lib/onboard.ts 20%

Updated July 01, 2026 06:27 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: sandbox-operations, hermes-inference-switch, openclaw-inference-switch, hermes-sandbox-secret-boundary, openclaw-slack-pairing, tunnel-lifecycle, spark-install
Optional E2E: runtime-overrides, cloud-onboard, hermes-e2e

Dispatch hint: sandbox-operations,hermes-inference-switch,openclaw-inference-switch,hermes-sandbox-secret-boundary,openclaw-slack-pairing,tunnel-lifecycle,spark-install

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • sandbox-operations (high): Required because the sandbox-operations workflow boundary and shared Docker auth/cleanup path changed; this live job exercises real sandbox lifecycle operations through the affected workflow wiring.
  • hermes-inference-switch (high): Required because inference-switch workflow boundary logic changed for Hermes, including the hosted/Anthropic matrix and shared Docker auth ordering.
  • openclaw-inference-switch (high): Required because inference-switch workflow boundary logic changed for OpenClaw and specifically validates the Docker auth, live switch test, artifact upload, and cleanup order.
  • hermes-sandbox-secret-boundary (high): Required because the PR changes Docker credential handling for image-building Hermes jobs and touches Hermes secret-boundary workflow coverage; this job verifies real image/startup secret isolation under the new auth path.
  • openclaw-slack-pairing (high): Required because the OpenClaw Slack workflow boundary was touched and the live job depends on the changed image-pull/auth cleanup wiring for a real assistant integration flow.
  • tunnel-lifecycle (high): Required because tunnel lifecycle workflow-boundary tests were touched and the workflow changes can affect trusted setup, credential scoping, and cleanup for the live tunnel lifecycle flow.
  • spark-install (medium): Required because release-gate workflow coverage explicitly changed Spark image-pull authentication expectations; this verifies the shared guarded Docker auth path on the Spark install lane.

Optional E2E

  • runtime-overrides (medium): Useful representative image-consuming job for validating the new shared Docker auth and cleanup helper on a real sandbox image build without exercising a full assistant flow.
  • cloud-onboard (high): Useful broader confidence that the modified E2E workflow still supports hosted onboarding through the shared auth/cleanup wiring, although no onboarding state-machine source changed.
  • hermes-e2e (high): Useful full Hermes assistant smoke for broad coverage of the shared Docker auth change across real onboarding, sandbox startup, and assistant behavior.

New E2E recommendations

  • None.

Dispatch hint

  • Workflow: .github/workflows/e2e.yaml
  • jobs input: sandbox-operations,hermes-inference-switch,openclaw-inference-switch,hermes-sandbox-secret-boundary,openclaw-slack-pairing,tunnel-lifecycle,spark-install

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

E2E Target Recommendation

Required E2E targets: e2e-all
Optional E2E targets: None

Dispatch required E2E targets:

  • gh workflow run e2e.yaml --ref <pr-head-ref>

Workflow run

Full E2E target advisor summary

E2E Target Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E targets

  • e2e-all: The PR changes the canonical E2E workflow machinery in .github/workflows/e2e.yaml, adds shared Docker auth cleanup used by many E2E jobs, and updates workflow-boundary support tests/tools. These changes can affect dispatch, setup, authentication, cleanup, and execution across registry-driven and free-standing E2E jobs, so the full E2E fan-out is required.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref>

Optional E2E targets

  • None.

Relevant changed files

  • .github/scripts/docker-auth-cleanup.sh
  • .github/workflows/e2e.yaml
  • test/e2e-release-gate-workflow.test.ts
  • test/e2e/support/dockerhub-auth-workflow-boundary.test.ts
  • test/e2e/support/e2e-workflow.test.ts
  • test/e2e/support/hermes-secret-boundary-workflow.test.ts
  • test/e2e/support/inference-switch-workflow-boundary.test.ts
  • test/e2e/support/openclaw-slack-workflow-boundary.test.ts
  • test/e2e/support/sandbox-operations-workflow-boundary.test.ts
  • test/e2e/support/tunnel-lifecycle-workflow-boundary.test.ts
  • tools/e2e/inference-switch-workflow-boundary.mts
  • tools/e2e/sandbox-operations-workflow-boundary.mts
  • tools/e2e/workflow-boundary.mts

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor (Nemotron Ultra) — Informational

Merge posture: Informational / low confidence
Primary next action: Resolve or justify PRA-1: PR review advisor unavailable.
Open items: 0 required · 1 warning · 0 suggestions · 1 test follow-up
Top item: PR review advisor unavailable

Action checklist

  • PRA-1 Resolve or justify: PR review advisor unavailable
  • PRA-T1 Add or justify test follow-up: Runtime validation

Findings index

ID Severity Category Location Required action
PRA-1 Resolve/justify correctness Re-run the PR Review Advisor or perform a manual review.
Review findings by urgency: 0 required fixes, 1 item to resolve/justify, 0 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

PRA-1 Resolve/justify — PR review advisor unavailable

  • Location: not file-specific
  • Category: correctness
  • Problem: The automated advisor could not complete: timed out after 900000 ms
  • Impact: Automated review evidence is incomplete, so human review must cover the changed code manually.
  • Recommended action: Re-run the PR Review Advisor or perform a manual review.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the workflow logs and raw advisor artifact for the execution failure.
  • Missing regression test: No regression test recommendation is available because the advisor did not complete.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the workflow logs and raw advisor artifact for the execution failure.
  • Evidence: timed out after 900000 ms

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

  • None.
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Add or identify targeted runtime/integration validation for the changed behavior; do not report external E2E job pass/fail here.. Runtime/sandbox/infrastructure paths need behavioral runtime validation: .github/scripts/docker-auth-cleanup.sh, .github/workflows/e2e.yaml, tools/e2e/inference-switch-workflow-boundary.mts, tools/e2e/sandbox-operations-workflow-boundary.mts, tools/e2e/workflow-boundary.mts.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Blocked

Merge posture: Do not merge until addressed
Primary next action: Fix PRA-2: Docker Hub rate-limit failures still lack infra classification and rerun guidance; then add or justify PRA-T1.
Open items: 1 required · 2 warnings · 0 suggestions · 8 test follow-ups
Since last review: 1 prior item resolved · 3 still apply · 0 new items found

Action checklist

  • PRA-2 Fix: Docker Hub rate-limit failures still lack infra classification and rerun guidance in .github/workflows/e2e.yaml:285
  • PRA-1 Resolve or justify: Source-of-truth review needed: Docker Hub authentication and cleanup workaround for image-consuming E2E jobs
  • PRA-3 Resolve or justify: Docker Hub auth workaround still lacks a complete source-of-truth boundary
  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Runtime validation
  • PRA-T4 Add or justify test follow-up: Acceptance clause
  • PRA-T5 Add or justify test follow-up: Acceptance clause
  • PRA-T6 Add or justify test follow-up: Acceptance clause
  • PRA-T7 Add or justify test follow-up: Acceptance clause
  • PRA-T8 Add or justify test follow-up: Docker Hub authentication and cleanup workaround for image-consuming E2E jobs

Findings index

ID Severity Category Location Required action
PRA-1 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-2 Required acceptance .github/workflows/e2e.yaml:285 Add classification/reporting at the source that observes Docker pull/build failures, or add a trusted workflow/report step that detects Docker Hub rate-limit signatures and emits an infra classification plus actionable rerun/recovery guidance. Keep non-rate-limit Docker and product failures visible rather than retrying or swallowing them broadly.
PRA-3 Resolve/justify architecture Document or encode the source boundary for Docker Hub pull-rate-limit handling, explain why a controlled mirror/pre-pull/report-source fix is not part of this PR, and connect that rationale to the classifier regression tests requested above. Include a removal condition such as moving the builder dependency to a controlled mirror with digest pinning.

🚨 Required before merge

Address these before merging unless a maintainer explicitly overrides the advisor with rationale.

PRA-2 Required — Docker Hub rate-limit failures still lack infra classification and rerun guidance

  • Location: .github/workflows/e2e.yaml:285
  • Category: acceptance
  • Problem: Issue e2e: handle intermittent Docker Hub rate-limit failures #4036 explicitly requires: “If Docker Hub rate limiting still occurs, the job/report clearly classifies it as infra and includes actionable rerun/recovery guidance.” This PR adds guarded authentication and cleanup, but the changed workflow, helper, tests, and workflow-boundary validator do not add a path that recognizes Docker Hub `toomanyrequests` pull/build output and reports it as infra with concrete rerun or recovery guidance.
  • Impact: When authentication is unavailable, exhausted, misconfigured, or insufficient, a Docker Hub rate-limit failure can still surface as raw Docker output or an ordinary E2E failure. That leaves maintainers without the required infra classification and can continue to obscure real product/test failures.
  • Required action: Add classification/reporting at the source that observes Docker pull/build failures, or add a trusted workflow/report step that detects Docker Hub rate-limit signatures and emits an infra classification plus actionable rerun/recovery guidance. Keep non-rate-limit Docker and product failures visible rather than retrying or swallowing them broadly.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Search the changed workflow/test/tool files for `toomanyrequests`, `unauthenticated pull rate limit`, or equivalent Docker Hub rate-limit classifier strings, then inspect the reporting path to confirm it emits infra classification and rerun/recovery guidance.
  • Missing regression test: Add a focused test that feeds Docker pull/build output containing `toomanyrequests: You have reached your unauthenticated pull rate limit` through the classifier/report path and asserts an infra classification plus rerun/recovery text; add a paired negative test where a non-rate-limit Docker failure remains a normal product/test failure.
  • Done when: The required change is committed and verification passes: Search the changed workflow/test/tool files for `toomanyrequests`, `unauthenticated pull rate limit`, or equivalent Docker Hub rate-limit classifier strings, then inspect the reporting path to confirm it emits infra classification and rerun/recovery guidance.
  • Evidence: The linked issue’s acceptance criteria include the quoted classifier/rerun clause. Grep of `.github/workflows/e2e.yaml`, `test/e2e/support/dockerhub-auth-workflow-boundary.test.ts`, and `tools/e2e/workflow-boundary.mts` found Docker Hub auth/login checks but no `toomanyrequests` or Docker Hub pull-rate-limit classifier implementation.
Review findings by urgency: 1 required fix, 2 items to resolve/justify, 0 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

PRA-1 Resolve/justify — Source-of-truth review needed: Docker Hub authentication and cleanup workaround for image-consuming E2E jobs

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Auth/cleanup coverage exists, but there is no residual `toomanyrequests` classifier regression at the Docker pull/build failure reporting boundary.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: Changed tests cover guarded login and cleanup safety, but the changed files do not include a Docker Hub rate-limit classifier, source-fix rationale, or removal condition.

PRA-3 Resolve/justify — Docker Hub auth workaround still lacks a complete source-of-truth boundary

  • Location: not file-specific
  • Category: architecture
  • Problem: The PR implements a workflow-wide guarded Docker Hub login and cleanup workaround for anonymous pull rate limits, but the remaining failure/reporting source and removal condition are not encoded or documented. The invalid state is anonymous or insufficiently authenticated Docker Hub pulls for E2E image builds; the source boundary is the image-consuming E2E Docker pull/build paths.
  • Impact: Without an owned reporting boundary and removal condition, future workflow changes can duplicate or regress Docker Hub rate-limit handling. The workaround may prevent the common case while leaving the actual residual failure signal unowned.
  • Recommended action: Document or encode the source boundary for Docker Hub pull-rate-limit handling, explain why a controlled mirror/pre-pull/report-source fix is not part of this PR, and connect that rationale to the classifier regression tests requested above. Include a removal condition such as moving the builder dependency to a controlled mirror with digest pinning.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the changed workflow, helper, and boundary tests for a concrete invalid-state description, source boundary, reason a source fix is out of scope, regression test for residual `toomanyrequests` behavior, and a removal condition.
  • Missing regression test: Existing changed tests cover guarded login, isolated config, retries, and safe deletion; the missing source regression is a classifier test at the actual Docker pull/build failure reporting boundary for residual `toomanyrequests` output.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the changed workflow, helper, and boundary tests for a concrete invalid-state description, source boundary, reason a source fix is out of scope, regression test for residual `toomanyrequests` behavior, and a removal condition.
  • Evidence: `.github/workflows/e2e.yaml` and `.github/scripts/docker-auth-cleanup.sh` implement auth and cleanup, while the changed files contain no rate-limit classifier or stated removal condition for replacing this workaround with a mirror/pre-pull/report-source fix.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

  • None.
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Classify Docker Hub `toomanyrequests` pull/build output as infra and emit actionable rerun/recovery guidance in the job/report path maintainers read.. The changed behavior spans GitHub Actions workflow structure, shell credential cleanup, Docker authentication, and E2E infrastructure reporting. Static and helper-script tests are strong for auth/cleanup, but runtime-facing classification behavior for residual Docker Hub pull failures is still missing.
  • PRA-T2 Runtime validation — Do not classify a non-rate-limit Docker pull/build failure as infra; preserve the normal product/test failure signal.. The changed behavior spans GitHub Actions workflow structure, shell credential cleanup, Docker authentication, and E2E infrastructure reporting. Static and helper-script tests are strong for auth/cleanup, but runtime-facing classification behavior for residual Docker Hub pull failures is still missing.
  • PRA-T3 Runtime validation — If a report-summary step is added, assert the trusted workflow report includes the rate-limit classification and recovery text without exposing Docker Hub credentials.. The changed behavior spans GitHub Actions workflow structure, shell credential cleanup, Docker authentication, and E2E infrastructure reporting. Static and helper-script tests are strong for auth/cleanup, but runtime-facing classification behavior for residual Docker Hub pull failures is still missing.
  • PRA-T4 Acceptance clause — Nightly E2E no longer commonly fails from anonymous Docker Hub pull limits on `node:22-trixie-slim`. — add test evidence or identify existing coverage. The workflow now adds a shared `Authenticate to Docker Hub` step to the live matrix and image-consuming free-standing E2E jobs, with trusted-main scheduled/manual runs requiring Docker Hub credentials. Static review can verify the prevention mechanism and broad coverage, but the “no longer commonly fails” outcome is runtime infrastructure behavior.
  • PRA-T5 Acceptance clause — If Docker Hub rate limiting still occurs, the job/report clearly classifies it as infra and includes actionable rerun/recovery guidance. — add test evidence or identify existing coverage. No changed workflow, helper, or boundary-test path recognizes `toomanyrequests` or equivalent Docker Hub pull-rate-limit output and emits infra classification plus rerun/recovery guidance.
  • PRA-T6 Acceptance clause — The solution applies to all sandbox-build-heavy nightly jobs, not just one script. — add test evidence or identify existing coverage. `validateDockerHubAuthBoundary` classifies `live` plus E2E jobs except named no-image exemptions, and tests assert broad image-job coverage, missing auth/cleanup rejection, and future E2E jobs requiring auth/cleanup by default. Static review cannot prove every semantically sandbox-build-heavy job is correctly categorized without runtime inventory confirmation.
  • PRA-T7 Acceptance clause — Real product/test failures remain visible and are not swallowed by broad retries. — add test evidence or identify existing coverage. The added retry loop is bounded to `docker login` and trusted authentication fails closed rather than wrapping product tests in broad retries. The remaining classifier work still needs a negative regression test to prove non-rate-limit Docker/product failures are not reclassified or hidden.
  • PRA-T8 Docker Hub authentication and cleanup workaround for image-consuming E2E jobs — Auth/cleanup coverage exists, but there is no residual `toomanyrequests` classifier regression at the Docker pull/build failure reporting boundary.. Changed tests cover guarded login and cleanup safety, but the changed files do not include a Docker Hub rate-limit classifier, source-fix rationale, or removal condition.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Resolve/justify — Source-of-truth review needed: Docker Hub authentication and cleanup workaround for image-consuming E2E jobs

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Auth/cleanup coverage exists, but there is no residual `toomanyrequests` classifier regression at the Docker pull/build failure reporting boundary.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: Changed tests cover guarded login and cleanup safety, but the changed files do not include a Docker Hub rate-limit classifier, source-fix rationale, or removal condition.

PRA-2 Required — Docker Hub rate-limit failures still lack infra classification and rerun guidance

  • Location: .github/workflows/e2e.yaml:285
  • Category: acceptance
  • Problem: Issue e2e: handle intermittent Docker Hub rate-limit failures #4036 explicitly requires: “If Docker Hub rate limiting still occurs, the job/report clearly classifies it as infra and includes actionable rerun/recovery guidance.” This PR adds guarded authentication and cleanup, but the changed workflow, helper, tests, and workflow-boundary validator do not add a path that recognizes Docker Hub `toomanyrequests` pull/build output and reports it as infra with concrete rerun or recovery guidance.
  • Impact: When authentication is unavailable, exhausted, misconfigured, or insufficient, a Docker Hub rate-limit failure can still surface as raw Docker output or an ordinary E2E failure. That leaves maintainers without the required infra classification and can continue to obscure real product/test failures.
  • Required action: Add classification/reporting at the source that observes Docker pull/build failures, or add a trusted workflow/report step that detects Docker Hub rate-limit signatures and emits an infra classification plus actionable rerun/recovery guidance. Keep non-rate-limit Docker and product failures visible rather than retrying or swallowing them broadly.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Search the changed workflow/test/tool files for `toomanyrequests`, `unauthenticated pull rate limit`, or equivalent Docker Hub rate-limit classifier strings, then inspect the reporting path to confirm it emits infra classification and rerun/recovery guidance.
  • Missing regression test: Add a focused test that feeds Docker pull/build output containing `toomanyrequests: You have reached your unauthenticated pull rate limit` through the classifier/report path and asserts an infra classification plus rerun/recovery text; add a paired negative test where a non-rate-limit Docker failure remains a normal product/test failure.
  • Done when: The required change is committed and verification passes: Search the changed workflow/test/tool files for `toomanyrequests`, `unauthenticated pull rate limit`, or equivalent Docker Hub rate-limit classifier strings, then inspect the reporting path to confirm it emits infra classification and rerun/recovery guidance.
  • Evidence: The linked issue’s acceptance criteria include the quoted classifier/rerun clause. Grep of `.github/workflows/e2e.yaml`, `test/e2e/support/dockerhub-auth-workflow-boundary.test.ts`, and `tools/e2e/workflow-boundary.mts` found Docker Hub auth/login checks but no `toomanyrequests` or Docker Hub pull-rate-limit classifier implementation.

PRA-3 Resolve/justify — Docker Hub auth workaround still lacks a complete source-of-truth boundary

  • Location: not file-specific
  • Category: architecture
  • Problem: The PR implements a workflow-wide guarded Docker Hub login and cleanup workaround for anonymous pull rate limits, but the remaining failure/reporting source and removal condition are not encoded or documented. The invalid state is anonymous or insufficiently authenticated Docker Hub pulls for E2E image builds; the source boundary is the image-consuming E2E Docker pull/build paths.
  • Impact: Without an owned reporting boundary and removal condition, future workflow changes can duplicate or regress Docker Hub rate-limit handling. The workaround may prevent the common case while leaving the actual residual failure signal unowned.
  • Recommended action: Document or encode the source boundary for Docker Hub pull-rate-limit handling, explain why a controlled mirror/pre-pull/report-source fix is not part of this PR, and connect that rationale to the classifier regression tests requested above. Include a removal condition such as moving the builder dependency to a controlled mirror with digest pinning.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the changed workflow, helper, and boundary tests for a concrete invalid-state description, source boundary, reason a source fix is out of scope, regression test for residual `toomanyrequests` behavior, and a removal condition.
  • Missing regression test: Existing changed tests cover guarded login, isolated config, retries, and safe deletion; the missing source regression is a classifier test at the actual Docker pull/build failure reporting boundary for residual `toomanyrequests` output.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the changed workflow, helper, and boundary tests for a concrete invalid-state description, source boundary, reason a source fix is out of scope, regression test for residual `toomanyrequests` behavior, and a removal condition.
  • Evidence: `.github/workflows/e2e.yaml` and `.github/scripts/docker-auth-cleanup.sh` implement auth and cleanup, while the changed files contain no rate-limit classifier or stated removal condition for replacing this workaround with a mirror/pre-pull/report-source fix.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 81173e88-a869-42ca-b1dd-d8b92fcc6f2a

📥 Commits

Reviewing files that changed from the base of the PR and between c32ea59 and 4ede2f0.

📒 Files selected for processing (2)
  • test/e2e/support/dockerhub-auth-workflow-boundary.test.ts
  • tools/e2e/workflow-boundary.mts
🚧 Files skipped from review as they are similar to previous changes (2)
  • test/e2e/support/dockerhub-auth-workflow-boundary.test.ts
  • tools/e2e/workflow-boundary.mts

📝 Walkthrough

Walkthrough

This PR centralizes Docker Hub authentication and cleanup for E2E workflows, adds a shared cleanup script and shared auth step, and updates workflow boundary validation plus tests to enforce the new canonical shared-auth contract.

Changes

Docker Hub Auth Centralization

Layer / File(s) Summary
Guarded cleanup script
.github/scripts/docker-auth-cleanup.sh
Validates DOCKER_CONFIG, conditionally logs out of docker.io, and removes the isolated credentials directory.
Shared auth step and job wiring
.github/workflows/e2e.yaml
Adds the reusable auth anchor and replaces per-job auth/cleanup logic with the shared step and cleanup script.
Centralized Docker Hub boundary validator
tools/e2e/workflow-boundary.mts
Adds canonical auth/cleanup validators and cross-job enforcement, and wires the validator into E2E boundary checks.
Workflow-specific validator updates
tools/e2e/sandbox-operations-workflow-boundary.mts, tools/e2e/inference-switch-workflow-boundary.mts
Aligns those validators with shared auth ordering and removes mode-specific isolated auth checks.
New Docker Hub boundary test suite
test/e2e/support/dockerhub-auth-workflow-boundary.test.ts
Adds coverage for canonical auth/cleanup behavior, drift detection, and script execution with fake docker binaries.
Existing boundary test updates
test/e2e-release-gate-workflow.test.ts, test/e2e/support/e2e-workflow.test.ts, test/e2e/support/hermes-secret-boundary-workflow.test.ts, test/e2e/support/inference-switch-workflow-boundary.test.ts, test/e2e/support/openclaw-slack-workflow-boundary.test.ts, test/e2e/support/sandbox-operations-workflow-boundary.test.ts, test/e2e/support/tunnel-lifecycle-workflow-boundary.test.ts
Updates existing boundary tests to expect shared guarded Docker Hub auth and cleanup behavior.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant E2EJob
  participant dockerhub-auth
  participant DockerHub
  participant docker-auth-cleanup.sh
  E2EJob->>dockerhub-auth: run after checkout
  dockerhub-auth->>DockerHub: docker login or skip for untrusted refs
  E2EJob->>E2EJob: run build and test steps
  E2EJob->>docker-auth-cleanup.sh: bash .github/scripts/docker-auth-cleanup.sh
  docker-auth-cleanup.sh->>DockerHub: docker logout docker.io
Loading

Suggested labels: area: e2e, bug-fix
Suggested reviewers: jyaunches

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR covers broad authenticated pulls, guarded secret exposure, retries, and cleanup, but it does not show infra-rate-limit classification or rerun guidance from #4036. Add rate-limit detection/reporting that classifies remaining Docker Hub failures as infra and surfaces actionable rerun/recovery guidance.
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: restoring Docker Hub auth coverage for E2E jobs.
Out of Scope Changes check ✅ Passed The workflow, cleanup, and boundary-test changes all support the Docker Hub auth coverage objective and do not show unrelated scope creep.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-e2e-dockerhub-auth

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/e2e/workflow-boundary.mts`:
- Around line 2578-2595: The step validation in workflow-boundary.mts still
allows a later step to bypass the isolated Docker auth config by setting
env.DOCKER_CONFIG, even though job-level checks exist. Update the per-step loop
that builds stepName/stepEnv and runs the auth/cleanup checks to also reject any
step-level DOCKER_CONFIG override outside the auth step, using the existing
error collection pattern alongside requireEnvDoesNotExposeSecret and the
runScript validation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fc0192ee-1e55-4367-8416-ca576a4ad4c9

📥 Commits

Reviewing files that changed from the base of the PR and between e4b9111 and c32ea59.

📒 Files selected for processing (13)
  • .github/scripts/docker-auth-cleanup.sh
  • .github/workflows/e2e.yaml
  • test/e2e-release-gate-workflow.test.ts
  • test/e2e/support/dockerhub-auth-workflow-boundary.test.ts
  • test/e2e/support/e2e-workflow.test.ts
  • test/e2e/support/hermes-secret-boundary-workflow.test.ts
  • test/e2e/support/inference-switch-workflow-boundary.test.ts
  • test/e2e/support/openclaw-slack-workflow-boundary.test.ts
  • test/e2e/support/sandbox-operations-workflow-boundary.test.ts
  • test/e2e/support/tunnel-lifecycle-workflow-boundary.test.ts
  • tools/e2e/inference-switch-workflow-boundary.mts
  • tools/e2e/sandbox-operations-workflow-boundary.mts
  • tools/e2e/workflow-boundary.mts
💤 Files with no reviewable changes (2)
  • test/e2e/support/openclaw-slack-workflow-boundary.test.ts
  • test/e2e/support/tunnel-lifecycle-workflow-boundary.test.ts

Comment thread tools/e2e/workflow-boundary.mts
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv
cv merged commit 711cf94 into main Jul 1, 2026
45 checks passed
@cv
cv deleted the codex/fix-e2e-dockerhub-auth branch July 1, 2026 07:00
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary
<!-- 1-3 sentences: what this PR does and why. -->

Restores Docker Hub authentication across every image-consuming
scheduled/manual-main E2E job after the legacy nightly workflow
retirement left 27 jobs anonymous. Trusted main runs now use one guarded
login contract, while untrusted refs retain isolated anonymous Docker
configs.

## Related Issue
<!-- Fixes #NNN or Closes #NNN. Remove this section if none. -->

Fixes NVIDIA#4036

## Changes
<!-- Bullet list of key changes. -->

- Apply the canonical guarded login to all 67 image-consuming E2E jobs,
with six no-image jobs kept as explicit exemptions.
- Add hardened isolated cleanup, a trusted-login marker, bounded
retries, and fail-closed behavior for trusted runs.
- Centralize regression validation and behavior tests so future E2E jobs
cannot silently omit authentication.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check all that apply. For any "covered by existing tests", "not
applicable", or waiver entry, add a brief justification on the same line
or in the Changes section. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: Internal CI workflow/auth
boundary only; no user-facing behavior or documentation surface changes.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Credential scope and
cleanup boundaries were reviewed locally; guards restrict secrets to
NVIDIA/NemoClaw main schedule/manual runs, and focused adversarial tests
cover traversal and symlink attacks.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
<!-- Check each item you ran and confirmed. Leave unchecked items you
skipped. Doc-only changes do not require npm test unless you ran it. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Git hooks passed during commit and push, or `npx prek run
--from-ref main --to-ref HEAD` passes
- [x] Targeted tests pass for changed behavior
- [x] Full `npm test` passes (broad runtime changes only)
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Centralized Docker Hub authentication and cleanup across E2E workflows
for more consistent CI behavior.
* Standardized guarded login and cleanup so image jobs follow the same
audited process.
* **Bug Fixes**
* Reduced the risk of leftover Docker credentials by cleaning up auth
state reliably after runs.
* Improved handling when authentication is skipped or fails, preventing
unsafe credential persistence.
* **Tests**
* Expanded workflow boundary coverage to enforce the canonical
auth/cleanup steps, ordering, and secret-safety rules.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@wscurran wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure area: security Security controls, permissions, secrets, or hardening bug-fix PR fixes a bug or regression platform: container Affects Docker, containerd, Podman, or images labels Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure area: security Security controls, permissions, secrets, or hardening bug-fix PR fixes a bug or regression platform: container Affects Docker, containerd, Podman, or images

Projects

None yet

Development

Successfully merging this pull request may close these issues.

e2e: handle intermittent Docker Hub rate-limit failures

2 participants