Skip to content

fix(ci): support mode-sharded E2E evidence - #6944

Merged
ericksoa merged 4 commits into
mainfrom
fix/pr-e2e-matrix-shard-contract
Jul 15, 2026
Merged

fix(ci): support mode-sharded E2E evidence#6944
ericksoa merged 4 commits into
mainfrom
fix/pr-e2e-matrix-shard-contract

Conversation

@ericksoa

@ericksoa ericksoa commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

The trusted PR E2E controller rejects inference-switch jobs because their fixed include matrix shards by mode rather than agent. This change binds matrix-include evidence to the explicit NEMOCLAW_E2E_SHARD field and makes the Hermes and OpenClaw inference-switch jobs emit distinct hosted and anthropic evidence. It unblocks the credential-bearing E2E gate for PR 6898.

Changes

  • Derive a fixed include matrix's shard key from a strict ${{ matrix.<field> }} expression, retaining the existing agent fallback and all safe-identifier/uniqueness checks.
  • Set NEMOCLAW_E2E_SHARD from matrix.mode for both inference-switch jobs.
  • Extend the controller test and inference-switch workflow boundary to pin the two expected mode shards.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: internal CI evidence plumbing only; no user-facing or contributor-facing behavior changes.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: PR advisor, CodeRabbit, and maintainer review requested on this exact head.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — test/pr-e2e-gate.test.ts: 27 passed; complete e2e-support project: 1,022 passed and 2 skipped
  • Applicable broad gate passed — repository-sharded PR CI is the authoritative broad gate; local all-project execution was not used as passing evidence
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Aaron Erickson aerickson@nvidia.com

Summary by CodeRabbit

  • Bug Fixes
    • Improved end-to-end validation for inference-switch workflows by wiring shard selection from the workflow matrix mode.
    • Strengthened PR shard-policy parsing to derive shard IDs from configurable environment-to-matrix selectors with stricter failure-closed validation.
  • Tests
    • Expanded PR E2E shard policy coverage to include both inference-switch job variants.
    • Added security-focused E2E tests covering malformed shard selectors and invalid matrix values, plus boundary validation for missing/misconfigured shard mappings.
  • Chores
    • Updated CI source-shape test budget to categorize the new security test.

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 863e08ee-8803-481f-a911-ecd4c403bbbc

📥 Commits

Reviewing files that changed from the base of the PR and between dc4332f and f8165ec.

📒 Files selected for processing (1)
  • test/e2e/support/inference-switch-workflow-boundary.test.ts

📝 Walkthrough

Walkthrough

The E2E workflows expose inference-switch matrix modes through NEMOCLAW_E2E_SHARD. Boundary validation enforces this mapping, shard policy derivation uses the configured matrix field, and tests cover valid and malformed shard selectors.

Changes

Inference switch sharding

Layer / File(s) Summary
Workflow shard environment contract
.github/workflows/e2e.yaml, tools/e2e/inference-switch-workflow-boundary.mts, test/e2e/support/inference-switch-workflow-boundary.test.ts
Both inference-switch jobs map NEMOCLAW_E2E_SHARD to ${{ matrix.mode }}, and boundary validation plus tests require that mapping.
Dynamic shard policy derivation
tools/e2e/pr-e2e-gate.mts, test/pr-e2e-gate.test.ts
expectedSignalShards() reads shard IDs from the matrix field referenced by NEMOCLAW_E2E_SHARD; tests expect ["hosted", "anthropic"] for both jobs.
Malformed selector coverage
test/pr-e2e-gate-shards.test.ts, ci/source-shape-test-budget.json
Tests verify rejection of malformed, missing, and non-string configured shard selectors, with the security test recorded as a source-shape exception.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested labels: bug-fix, area: ci, area: e2e, area: security

Suggested reviewers: cv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: adding mode-based sharding support for E2E evidence in CI.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/pr-e2e-matrix-shard-contract

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage remains at 96%, unchanged from the main branch.

TypeScript / code-coverage/cli

The overall coverage in the fix/pr-e2e-matrix-sh... branch remains at 79%, unchanged from the main branch.

Show a code coverage summary of the most impacted files.
File main 6ef2df9 fix/pr-e2e-matrix-sh... f8165ec +/-
src/lib/messagi...ate-resolver.ts 89% 67% -22%
src/lib/messagi...ate-resolver.ts 94% 82% -12%
src/lib/onboard...reachability.ts 72% 63% -9%
src/lib/messagi...parser-utils.ts 95% 86% -9%
src/lib/messagi...onfig-parser.ts 100% 93% -7%
src/lib/messagi...ate-resolver.ts 100% 95% -5%
src/lib/messagi...onfig-parser.ts 97% 93% -4%
src/lib/adapter.../docker/pull.ts 86% 83% -3%
src/lib/adapter...ocker/volume.ts 70% 85% +15%
src/lib/messagi...onfig-parser.ts 77% 92% +15%

Updated July 15, 2026 15:29 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized E2E selections match; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, inference-routing, network-policy

2 optional E2E recommendations
  • hermes-inference-switch
  • openclaw-inference-switch

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@ericksoa

Copy link
Copy Markdown
Contributor Author

Addressed PRA-1 on exact head 2c8a5179c29f1742ffcebae36faaa79864a65a5e: added focused negative coverage for a malformed NEMOCLAW_E2E_SHARD expression, a missing selected matrix field, and a non-string selected field. All cases assert fail-closed controller errors; focused controller tests are 28/28, and the repo source-shape, test-size, title, typecheck, pre-commit, commitlint, and pre-push gates pass locally.

@ericksoa

Copy link
Copy Markdown
Contributor Author

Addressed the current PRA-1 on exact head f8165ec8528e3f7ef106228023a3b49132fcbc96. test/e2e/support/inference-switch-workflow-boundary.test.ts now removes the Hermes shard mapping and hardcodes the OpenClaw mapping, asserting both validators reject the drift. The focused workflow-boundary suite is 6/6, and all diff-scoped pre-commit, commitlint, and pre-push checks pass against current main.

@wscurran wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure bug-fix PR fixes a bug or regression integration: hermes Hermes integration behavior integration: openclaw OpenClaw integration behavior labels Jul 15, 2026
@ericksoa
ericksoa merged commit 63617df into main Jul 15, 2026
81 of 82 checks passed
@ericksoa
ericksoa deleted the fix/pr-e2e-matrix-shard-contract branch July 15, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure bug-fix PR fixes a bug or regression integration: hermes Hermes integration behavior integration: openclaw OpenClaw integration behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants