Skip to content
Draft
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 19 additions & 11 deletions .github/workflows/hourly-commercialization-loop.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,20 +35,28 @@ jobs:
repair-review-feedback:
needs: inspect-pr-queue
if: ${{ always() }}
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
actions: write
contents: read
issues: write
pull-requests: read
statuses: read
uses: ContextualWisdomLab/.github/.github/workflows/pr-review-fix-scheduler.yml@21397126d708d2d536ccc1d68b0d333653ce9315
with:
target_repository: ContextualWisdomLab/RankWeave
base_branch: main
max_prs: "50"
max_dispatches: "1"
retry_hours: "1"
secrets: inherit
env:
TARGET_REPOSITORY: ContextualWisdomLab/RankWeave
steps:
- name: Keep review repair fail-closed until protected NVIDIA repair is available
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
open_pr_count="$(
gh api "/repos/${TARGET_REPOSITORY}/pulls?state=open&per_page=1" \
--jq 'length'
)"
if [ "$open_pr_count" -eq 0 ]; then
Comment thread
seonghobae marked this conversation as resolved.
echo "No pull request requires review repair."
exit 0
fi
echo "::notice::Review repair remains fail-closed while the protected central NVIDIA NIM scheduler is pending. Existing independent review agents and the central merge scheduler remain unchanged."
Comment thread
seonghobae marked this conversation as resolved.

revalidate-pr-queue:
needs: repair-review-feedback
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,17 @@ All notable changes to rankweave are documented here. The format follows [Keep a

## [Unreleased]

### Fixed
- Replaced an unreachable central review-fix reusable-workflow SHA that caused
scheduled commercialization runs to fail before job creation with a local
read-only, provider-neutral hold job.
- Kept review repair fail-closed until the protected central NVIDIA NIM/OpenCode
scheduler is merged, without falling back to GitHub Models,
`COPILOT_GITHUB_TOKEN`, inherited repair secrets, or mutable central code.
- Preserved hourly PR inspection, exact-policy revalidation, and the existing
NVIDIA NIM product-development stage while preventing a single unavailable
repair engine from disabling the entire loop.
Comment thread
seonghobae marked this conversation as resolved.

## [0.18.0] - 2026-08-05

### Added
Expand Down
50 changes: 50 additions & 0 deletions docs/adr/0006-fail-closed-hourly-repair-bridge.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# ADR 0006: Fail closed when the central repair workflow is unreachable

- **Status:** Accepted
- **Date:** 2026-08-07

## Context

The hourly RankWeave workflow composed central inspection, review repair,
revalidation, and local NVIDIA NIM product development. Its review-repair call
was pinned to a central commit that became unreachable from protected central
history. GitHub rejected each scheduled caller before creating jobs, disabling
the whole loop.

The current protected central repair implementation still uses GitHub Models,
while a reviewed NVIDIA NIM replacement remains outside protected main. Calling
either the orphaned SHA, mutable central `main`, or an unmerged branch would
violate the product's credential and immutable-source boundaries.

## Decision

Keep the two immutable reachable merge-scheduler calls. Replace review repair
with a local read-only hold job until the protected central NVIDIA NIM repair
engine is available at a reachable immutable SHA. The hold job may inspect only
the open-PR count and must not receive mutation, OIDC, provider, or inherited
secret permissions.

## Consequences

- The hourly workflow executes instead of failing during reusable-workflow
resolution.
- PR inspection and revalidation continue each hour.
- Product development can proceed when all governance jobs succeed and the PR
queue is empty.
- Review repair remains unavailable rather than silently routing through an
unapproved provider or mutable control plane.
- Re-enabling repair requires a focused PR that pins the protected central
NVIDIA scheduler and updates tests, operations documentation, and this ADR's
supersession record.

## Diagram

```mermaid
flowchart LR
S[Hourly schedule] --> I[Immutable central inspection]
I --> H[Local read-only repair hold]
H --> R[Immutable central revalidation]
R -->|PR queue empty| N[NVIDIA NIM product development]
R -->|PR open| Q[Ordinary review and checks]
C[Protected central NVIDIA repair] -. future reachable SHA .-> H
```
53 changes: 53 additions & 0 deletions docs/doctoring/hourly-reusable-workflow-reachability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Hourly reusable-workflow reachability incident

- **Date:** 2026-08-07
- **Component:** `.github/workflows/hourly-commercialization-loop.yml`
- **Failure:** scheduled workflow concluded `failure` before GitHub created any
jobs.

## Root cause

RankWeave pinned the central review-fix reusable workflow to commit
`21397126d708d2d536ccc1d68b0d333653ce9315`. That commit later diverged from the
protected central history, so the caller could no longer resolve the reusable
workflow. Recent failed runs contained zero jobs, while the last successful
hourly run used the same RankWeave caller before the central ref became
unreachable.

## Remediation

The local hourly workflow now retains its reachable immutable merge-scheduler
calls and replaces the unavailable repair call with a read-only local hold job.
The bridge checks whether an open PR exists and records the fail-closed repair
state, but it has no write, OIDC, issue, provider, or model credential. It does
not copy the repair engine and does not fall back to GitHub Models.

The central repair call may return only after a protected central NVIDIA
NIM/OpenCode scheduler has merged and RankWeave pins its reachable immutable
commit. The existing independent review workflows and their credentials remain
unchanged.

## Verification

- Contract tests reject any `pr-review-fix-scheduler.yml@...` reference in the
temporary bridge state.
- Contract tests require two immutable merge-scheduler calls.
- Contract tests require the bridge to remain local, read-only, secret-free,
provider-neutral, and bounded.
- Full Python 3.10-3.13 CI, package smoke, Security Scan, and SAST must pass on
the exact PR head before merge.

## Rollback

Restore a central review-repair call only with a protected, reachable, reviewed
commit SHA whose workflow uses NVIDIA NIM/OpenCode and preserves the existing
review-agent credential boundary. Never restore the orphaned SHA or substitute
a mutable branch.

## References

GitHub. (2026). *Reusing workflow configurations*. GitHub Docs.
https://docs.github.com/en/actions/reference/workflows-and-actions/reusing-workflow-configurations

GitHub. (2026). *GITHUB_TOKEN*. GitHub Docs.
https://docs.github.com/en/actions/concepts/security/github_token
40 changes: 31 additions & 9 deletions docs/operations/hourly-commercialization-loop.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,11 @@ Each run performs four jobs in order:
1. **Inspect the PR queue.** Call the central PR review/merge scheduler to
request missing current-head reviews, update eligible behind branches, and
merge or enable auto-merge only when repository policy is satisfied.
2. **Repair review feedback.** Call the central review-fix scheduler with one
dispatch of budget and a one-hour same-head retry interval.
2. **Hold repair fail-closed when the protected repair engine is unavailable.**
Inspect the open-PR queue without a mutation credential. Until the protected
central NVIDIA NIM repair scheduler is merged, do not call an orphaned or
GitHub-Models-backed repair ref; independent review agents and the merge
scheduler continue to operate normally.
3. **Revalidate the PR queue.** Call the merge scheduler again so a repaired or
newly approved current head is reconsidered under the same checks.
4. **Develop the next product gap.** Only when every governance job succeeded,
Expand All @@ -24,17 +27,36 @@ Each run performs four jobs in order:
The reusable workflows are referenced at immutable commits:

- merge/revalidation policy:
`5983b41ace75040c1d81818171ca7d0f3653254e`;
- hourly review-repair policy with called-workflow source bound to
`job.workflow_repository` and `job.workflow_sha`:
`21397126d708d2d536ccc1d68b0d333653ce9315`.
`5983b41ace75040c1d81818171ca7d0f3653254e`.

This prevents a privileged scheduled run from silently changing behavior
because the central `main` branch moved. Updating either central policy
requires an explicit reviewed SHA change in RankWeave.
The former review-repair SHA, `21397126d708d2d536ccc1d68b0d333653ce9315`,
was no longer reachable from the protected central history. GitHub rejected the
caller before creating any jobs, so every scheduled run failed without doing PR
maintenance or product development. RankWeave now uses a local read-only hold
job until the protected central NVIDIA NIM repair scheduler is available. This
keeps the hourly workflow executable without routing repairs through GitHub
Models, a mutable branch, or an unmerged central change.

Updating the central merge policy or re-enabling review repair requires an
explicit reviewed reachable SHA change in RankWeave.

## Product-development trust zones

## Reusable-workflow reachability incident
Comment thread
seonghobae marked this conversation as resolved.

GitHub Actions run `31124811165` and its immediate predecessors failed before
job creation. The caller still pinned the review-fix workflow to commit
`21397126d708d2d536ccc1d68b0d333653ce9315`, which had diverged from the
protected central history. The same caller had last succeeded before that
central ref became unreachable.

The repair is deliberately narrower than copying the central engine into this
repository. The local bridge is read-only and does not invoke a model or mutate
a PR. Once the protected central scheduler provides the reviewed NVIDIA NIM
boundary, RankWeave can replace the bridge with a new immutable reachable SHA.
This preserves the standalone repository, the central MSA control plane, and
the existing independent-review credential system.

The local development job is separated into four trust zones.

### 1. Trusted base verification
Expand Down
57 changes: 41 additions & 16 deletions tests/test_hourly_commercialization_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
PROJECT_ROOT = Path(__file__).resolve().parents[1]
WORKFLOW_PATH = PROJECT_ROOT / ".github/workflows/hourly-commercialization-loop.yml"
MERGE_WORKFLOW_SHA = "5983b41ace75040c1d81818171ca7d0f3653254e"
FIX_WORKFLOW_SHA = "21397126d708d2d536ccc1d68b0d333653ce9315"


def _workflow_text() -> str:
Expand All @@ -24,21 +23,44 @@ def test_commercialization_loop_runs_once_each_hour():
assert "cancel-in-progress: true" in workflow


def test_commercialization_loop_uses_pinned_central_pr_governance():
def test_commercialization_loop_uses_reachable_merge_governance():
workflow = _workflow_text()

merge_reference = (
"ContextualWisdomLab/.github/.github/workflows/"
f"pr-review-merge-scheduler.yml@{MERGE_WORKFLOW_SHA}"
)
fix_reference = (
"ContextualWisdomLab/.github/.github/workflows/"
f"pr-review-fix-scheduler.yml@{FIX_WORKFLOW_SHA}"
)
assert workflow.count(merge_reference) == 2
assert workflow.count(fix_reference) == 1
assert 'retry_hours: "1"' in workflow
assert "secrets: inherit" in workflow
assert "pr-review-fix-scheduler.yml@" not in workflow
assert workflow.count("secrets: inherit") == 2


def test_review_repair_bridge_is_local_read_only_and_provider_neutral():
workflow = _workflow_text()
repair = _job_section(
workflow,
"repair-review-feedback",
"revalidate-pr-queue",
)

assert "runs-on: ubuntu-latest" in repair
assert "contents: read" in repair
assert "pull-requests: read" in repair
for forbidden in (
"actions: write",
"contents: write",
"id-token: write",
"issues: write",
"statuses: read",
"secrets: inherit",
"github-models/",
"STRIX_GITHUB_MODELS_TOKEN",
"COPILOT_GITHUB_TOKEN",
"NVIDIA_NIM_API_KEY",
):
assert forbidden not in repair
assert "protected central NVIDIA NIM scheduler is pending" in repair
assert "/pulls?state=open&per_page=1" in repair


def test_product_development_uses_nvidia_nim_and_fails_closed():
Expand All @@ -47,7 +69,7 @@ def test_product_development_uses_nvidia_nim_and_fails_closed():
assert "NVIDIA_NIM_API_KEY" in workflow
assert "COPILOT_GITHUB_TOKEN" not in workflow
assert "/agents/repos" not in workflow
assert workflow.count("/pulls?state=open&per_page=1") == 3
assert workflow.count("/pulls?state=open&per_page=1") == 4
assert (
"NVIDIA_NIM_API_KEY is not configured; product development remains "
"fail-closed" in workflow
Expand Down Expand Up @@ -209,7 +231,7 @@ def test_queue_and_base_are_checked_before_and_after_token_exchange():
def test_final_queue_and_base_are_rechecked_before_pr_creation():
workflow = _workflow_text()

assert workflow.count("/pulls?state=open&per_page=1") == 3
assert workflow.count("/pulls?state=open&per_page=1") == 4
assert workflow.count("/commits/${BASE_BRANCH}") == 2
assert "The base branch moved during authoring" in workflow
assert "Another pull request acquired the queue" in workflow
Expand Down Expand Up @@ -263,15 +285,18 @@ def test_governance_permissions_are_scoped_per_calling_job():
assert "statuses: read" not in merge_job

for permission in (
"actions: write",
"contents: read",
"issues: write",
"pull-requests: read",
"statuses: read",
):
assert permission in repair
assert "contents: write" not in repair
assert "id-token: write" not in repair
for forbidden_permission in (
"actions: write",
"contents: write",
"id-token: write",
"issues: write",
"statuses: read",
):
assert forbidden_permission not in repair


def test_opencode_binary_and_models_are_pinned():
Expand Down
59 changes: 59 additions & 0 deletions tests/test_hourly_secret_boundaries.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
from pathlib import Path

PROJECT_ROOT = Path(__file__).resolve().parents[1]
WORKFLOW_PATH = PROJECT_ROOT / ".github/workflows/hourly-commercialization-loop.yml"


def _workflow_text() -> str:
return WORKFLOW_PATH.read_text(encoding="utf-8")


def _job_section(workflow: str, job_name: str, next_job_name: str) -> str:
start = workflow.index(f" {job_name}:\n")
end = workflow.index(f" {next_job_name}:\n", start)
return workflow[start:end]


def _step_section(workflow: str, step_name: str, next_step_name: str) -> str:
start = workflow.index(f" - name: {step_name}\n")
end = workflow.index(f" - name: {next_step_name}\n", start)
return workflow[start:end]


def test_merge_governance_does_not_inherit_all_repository_secrets():
workflow = _workflow_text()
inspect = _job_section(workflow, "inspect-pr-queue", "repair-review-feedback")
revalidate = _job_section(
workflow,
"revalidate-pr-queue",
"develop-next-product-gap",
)

# The pinned merge scheduler has same-repository GITHUB_TOKEN authority and
# an OIDC app-token path. RankWeave must not forward every repository secret
# merely to call that reusable governance workflow.
assert "secrets: inherit" not in inspect
assert "secrets: inherit" not in revalidate


def test_nvidia_secret_materialization_follows_the_deterministic_queue_gate():
workflow = _workflow_text()
gate = _step_section(
workflow,
"Determine whether product development may start",
"Check out the current base without persisted credentials",
)
red_authoring = _step_section(
workflow,
"Author one design and failing regression test",
"Verify test-only scope and observe the red state",
)

# An open PR is a deterministic stop and must be decided before any model
# credential is materialized. NVIDIA credentials belong only to the actual
# model-backed authoring step after that decision.
assert "/pulls?state=open&per_page=1" in gate
assert "NVIDIA_NIM_API_KEY" not in gate
assert "NVIDIA_API_KEY" not in gate
assert "NVIDIA_API_KEY: ${{ secrets.NVIDIA_NIM_API_KEY }}" in red_authoring
assert 'if [ -z "${NVIDIA_API_KEY:-}" ]; then' in red_authoring
Loading