Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 31 additions & 16 deletions .github/workflows/hourly-commercialization-loop.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,15 +31,32 @@ jobs:
merge_mode: direct_or_auto
update_branches: true

Comment thread
seonghobae marked this conversation as resolved.
# Review-feedback repair is dispatched by the central, always-current
# rankweave-hourly-review-repair.yml caller in ContextualWisdomLab/.github
# (uses: ./.github/workflows/pr-review-fix-scheduler.yml, a same-repository
# reference). A local repair-review-feedback job here previously called
# that reusable workflow cross-repository at a pinned commit SHA; that
# shape can never satisfy pr-review-fix-scheduler.yml's same-repository
# trusted-source check (github.repository == ContextualWisdomLab/.github),
# so every run failed before any job was scheduled ("workflow file issue",
# zero jobs created) for as long as that hardening has been in place.
repair-review-feedback:
needs: inspect-pr-queue
if: ${{ always() }}
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
pull-requests: read
env:
TARGET_REPOSITORY: ContextualWisdomLab/RankWeave
steps:
- name: Keep review repair fail-closed until protected NVIDIA repair is available
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
open_pr_count="$(
gh api "/repos/${TARGET_REPOSITORY}/pulls?state=open&per_page=1" \
--jq 'length'
)"
if [ "$open_pr_count" -eq 0 ]; then
Comment on lines +50 to +54

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Hold job can flake if the PR-count query returns empty

if [ "$open_pr_count" -eq 0 ] raises "integer expression expected" and fails the job if the gh api call yields an empty string on a transient error. Mirrors the existing gate pattern, so not a new regression, but the read-only hold job can flake on API hiccups.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

echo "No pull request requires review repair."
exit 0
fi
echo "::notice::Review repair remains fail-closed while the protected central NVIDIA NIM scheduler is pending. Existing independent review agents and the central merge scheduler remain unchanged."
Comment thread
seonghobae marked this conversation as resolved.

revalidate-pr-queue:
needs: inspect-pr-queue
if: ${{ always() }}
Comment on lines 60 to 62

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Repair hold runs parallel to revalidation, not before it

repair-review-feedback and revalidate-pr-queue both declare only needs: inspect-pr-queue, so they run concurrently, whereas the operations doc describes repair as step 2 before revalidation step 3. Harmless here because the repair job is a read-only no-op hold, but the sequencing does not match the documented order.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Expand Down Expand Up @@ -93,16 +110,9 @@ jobs:
id: gate
env:
GH_TOKEN: ${{ github.token }}
NVIDIA_API_KEY: ${{ secrets.NVIDIA_NIM_API_KEY }}
run: |
set -euo pipefail

if [ -z "${NVIDIA_API_KEY:-}" ]; then
echo "::warning::NVIDIA_NIM_API_KEY is not configured; product development remains fail-closed."
echo "eligible=false" >>"$GITHUB_OUTPUT"
exit 0
fi

open_pr_count="$(
gh api "/repos/${TARGET_REPOSITORY}/pulls?state=open&per_page=1" \
--jq 'length'
Expand Down Expand Up @@ -231,6 +241,11 @@ jobs:
run: |
set -euo pipefail

if [ -z "${NVIDIA_API_KEY:-}" ]; then
echo "::error::NVIDIA_NIM_API_KEY is not configured; product development remains fail-closed."
exit 1
fi
Comment thread
seonghobae marked this conversation as resolved.

prompt="$(cat <<'PROMPT'
Work only from the trusted files on the checked-out RankWeave main branch.
Do not read GitHub issues, pull requests, external web pages, environment
Expand Down
30 changes: 15 additions & 15 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,27 @@ All notable changes to rankweave are documented here. The format follows [Keep a

## [Unreleased]

### Fixed
- Replaced an unreachable central review-fix reusable-workflow SHA that caused
scheduled commercialization runs to fail before job creation with a local
read-only, provider-neutral hold job.
- Kept review repair fail-closed until the protected central NVIDIA NIM/OpenCode
scheduler is merged, without falling back to GitHub Models,
`COPILOT_GITHUB_TOKEN`, inherited repair secrets, or mutable central code.
- Preserved hourly PR inspection, exact-policy revalidation, and the existing
NVIDIA NIM product-development stage while preventing a single unavailable
repair engine from disabling the entire loop.
Comment thread
seonghobae marked this conversation as resolved.

- Restricted both autonomous OpenCode phases to explicit repository read paths
and removed agent-authored pull-request metadata, preventing workspace-external
reads or generated text from becoming a pull-request title or body.

### Added

- Classic reciprocal-rank fusion results now expose the exact per-channel
Cormack contribution beside each owned input rank, so consumers do not need
to duplicate the fusion arithmetic.

### Fixed
- Restricted both autonomous OpenCode phases to explicit repository read paths
and removed agent-authored pull-request metadata, preventing workspace-external
reads or generated text from becoming a pull-request title or body.
- Removed the `repair-review-feedback` job from
`hourly-commercialization-loop.yml`: it called
`ContextualWisdomLab/.github`'s `pr-review-fix-scheduler.yml`
cross-repository at a pinned commit SHA, a shape that reusable workflow's
same-repository trusted-source hardening can never satisfy
(`github.repository == ContextualWisdomLab/.github`), so every hourly run
failed before any job was scheduled for as long as that hardening has been
in place. Review-feedback repair is now dispatched by a central,
always-current `rankweave-hourly-review-repair.yml` caller added to
`ContextualWisdomLab/.github`, matching the pattern already used by every
other product repository in the organization.
### Changed
- Bumped the pinned `uv` version from `0.11.29` to `0.12.1` in `pyproject.toml`
and every `astral-sh/setup-uv` workflow step (`ci.yml`, `create-release.yml`,
Expand Down
50 changes: 50 additions & 0 deletions docs/adr/0006-fail-closed-hourly-repair-bridge.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# ADR 0006: Fail closed when the central repair workflow is unreachable

- **Status:** Accepted
- **Date:** 2026-08-07

## Context

The hourly RankWeave workflow composed central inspection, review repair,
revalidation, and local NVIDIA NIM product development. Its review-repair call
was pinned to a central commit that became unreachable from protected central
history. GitHub rejected each scheduled caller before creating jobs, disabling
the whole loop.

The current protected central repair implementation still uses GitHub Models,
while a reviewed NVIDIA NIM replacement remains outside protected main. Calling
either the orphaned SHA, mutable central `main`, or an unmerged branch would
violate the product's credential and immutable-source boundaries.

## Decision

Keep the two immutable reachable merge-scheduler calls. Replace review repair
with a local read-only hold job until the protected central NVIDIA NIM repair
engine is available at a reachable immutable SHA. The hold job may inspect only
the open-PR count and must not receive mutation, OIDC, provider, or inherited
secret permissions.

## Consequences

- The hourly workflow executes instead of failing during reusable-workflow
resolution.
- PR inspection and revalidation continue each hour.
- Product development can proceed when all governance jobs succeed and the PR
queue is empty.
- Review repair remains unavailable rather than silently routing through an
unapproved provider or mutable control plane.
- Re-enabling repair requires a focused PR that pins the protected central
NVIDIA scheduler and updates tests, operations documentation, and this ADR's
supersession record.

## Diagram

```mermaid
flowchart LR
S[Hourly schedule] --> I[Immutable central inspection]
I --> H[Local read-only repair hold]
H --> R[Immutable central revalidation]
R -->|PR queue empty| N[NVIDIA NIM product development]
R -->|PR open| Q[Ordinary review and checks]
C[Protected central NVIDIA repair] -. future reachable SHA .-> H
```
53 changes: 53 additions & 0 deletions docs/doctoring/hourly-reusable-workflow-reachability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Hourly reusable-workflow reachability incident

- **Date:** 2026-08-07
- **Component:** `.github/workflows/hourly-commercialization-loop.yml`
- **Failure:** scheduled workflow concluded `failure` before GitHub created any
jobs.

## Root cause

RankWeave pinned the central review-fix reusable workflow to commit
`21397126d708d2d536ccc1d68b0d333653ce9315`. That commit later diverged from the
protected central history, so the caller could no longer resolve the reusable
workflow. Recent failed runs contained zero jobs, while the last successful
hourly run used the same RankWeave caller before the central ref became
unreachable.

## Remediation

The local hourly workflow now retains its reachable immutable merge-scheduler
calls and replaces the unavailable repair call with a read-only local hold job.
The bridge checks whether an open PR exists and records the fail-closed repair
state, but it has no write, OIDC, issue, provider, or model credential. It does
not copy the repair engine and does not fall back to GitHub Models.

The central repair call may return only after a protected central NVIDIA
NIM/OpenCode scheduler has merged and RankWeave pins its reachable immutable
commit. The existing independent review workflows and their credentials remain
unchanged.

## Verification

- Contract tests reject any `pr-review-fix-scheduler.yml@...` reference in the
temporary bridge state.
- Contract tests require two immutable merge-scheduler calls.
- Contract tests require the bridge to remain local, read-only, secret-free,
provider-neutral, and bounded.
- Full Python 3.10-3.13 CI, package smoke, Security Scan, and SAST must pass on
the exact PR head before merge.

## Rollback

Restore a central review-repair call only with a protected, reachable, reviewed
commit SHA whose workflow uses NVIDIA NIM/OpenCode and preserves the existing
review-agent credential boundary. Never restore the orphaned SHA or substitute
a mutable branch.

## References

GitHub. (2026). *Reusing workflow configurations*. GitHub Docs.
https://docs.github.com/en/actions/reference/workflows-and-actions/reusing-workflow-configurations

GitHub. (2026). *GITHUB_TOKEN*. GitHub Docs.
https://docs.github.com/en/actions/concepts/security/github_token
42 changes: 32 additions & 10 deletions docs/operations/hourly-commercialization-loop.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,11 @@ Each run performs four jobs in order:
1. **Inspect the PR queue.** Call the central PR review/merge scheduler to
request missing current-head reviews, update eligible behind branches, and
merge or enable auto-merge only when repository policy is satisfied.
2. **Repair review feedback.** Call the central review-fix scheduler with one
dispatch of budget and a one-hour same-head retry interval.
2. **Hold repair fail-closed when the protected repair engine is unavailable.**
Inspect the open-PR queue without a mutation credential. Until the protected
central NVIDIA NIM repair scheduler is merged, do not call an orphaned or
GitHub-Models-backed repair ref; independent review agents and the merge
scheduler continue to operate normally.
3. **Revalidate the PR queue.** Call the merge scheduler again so a repaired or
newly approved current head is reconsidered under the same checks.
4. **Develop the next product gap.** Only when every governance job succeeded,
Expand All @@ -24,14 +27,33 @@ Each run performs four jobs in order:
The reusable workflows are referenced at immutable commits:

- merge/revalidation policy:
`5983b41ace75040c1d81818171ca7d0f3653254e`;
- hourly review-repair policy with called-workflow source bound to
`job.workflow_repository` and `job.workflow_sha`:
`21397126d708d2d536ccc1d68b0d333653ce9315`.

This prevents a privileged scheduled run from silently changing behavior
because the central `main` branch moved. Updating either central policy
requires an explicit reviewed SHA change in RankWeave.
`5983b41ace75040c1d81818171ca7d0f3653254e`.

The former review-repair SHA, `21397126d708d2d536ccc1d68b0d333653ce9315`,
was no longer reachable from the protected central history. GitHub rejected the
caller before creating any jobs, so every scheduled run failed without doing PR
maintenance or product development. RankWeave now uses a local read-only hold
job until the protected central NVIDIA NIM repair scheduler is available. This
keeps the hourly workflow executable without routing repairs through GitHub
Models, a mutable branch, or an unmerged central change.

Updating the central merge policy or re-enabling review repair requires an
explicit reviewed reachable SHA change in RankWeave.

## Reusable-workflow reachability incident
Comment thread
seonghobae marked this conversation as resolved.

GitHub Actions run `31124811165` and its immediate predecessors failed before
job creation. The caller still pinned the review-fix workflow to commit
`21397126d708d2d536ccc1d68b0d333653ce9315`, which had diverged from the
protected central history. The same caller had last succeeded before that
central ref became unreachable.

The repair is deliberately narrower than copying the central engine into this
repository. The local bridge is read-only and does not invoke a model or mutate
a PR. Once the protected central scheduler provides the reviewed NVIDIA NIM
boundary, RankWeave can replace the bridge with a new immutable reachable SHA.
This preserves the standalone repository, the central MSA control plane, and
the existing independent-review credential system.

## Product-development trust zones

Expand Down
61 changes: 55 additions & 6 deletions tests/test_hourly_commercialization_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,16 @@ def test_commercialization_loop_runs_once_each_hour():
assert "cancel-in-progress: true" in workflow


def test_commercialization_loop_uses_pinned_central_pr_governance():
def test_commercialization_loop_uses_reachable_merge_governance():
workflow = _workflow_text()

merge_reference = (
"ContextualWisdomLab/.github/.github/workflows/"
f"pr-review-merge-scheduler.yml@{MERGE_WORKFLOW_SHA}"
)
assert workflow.count(merge_reference) == 2
assert "pr-review-fix-scheduler.yml@" not in workflow
assert "secrets: inherit" not in workflow
uses_references = [
line.split("uses:", maxsplit=1)[1].strip()
for line in workflow.splitlines()
Expand All @@ -39,7 +41,37 @@ def test_commercialization_loop_uses_pinned_central_pr_governance():
assert all("pr-review-fix-scheduler.yml" not in ref for ref in uses_references)
# Review-feedback repair is dispatched by the central, always-current
# rankweave-hourly-review-repair.yml caller in ContextualWisdomLab/.github
# instead of a cross-repository pinned-SHA job here.
# instead of a cross-repository pinned-SHA job here; a local read-only
# fail-closed hold job below keeps the repair lane visible until the
# protected central NVIDIA NIM scheduler is reachable.


def test_review_repair_bridge_is_local_read_only_and_provider_neutral():
workflow = _workflow_text()
repair = _job_section(
workflow,
"repair-review-feedback",
"revalidate-pr-queue",
)

assert "runs-on: ubuntu-latest" in repair
assert "contents: read" in repair
assert "pull-requests: read" in repair
for forbidden in (
"actions: write",
"contents: write",
"id-token: write",
"issues: write",
"statuses: read",
"secrets: inherit",
"github-models/",
"STRIX_GITHUB_MODELS_TOKEN",
"COPILOT_GITHUB_TOKEN",
"NVIDIA_NIM_API_KEY",
):
assert forbidden not in repair
assert "protected central NVIDIA NIM scheduler is pending" in repair
assert "/pulls?state=open&per_page=1" in repair


def test_product_development_uses_nvidia_nim_and_fails_closed():
Expand All @@ -48,7 +80,7 @@ def test_product_development_uses_nvidia_nim_and_fails_closed():
assert "NVIDIA_NIM_API_KEY" in workflow
assert "COPILOT_GITHUB_TOKEN" not in workflow
assert "/agents/repos" not in workflow
assert workflow.count("/pulls?state=open&per_page=1") == 3
assert workflow.count("/pulls?state=open&per_page=1") == 4
assert (
"NVIDIA_NIM_API_KEY is not configured; product development remains "
"fail-closed" in workflow
Expand All @@ -64,7 +96,7 @@ def test_nvidia_secret_is_step_scoped_and_agent_has_no_github_credential():
job_environment = develop.split(" steps:\n", maxsplit=1)[0]

assert "NVIDIA_API_KEY" not in job_environment
assert workflow.count("NVIDIA_API_KEY: ${{ secrets.NVIDIA_NIM_API_KEY }}") == 3
assert workflow.count("NVIDIA_API_KEY: ${{ secrets.NVIDIA_NIM_API_KEY }}") == 2
assert "persist-credentials: false" in workflow
assert workflow.count("env -u GH_TOKEN -u GITHUB_TOKEN") == 2
assert workflow.count("-u ACTIONS_ID_TOKEN_REQUEST_TOKEN") == 2
Expand Down Expand Up @@ -222,7 +254,7 @@ def test_queue_and_base_are_checked_before_and_after_token_exchange():
def test_final_queue_and_base_are_rechecked_before_pr_creation():
workflow = _workflow_text()

assert workflow.count("/pulls?state=open&per_page=1") == 3
assert workflow.count("/pulls?state=open&per_page=1") == 4
assert workflow.count("/commits/${BASE_BRANCH}") == 2
assert "The base branch moved during authoring" in workflow
assert "Another pull request acquired the queue" in workflow
Expand All @@ -243,7 +275,8 @@ def test_product_development_requires_successful_pr_governance():
def test_governance_permissions_are_scoped_per_calling_job():
workflow = _workflow_text()
workflow_default = workflow.split("concurrency:", maxsplit=1)[0]
inspect = _job_section(workflow, "inspect-pr-queue", "revalidate-pr-queue")
inspect = _job_section(workflow, "inspect-pr-queue", "repair-review-feedback")
repair = _job_section(workflow, "repair-review-feedback", "revalidate-pr-queue")
revalidate = _job_section(
workflow, "revalidate-pr-queue", "develop-next-product-gap"
)
Expand All @@ -269,8 +302,24 @@ def test_governance_permissions_are_scoped_per_calling_job():
assert permission in merge_job
assert "issues: write" not in merge_job
assert "statuses: read" not in merge_job
assert "issues: write" not in merge_job
assert "statuses: read" not in merge_job
assert "secrets: inherit" not in merge_job

for permission in (
"contents: read",
"pull-requests: read",
):
assert permission in repair
for forbidden_permission in (
"actions: write",
"contents: write",
"id-token: write",
"issues: write",
"statuses: read",
):
assert forbidden_permission not in repair


def test_opencode_binary_and_models_are_pinned():
workflow = _workflow_text()
Expand Down
Loading
Loading