Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/review-fabric-policy.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"schema": "cmux.review-fabric-policy/v1",
"minimum_independent_runs": 2,
"quorum_roles": ["reviewer"],
"required_capability_classes": {
"frontier": 1
},
"blocking_run_dispositions": ["hold", "execute", "reject"],
"satisfied_finding_dispositions": [
"fixed",
"declined_with_rationale",
"not_actionable"
],
"blocking_severities": ["P0", "P1"]
}
96 changes: 96 additions & 0 deletions .github/review-fabric.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# Review fabric receipts

The review fabric owns review state independently of any one provider.

A receipt is bound to one exact pull-request head and records independent review runs plus the findings they published. `.github/scripts/review_fabric.py` evaluates those receipts against `.github/review-fabric-policy.json`.

## Identity

Reviewer independence is keyed by `session_id`, not GitHub login, provider, or model name. One session invoking multiple models still contributes one quorum vote.

Every run records:

- worker/session provenance;
- provider, harness, model, and capability class;
- role;
- exact head SHA;
- rules version;
- status and disposition;
- optional evidence class.

Old-head runs remain auditable but never count toward a current-head quorum.

## Findings

Published actionable findings must have an accepted disposition and a reply after the latest reviewer message. A `declined_with_rationale` finding must include a rationale.

The schema can also retain truthful nonterminal/audit states such as `answered_unverified`, `resolved_unverified`, `resolved_unanswered`, `outdated`, and `unavailable`. Those states remain visible without pretending that a reply or GitHub thread resolution proved the finding was fixed.

The v1 evidence classes follow the Fieldwork distinction:

- `source-read`
- `model-executed`
- `target-test-prepared`
- `target-executed`
- `integration-executed`
- `full-gate`

Synthesis must not silently upgrade one evidence class into another. A finding may use severity `unknown` when a provider does not expose enough structured evidence to classify it without guessing.

## Default policy

The repository policy currently requires:

- two independent completed `reviewer` sessions;
- at least one `frontier` capability-class session;
- no completed `hold`, `execute`, or `reject` run;
- all published actionable findings disposed and answered;
- complete capture.

The policy intentionally names no providers or models. Greptile, CodeRabbit, Claude Code, Codex, local GPU agents, or future reviewers can all produce the same receipt shape.

## CLI

Evaluate a receipt from a file:

```sh
python3 .github/scripts/review_fabric.py \
--input /path/to/receipt.json \
--json
```

Or pipe the receipt on stdin:

```sh
cat receipt.json | python3 .github/scripts/review_fabric.py
```

Exit status is zero only when the policy passes.

## Next integration

This is the provider-neutral contract layer. Listener/dispatcher work should create these receipts from exact-head review runs. The existing agent PR obligation ledger can then consume the evaluation instead of treating any named review provider as the source of truth.


## GitHub review adapter

`.github/scripts/github_review_receipt.py` converts the existing trusted GitHub review ledger into the same provider-neutral receipt format.

The adapter deliberately stays conservative:

- structured GitHub review submissions become `external` reviewer runs;
- review commit identities remain exact, so an old review never becomes current-head coverage;
- Greptile's mutable summary comment may provide an exact reviewed head through the trusted existing parser;
- active inline findings remain `pending`, `answered_unverified`, `resolved_unverified`, or `resolved_unanswered` according to the existing ledger;
- provider findings use severity `unknown` unless a later verifier supplies stronger evidence;
- a current inline thread without a structured review provenance makes capture incomplete instead of fabricating a review run.

External review lanes are useful evidence but cannot satisfy the default fabric policy alone: the repository still requires two independent sessions and at least one frontier-class lane.

The adapter can emit a receipt in a trusted GitHub Actions context with:

```sh
python3 .github/scripts/github_review_receipt.py > github-review-receipt.json
```

It reuses the existing agent PR gate's GitHub collector and classification rules; it does not create a second source of truth for review-thread semantics.
223 changes: 223 additions & 0 deletions .github/scripts/github_review_receipt.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,223 @@
#!/usr/bin/env python3
"""Convert GitHub review-bot data into provider-neutral review-fabric receipts."""
from __future__ import annotations

import datetime as dt
import hashlib
import importlib.util
import json
import os
import sys
from pathlib import Path
from typing import Any

SCRIPT_DIR = Path(__file__).resolve().parent
if str(SCRIPT_DIR) not in sys.path:
sys.path.insert(0, str(SCRIPT_DIR))

import review_fabric

DEFAULT_RULES_VERSION = "github-review-adapter-v1"


def load_gate() -> Any:
path = SCRIPT_DIR / "agent-pr-review-gate.py"
spec = importlib.util.spec_from_file_location("agent_pr_review_gate_for_fabric", path)
if spec is None or spec.loader is None:
raise RuntimeError("unable to load agent PR review gate")
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module


def parse_time(value: str | None) -> dt.datetime:
if not value:
return dt.datetime.min.replace(tzinfo=dt.timezone.utc)
return dt.datetime.fromisoformat(value.replace("Z", "+00:00"))


def stable_id(prefix: str, *parts: str) -> str:
digest = hashlib.sha256("\x00".join(parts).encode()).hexdigest()[:20]
return f"{prefix}:{digest}"


def _review_run(
*,
bot: str,
head: str,
submitted_at: str,
state: str,
rules_version: str,
harness: str = "github-review",
) -> dict[str, Any]:
run_id = stable_id("github-review", bot, head, submitted_at, harness)
session_id = stable_id("github-session", bot, head, submitted_at, harness)
disposition = "repair" if state == "CHANGES_REQUESTED" else "accept"
return {
"id": run_id,
"worker_id": bot,
"session_id": session_id,
"provider": bot,
"harness": harness,
"model": "external-opaque",
"role": "reviewer",
"capability_class": "external",
"head_sha": head,
"status": "completed",
"disposition": disposition,
"rules_version": rules_version,
"evidence_class": "source-read",
"submitted_at": submitted_at,
}


def _finding_disposition(item: Any) -> str:
mapping = {
"pending_reply": "pending",
"answered_unverified": "answered_unverified",
"resolved_unverified": "resolved_unverified",
"resolved_unanswered": "resolved_unanswered",
"outdated": "outdated",
"informational": "not_actionable",
"unavailable": "unavailable",
}
return mapping.get(str(item.disposition), "pending")


def receipt_from_pr(
pr: dict[str, Any],
*,
bots: tuple[str, ...],
reply_actors: tuple[str, ...],
gate: Any,
rules_version: str = DEFAULT_RULES_VERSION,
) -> dict[str, Any]:
head = str(pr.get("headRefOid") or "")
if len(head) != 40:
raise ValueError("pull request headRefOid must be a full commit SHA")

runs: list[dict[str, Any]] = []
review_records: list[tuple[str, dt.datetime, dict[str, Any]]] = []
by_run_id: dict[str, dict[str, Any]] = {}

for review in (pr.get("reviews") or {}).get("nodes") or []:
state = str(review.get("state") or "")
if state not in {"COMMENTED", "APPROVED", "CHANGES_REQUESTED"}:
continue
bot = gate.canonical_bot(gate.login(review.get("author")), bots)
review_head = str((review.get("commit") or {}).get("oid") or "")
if bot is None or len(review_head) != 40:
continue
submitted_at = str(review.get("submittedAt") or "")
run = _review_run(
bot=bot,
head=review_head,
submitted_at=submitted_at,
state=state,
rules_version=rules_version,
)
if run["id"] not in by_run_id:
runs.append(run)
by_run_id[run["id"]] = run
review_records.append((bot, parse_time(submitted_at), run))

# Greptile reuses one summary comment across re-reviews. The trusted gate
# already extracts its exact "Last reviewed commit" identity.
if "greptile-apps" in bots:
summary_head = gate.greptile_summary_head(pr)
if summary_head:
run = _review_run(
bot="greptile-apps",
head=summary_head,
submitted_at=f"summary:{summary_head}",
state="COMMENTED",
rules_version=rules_version,
harness="github-summary",
)
if run["id"] not in by_run_id:
runs.append(run)
by_run_id[run["id"]] = run
review_records.append(
("greptile-apps", dt.datetime.max.replace(tzinfo=dt.timezone.utc), run)
)

review_records.sort(key=lambda row: row[1])
capture_complete = pr.get("captureComplete", True) is not False
findings: list[dict[str, Any]] = []

for item in gate.review_ledger(pr, bots, reply_actors):
candidates = [
(submitted_at, run)
for bot, submitted_at, run in review_records
if bot == item.bot and submitted_at <= parse_time(item.latest_bot_comment_at)
]
if not candidates:
# Do not fabricate a current-head review session merely because an
# inline thread exists.
capture_complete = False
continue

_, source_run = max(candidates, key=lambda row: row[0])
disposition = _finding_disposition(item)
finding = {
"id": f"github-thread:{item.thread_id}",
"run_id": source_run["id"],
"head_sha": head,
"severity": "unknown",
"disposition": disposition,
"actionable": bool(item.active),
"published": True,
"verified": False,
"latest_reviewer_at": item.latest_bot_comment_at,
"latest_reply_at": item.latest_reply_at,
"path": item.path,
"line": item.line,
"reply_actor": item.reply_actor,
"source_disposition": item.disposition,
}
findings.append(finding)

if item.active:
source_run["disposition"] = "repair"

return {
"schema": review_fabric.RECEIPT_SCHEMA,
"head_sha": head,
"capture_complete": capture_complete,
"source": "github-review-ledger",
"pr_number": pr.get("number"),
"runs": runs,
"findings": findings,
}


def main() -> int:
gate = load_gate()
try:
pr = gate.fetch_pr()
bots = tuple(
bot.strip().lower()
for bot in os.environ.get(
"REVIEW_BOTS",
",".join(gate.DEFAULT_REVIEW_BOTS),
).split(",")
if bot.strip()
)
actors = gate.configured_reply_actors(pr)
receipt = receipt_from_pr(
pr,
bots=bots,
reply_actors=actors,
gate=gate,
rules_version=os.environ.get("REVIEW_RULES_VERSION", DEFAULT_RULES_VERSION),
)
print(json.dumps(receipt, indent=2, sort_keys=True))
return 0
except Exception as error:
print(f"github-review-receipt: ERROR: {error}", file=sys.stderr)
return 1


if __name__ == "__main__":
raise SystemExit(main())
Loading
Loading