Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 4 additions & 6 deletions .github/workflows/noema-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,10 @@ on:
concurrency:
group: >-
noema-review-${{
github.event_name == 'pull_request_target' && github.event.pull_request.base.repo.full_name ||
github.event_name == 'repository_dispatch' && github.event.client_payload.target_repository ||
github.repository }}-${{ github.event_name }}-${{
github.event_name == 'pull_request_target' && format('pr-{0}', github.event.pull_request.number) ||
github.event_name == 'workflow_run' && github.event.workflow_run.pull_requests[0].number && format('pr-{0}', github.event.workflow_run.pull_requests[0].number) ||
github.event_name == 'repository_dispatch' && github.event.client_payload.pr_number ||
github.event.pull_request.base.repo.full_name ||
github.event.client_payload.target_repository || github.repository }}-${{
github.event.pull_request.number || github.event.workflow_run.pull_requests[0].number ||
github.event.client_payload.pr_number ||
github.run_id }}
Comment on lines +23 to 27

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Follow-ups cancel required Noema checks

When OpenCode or Strix finishes during a pull-request Noema run, the shared concurrency.group cancels that required run. Its workflow-run replacement cannot restore the cancelled pull-request check.

Prompt for agents
Keep pull_request_target Noema runs in a concurrency group that workflow_run and repository_dispatch follow-ups cannot cancel, because only the pull-request-triggered run supplies the required PR check. Follow-up triggers can still be deduplicated separately, and the close-event run must retain a way to cancel the active pull-request run. Update the queue contract tests to assert both required-check isolation and close-event cancellation.
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

cancel-in-progress: true

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ all five, and auto-optimize routing by cost.

1. **Vendoring, pinned**: `scripts/ci/contextual_orchestrator_review_sidecar.sh`
clones `ContextualWisdomLab/contextual-orchestrator` at an exact SHA
(`c107e3e52371993aa9c326fcc245e01c41fc3850` today) into `RUNNER_TEMP`. The
(`8cd99f139915131ba0239bce12a5d6a5fd85394e` today) into `RUNNER_TEMP`. The
source's `requirements.lock` is installed with `--require-hashes` and
`--no-deps`, so dependency resolution cannot silently move the reviewed
runtime.
Expand Down Expand Up @@ -190,3 +190,10 @@ all five, and auto-optimize routing by cost.
amendment" (above) are closed, without requiring a manual re-audit.
`docs/doctoring/contextual-orchestrator-strix-free-diversity-evidence.md`
records that PR's own reasoning trail.
- **2026-08-31 amendment: Noema reviews independently of OpenCode.** Noema no
longer waits for an OpenCode approval, review-thread state, or other check
conclusions before calling the gateway and submitting its current-head
review. A colliding OpenCode reviewer credential fails closed. The Noema LLM
response must include a non-empty summary and an object-list `findings`
field; `request_changes` additionally requires a substantive finding, so a
bare decision cannot synthesize an evidence-free green review.
2 changes: 1 addition & 1 deletion scripts/ci/contextual_orchestrator_review_sidecar.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# (fail-closed zero-cost) pool.
set -euo pipefail

ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-c107e3e52371993aa9c326fcc245e01c41fc3850}"
ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-8cd99f139915131ba0239bce12a5d6a5fd85394e}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 External pin compatibility unverified

The runtime, ADR, and test pins agree. The vendored commit is absent locally, so its API and lock compatibility still depend on external integration evidence.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified: the pinned commit is the merged contextual-orchestrator #949 SHA 8cd99f139915131ba0239bce12a5d6a5fd85394e. Its exact-head full suite (2830 passed, 1 skipped), Atheris, supply-chain, CodeQL, Semgrep, Trivy and OSV checks passed before merge; this PR's sidecar contracts and full central suite also pass.

ORCHESTRATOR_GIT_URL="${ORCHESTRATOR_GIT_URL:-https://github.com/ContextualWisdomLab/contextual-orchestrator.git}"
# The Strix gate and Noema SSRF guard accept this one process-local origin.
# Keep it fixed so an environment override cannot create an unvalidated sidecar.
Expand Down
133 changes: 24 additions & 109 deletions scripts/ci/noema_review_gate.py
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Obsolete check data remains queried

PR_QUERY still fetches reviewDecision and the status-check rollup after their consumers were removed. Drop them to match the independent-review boundary and reduce GraphQL cost.

(Refers to this code)

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
Expand Up @@ -23,19 +23,6 @@
"opencode-agent[bot]",
"opencode-agent",
}
PRIMARY_REVIEW_MARKERS = (
"OpenCode reviewed the current-head bounded evidence and found no blocking issues.",
"Result: APPROVE",
"opencode-review-control-v1",
)
REVIEW_BODY_HEAD_SHA_RE = re.compile(r"Head SHA:\s*`([0-9a-fA-F]{40})`")
IGNORED_RUNNING_CHECKS = {
"approve-after-primary-review",
"noema-review",
"Required Noema Review",
}
FAILED_CONCLUSIONS = {"FAILURE", "ERROR", "CANCELLED", "TIMED_OUT", "ACTION_REQUIRED", "STARTUP_FAILURE"}
RUNNING_STATES = {"QUEUED", "IN_PROGRESS", "PENDING", "REQUESTED", "WAITING", "EXPECTED"}
MAX_DIFF_CHARS = 60000
MAX_CONTEXT_FILES = 12
MAX_FILE_CONTEXT_CHARS = 4000
Expand Down Expand Up @@ -183,83 +170,6 @@ def review_commit(review: dict[str, Any]) -> str:
return ((review.get("commit") or {}).get("oid") or "").strip()


def review_body_head_sha(review: dict[str, Any]) -> str | None:
"""Return the last explicit current-head SHA recorded in a review body."""
matches = REVIEW_BODY_HEAD_SHA_RE.findall(str(review.get("body") or ""))
return matches[-1] if matches else None


def review_matches_current_head(review: dict[str, Any], head_sha: str) -> bool:
"""Return whether commit and explicit review-body evidence match the live head."""
if not head_sha or review_commit(review) != head_sha:
return False
body_head = review_body_head_sha(review)
return body_head is None or body_head.lower() == head_sha.lower()


def current_primary_approval(pr: dict[str, Any]) -> dict[str, Any] | None:
"""Return the current-head OpenCode approval when it matches the contract."""
head_sha = str(pr.get("headRefOid") or "")
reviews = (((pr.get("reviews") or {}).get("nodes")) or [])
for review in reversed(reviews):
if not review_matches_current_head(review, head_sha):
continue
if str(review.get("state") or "").upper() != "APPROVED":
continue
body = str(review.get("body") or "")
author = review_author(review)
if author in PRIMARY_REVIEW_AUTHORS and any(marker in body for marker in PRIMARY_REVIEW_MARKERS):
return review
return None


def has_current_changes_requested(pr: dict[str, Any]) -> bool:
"""Return whether the current head has any changes-requested review."""
head_sha = str(pr.get("headRefOid") or "")
reviews = (((pr.get("reviews") or {}).get("nodes")) or [])
for review in reversed(reviews):
if review_matches_current_head(review, head_sha) and str(review.get("state") or "").upper() == "CHANGES_REQUESTED":
return True
return False


def has_unresolved_threads(pr: dict[str, Any]) -> bool:
"""Return whether any non-outdated review thread is unresolved."""
threads = (((pr.get("reviewThreads") or {}).get("nodes")) or [])
return any(not thread.get("isResolved") and not thread.get("isOutdated") for thread in threads)


def check_label(node: dict[str, Any]) -> str:
"""Return a human-readable label for a status context or check run."""
if node.get("__typename") == "StatusContext":
return str(node.get("context") or "")
workflow = ((((node.get("checkSuite") or {}).get("workflowRun") or {}).get("workflow") or {}).get("name") or "")
name = str(node.get("name") or "")
return f"{workflow} / {name}" if workflow else name


def blocking_checks(pr: dict[str, Any]) -> list[str]:
"""Return check contexts that should block Noema review."""
contexts = ((((pr.get("statusCheckRollup") or {}).get("contexts") or {}).get("nodes")) or [])
blockers: list[str] = []
for node in contexts:
label = check_label(node)
if label in IGNORED_RUNNING_CHECKS or str(node.get("name") or "") in IGNORED_RUNNING_CHECKS:
continue
if node.get("__typename") == "StatusContext":
state = str(node.get("state") or "").upper()
if state not in {"SUCCESS", "NEUTRAL"}:
blockers.append(f"{label}: {state}")
continue
status = str(node.get("status") or "").upper()
conclusion = str(node.get("conclusion") or "").upper()
if conclusion in FAILED_CONCLUSIONS:
blockers.append(f"{label}: {conclusion}")
elif status in RUNNING_STATES and conclusion not in {"SUCCESS", "NEUTRAL", "SKIPPED"}:
blockers.append(f"{label}: {status}")
return blockers


def existing_noema_review(pr: dict[str, Any], actor: str) -> bool:
"""Return whether Noema already reviewed the current head."""
head_sha = str(pr.get("headRefOid") or "")
Expand Down Expand Up @@ -588,6 +498,25 @@ def call_llm(
decision = str(verdict.get("decision") or "").strip().lower()
if decision not in {"approve", "request_changes", "comment"}:
raise RuntimeError(f"Noema LLM returned unsupported decision: {decision!r}")
summary = verdict.get("summary")
if not isinstance(summary, str) or not summary.strip():
raise RuntimeError("Noema LLM response did not contain a substantive summary")
findings = verdict.get("findings")
if not isinstance(findings, list) or any(not isinstance(finding, dict) for finding in findings):
raise RuntimeError("Noema LLM response findings must be a list of objects")
for finding in findings:
if (
finding.get("severity") not in {"high", "medium", "low"}
or not isinstance(finding.get("file"), str)
or not finding["file"].strip()
or type(finding.get("line")) is not int
or finding["line"] <= 0
or not isinstance(finding.get("message"), str)
or not finding["message"].strip()
):
raise RuntimeError("Noema LLM response contained a malformed finding")
if decision == "request_changes" and not findings:
raise RuntimeError("Noema LLM request_changes response did not contain a substantive finding")
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.
return verdict


Expand Down Expand Up @@ -647,36 +576,22 @@ def submit_review(repo: str, number: int, pr: dict[str, Any], actor: str, verdic


def inspect_and_review(repo: str, number: int) -> int:
"""Inspect PR state and submit Noema's LLM review when gates are clean."""
"""Inspect PR state and submit Noema's independent LLM review."""
pr = fetch_pr(repo, number)
actor = current_actor()
if not actor:
raise RuntimeError("Noema reviewer identity could not be verified")
if actor in PRIMARY_REVIEW_AUTHORS:
print(
raise RuntimeError(
f"Current token actor {actor!r} is already a primary review actor; "
"Noema review skipped so GitHub receives an independent reviewer."
"Noema requires an independent reviewer credential."
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.
)
return 0
if pr.get("isDraft"):
print("PR is draft; Noema review skipped.")
return 0
if existing_noema_review(pr, actor):
print("Current head already has a Noema review; nothing to do.")
return 0
if not current_primary_approval(pr):
print("Current head does not have a primary OpenCode approval; Noema review skipped.")
return 0
if has_current_changes_requested(pr):
print("Current head has requested changes; Noema review skipped.")
return 0
if has_unresolved_threads(pr):
print("PR has unresolved review threads; Noema review skipped.")
return 0
blockers = blocking_checks(pr)
if blockers:
print("Blocking checks remain; Noema review skipped:")
for blocker in blockers:
print(f"- {blocker}")
return 0
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.
diff, truncated = fetch_diff(repo, number)
review_context = build_review_context(repo, number, pr)
verdict = call_llm(repo, number, pr, diff, truncated, review_context)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
)

GATEWAY_MODEL = "contextual-orchestrator/orchestrator/free"
ORCH_PIN_SHA = "c107e3e52371993aa9c326fcc245e01c41fc3850"
ORCH_PIN_SHA = "8cd99f139915131ba0239bce12a5d6a5fd85394e"


def _read(path: Path) -> str:
Expand Down
Loading
Loading