fix: bind Noema App slug to review identity by seonghobae · Pull Request #1483 · ContextualWisdomLab/.github

seonghobae · 2026-08-31T05:46:14Z

Summary

pass the pinned GitHub App action app-slug output to Noema as its verified bot actor
retain /user and /installation discovery for PAT/OIDC-compatible credentials
define current-head runtime acceptance evidence

Runtime evidence

The merged independent gate reached the LLM review step twice without OpenCode approval. Both runs failed closed only because installation tokens expose identity through the token-mint action output rather than /user or /installation in this environment.

Verification

Noema/workflow contracts: 102 passed
acceptance still requires a substantive exact-head Noema GitHub review after merge.

Signed-off-by: Seongho Bae <me@seonghobae.me>

coderabbitai · 2026-08-31T05:46:24Z

Warning

Review limit reached

Next included review available in 39 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 023793c8-4600-4c03-bbb6-927fdef33af1

📥 Commits

Reviewing files that changed from the base of the PR and between 71633cd and 8dd4334.

📒 Files selected for processing (8)

.github/workflows/noema-review.yml
docs/doctoring/noema-orchestrator-free-zdr.md
scripts/ci/noema_review_gate.py
scripts/ci/noema_review_handoff.py
tests/test_noema_review_gate.py
tests/test_noema_review_handoff.py
tests/test_repository_branch_coverage_review_schedulers.py
tests/test_required_workflow_queue_contract.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Signed-off-by: Seongho Bae <me@seonghobae.me>

devin-ai-integration

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 3 new potential issues.

devin-ai-integration · 2026-08-31T05:58:56Z

+        if len(marker_heads) != 1 or len(body_heads) != 1:
+            continue
+        if marker_heads[0].lower() != head_sha.lower() or body_heads[0].lower() != head_sha.lower():
+            continue


🟡 Rejected reviews block every redispatch

A malformed or duplicate head binding makes noema_review_state reject a review that the publication gate still reuses. Every redispatch then skips publication, so the handoff exhausts its polls.

Prompt for agents

Align the review producer's idempotency check with the handoff's new acceptance contract. In scripts/ci/noema_review_gate.py, existing_noema_review currently accepts any same-commit, same-actor terminal review containing the generic marker, including every malformed body rejected by noema_review_state. Make the producer reuse only reviews with exactly one marker head and one displayed head, both matching the current head, or otherwise publish a corrected review. Share the parsing logic if practical, and add an integration-style regression test showing a malformed existing review is replaced after repository_dispatch and then accepted by the handoff.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-08-31T05:58:57Z

+    action_actor = os.environ.get("NOEMA_REVIEW_ACTOR", "").strip()
+    installation_id = os.environ.get("NOEMA_REVIEW_INSTALLATION_ID", "").strip()
+    if action_actor or installation_id:
+        if (
+            os.environ.get("NOEMA_REVIEW_TOKEN_SOURCE") != "noema-review-github-app"
+            or not GITHUB_APP_BOT_RE.fullmatch(action_actor)
+            or not installation_id.isdigit()
+        ):
+            raise RuntimeError("Noema GitHub App identity binding is invalid")
+        return action_actor


📝 Info: Installation binding is syntactic

current_actor checks only that the installation ID is numeric. Identity trust still rests entirely on outputs from the pinned token-mint action.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-08-31T05:58:58Z

+        body = str(review.get("body") or "")
+        marker_heads = NOEMA_MARKER_HEAD_RE.findall(body)
+        body_heads = NOEMA_BODY_HEAD_RE.findall(body)
+        if len(marker_heads) != 1 or len(body_heads) != 1:
+            continue
+        if marker_heads[0].lower() != head_sha.lower() or body_heads[0].lower() != head_sha.lower():
+            continue
        state = str(review.get("state") or "").upper()
        if state in TERMINAL_NOEMA_STATES:
            return state


📝 Info: Marker decision is not cross-checked

noema_review_state validates both head bindings but never compares the marker decision with the GitHub review state. The authenticated review state remains authoritative.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 1 new potential issue.

devin-ai-integration · 2026-08-31T06:00:52Z

existing

+        marker_heads = NOEMA_MARKER_HEAD_RE.findall(body)
+        body_heads = NOEMA_BODY_HEAD_RE.findall(body)
+        if len(marker_heads) != 1 or len(body_heads) != 1:
+            continue


🟡 Valid reviews become unrecoverable

When model text repeats Head SHA, noema_review_state rejects the generated review for duplicate bindings. Later dispatches treat it as existing, so every handoff poll fails.

Prompt for agents

Align review deduplication with runtime receipt acceptance. noema_review_gate.py formats untrusted LLM summary and findings before its canonical Head SHA evidence, so those fields can add another matching Head SHA. noema_review_handoff.py then rejects the review because NOEMA_BODY_HEAD_RE finds multiple matches, while existing_noema_review accepts any current-head marker and prevents a replacement on redispatch. Preserve fail-closed exact-head validation, but make generated canonical evidence unambiguous or ensure rejected receipts can be replaced rather than permanently suppressing review generation. Add an end-to-end regression covering an LLM summary or finding that contains a Head SHA line followed by redispatch.

Was this helpful? React with 👍 or 👎 to provide feedback.

Signed-off-by: Seongho Bae <me@seonghobae.me>

devin-ai-integration

Devin Review found 1 new potential issue.

devin-ai-integration · 2026-08-31T06:02:35Z

+def test_noema_state_skips_newer_nonterminal_review():
+    pending = noema_review("PENDING", HEAD)
+    assert handoff.noema_review_state([noema_review("APPROVED", HEAD), pending], HEAD) == "APPROVED"


📝 Info: Pending reviews preserve prior verdicts

A newer PENDING review remains unsubmitted. The latest authenticated terminal verdict for the same head therefore remains authoritative.

Was this helpful? React with 👍 or 👎 to provide feedback.

Devin's follow-up review of PR #1500 found a real remaining gap: the whole-line MULTILINE anchor on NOEMA_BODY_HEAD_RE narrowed the collision surface from "anywhere in the body" to "any full line before the end marker" — but that still lets LLM-generated summary/findings text spoof a rejection if it happens to contain a complete, correctly-formatted standalone "- Head SHA: `<sha>`" line of its own (plausible in exactly the self-referential scenario that makes the underlying bug likely: Noema discussing or quoting this exact review-body format). A content pattern can never fully close this off, since nothing stops the LLM's own unsanitized text from reproducing any fixed string. So instead of tightening the pattern further, submit_review() now wraps the start of its fixed-format footer in a second, purpose-specific HTML marker (NOEMA_REVIEW_FOOTER_MARKER) that — like the existing closing "noema-review-gate head_sha=..." comment — only submit_review() itself ever emits. noema_review_state() (via the new _isolate_trusted_footer() helper) now isolates the footer by *position*: the span strictly between that footer-start marker and the closing marker, both machine-emitted and never reachable by the LLM's own generated text. Only that trusted span is searched for the "- Head SHA:" bullet, so no amount of LLM text anywhere else in the body — mid-sentence or a complete standalone bullet, a different SHA or the same one — can ever count. The MULTILINE whole-line anchor is kept as defense-in-depth inside that now-trusted span. The dual-binding property #1480/#1483 introduced is unchanged: the marker comment and the human-readable bullet must still independently agree with the live head, exactly as before. Extended the regression tests: the existing mid-sentence-prose case is updated for the new required footer marker, and a new parametrized test (test_noema_state_ignores_standalone_body_head_bullet_before_footer) proves a complete standalone bullet line before the real footer — the exact case Devin flagged, both a different SHA and the same SHA — no longer causes a false rejection. The negative-control tests (missing/stale/duplicate/ conflicting bindings) are extended to exercise those failure modes from inside the new trusted footer span and still correctly reject. Also updates the two other hand-rolled Noema review body fixtures (test_repository_branch_coverage_review_schedulers.py) to include the new footer marker so they keep testing the intended happy path rather than incidentally exercising the new fail-closed missing-marker case. Full suite (2122 tests) passes; 100% coverage and 100% docstrings on scripts/ci/ maintained. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX

docs: define Noema runtime acceptance

3bcdf8b

Signed-off-by: Seongho Bae <me@seonghobae.me>

This comment was marked as resolved.

Sign in to view

fix: bind Noema App slug to review identity

255280d

Signed-off-by: Seongho Bae <me@seonghobae.me>

seonghobae changed the title ~~docs: define Noema runtime acceptance~~ fix: bind Noema App slug to review identity Aug 31, 2026

This comment was marked as resolved.

Sign in to view

seonghobae added 2 commits August 31, 2026 14:56

fix: bind Noema identity and head evidence

851e782

Signed-off-by: Seongho Bae <me@seonghobae.me>

test: require dual-bound Noema receipt

cc74dcf

Signed-off-by: Seongho Bae <me@seonghobae.me>

devin-ai-integration Bot reviewed Aug 31, 2026

View reviewed changes

test: cover nonterminal Noema receipt

8dd4334

Signed-off-by: Seongho Bae <me@seonghobae.me>

devin-ai-integration Bot reviewed Aug 31, 2026

View reviewed changes

seonghobae merged commit e4797ec into main Aug 31, 2026
49 of 52 checks passed

seonghobae deleted the docs/noema-current-head-verification branch August 31, 2026 06:03

This was referenced Aug 31, 2026

fix(strix): break contextual-orchestrator sidecar bootstrap deadlock #1399

Open

fix(noema): stop LLM prose from spoofing the body-side head SHA match #1500

Merged

Navigation Menu

Uh oh!

fix: bind Noema App slug to review identity - #1483

fix: bind Noema App slug to review identity#1483
seonghobae merged 5 commits into
mainfrom
docs/noema-current-head-verification

seonghobae commented Aug 31, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Aug 31, 2026 •

edited

Loading

Review limit reached

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

devin-ai-integration Bot Aug 31, 2026

Uh oh!

devin-ai-integration Bot Aug 31, 2026

Uh oh!

devin-ai-integration Bot Aug 31, 2026

Uh oh!

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

devin-ai-integration Bot Aug 31, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Aug 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

seonghobae commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Runtime evidence

Verification

Uh oh!

coderabbitai Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

seonghobae commented Aug 31, 2026 •

edited

Loading

coderabbitai Bot commented Aug 31, 2026 •

edited

Loading

devin-ai-integration Bot left a comment •

edited

Loading

devin-ai-integration Bot left a comment •

edited

Loading