From 2e249a983b380d041a20f6ed5bcc9ce980eb07e5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 3 Sep 2026 10:58:22 +0900 Subject: [PATCH 1/9] docs(doctoring): verify item 13's stale-head-cancellation hypothesis is refuted Investigated with a 9-agent workflow (4 independent audits of strix.yml, opencode-review.yml, noema-review.yml, pr-review-merge-scheduler.yml + 1 direct pull of the item's own cited naruon PR#1528 evidence + 4 adversarial re-verification passes, all refuted:false). All four central workflows already reliably cancel a stale prior-head run on a new push, via correctly scoped native concurrency or purpose-built same-file API-driven cancellation jobs. The cited evidence instead shows pure queue starvation on a single unchanged SHA (23h22m queue wait before a Strix run even started; a paired OpenCode Review run still queued 24+ hours later) -- corroborating the already-recorded plan-level concurrency ceiling with a concrete example rather than a cancellation bug. No fix applied; the standing chicken-and-egg bypass-merge authorization was not used since no fixable bug was found -- forcing one would mean inventing a problem the evidence doesn't support. Co-Authored-By: Claude Sonnet 5 --- ...-stale-head-cancellation-audit-20260903.md | 112 ++++++++++++++++++ docs/product-technical-gap-baseline.md | 10 ++ 2 files changed, 122 insertions(+) create mode 100644 docs/doctoring/item13-stale-head-cancellation-audit-20260903.md diff --git a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md new file mode 100644 index 0000000000..977c7bf49b --- /dev/null +++ b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md @@ -0,0 +1,112 @@ +# Doctoring record: backlog item 13's stale-head-cancellation hypothesis is refuted; the real evidence is queue depth itself (2026-09-03) + +- **Date:** 2026-09-03 +- **Subject:** backlog item 13 states "Strix, OpenCode Review, Noema가 Concurrency에 이슈가 없을 것. 한 PR 안에서 + Push가 발생했을 때 이전 HEAD에 관한 Cancel이 발생할 것" (Strix/OpenCode Review/Noema must have no concurrency + issues; a push within a PR must cancel the previous HEAD's run), citing + `ContextualWisdomLab/naruon` run `33581213829` / job `100095712154` / PR `#1528` as evidence. The user + separately directed: if the org's ~60-concurrent-job ceiling (`docs/doctoring/actions-plan-concurrency-ceiling-20260903.md`) + is blocking work, trace and resolve the workflow issues that create it, authorizing bypass-merge for this + specific chicken-and-egg case (a queue-congestion fix that would itself be blocked by queue congestion). + This record is that trace — and its answer is not the one the hypothesis expected. +- **Decision record:** none in `docs/adr/` — this is a verified negative/confirmatory finding for one specific + hypothesis, plus a positive, evidence-strengthening finding for a different, already-recorded root cause. +- **PR:** see the PR that carries this commit. + +## Method + +A 9-agent workflow (4 investigate + 1 direct evidence pull + 4 adversarial verify; `wf_eb15dd2b-ad1`) fetched +`strix.yml`, `opencode-review.yml`, `noema-review.yml`, and `pr-review-merge-scheduler.yml` fresh from +`raw.githubusercontent.com` (not from memory or a prior session's notes), extracted each workflow's exact +`concurrency:` group expression and `cancel-in-progress` value verbatim, searched each file end-to-end for +any supplementary same-file mechanism that cancels a stale prior-head run via the GitHub Actions API, and +reached a verdict on whether a new push to an open PR reliably retires the now-stale run for the previous +head SHA. A separate agent pulled the exact cited evidence (`naruon` run `33581213829`, its job, and PR +`#1528`'s full run history) directly from the GitHub API. Every one of the four workflow findings was then +independently re-verified by a second agent instructed to actively try to refute it — re-fetching the same +file fresh, checking for companion cancellation workflows, per-job (not just workflow-level) concurrency +blocks, and verbatim accuracy of every quoted line — before being accepted. + +## Result 1: item 13's hypothesis is refuted for all four central workflows — verified, not assumed + +| Workflow | Native concurrency scoped by SHA? | Stale-head run gets cancelled? | Mechanism | +|---|---|---|---| +| `strix.yml` | No — group is `strix--` only; `cancel-in-progress: false` (deliberate, to preserve scanner logs) | **Yes** | Separate `cancel-superseded-pr-runs` job, same file, fires on `synchronize`/`closed`, lists active runs via the Actions API, matches by workflow name + PR number + head SHA (via `display_title` and `pull_requests[].head.sha`), and POSTs cancel/force-cancel | +| `opencode-review.yml` | Yes — group includes both PR number and exact head SHA (`opencode-review-bootstrap---`), `cancel-in-progress: true` | **Yes** | The SHA-scoped group means native cancellation never even needs to fire cross-SHA (a design fix for a real prior incident, `#1568`, where SHA-agnostic grouping let a stale run wrongly cancel a *newer* one); a dedicated `cancel-superseded-opencode-review-runs` job plus an in-loop live-head self-retirement check (60s poll) provide defense-in-depth | +| `noema-review.yml` | No — group is `noema-review--` (PR number only); `cancel-in-progress: true` for `synchronize`/`closed` | **Yes** | Native cancellation alone suffices here (same group, cancel-in-progress true), *and* a same-job "Cancel superseded Noema runs after live-head validation" step independently re-verifies and cancels via the API as defense-in-depth | +| `pr-review-merge-scheduler.yml` | No (PR-number only) for the scheduler's own runs; native cancellation handles those | **Yes, for every repo except `.github` itself** | The `org-queue-sweep` job's hourly cross-repo sweep lists every queued/in-progress run of *any* workflow (reaching Strix/OpenCode/Noema runs directly, not just this scheduler's own), classifies by `head_sha` mismatch against the PR's live head, re-validates immediately before acting, and cancels. Explicitly excludes `ContextualWisdomLab/.github` from its target list — this repo's own PRs rely on Strix/OpenCode/Noema's own (separately verified, correct) mechanisms plus a same-head duplicate-run coalescer (`current-head-run-coalescer.yml`), not this sweep | + +All four adversarial verification passes returned `refuted: false` after independently re-fetching the +live files and checking specifically for missed per-job concurrency blocks, companion cancellation +workflows, and misquoted YAML — none were found. One cosmetic inaccuracy was caught and is worth recording +for anyone re-reading `strix.yml`: the investigating agent described a design-rationale comment ("Strix +runs intentionally do not cancel in progress because a pre-job cancellation leaves no scanner log to +review") as adjacent to the `cancel-in-progress: false` line; it is actually ~150 lines earlier, in the +trigger block's `paths-ignore` comment. The design rationale itself is accurate and real — only its +in-file location was misdescribed. This does not change the substantive verdict. + +**Conclusion: there is no stale-head-cancellation bug to fix.** Every one of the four central, +required-workflow-ruleset workflows this org's own PR pipeline depends on already reliably retires a +superseded-head run on a new push, through a combination of correctly-scoped native GitHub concurrency and +purpose-built, independently-verified supplementary cancellation jobs — several of which carry their own +design-rationale comments citing prior incidents (`#1568`) that already taught this exact lesson once. +Forcing a "fix" here on the strength of item 13's hypothesis alone, without this evidence, would have meant +inventing a problem that does not exist — the throttle this session has held all along (do not force a +consolidation, or here a fix, that a real look shows is not actually needed) applies. + +## Result 2: the cited evidence shows a different, real, and more severe problem — pure queue starvation + +The naruon PR `#1528` run history (all 17 recorded runs, pulled live from the GitHub API) shows **zero** +occurrences of two different head SHAs being simultaneously active — every run, across the whole history, +shares the PR's one unchanged head SHA (`cf472cf77fb93325858f485a22e967449d7c387a`). The multi-SHA race +item 13 hypothesized is not what happened here. What actually happened, quoted directly from the API: + +- The cited Strix run (`33581213829`) was **created at `2026-09-02T01:54:46Z` but its job did not start + until `2026-09-03T01:17:10Z`** — a **23-hour-22-minute queue wait** before it even began running, then + ran for ~14 minutes and was cancelled (superseded by this same investigation's live re-check, not by a + bug). +- The paired "Required OpenCode Review" run for the identical SHA (`33581213805`), created at the same + timestamp, **was still `status: queued`, `conclusion: null` when re-checked live on 2026-09-03** — stuck + queued for **24+ hours with no run at all.** +- Six separate "PR Governance" workflow runs fired for this one unchanged SHA (five `pull_request_target` + events, one `pull_request_review`) — naruon's own `pr-governance.yml` watches `labeled`/`unlabeled` + PR-target actions plus completion of four other workflows (`Application CI`, `Bandit Security Scan`, + `Build and Publish Docker Images`, `Strix Security Scan`) plus `check_run: completed`, any of which can + independently re-trigger it. Whether this specific PR's six firings were legitimate repeated + label/status activity or avoidable redundant triggering was **not verified** in this pass — flagged as + an open, unconfirmed lead, not a finding, since asserting it without checking the PR's actual event + history would repeat exactly the kind of unverified claim this session's own norms exist to catch. + +This is the same root cause `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` already +identified (a plan-level concurrent-job ceiling), now corroborated with a concrete, painful, individually +named example instead of aggregate counts: a real open PR's real review evidence sat queued for over a day +— not because anything is misconfigured, but because there was nowhere for it to run sooner. This +strengthens, rather than changes, that record's conclusion and its recommendation (a plan-tier decision or +added runner capacity is the actual fix; workflow-file consolidation reduces total triggered runs at the +margin but cannot lift the ceiling). + +## What this resolves, and what it does not + +- **Resolves:** whether item 13's specific "no cancellation on push" complaint reflects a real + configuration bug in the four central workflows. It does not — verified, not assumed, across all four, + with adversarial re-checking. Item 13 should be marked accordingly in `docs/product-technical-gap-baseline.md`. +- **Does not resolve:** why the org's real capacity is saturated to the point of 23-24+ hour stalls — that + is the plan-level ceiling question `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` already + raises as an org-owner billing/plan decision, now with stronger evidence, not a new answer. +- **Open, unverified lead, not a finding:** whether naruon's `pr-governance.yml` fires more often than + necessary per PR (six runs on one SHA in this one case) is worth a dedicated, evidence-first follow-up + investigation of that PR's actual label/review event history before concluding anything — recorded here + so it is not lost, not asserted as confirmed. +- **Bypass-merge authorization:** the user authorized bypass-merge for this investigation as a genuine + chicken-and-egg case. It is not used here because no fix was found that needed it — this record is itself + a normal docs-only PR, subject to normal review like any other. + +## Audit trail + +- Workflow run `wf_eb15dd2b-ad1` (9 agents: 4 investigate, 1 direct-evidence pull, 4 adversarial verify) — + full per-agent transcripts and the complete unredacted findings/verification JSON live in that run's + journal. +- `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` — the root-cause record this evidence + corroborates. +- `docs/product-technical-gap-baseline.md` — backlog item 13's original text and citation, to be updated + to reference this record's verdict. diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 7888a5e04a..679d003645 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2627,3 +2627,13 @@ Higgins, S. S., Crepalde, N., & Fernandes, L. (2021). Segmented multiplexity: A **900-second clarification.** The historical `NoemaRepairDeadlineExceeded` from the html4tree incident came from the retired caller repair path. The three literal `timeout --kill-after=20 900` invocations still present in `opencode-review-dispatch.yml` are separate containment limits for untrusted test-measurement commands; they are not model or Noema inference timeouts. Telemetry and runbooks must report the command class and phase separately. **Evidence / acceptance.** Permanent tests forbid retry/deadline/sampling symbols in the caller and prove one gateway request, one attempt annotation, control-character-safe telemetry, missing-value rejection, valid trailing-comma normalization, and exact changed-line guidance. Fresh exact-head repository checks and reviews remain the admission authority; predecessor-head evidence is not transferable. The remaining runtime work is to preserve distinct `request_too_large`, discovery, rate-limit, provider transport, malformed-output, stale-head, and sandbox-command-timeout categories in hosted logs. + +## Backlog item 13 (Strix/OpenCode/Noema stale-head cancellation) — verified NOT a bug — 2026-09-03 + +**Status:** Investigated with a 9-agent workflow (4 independent file audits + 1 direct-evidence pull against the item's own cited example + 4 adversarial re-verification passes, all `refuted: false`), per `docs/doctoring/item13-stale-head-cancellation-audit-20260903.md`. Item 13 asks that Strix/OpenCode Review/Noema reliably cancel a PR's previous-head run when a new push supersedes it, citing `naruon` run `33581213829` / PR `#1528` as evidence of a gap. + +**Verdict: the hypothesis is refuted.** All four central required workflows (`strix.yml`, `opencode-review.yml`, `noema-review.yml`, `pr-review-merge-scheduler.yml`) already reliably retire a stale prior-head run on a new push — via correctly SHA-scoped native `concurrency:` groups where that's the right tool (`opencode-review.yml`, fixed after a real prior incident, `#1568`), correctly PR-scoped native cancellation where SHA-scoping would have been wrong (`noema-review.yml`), and purpose-built same-file jobs that call the GitHub Actions API directly to find and cancel stale-head runs by exact `head_sha` match where native concurrency alone can't reach (`strix.yml`'s `cancel-superseded-pr-runs`, `pr-review-merge-scheduler.yml`'s hourly `org-queue-sweep`, which covers every repo except `.github` itself). Every quoted line and mechanism was independently re-fetched and re-checked before acceptance. + +**The cited evidence shows a different, real problem instead: pure queue starvation, not a cancellation gap.** `naruon` PR #1528's full 17-run history (pulled live) shows every run sharing one unchanged head SHA — no multi-SHA race ever occurred. What did happen: the cited Strix run sat **23h22m queued before it even started running**, and the paired OpenCode Review run for the same commit was **still queued 24+ hours later with no run at all** at time of check. This corroborates `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md`'s plan-level-ceiling finding with a concrete, individually-named example rather than aggregate counts — the fix is capacity (a plan decision or added runner capacity), not a workflow-config bug. + +**Not acted on further, deliberately.** No fix was applied and the standing chicken-and-egg bypass-merge authorization was not used, because no fixable bug was found — forcing one here would have meant inventing a problem the evidence does not support, which this session's standing practice avoids. One unconfirmed lead is recorded for a future pass: `naruon`'s own `pr-governance.yml` fired six times for PR #1528's single unchanged SHA (broad `workflow_run`/`check_run`/label-toggle triggers); whether that specific firing pattern was legitimate repeated PR activity or avoidable redundant triggering was not checked and should not be assumed either way. From 087646d2797d5853521c5428cf0420a0d673be0f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 3 Sep 2026 11:02:35 +0900 Subject: [PATCH 2/9] docs(doctoring): correct and deepen the naruon pr-governance.yml lead A peer session's check_run-based waste claim missed that the job's own if: already restricts that trigger path to CodeRabbit checks -- GitHub Actions requests no runner for a skipped job, so that vector isn't job-slot waste. Read pr_governance_gate.sh in full: it evaluates live required-check/ review-thread/CodeRabbit state on every run, not a pure function of head SHA, so a same-head debounce (the natural reading of the proposed fix) would be unsafe -- it could leave the gate reporting a stale blocker list after a check finishes or a review lands. No fix implemented; recorded as still open rather than forced. Co-Authored-By: Claude Sonnet 5 --- ...-stale-head-cancellation-audit-20260903.md | 25 +++++++++++++------ docs/product-technical-gap-baseline.md | 2 +- 2 files changed, 19 insertions(+), 8 deletions(-) diff --git a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md index 977c7bf49b..0843165ac6 100644 --- a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md +++ b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md @@ -69,13 +69,24 @@ item 13 hypothesized is not what happened here. What actually happened, quoted d timestamp, **was still `status: queued`, `conclusion: null` when re-checked live on 2026-09-03** — stuck queued for **24+ hours with no run at all.** - Six separate "PR Governance" workflow runs fired for this one unchanged SHA (five `pull_request_target` - events, one `pull_request_review`) — naruon's own `pr-governance.yml` watches `labeled`/`unlabeled` - PR-target actions plus completion of four other workflows (`Application CI`, `Bandit Security Scan`, - `Build and Publish Docker Images`, `Strix Security Scan`) plus `check_run: completed`, any of which can - independently re-trigger it. Whether this specific PR's six firings were legitimate repeated - label/status activity or avoidable redundant triggering was **not verified** in this pass — flagged as - an open, unconfirmed lead, not a finding, since asserting it without checking the PR's actual event - history would repeat exactly the kind of unverified claim this session's own norms exist to catch. + events, one `pull_request_review`). Investigated further after a peer session flagged this as a likely + redundant-trigger source: `naruon`'s `pr-governance.yml` and `scripts/ci/pr_governance_gate.sh` were + fetched and read in full (not assumed). Two corrections to the initial framing: (1) the `governance` job + carries a job-level `if:` that restricts its `check_run`-triggered case to CodeRabbit-named checks only + — GitHub Actions genuinely cannot filter `check_run` by name at the `on:` trigger level, but the job + itself is *skipped* (no runner requested) for every non-CodeRabbit check-run completion, so that specific + vector is not the job-slot waste it first appeared to be; (2) the five observed `pull_request_target` + firings on one unchanged SHA are near-certainly `labeled`/`unlabeled` (or similar non-`synchronize`) + events — `synchronize` is the only `pull_request_target` type tied to a new commit, and the SHA never + changed. More importantly, `pr_governance_gate.sh` evaluates **live** state at the current head on every + run (required-check states via `gh pr checks`, unresolved review-thread count, CodeRabbit findings via + check-runs and commit status) — it is explicitly not a pure function of `(head_sha, base_sha)`, so a + same-head debounce ("skip if nothing changed since the last run at this SHA") would be actively wrong: it + could leave the gate reporting a stale blocker list from before a required check finished or a review + landed, a real correctness regression in merge-gating, not merely a missed optimization. No fix was + attempted for this reason — a safe one needs either confirming which specific labels toggled five times + on this PR and whether they are governance-irrelevant, or a considered design for distinguishing genuinely + new gate-relevant information from a redundant re-trigger. Recorded as still open, not fixed. This is the same root cause `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` already identified (a plan-level concurrent-job ceiling), now corroborated with a concrete, painful, individually diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 679d003645..53bebcb272 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2636,4 +2636,4 @@ Higgins, S. S., Crepalde, N., & Fernandes, L. (2021). Segmented multiplexity: A **The cited evidence shows a different, real problem instead: pure queue starvation, not a cancellation gap.** `naruon` PR #1528's full 17-run history (pulled live) shows every run sharing one unchanged head SHA — no multi-SHA race ever occurred. What did happen: the cited Strix run sat **23h22m queued before it even started running**, and the paired OpenCode Review run for the same commit was **still queued 24+ hours later with no run at all** at time of check. This corroborates `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md`'s plan-level-ceiling finding with a concrete, individually-named example rather than aggregate counts — the fix is capacity (a plan decision or added runner capacity), not a workflow-config bug. -**Not acted on further, deliberately.** No fix was applied and the standing chicken-and-egg bypass-merge authorization was not used, because no fixable bug was found — forcing one here would have meant inventing a problem the evidence does not support, which this session's standing practice avoids. One unconfirmed lead is recorded for a future pass: `naruon`'s own `pr-governance.yml` fired six times for PR #1528's single unchanged SHA (broad `workflow_run`/`check_run`/label-toggle triggers); whether that specific firing pattern was legitimate repeated PR activity or avoidable redundant triggering was not checked and should not be assumed either way. +**Not acted on further, deliberately.** No fix was applied and the standing chicken-and-egg bypass-merge authorization was not used, because no fixable bug was found — forcing one here would have meant inventing a problem the evidence does not support, which this session's standing practice avoids. A peer session's lead on `naruon`'s `pr-governance.yml` (six runs on PR #1528's one unchanged SHA) was investigated further by fetching and reading the workflow and its gate script in full: a `check_run`-triggered job-slot-waste claim was corrected (the job's own `if:` restricts that path to CodeRabbit checks only — GitHub Actions requests no runner for a skipped job), and a proposed same-head debounce fix was found to be unsafe rather than implemented — `scripts/ci/pr_governance_gate.sh` evaluates live required-check/review-thread/CodeRabbit state on every run, not a pure function of head SHA, so skipping re-evaluation whenever the SHA is unchanged would leave the gate reporting a stale blocker list after a check finishes or a review lands. See `docs/doctoring/item13-stale-head-cancellation-audit-20260903.md` for the full trace; recorded as still open, not fixed. From 5a08116557718069bf15c0b0d04b75537b64eec9 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 3 Sep 2026 02:30:52 +0000 Subject: [PATCH 3/9] fix(doctoring): address CodeRabbit and Devin review findings on item 13 audit - CodeRabbit: "with no run at all" contradicted the same sentence's own status: queued evidence -- a run existed, its job just never started. Reworded to "with no job started" in both the doctoring record and the gap-baseline entry. - Devin (analysis): qualify the bare naruon PR #1528 references with the ContextualWisdomLab/ owner prefix so they resolve as cross-repo links from this repo, consistent with this cycle's established convention. - Devin (bug): soften the unverified claim that the five pull_request_target firings on one unchanged SHA were "near-certainly labeled/unlabeled" -- only synchronize was ruled out; the specific event types were not actually checked. Say so. - Devin (bug, most substantive): the audit's noema-review.yml row claimed unconditional reliability for native cancel-in-progress, but that mechanism cancels whichever run most recently entered the concurrency group -- keyed on run-creation order, not head-SHA recency. An older push's synchronize event processed after a newer one's could in principle cancel the current run instead of the stale one; existing tests (tests/test_noema_review_gate.py) prove the *explicit* cancellation step can't do this, but don't cover native cancel-in-progress itself. No evidence this has ever happened, and no workflow change is made here -- recorded as an open, unverified risk (footnoted in the table, and as a new bullet in "What this resolves, and what it does not") rather than silently dismissed or asserted as a confirmed bug, consistent with this record's own practice for its other open leads. Grepped tests/ for the exact strings touched here; only tests/test_product_technical_gap_baseline.py pins this file's content, and it still passes. Full suite: 2630 passed, 1 skipped (two pre-existing Python-3.12-only test files not collectible under this sandbox's 3.11). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4 --- ...-stale-head-cancellation-audit-20260903.md | 56 +++++++++++++------ docs/product-technical-gap-baseline.md | 6 +- 2 files changed, 43 insertions(+), 19 deletions(-) diff --git a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md index 0843165ac6..bec30dcb6e 100644 --- a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md +++ b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md @@ -4,7 +4,7 @@ - **Subject:** backlog item 13 states "Strix, OpenCode Review, Noema가 Concurrency에 이슈가 없을 것. 한 PR 안에서 Push가 발생했을 때 이전 HEAD에 관한 Cancel이 발생할 것" (Strix/OpenCode Review/Noema must have no concurrency issues; a push within a PR must cancel the previous HEAD's run), citing - `ContextualWisdomLab/naruon` run `33581213829` / job `100095712154` / PR `#1528` as evidence. The user + `ContextualWisdomLab/naruon` run `33581213829` / job `100095712154` / PR ContextualWisdomLab/naruon#1528 as evidence. The user separately directed: if the org's ~60-concurrent-job ceiling (`docs/doctoring/actions-plan-concurrency-ceiling-20260903.md`) is blocking work, trace and resolve the workflow issues that create it, authorizing bypass-merge for this specific chicken-and-egg case (a queue-congestion fix that would itself be blocked by queue congestion). @@ -22,7 +22,7 @@ A 9-agent workflow (4 investigate + 1 direct evidence pull + 4 adversarial verif any supplementary same-file mechanism that cancels a stale prior-head run via the GitHub Actions API, and reached a verdict on whether a new push to an open PR reliably retires the now-stale run for the previous head SHA. A separate agent pulled the exact cited evidence (`naruon` run `33581213829`, its job, and PR -`#1528`'s full run history) directly from the GitHub API. Every one of the four workflow findings was then +ContextualWisdomLab/naruon#1528's full run history) directly from the GitHub API. Every one of the four workflow findings was then independently re-verified by a second agent instructed to actively try to refute it — re-fetching the same file fresh, checking for companion cancellation workflows, per-job (not just workflow-level) concurrency blocks, and verbatim accuracy of every quoted line — before being accepted. @@ -33,9 +33,25 @@ blocks, and verbatim accuracy of every quoted line — before being accepted. |---|---|---|---| | `strix.yml` | No — group is `strix--` only; `cancel-in-progress: false` (deliberate, to preserve scanner logs) | **Yes** | Separate `cancel-superseded-pr-runs` job, same file, fires on `synchronize`/`closed`, lists active runs via the Actions API, matches by workflow name + PR number + head SHA (via `display_title` and `pull_requests[].head.sha`), and POSTs cancel/force-cancel | | `opencode-review.yml` | Yes — group includes both PR number and exact head SHA (`opencode-review-bootstrap---`), `cancel-in-progress: true` | **Yes** | The SHA-scoped group means native cancellation never even needs to fire cross-SHA (a design fix for a real prior incident, `#1568`, where SHA-agnostic grouping let a stale run wrongly cancel a *newer* one); a dedicated `cancel-superseded-opencode-review-runs` job plus an in-loop live-head self-retirement check (60s poll) provide defense-in-depth | -| `noema-review.yml` | No — group is `noema-review--` (PR number only); `cancel-in-progress: true` for `synchronize`/`closed` | **Yes** | Native cancellation alone suffices here (same group, cancel-in-progress true), *and* a same-job "Cancel superseded Noema runs after live-head validation" step independently re-verifies and cancels via the API as defense-in-depth | +| `noema-review.yml` | No — group is `noema-review--` (PR number only); `cancel-in-progress: true` for `synchronize`/`closed` | **Yes\*** | Native cancellation alone suffices here (same group, cancel-in-progress true), *and* a same-job "Cancel superseded Noema runs after live-head validation" step independently re-verifies and cancels via the API as defense-in-depth | | `pr-review-merge-scheduler.yml` | No (PR-number only) for the scheduler's own runs; native cancellation handles those | **Yes, for every repo except `.github` itself** | The `org-queue-sweep` job's hourly cross-repo sweep lists every queued/in-progress run of *any* workflow (reaching Strix/OpenCode/Noema runs directly, not just this scheduler's own), classifies by `head_sha` mismatch against the PR's live head, re-validates immediately before acting, and cancels. Explicitly excludes `ContextualWisdomLab/.github` from its target list — this repo's own PRs rely on Strix/OpenCode/Noema's own (separately verified, correct) mechanisms plus a same-head duplicate-run coalescer (`current-head-run-coalescer.yml`), not this sweep | +**\*Caveat on `noema-review.yml`, raised by Devin Review on this PR and not fully closed here:** GitHub's +native `cancel-in-progress` cancels whichever run most recently *entered* the concurrency group — it keys +on run-creation order, not head-SHA recency. If GitHub ever processed an older push's `synchronize` event +after a newer push's `synchronize` event had already created its run, native cancellation would retire the +newer, correct run instead of the stale one; the older run's own "Reject a stale trigger before credential +or model setup" step would then correctly reject itself, but only after the current head's run was already +gone. `tests/test_noema_review_gate.py` proves the *explicit* "Cancel superseded Noema runs" step cannot do +this (its `.id < $current` ordering guard is pinned end to end), and proves a delayed +`workflow_run`/`repository_dispatch` retry cannot reach that step at all — but neither covers native +`cancel-in-progress` deciding between two live `pull_request_target` events processed out of creation +order, which happens above any step in this workflow and is not something workflow YAML can independently +guard against. This investigation found no evidence this has ever occurred (GitHub's own event ordering for +a single PR's webhooks is the only thing preventing it, not this repo's code) and did not attempt to +reproduce it — recorded as an open, unverified risk distinct from item 13's own (refuted) hypothesis, not +asserted away. + All four adversarial verification passes returned `refuted: false` after independently re-fetching the live files and checking specifically for missed per-job concurrency blocks, companion cancellation workflows, and misquoted YAML — none were found. One cosmetic inaccuracy was caught and is worth recording @@ -45,18 +61,20 @@ review") as adjacent to the `cancel-in-progress: false` line; it is actually ~15 trigger block's `paths-ignore` comment. The design rationale itself is accurate and real — only its in-file location was misdescribed. This does not change the substantive verdict. -**Conclusion: there is no stale-head-cancellation bug to fix.** Every one of the four central, -required-workflow-ruleset workflows this org's own PR pipeline depends on already reliably retires a -superseded-head run on a new push, through a combination of correctly-scoped native GitHub concurrency and -purpose-built, independently-verified supplementary cancellation jobs — several of which carry their own -design-rationale comments citing prior incidents (`#1568`) that already taught this exact lesson once. -Forcing a "fix" here on the strength of item 13's hypothesis alone, without this evidence, would have meant -inventing a problem that does not exist — the throttle this session has held all along (do not force a -consolidation, or here a fix, that a real look shows is not actually needed) applies. +**Conclusion: there is no stale-head-cancellation bug to fix for item 13's own hypothesis and cited +evidence.** Every one of the four central, required-workflow-ruleset workflows this org's own PR pipeline +depends on already reliably retires a superseded-head run on a new push, through a combination of +correctly-scoped native GitHub concurrency and purpose-built, independently-verified supplementary +cancellation jobs — several of which carry their own design-rationale comments citing prior incidents +(`#1568`) that already taught this exact lesson once — with one caveat (`noema-review.yml`'s native-ordering +assumption, above) recorded as open rather than verified. Forcing a "fix" here on the strength of item 13's +hypothesis alone, without this evidence, would have meant inventing a problem that does not exist — the +throttle this session has held all along (do not force a consolidation, or here a fix, that a real look +shows is not actually needed) applies. ## Result 2: the cited evidence shows a different, real, and more severe problem — pure queue starvation -The naruon PR `#1528` run history (all 17 recorded runs, pulled live from the GitHub API) shows **zero** +The ContextualWisdomLab/naruon#1528 run history (all 17 recorded runs, pulled live from the GitHub API) shows **zero** occurrences of two different head SHAs being simultaneously active — every run, across the whole history, shares the PR's one unchanged head SHA (`cf472cf77fb93325858f485a22e967449d7c387a`). The multi-SHA race item 13 hypothesized is not what happened here. What actually happened, quoted directly from the API: @@ -67,7 +85,7 @@ item 13 hypothesized is not what happened here. What actually happened, quoted d bug). - The paired "Required OpenCode Review" run for the identical SHA (`33581213805`), created at the same timestamp, **was still `status: queued`, `conclusion: null` when re-checked live on 2026-09-03** — stuck - queued for **24+ hours with no run at all.** + queued for **24+ hours with no job started.** - Six separate "PR Governance" workflow runs fired for this one unchanged SHA (five `pull_request_target` events, one `pull_request_review`). Investigated further after a peer session flagged this as a likely redundant-trigger source: `naruon`'s `pr-governance.yml` and `scripts/ci/pr_governance_gate.sh` were @@ -76,9 +94,9 @@ item 13 hypothesized is not what happened here. What actually happened, quoted d — GitHub Actions genuinely cannot filter `check_run` by name at the `on:` trigger level, but the job itself is *skipped* (no runner requested) for every non-CodeRabbit check-run completion, so that specific vector is not the job-slot waste it first appeared to be; (2) the five observed `pull_request_target` - firings on one unchanged SHA are near-certainly `labeled`/`unlabeled` (or similar non-`synchronize`) - events — `synchronize` is the only `pull_request_target` type tied to a new commit, and the SHA never - changed. More importantly, `pr_governance_gate.sh` evaluates **live** state at the current head on every + firings on one unchanged SHA came from non-`synchronize` events — `synchronize` is the only + `pull_request_target` type tied to a new commit, and the SHA never changed. The specific event types were + not verified. More importantly, `pr_governance_gate.sh` evaluates **live** state at the current head on every run (required-check states via `gh pr checks`, unresolved review-thread count, CodeRabbit findings via check-runs and commit status) — it is explicitly not a pure function of `(head_sha, base_sha)`, so a same-head debounce ("skip if nothing changed since the last run at this SHA") would be actively wrong: it @@ -108,6 +126,12 @@ margin but cannot lift the ceiling). necessary per PR (six runs on one SHA in this one case) is worth a dedicated, evidence-first follow-up investigation of that PR's actual label/review event history before concluding anything — recorded here so it is not lost, not asserted as confirmed. +- **Open, unverified lead, not a finding (raised by Devin Review on this PR):** whether `noema-review.yml`'s + native `cancel-in-progress` could ever cancel a genuinely current run because GitHub processed an older + push's `synchronize` event after a newer one — see the caveat under Result 1's table. No evidence this has + happened was found or sought; closing it would need either a reproduction attempt or an explicit + ordering guard (analogous to the existing `.id < $current` guard on the explicit cancellation step, but + for native `cancel-in-progress` itself, which workflow YAML cannot directly condition on SHA age). - **Bypass-merge authorization:** the user authorized bypass-merge for this investigation as a genuine chicken-and-egg case. It is not used here because no fix was found that needed it — this record is itself a normal docs-only PR, subject to normal review like any other. diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 53bebcb272..0a712098c9 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2630,10 +2630,10 @@ Higgins, S. S., Crepalde, N., & Fernandes, L. (2021). Segmented multiplexity: A ## Backlog item 13 (Strix/OpenCode/Noema stale-head cancellation) — verified NOT a bug — 2026-09-03 -**Status:** Investigated with a 9-agent workflow (4 independent file audits + 1 direct-evidence pull against the item's own cited example + 4 adversarial re-verification passes, all `refuted: false`), per `docs/doctoring/item13-stale-head-cancellation-audit-20260903.md`. Item 13 asks that Strix/OpenCode Review/Noema reliably cancel a PR's previous-head run when a new push supersedes it, citing `naruon` run `33581213829` / PR `#1528` as evidence of a gap. +**Status:** Investigated with a 9-agent workflow (4 independent file audits + 1 direct-evidence pull against the item's own cited example + 4 adversarial re-verification passes, all `refuted: false`), per `docs/doctoring/item13-stale-head-cancellation-audit-20260903.md`. Item 13 asks that Strix/OpenCode Review/Noema reliably cancel a PR's previous-head run when a new push supersedes it, citing `naruon` run `33581213829` / PR ContextualWisdomLab/naruon#1528 as evidence of a gap. -**Verdict: the hypothesis is refuted.** All four central required workflows (`strix.yml`, `opencode-review.yml`, `noema-review.yml`, `pr-review-merge-scheduler.yml`) already reliably retire a stale prior-head run on a new push — via correctly SHA-scoped native `concurrency:` groups where that's the right tool (`opencode-review.yml`, fixed after a real prior incident, `#1568`), correctly PR-scoped native cancellation where SHA-scoping would have been wrong (`noema-review.yml`), and purpose-built same-file jobs that call the GitHub Actions API directly to find and cancel stale-head runs by exact `head_sha` match where native concurrency alone can't reach (`strix.yml`'s `cancel-superseded-pr-runs`, `pr-review-merge-scheduler.yml`'s hourly `org-queue-sweep`, which covers every repo except `.github` itself). Every quoted line and mechanism was independently re-fetched and re-checked before acceptance. +**Verdict: the hypothesis is refuted for the item's own cited evidence.** All four central required workflows (`strix.yml`, `opencode-review.yml`, `noema-review.yml`, `pr-review-merge-scheduler.yml`) already reliably retire a stale prior-head run on a new push — via correctly SHA-scoped native `concurrency:` groups where that's the right tool (`opencode-review.yml`, fixed after a real prior incident, `#1568`), correctly PR-scoped native cancellation where SHA-scoping would have been wrong (`noema-review.yml`), and purpose-built same-file jobs that call the GitHub Actions API directly to find and cancel stale-head runs by exact `head_sha` match where native concurrency alone can't reach (`strix.yml`'s `cancel-superseded-pr-runs`, `pr-review-merge-scheduler.yml`'s hourly `org-queue-sweep`, which covers every repo except `.github` itself). Every quoted line and mechanism was independently re-fetched and re-checked before acceptance. One caveat, raised by Devin Review and not fully closed: `noema-review.yml`'s native `cancel-in-progress` keys on run-creation order, not head-SHA recency, so an older push's `synchronize` event processed after a newer one's — never observed, not reproduced, but not ruled out by workflow text alone — could in principle cancel the current run instead of the stale one; see the doctoring record's caveat for detail. This is recorded as an open, unverified risk distinct from item 13's own (refuted) hypothesis. -**The cited evidence shows a different, real problem instead: pure queue starvation, not a cancellation gap.** `naruon` PR #1528's full 17-run history (pulled live) shows every run sharing one unchanged head SHA — no multi-SHA race ever occurred. What did happen: the cited Strix run sat **23h22m queued before it even started running**, and the paired OpenCode Review run for the same commit was **still queued 24+ hours later with no run at all** at time of check. This corroborates `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md`'s plan-level-ceiling finding with a concrete, individually-named example rather than aggregate counts — the fix is capacity (a plan decision or added runner capacity), not a workflow-config bug. +**The cited evidence shows a different, real problem instead: pure queue starvation, not a cancellation gap.** ContextualWisdomLab/naruon#1528's full 17-run history (pulled live) shows every run sharing one unchanged head SHA — no multi-SHA race ever occurred. What did happen: the cited Strix run sat **23h22m queued before it even started running**, and the paired OpenCode Review run for the same commit was **still queued 24+ hours later with no job started** at time of check. This corroborates `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md`'s plan-level-ceiling finding with a concrete, individually-named example rather than aggregate counts — the fix is capacity (a plan decision or added runner capacity), not a workflow-config bug. **Not acted on further, deliberately.** No fix was applied and the standing chicken-and-egg bypass-merge authorization was not used, because no fixable bug was found — forcing one here would have meant inventing a problem the evidence does not support, which this session's standing practice avoids. A peer session's lead on `naruon`'s `pr-governance.yml` (six runs on PR #1528's one unchanged SHA) was investigated further by fetching and reading the workflow and its gate script in full: a `check_run`-triggered job-slot-waste claim was corrected (the job's own `if:` restricts that path to CodeRabbit checks only — GitHub Actions requests no runner for a skipped job), and a proposed same-head debounce fix was found to be unsafe rather than implemented — `scripts/ci/pr_governance_gate.sh` evaluates live required-check/review-thread/CodeRabbit state on every run, not a pure function of head SHA, so skipping re-evaluation whenever the SHA is unchanged would leave the gate reporting a stale blocker list after a check finishes or a review lands. See `docs/doctoring/item13-stale-head-cancellation-audit-20260903.md` for the full trace; recorded as still open, not fixed. From b1e585c4b5c185525ffaad59c9ec09b2675e49c0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 3 Sep 2026 12:09:13 +0900 Subject: [PATCH 4/9] fix(doctoring): address Devin/CodeRabbit findings on the item13 audit Three fixes: (1) the naruon PR reference lacked the ContextualWisdomLab/ owner prefix, breaking the cross-repo timeline backlink; (2) the labeled/unlabeled attribution for the five non-synchronize pull_request_target firings was speculation presented as near-certain -- softened to state plainly that the specific event types were not verified; (3) "with no run at all" imprecisely described a workflow run that existed (status: queued) but whose job never started -- corrected to "no job started" in both this record and the gap-baseline entry it feeds. Co-Authored-By: Claude Sonnet 5 --- .../item13-stale-head-cancellation-audit-20260903.md | 6 ++++-- docs/product-technical-gap-baseline.md | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md index bec30dcb6e..94e3193437 100644 --- a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md +++ b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md @@ -4,7 +4,7 @@ - **Subject:** backlog item 13 states "Strix, OpenCode Review, Noema가 Concurrency에 이슈가 없을 것. 한 PR 안에서 Push가 발생했을 때 이전 HEAD에 관한 Cancel이 발생할 것" (Strix/OpenCode Review/Noema must have no concurrency issues; a push within a PR must cancel the previous HEAD's run), citing - `ContextualWisdomLab/naruon` run `33581213829` / job `100095712154` / PR ContextualWisdomLab/naruon#1528 as evidence. The user + `ContextualWisdomLab/naruon#1528` (run `33581213829`, job `100095712154`) as evidence. The user separately directed: if the org's ~60-concurrent-job ceiling (`docs/doctoring/actions-plan-concurrency-ceiling-20260903.md`) is blocking work, trace and resolve the workflow issues that create it, authorizing bypass-merge for this specific chicken-and-egg case (a queue-congestion fix that would itself be blocked by queue congestion). @@ -96,7 +96,9 @@ item 13 hypothesized is not what happened here. What actually happened, quoted d vector is not the job-slot waste it first appeared to be; (2) the five observed `pull_request_target` firings on one unchanged SHA came from non-`synchronize` events — `synchronize` is the only `pull_request_target` type tied to a new commit, and the SHA never changed. The specific event types were - not verified. More importantly, `pr_governance_gate.sh` evaluates **live** state at the current head on every + not verified (an earlier draft attributed them specifically to `labeled`/`unlabeled`, which is one + plausible explanation among several non-`synchronize` types and was not confirmed against the PR's actual + event history — corrected per Devin Review). More importantly, `pr_governance_gate.sh` evaluates **live** state at the current head on every run (required-check states via `gh pr checks`, unresolved review-thread count, CodeRabbit findings via check-runs and commit status) — it is explicitly not a pure function of `(head_sha, base_sha)`, so a same-head debounce ("skip if nothing changed since the last run at this SHA") would be actively wrong: it diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 0a712098c9..203e563dd0 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2630,10 +2630,10 @@ Higgins, S. S., Crepalde, N., & Fernandes, L. (2021). Segmented multiplexity: A ## Backlog item 13 (Strix/OpenCode/Noema stale-head cancellation) — verified NOT a bug — 2026-09-03 -**Status:** Investigated with a 9-agent workflow (4 independent file audits + 1 direct-evidence pull against the item's own cited example + 4 adversarial re-verification passes, all `refuted: false`), per `docs/doctoring/item13-stale-head-cancellation-audit-20260903.md`. Item 13 asks that Strix/OpenCode Review/Noema reliably cancel a PR's previous-head run when a new push supersedes it, citing `naruon` run `33581213829` / PR ContextualWisdomLab/naruon#1528 as evidence of a gap. +**Status:** Investigated with a 9-agent workflow (4 independent file audits + 1 direct-evidence pull against the item's own cited example + 4 adversarial re-verification passes, all `refuted: false`), per `docs/doctoring/item13-stale-head-cancellation-audit-20260903.md`. Item 13 asks that Strix/OpenCode Review/Noema reliably cancel a PR's previous-head run when a new push supersedes it, citing `ContextualWisdomLab/naruon#1528` (run `33581213829`) as evidence of a gap. **Verdict: the hypothesis is refuted for the item's own cited evidence.** All four central required workflows (`strix.yml`, `opencode-review.yml`, `noema-review.yml`, `pr-review-merge-scheduler.yml`) already reliably retire a stale prior-head run on a new push — via correctly SHA-scoped native `concurrency:` groups where that's the right tool (`opencode-review.yml`, fixed after a real prior incident, `#1568`), correctly PR-scoped native cancellation where SHA-scoping would have been wrong (`noema-review.yml`), and purpose-built same-file jobs that call the GitHub Actions API directly to find and cancel stale-head runs by exact `head_sha` match where native concurrency alone can't reach (`strix.yml`'s `cancel-superseded-pr-runs`, `pr-review-merge-scheduler.yml`'s hourly `org-queue-sweep`, which covers every repo except `.github` itself). Every quoted line and mechanism was independently re-fetched and re-checked before acceptance. One caveat, raised by Devin Review and not fully closed: `noema-review.yml`'s native `cancel-in-progress` keys on run-creation order, not head-SHA recency, so an older push's `synchronize` event processed after a newer one's — never observed, not reproduced, but not ruled out by workflow text alone — could in principle cancel the current run instead of the stale one; see the doctoring record's caveat for detail. This is recorded as an open, unverified risk distinct from item 13's own (refuted) hypothesis. -**The cited evidence shows a different, real problem instead: pure queue starvation, not a cancellation gap.** ContextualWisdomLab/naruon#1528's full 17-run history (pulled live) shows every run sharing one unchanged head SHA — no multi-SHA race ever occurred. What did happen: the cited Strix run sat **23h22m queued before it even started running**, and the paired OpenCode Review run for the same commit was **still queued 24+ hours later with no job started** at time of check. This corroborates `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md`'s plan-level-ceiling finding with a concrete, individually-named example rather than aggregate counts — the fix is capacity (a plan decision or added runner capacity), not a workflow-config bug. +**The cited evidence shows a different, real problem instead: pure queue starvation, not a cancellation gap.** `ContextualWisdomLab/naruon#1528`'s full 17-run history (pulled live) shows every run sharing one unchanged head SHA — no multi-SHA race ever occurred. What did happen: the cited Strix run sat **23h22m queued before it even started running**, and the paired OpenCode Review run for the same commit was **still queued 24+ hours later with no job started** at time of check. This corroborates `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md`'s plan-level-ceiling finding with a concrete, individually-named example rather than aggregate counts — the fix is capacity (a plan decision or added runner capacity), not a workflow-config bug. **Not acted on further, deliberately.** No fix was applied and the standing chicken-and-egg bypass-merge authorization was not used, because no fixable bug was found — forcing one here would have meant inventing a problem the evidence does not support, which this session's standing practice avoids. A peer session's lead on `naruon`'s `pr-governance.yml` (six runs on PR #1528's one unchanged SHA) was investigated further by fetching and reading the workflow and its gate script in full: a `check_run`-triggered job-slot-waste claim was corrected (the job's own `if:` restricts that path to CodeRabbit checks only — GitHub Actions requests no runner for a skipped job), and a proposed same-head debounce fix was found to be unsafe rather than implemented — `scripts/ci/pr_governance_gate.sh` evaluates live required-check/review-thread/CodeRabbit state on every run, not a pure function of head SHA, so skipping re-evaluation whenever the SHA is unchanged would leave the gate reporting a stale blocker list after a check finishes or a review lands. See `docs/doctoring/item13-stale-head-cancellation-audit-20260903.md` for the full trace; recorded as still open, not fixed. From 659a308bbd4519189e8701ed5ae339c4e3808c2d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 3 Sep 2026 12:13:09 +0900 Subject: [PATCH 5/9] fix(doctoring): soften item 13's queue-diagnosis-exceeds-evidence overreach Devin Review correctly noted the 23h22m/24+ hour waits prove queueing occurred, not exclusively that a plan-level ceiling is the cause. Added a precision paragraph: what actually narrows the explanation here is that Result 1 already ruled out a workflow-configuration bug in this PR's own review pipeline, not elimination of every other conceivable cause. Does not change the conclusion, states its actual evidentiary basis correctly. Co-Authored-By: Claude Sonnet 5 --- ...-stale-head-cancellation-audit-20260903.md | 26 ++++++++++++++----- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md index 94e3193437..cc86f32df7 100644 --- a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md +++ b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md @@ -108,13 +108,25 @@ item 13 hypothesized is not what happened here. What actually happened, quoted d on this PR and whether they are governance-irrelevant, or a considered design for distinguishing genuinely new gate-relevant information from a redundant re-trigger. Recorded as still open, not fixed. -This is the same root cause `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` already -identified (a plan-level concurrent-job ceiling), now corroborated with a concrete, painful, individually -named example instead of aggregate counts: a real open PR's real review evidence sat queued for over a day -— not because anything is misconfigured, but because there was nowhere for it to run sooner. This -strengthens, rather than changes, that record's conclusion and its recommendation (a plan-tier decision or -added runner capacity is the actual fix; workflow-file consolidation reduces total triggered runs at the -margin but cannot lift the ceiling). +**Precision on what this evidence actually establishes (Devin Review):** the 23h22m and 24+ hour waits prove +queueing occurred; on their own they do not prove a plan-level concurrent-job ceiling is the *exclusive* +cause, only that they are consistent with one. `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` +treats its own live API counts (jobs `in_progress` vs. `queued`) the same way — as corroboration for that +theory, not as independent proof of it; that record does not claim otherwise, and neither does this one. A +misconfigured scheduler, a starved runner label (a real, separately-documented org history — see this +repository's own `ubuntu-latest` floating-image finding), or some other single-repository cause could in +principle also produce a multi-hour wait for one PR. What rules those out *here*, specifically, is that +Result 1 above already verified all four central workflows' own cancellation/scheduling logic is correct — +so the wait is not explained by a bug in this PR's own review pipeline, which narrows the remaining +explanation toward capacity rather than proving it by elimination of every other conceivable cause. + +With that precision stated, this evidence is consistent with, and corroborates, the root cause +`docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` already identified (a plan-level concurrent-job +ceiling) — now with a concrete, individually named example instead of only aggregate counts: a real open +PR's real review evidence sat queued for over a day, with no workflow-configuration defect found to explain +it. This strengthens, rather than changes, that record's conclusion and its recommendation (a plan-tier +decision or added runner capacity is the actual fix; workflow-file consolidation reduces total triggered +runs at the margin but cannot lift the ceiling). ## What this resolves, and what it does not From 15396678e871f52c321888a8faff000450335dc1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 3 Sep 2026 12:18:42 +0900 Subject: [PATCH 6/9] fix(doctoring): upgrade item13's Noema caveat to a confirmed bug A dedicated 4-agent follow-up (2 investigate + 2 adversarial verify, wf_68f78449-bb6) re-examined the two substantive Devin Review claims with fresh rigor. Result: the noema-review.yml concurrency-ordering hazard is a CONFIRMED real bug, not the "open, unverified risk" a concurrent edit had already merged in -- both the investigation and a dedicated refutation attempt found no flaw in the mechanism (GitHub cancels the currently-active run in a group unconditionally when a new run enters it, before any of the new run's own steps execute; noema-review.yml's group has no head-SHA component, so an out-of-order synchronize event can kill a valid current-head review before its own stale-check step ever runs). Strong corroborating evidence: strix.yml and opencode-review.yml both deliberately avoid this exact pattern already, for exactly this reason. The strix.yml paths-ignore claim was also fully re-verified: real internal gap in that file alone, but pr-review-merge-scheduler.yml's scan-pr-queue (no paths-ignore, same-event trigger) closes it in practice -- confirmed not a real bug via full adversarial re-check. Updated the doctoring record's table, caveat, conclusion, and "what this resolves" section, plus the gap-baseline summary, to state the confirmed bug plainly rather than hedge it. Not fixing the noema-review.yml bug in this PR -- a live CI concurrency change deserves its own dedicated PR with a regression test, not a same-breath edit to documentation. Co-Authored-By: Claude Sonnet 5 --- ...-stale-head-cancellation-audit-20260903.md | 116 ++++++++++++------ docs/product-technical-gap-baseline.md | 8 +- 2 files changed, 81 insertions(+), 43 deletions(-) diff --git a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md index cc86f32df7..fbb0971c25 100644 --- a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md +++ b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md @@ -33,24 +33,41 @@ blocks, and verbatim accuracy of every quoted line — before being accepted. |---|---|---|---| | `strix.yml` | No — group is `strix--` only; `cancel-in-progress: false` (deliberate, to preserve scanner logs) | **Yes** | Separate `cancel-superseded-pr-runs` job, same file, fires on `synchronize`/`closed`, lists active runs via the Actions API, matches by workflow name + PR number + head SHA (via `display_title` and `pull_requests[].head.sha`), and POSTs cancel/force-cancel | | `opencode-review.yml` | Yes — group includes both PR number and exact head SHA (`opencode-review-bootstrap---`), `cancel-in-progress: true` | **Yes** | The SHA-scoped group means native cancellation never even needs to fire cross-SHA (a design fix for a real prior incident, `#1568`, where SHA-agnostic grouping let a stale run wrongly cancel a *newer* one); a dedicated `cancel-superseded-opencode-review-runs` job plus an in-loop live-head self-retirement check (60s poll) provide defense-in-depth | -| `noema-review.yml` | No — group is `noema-review--` (PR number only); `cancel-in-progress: true` for `synchronize`/`closed` | **Yes\*** | Native cancellation alone suffices here (same group, cancel-in-progress true), *and* a same-job "Cancel superseded Noema runs after live-head validation" step independently re-verifies and cancels via the API as defense-in-depth | +| `noema-review.yml` | No — group is `noema-review--` (PR number only); `cancel-in-progress: true` for `synchronize`/`closed` | **No\*** | The same-job "Cancel superseded Noema runs after live-head validation" step is real and correctly implemented, but it runs too late to prevent the specific failure mode below — this is a **confirmed, unfixed bug**, not a caveat | | `pr-review-merge-scheduler.yml` | No (PR-number only) for the scheduler's own runs; native cancellation handles those | **Yes, for every repo except `.github` itself** | The `org-queue-sweep` job's hourly cross-repo sweep lists every queued/in-progress run of *any* workflow (reaching Strix/OpenCode/Noema runs directly, not just this scheduler's own), classifies by `head_sha` mismatch against the PR's live head, re-validates immediately before acting, and cancels. Explicitly excludes `ContextualWisdomLab/.github` from its target list — this repo's own PRs rely on Strix/OpenCode/Noema's own (separately verified, correct) mechanisms plus a same-head duplicate-run coalescer (`current-head-run-coalescer.yml`), not this sweep | -**\*Caveat on `noema-review.yml`, raised by Devin Review on this PR and not fully closed here:** GitHub's -native `cancel-in-progress` cancels whichever run most recently *entered* the concurrency group — it keys -on run-creation order, not head-SHA recency. If GitHub ever processed an older push's `synchronize` event -after a newer push's `synchronize` event had already created its run, native cancellation would retire the -newer, correct run instead of the stale one; the older run's own "Reject a stale trigger before credential -or model setup" step would then correctly reject itself, but only after the current head's run was already -gone. `tests/test_noema_review_gate.py` proves the *explicit* "Cancel superseded Noema runs" step cannot do -this (its `.id < $current` ordering guard is pinned end to end), and proves a delayed -`workflow_run`/`repository_dispatch` retry cannot reach that step at all — but neither covers native -`cancel-in-progress` deciding between two live `pull_request_target` events processed out of creation -order, which happens above any step in this workflow and is not something workflow YAML can independently -guard against. This investigation found no evidence this has ever occurred (GitHub's own event ordering for -a single PR's webhooks is the only thing preventing it, not this repo's code) and did not attempt to -reproduce it — recorded as an open, unverified risk distinct from item 13's own (refuted) hypothesis, not -asserted away. +**\*`noema-review.yml` has a confirmed, real concurrency bug, raised by Devin Review and independently +adversarially re-verified twice (both the initial investigation and a dedicated refutation attempt failed +to find any flaw) — this is not a hedge, it is a confirmed finding requiring correction to the table row +above and the session's earlier premature "no bug to fix" framing.** GitHub evaluates a workflow's top-level +`concurrency:` block at run-creation time, before any job or step of that run executes, using only the +triggering event's payload. When a new run enters a busy group with `cancel-in-progress: true`, GitHub +cancels whatever is *currently active* in that group unconditionally — as a side effect of the new run +merely starting, not as a result of anything the new run's own logic decides. `noema-review.yml`'s group +(`noema-review--`, no head SHA component) means **every** push to a PR shares one group with every +other push to that same PR. If GitHub's webhook/dispatch pipeline ever processes an older push's +`synchronize` event *after* a newer push's `synchronize` event has already started its run — GitHub does +not guarantee delivery order — the older run's mere entry into the group cancels the newer, valid, +current-head run immediately, **before** the older run ever reaches its own "Reject a stale trigger before +credential or model setup" step. That step then correctly identifies itself as stale and self-aborts — but +only after it has already destroyed the one valid review in flight, leaving the actual current head with no +review at all. Neither the in-job "Cancel superseded Noema runs" step (which only mops up runs with a +strictly *smaller* run id, i.e. genuinely earlier-dispatched ones — it cannot protect a run from a +later-dispatched cancellation) nor any pre-flight gate (none can exist here: GitHub evaluates +`concurrency:` before any job step runs, full stop) closes this. **Strong corroborating evidence that this +is a real, known-avoidable hazard, not a theoretical nitpick:** `strix.yml`'s own `strix` job explicitly sets +`cancel-in-progress: false` specifically to avoid this exact class of problem, with an inline comment +explaining the reasoning, and `opencode-review.yml` closes the identical hazard by scoping its group with +the exact head SHA (a fix already shipped for a real prior incident, `#1568`) rather than relying on native +cancel-in-progress at all. `noema-review.yml` uses neither established mitigation — it is the one central +workflow in this org that still uses the blunt, unguarded pattern the other two deliberately moved away +from. No evidence this has actually fired in production was found or sought (GitHub's own typical event +ordering, not any code in this repository, is the only thing that has prevented it so far) — but "not yet +observed" is not the same claim as "not a bug," and this record's own initial draft conflated the two before +this correction. **Not fixed in this PR** — the safe, precedented fix (adopt `opencode-review.yml`'s +SHA-scoped-group pattern, or an equivalent live-head pre-validation before group entry) is a code change to +a live, security-critical CI workflow gating every PR's required review, and deserves its own focused PR +with a regression test, not a same-breath edit alongside this documentation correction. All four adversarial verification passes returned `refuted: false` after independently re-fetching the live files and checking specifically for missed per-job concurrency blocks, companion cancellation @@ -61,16 +78,17 @@ review") as adjacent to the `cancel-in-progress: false` line; it is actually ~15 trigger block's `paths-ignore` comment. The design rationale itself is accurate and real — only its in-file location was misdescribed. This does not change the substantive verdict. -**Conclusion: there is no stale-head-cancellation bug to fix for item 13's own hypothesis and cited -evidence.** Every one of the four central, required-workflow-ruleset workflows this org's own PR pipeline -depends on already reliably retires a superseded-head run on a new push, through a combination of -correctly-scoped native GitHub concurrency and purpose-built, independently-verified supplementary -cancellation jobs — several of which carry their own design-rationale comments citing prior incidents -(`#1568`) that already taught this exact lesson once — with one caveat (`noema-review.yml`'s native-ordering -assumption, above) recorded as open rather than verified. Forcing a "fix" here on the strength of item 13's -hypothesis alone, without this evidence, would have meant inventing a problem that does not exist — the -throttle this session has held all along (do not force a consolidation, or here a fix, that a real look -shows is not actually needed) applies. +**Conclusion, corrected:** three of the four central, required-workflow-ruleset workflows (`strix.yml`, +`opencode-review.yml`, `pr-review-merge-scheduler.yml`) already reliably retire a superseded-head run on a +new push, through a combination of correctly-scoped native GitHub concurrency and purpose-built, +independently-verified supplementary cancellation jobs. `noema-review.yml` does not — it has the one +confirmed, real, currently-unfixed concurrency bug found in this investigation (above), distinct from item +13's own hypothesis and cited evidence, which remains refuted (naruon's PR #1528 never exhibited a multi-SHA +race; see Result 2). Forcing a fix on the strength of item 13's *own* hypothesis and cited evidence alone +would have meant inventing a problem that does not exist there — but this investigation surfaced a real one +elsewhere in the same file family, and reporting it accurately, not softening it into an "unverified risk," +is the correct application of the same throttle-agreement discipline (don't force what isn't real; don't +minimize what is). ## Result 2: the cited evidence shows a different, real, and more severe problem — pure queue starvation @@ -131,30 +149,50 @@ runs at the margin but cannot lift the ceiling). ## What this resolves, and what it does not - **Resolves:** whether item 13's specific "no cancellation on push" complaint reflects a real - configuration bug in the four central workflows. It does not — verified, not assumed, across all four, - with adversarial re-checking. Item 13 should be marked accordingly in `docs/product-technical-gap-baseline.md`. -- **Does not resolve:** why the org's real capacity is saturated to the point of 23-24+ hour stalls — that - is the plan-level ceiling question `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` already - raises as an org-owner billing/plan decision, now with stronger evidence, not a new answer. + configuration bug *as evidenced by its own cited example* (`naruon`'s PR #1528). It does not — that PR + never exhibited a multi-SHA race; see Result 2. Item 13 should be marked accordingly in + `docs/product-technical-gap-baseline.md`, alongside the confirmed finding below rather than instead of it. +- **Confirmed, not-yet-fixed finding (raised by Devin Review, adversarially re-verified twice with no + refutation found):** `noema-review.yml`'s native `cancel-in-progress` can cancel a genuinely current run + when GitHub processes an older push's `synchronize` event after a newer one — GitHub does not guarantee + webhook/dispatch delivery order, and this workflow's concurrency group has no head-SHA component to make + such an inversion harmless. See the corrected caveat under Result 1's table for the full mechanism and the + corroborating evidence that `strix.yml` and `opencode-review.yml` both deliberately avoid this exact + pattern already. Not fixed here — this is a code change to live, security-critical CI configuration and + deserves a dedicated PR with its own regression test, not a same-breath edit to a documentation PR. - **Open, unverified lead, not a finding:** whether naruon's `pr-governance.yml` fires more often than necessary per PR (six runs on one SHA in this one case) is worth a dedicated, evidence-first follow-up investigation of that PR's actual label/review event history before concluding anything — recorded here so it is not lost, not asserted as confirmed. -- **Open, unverified lead, not a finding (raised by Devin Review on this PR):** whether `noema-review.yml`'s - native `cancel-in-progress` could ever cancel a genuinely current run because GitHub processed an older - push's `synchronize` event after a newer one — see the caveat under Result 1's table. No evidence this has - happened was found or sought; closing it would need either a reproduction attempt or an explicit - ordering guard (analogous to the existing `.id < $current` guard on the explicit cancellation step, but - for native `cancel-in-progress` itself, which workflow YAML cannot directly condition on SHA age). +- **Investigated and refuted (raised by Devin Review, adversarially re-verified with no refutation found):** + a claim that `strix.yml`'s `pull_request_target: paths-ignore:` list suppresses `cancel-superseded-pr-runs` + (a job in the same file, sharing the same trigger) for a push whose diff touches only ignored paths, + leaving the previous head's Strix scan running indefinitely. `strix.yml`'s own internal gap is real — that + half of the claim is correct, and there is no escape hatch inside that file. But a sibling required + workflow, `pr-review-merge-scheduler.yml`, has no `paths-ignore` at all and fires unconditionally on the + same event; its `scan-pr-queue` job unconditionally calls `cancel_stale_pr_runs()` + (`scripts/ci/pr_review_merge_scheduler.py`), which cancels any active run in the repository whose + `head_sha` no longer matches the PR's live head — regardless of which workflow created that run — + typically within the same push event, with a 30-minute local-cron backstop specifically for + `ContextualWisdomLab/.github` (whose own comment already documents this as the reason `org-queue-sweep`'s + `.github` exclusion is safe) and an hourly org-wide sweep backstop for every sibling repository. The + scenario does not leave a stale Strix scan running indefinitely anywhere. - **Bypass-merge authorization:** the user authorized bypass-merge for this investigation as a genuine - chicken-and-egg case. It is not used here because no fix was found that needed it — this record is itself - a normal docs-only PR, subject to normal review like any other. + chicken-and-egg case. It is not used here because no fix was found that needed it for item 13's own + hypothesis or the paths-ignore claim; the one confirmed bug found (`noema-review.yml`'s concurrency + ordering hazard, above) is deliberately left for its own dedicated fix PR rather than bypass-merged in + alongside documentation. This record is itself a normal docs-only PR, subject to normal review like any + other. ## Audit trail - Workflow run `wf_eb15dd2b-ad1` (9 agents: 4 investigate, 1 direct-evidence pull, 4 adversarial verify) — full per-agent transcripts and the complete unredacted findings/verification JSON live in that run's journal. +- Workflow run `wf_68f78449-bb6` (4 agents: 2 investigate, 2 adversarial verify) — the follow-up + investigation of the two substantive Devin Review findings above (`noema-review.yml`'s confirmed + concurrency bug, `strix.yml`'s refuted paths-ignore claim); full per-agent transcripts and the complete + unredacted findings/verification JSON live in that run's journal. - `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` — the root-cause record this evidence corroborates. - `docs/product-technical-gap-baseline.md` — backlog item 13's original text and citation, to be updated diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 203e563dd0..1199336f5a 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2628,12 +2628,12 @@ Higgins, S. S., Crepalde, N., & Fernandes, L. (2021). Segmented multiplexity: A **Evidence / acceptance.** Permanent tests forbid retry/deadline/sampling symbols in the caller and prove one gateway request, one attempt annotation, control-character-safe telemetry, missing-value rejection, valid trailing-comma normalization, and exact changed-line guidance. Fresh exact-head repository checks and reviews remain the admission authority; predecessor-head evidence is not transferable. The remaining runtime work is to preserve distinct `request_too_large`, discovery, rate-limit, provider transport, malformed-output, stale-head, and sandbox-command-timeout categories in hosted logs. -## Backlog item 13 (Strix/OpenCode/Noema stale-head cancellation) — verified NOT a bug — 2026-09-03 +## Backlog item 13 (Strix/OpenCode/Noema stale-head cancellation) — own hypothesis refuted, but a real bug was found in the process — 2026-09-03 -**Status:** Investigated with a 9-agent workflow (4 independent file audits + 1 direct-evidence pull against the item's own cited example + 4 adversarial re-verification passes, all `refuted: false`), per `docs/doctoring/item13-stale-head-cancellation-audit-20260903.md`. Item 13 asks that Strix/OpenCode Review/Noema reliably cancel a PR's previous-head run when a new push supersedes it, citing `ContextualWisdomLab/naruon#1528` (run `33581213829`) as evidence of a gap. +**Status:** Investigated with a 9-agent workflow (4 independent file audits + 1 direct-evidence pull against the item's own cited example + 4 adversarial re-verification passes) plus a 4-agent follow-up (2 investigate + 2 adversarial verify) triggered by Devin Review findings, per `docs/doctoring/item13-stale-head-cancellation-audit-20260903.md`. Item 13 asks that Strix/OpenCode Review/Noema reliably cancel a PR's previous-head run when a new push supersedes it, citing `ContextualWisdomLab/naruon#1528` (run `33581213829`) as evidence of a gap. -**Verdict: the hypothesis is refuted for the item's own cited evidence.** All four central required workflows (`strix.yml`, `opencode-review.yml`, `noema-review.yml`, `pr-review-merge-scheduler.yml`) already reliably retire a stale prior-head run on a new push — via correctly SHA-scoped native `concurrency:` groups where that's the right tool (`opencode-review.yml`, fixed after a real prior incident, `#1568`), correctly PR-scoped native cancellation where SHA-scoping would have been wrong (`noema-review.yml`), and purpose-built same-file jobs that call the GitHub Actions API directly to find and cancel stale-head runs by exact `head_sha` match where native concurrency alone can't reach (`strix.yml`'s `cancel-superseded-pr-runs`, `pr-review-merge-scheduler.yml`'s hourly `org-queue-sweep`, which covers every repo except `.github` itself). Every quoted line and mechanism was independently re-fetched and re-checked before acceptance. One caveat, raised by Devin Review and not fully closed: `noema-review.yml`'s native `cancel-in-progress` keys on run-creation order, not head-SHA recency, so an older push's `synchronize` event processed after a newer one's — never observed, not reproduced, but not ruled out by workflow text alone — could in principle cancel the current run instead of the stale one; see the doctoring record's caveat for detail. This is recorded as an open, unverified risk distinct from item 13's own (refuted) hypothesis. +**Verdict: the hypothesis is refuted for the item's own cited evidence, but `noema-review.yml` has a separate, confirmed, unfixed concurrency bug.** `strix.yml`, `opencode-review.yml`, and `pr-review-merge-scheduler.yml` already reliably retire a stale prior-head run on a new push — via correctly SHA-scoped native `concurrency:` groups where that's the right tool (`opencode-review.yml`, fixed after a real prior incident, `#1568`), and purpose-built same-file jobs that call the GitHub Actions API directly to find and cancel stale-head runs by exact `head_sha` match where native concurrency alone can't reach (`strix.yml`'s `cancel-superseded-pr-runs`, `pr-review-merge-scheduler.yml`'s hourly `org-queue-sweep`). `noema-review.yml` does not: its concurrency group has no head-SHA component, so if GitHub ever processes an older push's `synchronize` event after a newer one's (GitHub does not guarantee delivery order), native `cancel-in-progress` cancels the newer, valid, current-head run immediately — before the older run's own stale-trigger check ever executes, and nothing in the file can prevent this since GitHub evaluates `concurrency:` before any job step runs. Confirmed via two independent adversarial re-verification passes, neither of which found a refutation; corroborated by `strix.yml` and `opencode-review.yml` both deliberately using different patterns specifically to avoid this exact hazard. Not fixed here — a live CI concurrency-scoping change deserves its own dedicated PR with a regression test, not a same-breath edit to documentation. See the doctoring record for the full mechanism and evidence. **The cited evidence shows a different, real problem instead: pure queue starvation, not a cancellation gap.** `ContextualWisdomLab/naruon#1528`'s full 17-run history (pulled live) shows every run sharing one unchanged head SHA — no multi-SHA race ever occurred. What did happen: the cited Strix run sat **23h22m queued before it even started running**, and the paired OpenCode Review run for the same commit was **still queued 24+ hours later with no job started** at time of check. This corroborates `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md`'s plan-level-ceiling finding with a concrete, individually-named example rather than aggregate counts — the fix is capacity (a plan decision or added runner capacity), not a workflow-config bug. -**Not acted on further, deliberately.** No fix was applied and the standing chicken-and-egg bypass-merge authorization was not used, because no fixable bug was found — forcing one here would have meant inventing a problem the evidence does not support, which this session's standing practice avoids. A peer session's lead on `naruon`'s `pr-governance.yml` (six runs on PR #1528's one unchanged SHA) was investigated further by fetching and reading the workflow and its gate script in full: a `check_run`-triggered job-slot-waste claim was corrected (the job's own `if:` restricts that path to CodeRabbit checks only — GitHub Actions requests no runner for a skipped job), and a proposed same-head debounce fix was found to be unsafe rather than implemented — `scripts/ci/pr_governance_gate.sh` evaluates live required-check/review-thread/CodeRabbit state on every run, not a pure function of head SHA, so skipping re-evaluation whenever the SHA is unchanged would leave the gate reporting a stale blocker list after a check finishes or a review lands. See `docs/doctoring/item13-stale-head-cancellation-audit-20260903.md` for the full trace; recorded as still open, not fixed. +**Not acted on further, deliberately, except for the confirmed `noema-review.yml` bug which is deferred to its own PR.** No fix was applied to item 13's own hypothesis or the (also-refuted) `strix.yml` paths-ignore claim, because no fixable bug was found there — forcing one would have meant inventing a problem the evidence does not support. The `noema-review.yml` concurrency bug is real and confirmed, but a live security-critical CI concurrency-scoping change was deliberately not bundled into this documentation PR; the standing chicken-and-egg bypass-merge authorization remains available for whichever PR carries that fix, once it exists. A peer session's lead on `naruon`'s `pr-governance.yml` (six runs on PR #1528's one unchanged SHA) was investigated further by fetching and reading the workflow and its gate script in full: a `check_run`-triggered job-slot-waste claim was corrected (the job's own `if:` restricts that path to CodeRabbit checks only — GitHub Actions requests no runner for a skipped job), and a proposed same-head debounce fix was found to be unsafe rather than implemented — `scripts/ci/pr_governance_gate.sh` evaluates live required-check/review-thread/CodeRabbit state on every run, not a pure function of head SHA, so skipping re-evaluation whenever the SHA is unchanged would leave the gate reporting a stale blocker list after a check finishes or a review lands. See `docs/doctoring/item13-stale-head-cancellation-audit-20260903.md` for the full trace; recorded as still open, not fixed. From a9fe75cff1d06eba24a45bcf2ac55cef592f74fc Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 3 Sep 2026 12:29:40 +0900 Subject: [PATCH 7/9] fix(doctoring): fix naruon reference format, resolve cancellation-verdict contradiction - Cite ContextualWisdomLab/naruon#1528 with owner prefix in both remaining spots (Devin Review: cross-repo reference format violation) - Fix the self-contradiction Devin caught: "Result 1 verified all four central workflows' cancellation logic is correct" directly contradicted the same document's confirmed noema-review.yml bug finding above it. Corrected to: three of four are fully correct; the fourth's confirmed bug has a different failure signature (wrongful cancellation of a live run) than what Result 2's evidence shows (runs stuck queued for 23h22m/24+ hours with no cancellation at all) -- so the bug still doesn't explain this specific wait, without overstating the workflows' correctness. Co-Authored-By: Claude Sonnet 5 --- ...3-stale-head-cancellation-audit-20260903.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md index fbb0971c25..9c75c3d743 100644 --- a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md +++ b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md @@ -83,8 +83,8 @@ in-file location was misdescribed. This does not change the substantive verdict. new push, through a combination of correctly-scoped native GitHub concurrency and purpose-built, independently-verified supplementary cancellation jobs. `noema-review.yml` does not — it has the one confirmed, real, currently-unfixed concurrency bug found in this investigation (above), distinct from item -13's own hypothesis and cited evidence, which remains refuted (naruon's PR #1528 never exhibited a multi-SHA -race; see Result 2). Forcing a fix on the strength of item 13's *own* hypothesis and cited evidence alone +13's own hypothesis and cited evidence, which remains refuted (`ContextualWisdomLab/naruon#1528` never +exhibited a multi-SHA race; see Result 2). Forcing a fix on the strength of item 13's *own* hypothesis and cited evidence alone would have meant inventing a problem that does not exist there — but this investigation surfaced a real one elsewhere in the same file family, and reporting it accurately, not softening it into an "unverified risk," is the correct application of the same throttle-agreement discipline (don't force what isn't real; don't @@ -133,10 +133,14 @@ treats its own live API counts (jobs `in_progress` vs. `queued`) the same way theory, not as independent proof of it; that record does not claim otherwise, and neither does this one. A misconfigured scheduler, a starved runner label (a real, separately-documented org history — see this repository's own `ubuntu-latest` floating-image finding), or some other single-repository cause could in -principle also produce a multi-hour wait for one PR. What rules those out *here*, specifically, is that -Result 1 above already verified all four central workflows' own cancellation/scheduling logic is correct — -so the wait is not explained by a bug in this PR's own review pipeline, which narrows the remaining -explanation toward capacity rather than proving it by elimination of every other conceivable cause. +principle also produce a multi-hour wait for one PR. What narrows toward capacity *here*, specifically, is +that Result 1 above already verified three of the four central workflows' cancellation/scheduling logic is +fully correct, and that the fourth's (`noema-review.yml`'s) confirmed bug has a different failure signature +than what this evidence shows: that bug wrongly *cancels* a still-current run outright, whereas Result 2's +runs sat *queued* for 23h22m/24+ hours with no cancellation at all. A run stuck queued that long, never +cancelled, is not the symptom the confirmed bug produces — so this specific wait is still not explained by a +known bug in this PR's own review pipeline, which narrows the remaining explanation toward capacity rather +than proving it by elimination of every other conceivable cause. With that precision stated, this evidence is consistent with, and corroborates, the root cause `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` already identified (a plan-level concurrent-job @@ -149,7 +153,7 @@ runs at the margin but cannot lift the ceiling). ## What this resolves, and what it does not - **Resolves:** whether item 13's specific "no cancellation on push" complaint reflects a real - configuration bug *as evidenced by its own cited example* (`naruon`'s PR #1528). It does not — that PR + configuration bug *as evidenced by its own cited example* (`ContextualWisdomLab/naruon#1528`). It does not — that PR never exhibited a multi-SHA race; see Result 2. Item 13 should be marked accordingly in `docs/product-technical-gap-baseline.md`, alongside the confirmed finding below rather than instead of it. - **Confirmed, not-yet-fixed finding (raised by Devin Review, adversarially re-verified twice with no From 0e54b05f867b69b8a49dee2ab88f7e9e6ae9eee7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 3 Sep 2026 12:43:57 +0900 Subject: [PATCH 8/9] docs(doctoring): record noema-review.yml fix status as proposed, not merged A peer session pushed 31e46db on ContextualWisdomLab/.github#1661 porting opencode-review.yml's SHA-scoping pattern to noema-review.yml, closing the confirmed bug this record documents. Independently verified the fix itself is correct. But their own gap-baseline.md entry (e2869c3) marked item 13 "fully resolved" -- the same premature-credit pattern Devin caught once already on bebd7c7 (PR #1765): 31e46db is unreachable from main (diverged, #1661 still open), and main's live noema-review.yml still has the pre-fix group with no head-SHA component. Recorded the fix's real status here (proposed, not yet merged) so this record doesn't imply main is safe when it isn't yet. Co-Authored-By: Claude Sonnet 5 --- ...-stale-head-cancellation-audit-20260903.md | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md index 9c75c3d743..0c5c041303 100644 --- a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md +++ b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md @@ -156,14 +156,19 @@ runs at the margin but cannot lift the ceiling). configuration bug *as evidenced by its own cited example* (`ContextualWisdomLab/naruon#1528`). It does not — that PR never exhibited a multi-SHA race; see Result 2. Item 13 should be marked accordingly in `docs/product-technical-gap-baseline.md`, alongside the confirmed finding below rather than instead of it. -- **Confirmed, not-yet-fixed finding (raised by Devin Review, adversarially re-verified twice with no - refutation found):** `noema-review.yml`'s native `cancel-in-progress` can cancel a genuinely current run - when GitHub processes an older push's `synchronize` event after a newer one — GitHub does not guarantee - webhook/dispatch delivery order, and this workflow's concurrency group has no head-SHA component to make - such an inversion harmless. See the corrected caveat under Result 1's table for the full mechanism and the - corroborating evidence that `strix.yml` and `opencode-review.yml` both deliberately avoid this exact - pattern already. Not fixed here — this is a code change to live, security-critical CI configuration and - deserves a dedicated PR with its own regression test, not a same-breath edit to a documentation PR. +- **Confirmed finding, fix proposed but not yet merged (raised by Devin Review, adversarially re-verified + twice with no refutation found):** `noema-review.yml`'s native `cancel-in-progress` can cancel a genuinely + current run when GitHub processes an older push's `synchronize` event after a newer one — GitHub does not + guarantee webhook/dispatch delivery order, and this workflow's concurrency group has no head-SHA component + to make such an inversion harmless. See the corrected caveat under Result 1's table for the full mechanism + and the corroborating evidence that `strix.yml` and `opencode-review.yml` both deliberately avoid this + exact pattern already. **Fix pushed as commit `31e46db` on `ContextualWisdomLab/.github#1661`** (a peer + session ported `opencode-review.yml`'s own `#1568` fix: the event's head SHA added as a third group-key + segment), independently re-verified against that branch — but `31e46db` is not reachable from `main` + (`git compare main...31e46db` reports `diverged`, `#1661` still open), and `main`'s live `noema-review.yml` + still has the pre-fix group with no head-SHA component. Do not mark this closed on `main` until `#1661` + merges — the same "proposed vs. landed" distinction Devin caught once already on this record's sibling PR + (`.github#1765`'s phase-labeling citation). - **Open, unverified lead, not a finding:** whether naruon's `pr-governance.yml` fires more often than necessary per PR (six runs on one SHA in this one case) is worth a dedicated, evidence-first follow-up investigation of that PR's actual label/review event history before concluding anything — recorded here From 0e2c4e4ba741b5946db964a939086e4193b68d71 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 3 Sep 2026 14:34:48 +0900 Subject: [PATCH 9/9] docs(doctoring): mark internal workflow-tool run IDs as non-durable evidence Devin Review caught it: wf_eb15dd2b-ad1 and wf_68f78449-bb6 are internal Claude Code orchestration run identifiers with no repository path or public URL -- a future reader has no way to open and inspect them, so citing them as "the evidence" for this record's findings was misleading. Added an explicit caveat: they document how the investigation was structured (agent counts, investigate-vs-verify split), not the evidence itself -- the actual checkable evidence is what each finding cites inline (exact file paths/line ranges, raw.githubusercontent.com fetches, gh api calls, named PR/commit references), which any future reader can independently re-run. Co-Authored-By: Claude Sonnet 5 --- ...3-stale-head-cancellation-audit-20260903.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md index 0c5c041303..e711aa7f88 100644 --- a/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md +++ b/docs/doctoring/item13-stale-head-cancellation-audit-20260903.md @@ -195,13 +195,23 @@ runs at the margin but cannot lift the ceiling). ## Audit trail +**Devin Review correctly flagged that the two run IDs below are not durable, externally checkable evidence +on their own.** `wf_eb15dd2b-ad1` and `wf_68f78449-bb6` are internal Claude Code orchestration-tool run +identifiers, local to the session that produced them — they have no repository path, no public URL, and no +way for a future reader (human or agent) to open and inspect them. They are recorded here only as an +internal audit trail of *how* this record's investigation was structured (agent counts, investigate-vs-verify +split), not as the evidence itself. The actual checkable evidence is what each finding above cites inline: +exact file paths and line ranges in this repository, `raw.githubusercontent.com` fetches of the live +workflow files, `gh api` calls against the GitHub REST API (rulesets, runs, jobs, PRs), and named PR/commit +references (`#1568`, `ContextualWisdomLab/naruon#1528`). Any future reader who doubts a finding above should +re-run those same file reads and API calls, not attempt to open these run IDs. + - Workflow run `wf_eb15dd2b-ad1` (9 agents: 4 investigate, 1 direct-evidence pull, 4 adversarial verify) — - full per-agent transcripts and the complete unredacted findings/verification JSON live in that run's - journal. + internal orchestration record only, per the caveat above. - Workflow run `wf_68f78449-bb6` (4 agents: 2 investigate, 2 adversarial verify) — the follow-up investigation of the two substantive Devin Review findings above (`noema-review.yml`'s confirmed - concurrency bug, `strix.yml`'s refuted paths-ignore claim); full per-agent transcripts and the complete - unredacted findings/verification JSON live in that run's journal. + concurrency bug, `strix.yml`'s refuted paths-ignore claim); internal orchestration record only, per the + caveat above. - `docs/doctoring/actions-plan-concurrency-ceiling-20260903.md` — the root-cause record this evidence corroborates. - `docs/product-technical-gap-baseline.md` — backlog item 13's original text and citation, to be updated