From 7e2f6b4d0db08ca86734da7614687a287a7f71b2 Mon Sep 17 00:00:00 2001 From: Aaron Stainback Date: Tue, 28 Apr 2026 10:19:55 -0400 Subject: [PATCH] =?UTF-8?q?memory:=20detect=20CodeQL=20umbrella=20NEUTRAL?= =?UTF-8?q?=20vs=20per-language=20SUCCESS=20=E2=80=94=20code=5Fquality=20r?= =?UTF-8?q?uleset=20BLOCKED=20diagnostic=20(Aaron=202026-04-28)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures detection pattern for the failure mode Aaron has seen across multiple projects: code_quality:severity=all ruleset returns 'pending for N analyzed languages' even though every per-language Analyze (X) leg succeeds. Actual signal is the umbrella CodeQL check (no language suffix) being NEUTRAL with '1 configuration not found' details. 30-second detection: gh pr view N --json statusCheckRollup --jq '.[]|select(.name=="CodeQL")|{conclusion}' — if NEUTRAL on a code_quality-BLOCKED PR, this is the failure mode. Industry-wide pattern per Aaron 2026-04-28 ('i've seen these before'). Open question deferred: why same default-setup state ('not-configured') yields umbrella SUCCESS on AceHack vs NEUTRAL on LFG. Aaron's hypothesis (2026-04-28T14:23Z): org- level Code Security policy on LFG creates inheritance expectation that AceHack's personal-account context lacks. Agency-Signature-Version: 1 Agent: otto Agent-Runtime: claude-code Agent-Model: claude-opus-4-7 Credential-Identity: AceHack-shared Credential-Mode: shared-with-aaron Human-Review: not-implied-by-credential Human-Review-Evidence: aaron-explicit-ask Action-Mode: autonomous-fail-open Task: codeql-umbrella-detection-memory --- memory/MEMORY.md | 1 + ...uage_detection_pattern_aaron_2026_04_28.md | 87 +++++++++++++++++++ 2 files changed, 88 insertions(+) create mode 100644 memory/feedback_codeql_umbrella_neutral_vs_per_language_detection_pattern_aaron_2026_04_28.md diff --git a/memory/MEMORY.md b/memory/MEMORY.md index 89b42cb42..6f2fb90a1 100644 --- a/memory/MEMORY.md +++ b/memory/MEMORY.md @@ -2,6 +2,7 @@ **📌 Fast path: read `CURRENT-aaron.md` and `CURRENT-amara.md` first.** These per-maintainer distillations show what's currently in force. Raw memories below are the history; CURRENT files are the projection. (`CURRENT-aaron.md` refreshed 2026-04-25 with the Otto-281..285 substrate cluster + factory-as-superfluid framing — sections 18-22; prior refresh 2026-04-24 covered sections 13-17.) +- [**CodeQL umbrella check NEUTRAL while per-language Analyze legs SUCCESS — code_quality ruleset BLOCKED detection pattern (Aaron 2026-04-28)**](feedback_codeql_umbrella_neutral_vs_per_language_detection_pattern_aaron_2026_04_28.md) — When `code_quality:severity=all` ruleset says "Code quality results are pending for N analyzed languages" despite per-language `Analyze (X)` legs SUCCESS, check the umbrella `CodeQL` check (no language suffix) for NEUTRAL conclusion + "1 configuration not found" details. Industry-wide pattern; Aaron seen across other projects. - [**`gh workflow run --ref` on PR branch overwrites latest-by-name check-runs — branch-protection collateral risk (Aaron 2026-04-28)**](feedback_workflow_dispatch_overwrites_latest_byname_check_runs_branch_protection_caveat_2026_04_28.md) — Empirical 2026-04-28 LFG #660: dispatched gate.yml to populate missing macos-26; macos-26 succeeded but ubuntu legs flaked + OVERWROTE PR-run successes via latest-by-name; preferred recovery for "missing required check on PR" is `gh run rerun --failed` on the EXISTING PR-event run, NOT `gh workflow run --ref`. - [**Reviewer false-positive pattern catalog — 7-class taxonomy + per-class resolution forms + ROI-ranked prevention (Aaron 2026-04-28)**](feedback_reviewer_false_positive_pattern_catalog_aaron_2026_04_28.md) — Stale-snapshot / carve-out blind spot / schema drift / wrong-language parser / convention conflict / broken xref / recursive-CI-new-threads; speeds future thread classification; high-ROI prevention candidates listed. - [**CALIBRATION — `requiredApprovingReviewCount=0` on both Zeta forks; BLOCKED ≠ reviewer; 5-class taxonomy + complete enum coverage (Aaron 2026-04-28)**](feedback_no_required_approval_on_zeta_BLOCKED_means_threads_or_ci_aaron_2026_04_28.md) — 5 BLOCKED classes (threads / failing-or-pending CI / merge conflicts / required-check-MISSING-from-rollup / repository-ruleset gates); failed-conclusion enum covers FAILURE/CANCELLED/TIMED_OUT/ACTION_REQUIRED/STARTUP_FAILURE/STALE; pending-status enum covers IN_PROGRESS/QUEUED/WAITING/REQUESTED/PENDING; CheckRun.name vs StatusContext.context union extraction; always-double-check-after-CI rule. diff --git a/memory/feedback_codeql_umbrella_neutral_vs_per_language_detection_pattern_aaron_2026_04_28.md b/memory/feedback_codeql_umbrella_neutral_vs_per_language_detection_pattern_aaron_2026_04_28.md new file mode 100644 index 000000000..afaa1d723 --- /dev/null +++ b/memory/feedback_codeql_umbrella_neutral_vs_per_language_detection_pattern_aaron_2026_04_28.md @@ -0,0 +1,87 @@ +--- +name: CodeQL umbrella check NEUTRAL while per-language Analyze legs SUCCESS — detection pattern for code_quality ruleset BLOCKED state (Aaron 2026-04-28) +description: When `code_quality:severity=all` ruleset gates a PR with "Code quality results are pending for N analyzed languages" and CI looks all-green, the actual signal is the umbrella `CodeQL` check NEUTRAL (often "1 configuration not found" details), distinct from per-language `Analyze (X)` legs that are SUCCESS. Default-setup state alone doesn't predict it. Spotted by Aaron 2026-04-28 on LFG #661; he's seen the pattern in other projects. +type: feedback +--- + +# CodeQL umbrella NEUTRAL detection pattern (Aaron 2026-04-28) + +## The pattern + +PR is BLOCKED. All visible CI checks are SUCCESS. Merge probe via REST returns: + +> "Repository rule violations found. Code quality results are pending for N analyzed languages." + +Diagnostic intuition says CodeQL is failing. But every per-language `Analyze (csharp)`, `Analyze (python)`, `Analyze (javascript-typescript)`, `Analyze (actions)` leg is SUCCESS. CodeQL workflow run completed SUCCESS. SARIF analyses are uploaded for `refs/pull/N/merge` with `results_count: 0` for every language. + +**The actual signal**: there's a separate `CodeQL` umbrella status check (distinct from the per-language legs) and it's `NEUTRAL` with details URL showing **"1 configuration not found"**. + +The `code_quality:severity=all` ruleset rule reads the **umbrella** check, not the per-language legs. NEUTRAL → ruleset says "pending." + +## Detection in <30 seconds + +```bash +# Pull the umbrella CodeQL check specifically: +gh pr view N --repo OWNER/REPO --json statusCheckRollup --jq \ + '.statusCheckRollup[] | select(.name == "CodeQL") | {name, conclusion, detailsUrl}' + +# If conclusion is NEUTRAL (not SUCCESS) on a PR that's BLOCKED with +# "Code quality results pending" — this is the failure mode. +``` + +The umbrella `CodeQL` check is named just `CodeQL` (no language suffix). Don't confuse it with `Analyze (csharp)` etc. + +## Empirical evidence collected on LFG #661 (2026-04-28T14:16Z) + +- Per-language Analyze legs: 4/4 SUCCESS, all `results_count: 0` +- CodeQL workflow runs: SUCCESS +- SARIF analyses uploaded to `refs/pull/661/merge`: 4 entries, all 4 languages +- `CodeQL` umbrella check: **NEUTRAL** with "Completed in 3s — 1 configuration not found" +- LFG default-setup state: `not-configured` +- AceHack default-setup state: `not-configured` (SAME) +- AceHack PR #92 (recent merge): `CodeQL` umbrella = **SUCCESS** despite same default-setup state + +**Default-setup state alone does NOT predict umbrella outcome.** Something else differs between AceHack and LFG that causes the umbrella to go NEUTRAL on LFG. + +## Industry-wide pattern (Aaron 2026-04-28) + +Aaron's exact framing: "i've seen these before" — across other projects he uses +Claude PR review on. Not a Zeta-specific config bug. The asymmetry between umbrella +NEUTRAL and per-language SUCCESS is a recurring GitHub Code Scanning surface +oddity. + +## Why this is hard to see + +1. The standard `gh pr checks N` output lists per-language Analyze legs but the + umbrella `CodeQL` check is in a different position visually. +2. Per-language SUCCESS + workflow SUCCESS gives a strong "everything passed" signal + that masks the umbrella NEUTRAL. +3. The merge-endpoint error message says "results pending for N analyzed languages" + which sounds like per-language pending, but actually means the umbrella. + +## Composes with + +- `feedback_no_required_approval_on_zeta_BLOCKED_means_threads_or_ci_aaron_2026_04_28.md` + — the 5-class BLOCKED taxonomy. This adds a 6th sub-class: + **class-5b**: ruleset blocking on umbrella-NEUTRAL despite per-language SUCCESS. +- `feedback_reviewer_false_positive_pattern_catalog_aaron_2026_04_28.md` + — ditto for the 7-class false-positive catalog; this is a meta-analyzer + failure mode rather than a reviewer false-positive. +- Otto-352 narrow-not-broad principle — "ruleset BLOCKED" is too broad; the + precise diagnostic check is the umbrella `CodeQL` conclusion. +- `feedback_otto_355_blocked_with_green_ci_means_investigate_review_threads_first_dont_wait_2026_04_27.md` + — extends Otto-355's BLOCKED-investigate-first to include the umbrella-check + interrogation when threads are clean and per-language CI is green. + +## Open question (deferred) + +Why does AceHack's umbrella check go SUCCESS while LFG's goes NEUTRAL despite +identical default-setup state? Suspects: +- Different repo settings under `code-scanning/configurations` (org-level vs + repo-level) +- Different `paths-ignore` / `paths` config consumed by GitHub's umbrella + evaluator vs the workflow's evaluator +- Some ingestion-side flag that differs + +When the next PR hits this on LFG (or any other repo), grep for this memory file +and run the detection pattern first; investigate the asymmetry second.