Repository navigation
Let catalog-freshness open a reviewable, off-by-default evidence PR - #95
Merged
seathatflowsinourveins merged 13 commits intoSep 23, 2026
Merged
Conversation
The freshness job now diffs against the newest published catalogs/sota-convergence/manifest-*.json (sorted by name) instead of a hard-coded dated filename, exposes a `drift` job output, and reports the test suite's skipped-test count in the job summary. A new `propose` job (needs: freshness, off by default; runs only on a manual open_pr:true dispatch or a scheduled run with the repository variable CATALOG_FRESHNESS_PROPOSE=true) turns a detected drift into a force-created automation/catalog-freshness branch and PR: it copies the run's drift.md/manifest artifact into evidence/artifacts/, writes a scripts/validate.py-shaped upstream_provenance receipt under evidence/receipts/, registers both via scripts/host_receipts.py's register_file (matched by path/id, never list position), conditionally rebuilds/rehashes docs/ecosystem/index.html only while it stays a tracked file, then commits and pushes with the job's own GITHUB_TOKEN via GIT_CONFIG_COUNT/KEY/VALUE (never persisted to disk) and dispatches validate.yml/token-report.yml on the branch, since a GITHUB_TOKEN push does not trigger pull_request-event runs. It never touches catalogs/sota-convergence/*, catalogs/landscape/*.json, manifests/stack.json, or layer-verdicts* -- those stay owned by the separate SOTA-convergence lane review. The receipt/registration logic is factored into scripts/freshness_propose.py (new) so it is unit-testable independent of the workflow YAML; tests/test_catalog_freshness_propose.py covers drift-table parsing, component-id selection with its stack-component fallback, an end-to-end fixture that runs scripts.validate.validate() against apply()'s output, list-order independence, and text-level checks of the committed workflow. docs/decisions/2026-09-23-bot-pr-dispatch.md records the evidence (GitHub's GITHUB_TOKEN and Actions-settings documentation), the alternatives (GitHub App token, PAT, stay report-only) and the overturn condition. docs/github-automation.md gets a matching section covering the one-time "Allow GitHub Actions to create and approve pull requests" setting, the CATALOG_FRESHNESS_PROPOSE variable, and why auto-merge stays off. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Independent Opus review (H1/H2/M1/L1-L6/T1-T3) and Codex cross-family
review (P1) on the prior commit's catalog-freshness propose job and
scripts/freshness_propose.py:
- H1: rebuild_explorer() no longer lets build_ecosystem.py's own stdout
leak into main()'s single-JSON-document stdout contract (capture_output
on both the --write and --check subprocess calls); verified with a real
subprocess integration test against a throwaway git-tracked repo copy.
- H2: propose's checkout now uses fetch-depth: 0, matching validate.yml,
so host_receipts.py validate can resolve every existing receipt's
pinned catalog_revision commit instead of failing on a shallow clone.
- M1: a rebuilt row with upstream.latest=None (a bounded --max-repos run
or an API error, not an observed change) is excluded from the drift
count and reported separately as "unfetched"; propose additionally
refuses to run unless this run's own fetch was unbounded and recorded
zero upstream errors.
- P1 (Codex, refuting the prior design): a workflow_dispatch run's checks
do not satisfy a required status check on a pull request at all
(GitHub's troubleshooting docs), so dispatching validate.yml/
token-report.yml after the push was a green-looking, not-actually-
required substitute. Removed. propose now relies on the PR's own
pull_request-triggered runs, which GitHub puts into an approval-required
state for a GITHUB_TOKEN-created PR (GITHUB_TOKEN docs); the job prints
the PR URL and instructs a write-access reviewer to approve them. The
REST approve-a-run endpoint is documented only for fork PRs, so this is
left as a manual UI step rather than an unverified automatic call.
docs/decisions/2026-09-23-bot-pr-dispatch.md records the corrected claim,
the quotes, and the overturn condition.
- Race guards (Codex P2): a single `${{ github.workflow }}` concurrency
group (no event_name, cancel-in-progress: false) serializes every run
regardless of trigger; the evidence-branch push uses
--force-with-lease against the remote tip `git ls-remote` just
observed, not a plain --force.
- L1: drift.md/PR-body table cells are rendered through a new md_cell()
helper (backtick-wrapped, `|`-escaped) so an upstream release tag
fetched from an external API can't break the Markdown table.
- L2: no fallback to an unrelated fixed component set when no drifted id
matches a stack component; raises instead, so the job fails loudly
rather than opening a PR with a misleading component_ids.
- L4: the basic-auth header is masked (::add-mask::) before use.
- L6: doc fixes (explorer is rewritten, not just rehashed; `gh variable
set` instead of a PATCH that fails on a new variable).
- Codex low: refuse to write through an existing symlink at any
destination this module creates (receipt, drift/manifest copies).
- T3: the diff logic moved into scripts/freshness_propose.py's
build_drift_report()/compute_drift()/render_drift_markdown(), imported
by catalog-freshness.yml's diff step instead of duplicated there, so
the drift-table header and the drift-vs-unfetched rule can't drift out
of sync between the workflow and the module.
tests/test_catalog_freshness_propose.py rewritten: 52 tests covering all
of the above, including a full normalized `if:` expression assertion (T2)
and a real subprocess suite against a throwaway git-tracked repository
copy for the stdout contract and explorer rehash paths (T1).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…h-pr-20260923 # Conflicts: # docs/ecosystem/index.html # manifests/evidence.json
Merging origin/main (796f759, #96: docs/ecosystem/index.html is no longer committed or tracked) left RebuildExplorerSubprocessTests' setUpClass asserting the file WAS tracked in its scratch copy, which no longer holds once the copied .gitignore excludes it. Split the class: RebuildExplorerSubprocessTests now builds its scratch copy the plain way (matching main's real, untracked-by-default state) and asserts rehashed_explorer is False; a new TrackedExplorerSubprocessTests force-adds a locally built index.html to keep exercising rebuild_explorer()'s --write/register/--check loop for forward compatibility, explicitly labeled as exercising a path that is currently dead code on main. This edit was made after the prior merge commit but not staged into it; committing it separately here rather than amending, since the merge commit already has other reviewers' context. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docs/github-automation.md's propose-job walkthrough now says plainly that step 4 (rebuild/rehash docs/ecosystem/index.html) is inert today, since #96 already made the file .gitignore'd and uncommitted; it is kept for the H1 regression it guards against if a future change ever tracks the file again, and TrackedExplorerSubprocessTests keeps exercising it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Independent Opus pass-with-findings review (N1/N2/N2b/N3/N4) and a Codex cross-family re-review (reproducing N1/N2, plus P2-3): - N1/N2 (compute_drift bug): the prior code skipped a row entirely whenever the *rebuilt* row's upstream.latest was None, before ever comparing pin. This hid real pin changes on any no-release repository (7 already exist in the published manifest: tavily-cli, skills-ref, poppler, ...) and, separately, treated a releases-endpoint failure papered over by a tags-endpoint fallback (partial_errors) as ordinary clean data. compute_drift() now returns three buckets -- drifted, unfetched, no_release -- comparing pin unconditionally and excluding a row as unfetched only when it lacks fetch evidence (upstream.pushed_at is None) or its raw github-freshness.json record shows a fetch problem (error or partial_errors, matched by URL or normalized GitHub slug). Regression tests reproduce both Opus's and Codex's exact scenarios. - N2 (job-level gate): freshness now also emits a partial_errors output; propose's if: requires it to be '0' alongside upstream_errors. - N2b: a missing/malformed github-freshness.json now fails closed (raises) instead of silently reading as zero errors. - N3: corrected the approval instructions (the "Awaiting approval" button near the PR's merge box opens the merge status panel, which holds "Approve workflows to run" -- not the Actions tab) and the GITHUB_TOKEN quote (current wording: "...will not create a new workflow run, with the following exceptions: ..."), and noted the 30-day auto-deletion of unapproved runs. - N4: render_drift_markdown() now receives only rebuilt_path.name, never the absolute $RUNNER_TEMP path that used to land in committed evidence. - P2-3 (Codex): the workflow-level concurrency group's default queue:single let a plain scheduled run silently replace a pending manual open_pr:true request. queue:max is the documented fix but is rejected by this repository's pinned actionlint 1.7.12 (checked directly). Moved concurrency to a job-scoped group on `propose` only, keyed on manual vs. scheduled, so the two categories can never replace each other's pending slot; --force-with-lease remains the actual data-safety guard for the resulting rare concurrent-execution case. - Also documented that force-creating automation/catalog-freshness from main on every run discards any commit a human pushed to it directly. docs/decisions/2026-09-23-bot-pr-dispatch.md records the corrected quotes/wording as corrections, not silent edits, plus the queue:max rejection and the chosen alternative with its overturn condition. tests/test_catalog_freshness_propose.py: 76 tests, including exact N1 (skills-ref 0.1.0->0.1.1) and N2 (503+tag-fallback) reproductions at both the compute_drift() and build_drift_report() levels, N2b fail-closed coverage, N4's relative-path assertion, and the job-scoped/keyed concurrency text checks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…h-pr-20260923 # Conflicts: # manifests/evidence.json
… claim (Opus verification) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ch head (Codex verification) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
52d136a) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… prose (Codex verification of 250adae) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seathatflowsinourveins
marked this pull request as ready for review
September 23, 2026 05:21
seathatflowsinourveins
deleted the
claude/catalog-refresh-pr-20260923
branch
September 23, 2026 05:21
seathatflowsinourveins
added a commit
that referenced
this pull request
Sep 23, 2026
…immutable releases, target ruleset (#108) * Close the catalog's GitHub automation: OSV/zizmor-online scans, gates, immutable releases, target ruleset Add security-scan.yml: an osv-scanner job (OSV-Scanner 2.6.0, checksum-verified) over every tracked lockfile in .github/osv-scanner-lockfiles.json with --no-resolve, failing on unignored vulnerabilities and uploading SARIF off PRs, plus a zizmor-online SARIF job. A unittest fails when a tracked lockfile is missing from the inventory or an ignore lacks a <=90-day expiry. Gate dependency review at high (warn-only removed) and grype at --fail-on high with a reviewed, empty .grype.yaml; upload Scorecard SARIF with a job-scoped security-events write; add a 7-day Dependabot cooldown. publish-catalog.yml gains a tag-only release job (contents: write only) that re-checks the attested digests and creates the immutable release with both files attached at creation. .github/main-ruleset.json becomes the target (dependency-review and osv-scanner required, strict checks, signatures, CodeQL code_scanning, squash only); the tag rulesets match live 23829417 and 23859358. Record the evidence, the CodeQL-vs-zizmor comparison on 168a3a8 and the superseded decisions in docs/decisions/2026-09-22-github-automation-closure.md; update SECURITY.md, automation.json, the docs, the regenerated verdicts/explorer and evidence hashes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Drop required_signatures from the target ruleset; gate .grype.yaml changes; fix stale gate docs A measured agent-lab run (2026-09-23) blocked PRs with unsigned branch commits under required_signatures even though GitHub signs the squash merge, so the target main ruleset leaves the rule out as keep-but-compare until every writer signs. supply-chain.yml now runs its grype gate when .grype.yaml changes, with a test. Scorecard, dependency-review and grype docs no longer call the new gates report-only, and the CodeQL default-setup row cites the re-read settings GET. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Resolve the catalog-automation review findings after rebasing on main - Rebase onto origin/main 8faca90 (#96, #99-#104, #106): take main's removal of docs/ecosystem/index.html, merge main's explorer build/attest/upload steps with the release job's digest outputs, and re-register manifest hashes with host_receipts.register_file. - OSV inventory: add a reasoned "excluded" list for the #101 grype positive-control fixture; the coverage test accepts only listed, fixture-scoped exclusions with an evidence path and still fails on any unlisted tracked lockfile. osv-scanner 2.6.0 exits 0 on the 37 listed files. - Target main ruleset: strict_required_status_checks_policy false (auto-merge without a merge queue would stall every open PR when main moves); required_signatures stays out; regression tests assert both. - automation.json: gating lanes move to security_gate_lanes with boolean required_check/target_required_check; fresh CodeQL default-setup GET cited. - foundation.json: restore ci-supply-chain lane gap text, replace stale automation.json line citations with JSON-path anchors at 92bb279; regenerate verdicts and the handbook section with build_verdicts. - Handbook automation summary and closure record updated (mlx branch dropped after #99, fixture alerts 7-15 dismissed and #105 closed, security updates kept on, strict decision, gap-ledger mapping). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Align merge-queue and osv-scanner catalog text with the strict-off target; record #104 CodeQL triage - automation.json merge-queue non-adoption no longer cites strict checks; it points at the strict-off decision (closure record section 10) with its overturn condition. - osv-scanner selection says "required in the target ruleset once applied" and names the 37 listed lockfiles, 2 covered manifests and 1 fixture exclusion. - Closure record section 2: the 10 first-analysis CodeQL alerts were resolved in #104, not left for owners. - Hashes re-registered in manifests/evidence.json. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Keep Dependabot security PRs off the grype positive-control fixture ignore: urllib3 on a pip entry scoped to the fixture directory (ignore applies to security updates; exclude-paths does not). Clarify that osv-scanner becomes a required check only after the target ruleset is applied. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Refresh evidence hashes after rebasing onto #95 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Fix the OSV SARIF path under bash -e; keep the zizmor write token away from the analyzer shell: bash implies -e, so a findings exit ended the OSV step before the SARIF run (reproduced; found by an independent-implementation comparison and the Codex cross-family review). zizmor now runs read-only and a tool-free upload job holds security-events: write. Tests cover both, plus OSV PackageOverrides and grype review-by dates. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: seathatflowsinourveins <234074349+seathatflowsinourveins@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
6 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, awaiting Opus and Codex review. Adds an off-by-default
proposejob to catalog-freshness.yml that opens an evidence-only PR, meaning no selection or pin changes; pin bumps require a qualified receipt. Its required checks are dispatched on the bot branch, because GITHUB_TOKEN events don't trigger pull_request runs. Decision record: docs/decisions/2026-09-23-bot-pr-dispatch.md. Local checks: validators pass, 2503 unit tests OK, zizmor, actionlint and gitleaks are clean. The live round trip will be tested after merge, once the Actions PR-creation setting is enabled.🤖 Generated with Claude Code