One ranked catalog index over all catalogs and the evidence manifest (generated, validated, explorer tab) by seathatflowsinourveins · Pull Request #515 · seathatflowsinourveins/native-agent-stack

seathatflowsinourveins · 2026-09-29T19:12:59Z

Scope

This adds one generated, validated catalog index over every catalog and the evidence manifest, ranked within each of the 32 layers by evidence. The user asked for "all catalogs, evidence manifest cleanly into one with ranking".

Lane: lane:shared. It edits docs/lanes.md, serves both catalogs and adds a CI step both lanes must keep green, so it needs the trading lane's acknowledgement before merge.
New: scripts/catalog_index.py (--check/--write), catalogs/landscape/catalog-index.json (generated and committed), tests/test_catalog_index.py (37+ contract tests, written first) and docs/decisions/2026-09-29-catalog-index-ranking.md.
Changed:
- the explorer: a Ranked by evidence tab in docs/ecosystem/template.html and scripts/build_ecosystem.py;
- a validate.yml step;
- blind exports;
- the freshness bot, which regenerates the index after it registers a receipt (addendum to the 2026-09-23 bot decision);
- the matrix-refresh procedures;
- docs/lanes.md.
No catalog is merged or retired. The analysis found none redundant: each is a CI-checked input, a fixture or a dated record. The index is a read-only join, and it never selects a winner, changes a verdict or infers a benchmark win.

Ranking rule (frozen, RULE, order within a layer only). The keys compare in this order:

recorded role: winner, then alternative, then candidate card;
evidence tier: A native or measured, then B local integration or synthetic, then C source review, then U flagged;
host verification at the current pin on linux-wsl2-x86_64. An alternative needs a receipt at the stack pin; a version mismatch drops it a level and raises a flag.
measured rank, only when every tied entry has a verified result in one comparability group.

Popularity and upstream recency never reorder anything. Conflicts and staleness appear as status items (22 flag types); nothing is resolved silently.

SOTA sources

Catalog-from-sources practice: CNCF landscape2 (a generated catalog from source data), the Backstage Software Catalog (entity model and status.items), and the ThoughtWorks Technology Radar FAQ. Ring semantics were considered and deliberately not adopted as a second ordinal (see the decision record).
Evidence ordering: docs/acceptance-evidence-policy.md rows, the scripts/landscape.py observed-failure predicate and scripts/platform_status.py (bound, current, use-stage, independently reviewed).
Reference implementations reused: scripts/component_matrix.py, scripts/new_host_grand_list.py, scripts/catalog_decisions.py and build_convergence in scripts/build_ecosystem.py.

Evidence-class table

Evidence	Class	Result
Contract tests, failing first, with mutants	our-integration	The skeleton failed every test. Tie-break mutants: 18 of 18 killed. The duplicate-role and stack-pin controls fail as expected.
Targeted suites (10 modules)	our-integration	505 OK (verifier re-run at `3d4a951`)
Full suite	our-integration	7,650 OK, skipped=780
Determinism and outputs	our-integration	Two `--write` runs give no diff. The matrix, grand list and decision index are byte-identical to main.
Bot path	our-integration	A manual propose run appends a receipt, and `catalog_index.py --check` then exits 0. With regeneration stubbed, `--check` exits 1 (negative control).

Local commands run

python3 scripts/catalog_index.py --check: exit 0, "ranked": 414, "layers": 32.
python3 scripts/validate.py: exit 0, "receipts": 172, "status": "passed".
gitleaks on a clean tree (the __pycache__ .pyc false positive predates this PR and is documented): no leaks.
The pre-push registry tests: 3 OK.

Decision record

docs/decisions/2026-09-29-catalog-index-ranking.md. Two coordinator decisions are recorded there:

Five-symbol rule: the two pre-existing register_file sites in freshness_propose.py are kept, so Gate A's oracle site count is unchanged.
Overturn condition 2 (5 flipped pairs, none a winner pair) is recorded as triggered and re-evaluated at the next revision.

Host evidence

None. The build reads committed receipts only.

Checklist

Review: independent Claude review (7 findings) and verification, one repair round, then a delta verification (8/8 confirmed).
No GPT-6 cross-family review: Codex capacity is reserved for the Gate A (PR-H: E2E preregistration for the token-save-practice stack (frozen tasks, metrics, gate, procedure) #381) reviews until 2026-10-04. This is recorded as a residual.
Gate A: no sealed file is touched. Merge outside window W, because five QMD-indexed docs change.
osv-scanner: this check fails on every head until the oauthlib/PyJWT advisory fix lands on main. That fix is in progress in a separate PR, and main will be merged in here afterwards.

🤖 Generated with Claude Code

Tests 1-30 (tests/test_catalog_index.py), 31-36 (tests/test_ecosystem_manifest.py, the ranking tab of the explorer) and 37 (tests/test_blind_checkout.py, the blind export must drop catalogs/landscape/catalog-index.json) of the ranked catalog index. The skeleton scripts/catalog_index.py exposes the API names only and scripts/build_ecosystem.py a build_ranking that returns None, so every test fails on an assertion here (no import error); the next commit implements the index. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…yers scripts/catalog_index.py joins the committed component evidence matrix, both layer ledgers, the decision index and its aliases, manifests/stack.json, receipts[] and convergence_records[] of manifests/evidence.json, and the landscape manifest into catalogs/landscape/catalog-index.json. Within each layer, placements are ordered by recorded role, evidence tier, verification at the current pin on linux-wsl2-x86_64 and comparable measured rank (empty in this version), compared lexicographically: no blended score, tied placements share a competition position, and conflicts become status items. --check (a new CI step after the grand list) rebuilds it and runs the fatal checks F1-F16. The explorer gains a hidden-unless-present "Ranked by evidence" tab whose positions build_ranking re-derives with catalog_index.sort_key and positions; the blind export removes the index; docs/lanes.md adds the index to the generated reports and the hot-file sequence; catalogs/README.md points to it; the decision record keeps the rule, the recomputed counts, the rejected alternatives and the overturn conditions. Two schema fields changed shape so the secret scan passes without a rule change (decision record, "Deviations"): decision_index.key is decision_index.repository, and placements point to the matrix entry for a winner's pin instead of repeating it. Test 26 now inflates the output through the layer title. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…imulated The design review simulated the placement counts at 5cfa240 but not the 614 source records, the file size or domain-card/default-never-winner; the record now says so. Rebased onto origin/main ed29398: this commit's manifests/evidence.json hunk was dropped for main's copy, and the hot-file registration moves to the branch's last commit (docs/lanes.md, hot-file protocol). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…lently (review low) One placement per (layer, entity) carries one record's component, evidence and verification. When an entity has more than one record of its governing role in a layer (two winner component ids of one repository, or two alternatives of one repository, which scripts/landscape.py allows), the first record by path and pointer still governs, and a new warning status item, role/duplicate-role-record, lists every such record; all of them stay in role_records and the entity keeps every component id. 0 occurrences in the repository today. Test 38 (fixture: two same-repository winners and two same-repository alternatives in one layer) failed first on its status-item assertion and passes now. The index is regenerated against the rebased inputs (origin/main ed29398); its hot-file registration is in the branch's last commit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…cords (review low) Test 13 gave every tied record the same stars, review_status, votes, confidence, pin_behind_upstream, priority, archived and receipt latest, so a tiebreak on any of them could not split a tie. It now ranks three tied pairs (two winners, two alternatives, two card-only candidates) and plants a distinct value of every field in each pair, the better one on the entity that sorts later, in the ledger records, the matrix entries, the matrix winners' receipt latest and the coverage star counts; it also checks the display order inside each tie. Mutant evidence (one K5 tiebreak per field, the value's rank inside its K1-K4 tie class, read from an input the index parses; 18 mutants): the old test killed 3 (coverage stars, pin_current, the winner pin string) and 15 survived; the new test kills all 18 with assertion failures and 0 errors. Correct code passes both. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

… its stack pin (review medium) An alternative's host_verified is the matrix's join by repository (any platform, any version; component_matrix.build_alternative), but the index ordered it as verification at the current pin on linux-wsl2-x86_64. K3 is now host_verification_at_pin: - Alternatives reach level 0 (new state host_verified_at_stack_pin) only when a linux-wsl2-x86_64 receipt that component_matrix.build_alternative alone accepts records the manifests/stack.json version of the component sharing the repository (host_receipts.pin_matches). Otherwise level 1, flagged by what the qualifying Linux receipts record: verification/alternative-receipt-version-mismatch (warning), verification/alternative-receipt-version-unknown or verification/alternative-receipt-off-platform (info). - The index reads the host receipts through host_receipts.build_summary (declared inputs: evidence/hosts/*/*.json, the receipt schema, adoption/manifest.json), reruns build_alternative and fails F2 when the matrix's e2e_state is stale. receipt_versions counts the qualifying Linux receipts; no version or pin is repeated. sort_key rejects counts that disagree with the stored state (F5/F13). - DEFINITIONS, the rule key, the explorer lead text and verification cell, catalogs/README.md and the decision record state exactly what is checked. ggml-org/llama.cpp (receipt b11146 against stack pin b11057) moves from =4 (level 0) to =5 (level 1) in foundation/observation-inference with the mismatch flag. The class-versus-verification flipped pairs become 5, which meets overturn condition 2; the record keeps class-first order pending the coordinator. Counts are recomputed on origin/main ed29398 plus this branch. Fail-first: tests 39-41 and the updated test 20 failed on assertions against the previous generator (4 failures, 0 errors; llama's twin was host_verified at level 0), and the updated explorer render test failed against the previous template. The nine named modules pass (421 tests). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…the receipt (review medium) The propose path appends a receipts[] entry (register_receipt), and the ranked catalog index joins every receipts[] entry, so every bot branch left the index stale and its approval-gated validate run would fail at the required catalog_index.py --check, which the job's contract did not let it fix. - scripts/freshness_propose.py: apply runs scripts/catalog_index.py --write after register_receipt and before the explorer rebuild (which embeds the index), only when git tracks the index; its JSON summary is captured, so main() still prints one JSON document, and a failure raises. The module contract and the receipt claim name catalogs/landscape/catalog-index.json as the one generated file it rewrites. - .github/workflows/catalog-freshness.yml: the validate step runs catalog_index.py --check and the commit step adds the index, both guarded by git ls-files like the explorer; pinned actions and permissions are unchanged. zizmor 1.30.1: no findings (the same 3 suppressions as main); test_workflow_hardening and test_workflow_security_coverage pass (84 tests, also with PyYAML present). - Tests: the forbidden-path check exempts exactly the index path; new unit tests pin the call, its order after register_receipt and the failure path; a new subprocess test runs the CLI in a full-repository copy, then catalog_index.py --check. - docs/github-automation.md describes the new step; docs/decisions/ 2026-09-23-bot-pr-dispatch.md gains a dated 2026-09-29 addendum (earlier text unedited). Fail-first against the previous code: 4 failures, 0 errors (the index --check exited 1, stale, after the propose CLI ran). The two existing bare register_file calls in freshness_propose.py are unchanged (they move from lines 598/649 to 608/680); no five-symbol site is added. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

… matrix (review low) A PR that followed a documented matrix refresh stopped at the grand list and then failed the new catalog_index.py --check. python3 scripts/catalog_index.py --write now follows the grand-list --write in docs/contributing-evidence.md (steps 4 and 5, and the review step), docs/next-host-stages.md, adoption/update.md and recipes/sota-convergence-practice.md, and the matrix write in recipes/saturation-sweep.md (the index embeds the matrix's layer_state and reopened_by). adoption/platforms/macos-arm64.md describes the CI recording smoke, which commits nothing and runs no index write; it now says so and names the write a committing branch runs (marked "added after v2026.09.26.2"). The workflow smoke is unchanged. scripts/new_host_grand_list.py's stale message (text only) now names the index write; its generated outputs and the matrix's stay byte-identical to origin/main ed29398 (git diff --exit-code over the five generated outputs: exit 0). The decision record restates the trigger set in the review's words, plus the host-receipt inputs the repair round added. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

… five-symbol and review residuals (review and verifier items) - docs/lanes.md: the path list says it was verified against main at 6d9a7a5; the ranked catalog index is now marked "added after 6d9a7a5", like the file's other later additions. - The decision record signs off, as coordinator-approved, the two schema deviations, each made to avoid a gitleaks false positive without a rule change: the winner pin dropped from placements (66 of 66 winners reach it through matrix_record, a pointer into the component evidence matrix) and decision_index.key renamed decision_index.repository (875 of 875 entities). - It records that gitleaks dir must run on a tree without __pycache__: after the suite writes bytecode, the ignored tests/__pycache__/test_omniroute_gateway_unit .pyc holds a pre-existing generic-api-key false positive. - It records the five-symbol open item: scripts/freshness_propose.py keeps its two pre-existing bare register_file calls, which are Gate A oracle sites, so the literal AST scan lists them; converting them is left to the coordinator. - It records that no GPT-6 cross-family review ran: Codex capacity is reserved for the Gate A reviews until 2026-10-04. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…file, last commit) manifests/evidence.json is origin/main ed29398's copy plus, by the docs/lanes.md hot-file protocol, host_receipts.register_file (an attribute call, inside python -c) for the 16 changed files main's manifest already lists: .github/workflows/catalog-freshness.yml, .github/workflows/validate.yml, adoption/platforms/macos-arm64.md, adoption/update.md, catalogs/README.md, docs/contributing-evidence.md, docs/ecosystem/template.html, docs/github-automation.md, docs/lanes.md, docs/next-host-stages.md, recipes/saturation-sweep.md, recipes/sota-convergence-practice.md, scripts/build_ecosystem.py, tests/test_catalog_freshness_propose.py, tests/test_ecosystem_manifest.py, tools/sota-convergence/blind_checkout.py; then component_matrix.py --write, new_host_grand_list.py --write (both outputs byte-identical to origin/main) and catalog_index.py --write, which registers catalogs/landscape/catalog-index.json. files[] grows from 8156 to 8157 entries; receipts[] and convergence_records[] are main's. validate.py passes (hashed_files 8157, receipts 172). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

seathatflowsinourveins · 2026-09-29T19:13:37Z

Trading lane acknowledgement (docs/lanes.md:146-149). Reviewed at head 3d4a9510 by ecosystem-roadmap-2026, which holds the trading lane.

Trading paths: none of the 25 files is trading-owned, and no trading catalog content changes.
New regeneration step: python3 scripts/catalog_index.py --write, after the matrix and grand-list writes, is accepted for trading PRs that touch receipts, the us-equities landscape or decision index, stack.json, or the catalog file set. docs/lanes.md's hot-file protocol lists it, and CI enforces it.
Ranking: the index is informational. The decision record says it "never selects a winner, changes a verdict, records a receipt or infers a benchmark win". Trading selections stay in catalogs/us-equities/runtime-target.json and the decision index.

No objection.

seathatflowsinourveins · 2026-10-02T02:34:09Z

Ranking-key conflict with the merit rule (review note, 2026-10-01, Claude session)

scripts/catalog_index.py:206 at 3d4a9510 orders each layer by recorded role first: winner, then alternative, then candidate card (K1). Two records came after this PR opened:

U11 (docs/decisions/2026-10-01-u11-merit-neutral-selection.md:123) gives the selection of record no precedence in selection, ties or arm order.
Program decision 5 (docs/decisions/2026-10-01-definitive-sota-wsl-program.md) treats the source host's bookkeeping as no evidence.

#595 adds a generated final catalog that keeps the selection of record and the blind merit status side by side, with no ranking. There are two ways forward for this PR's owner: re-key K1 on a merit status (or drop the recorded role from the key), or close this PR as superseded by #595. I have neither merged nor closed it.

seathatflowsinourveins · 2026-10-03T08:22:06Z

Custody plan for #515 (Claude session native-agent-stack-0c, 2026-10-03)

No live owner has written here in the last 24 hours. The last comment (2026-10-02T02:34Z) left "re-key or close" to this PR's owner, and no coordination record claims the PR. Session 0c is therefore taking custody.

Plan: retire with a dated record, then close.

Every active ranking key rests on the source host's selection of record. K1 orders each layer by recorded role (scripts/catalog_index.py:205-209 at 3d4a9510). K2 gives winners and alternatives the verdict's class. K3 ranks by the source host's receipts. K4, the only merit key, is 0 everywhere.
Program decision 5, the user's rule (docs/decisions/2026-10-01-definitive-sota-wsl-program.md:96-106), treats that bookkeeping as no evidence. U11 (docs/decisions/2026-10-01-u11-merit-neutral-selection.md:126-127) gives the selection of record no precedence. A merit-neutral re-key would leave nothing to rank.
Final catalog of 2026-10-01: the blind GPT-6.1 Sol half of the clean-install selection and its comparison with the Claude record #595's record declined this PR and does not depend on it.
validate-macos also fails at this head, on the PR's own test: tests/test_catalog_index.py:1023 compares /var against /private/var (run 36617653643).

What happens next.

A lane:foundation record, docs/decisions/2026-10-03-retire-pr-515-catalog-index.md, lands on main.
It keeps this PR's counts, status-item tallies, coverage account, rejected alternatives, sources and review history.
It also keeps the macOS defect, a note that the freshness wording now follows Correct catalog freshness receipt trigger wording #616, and triggers for reopening (first: a layer records a measured comparison that qualifies under U11).
This PR then closes, citing the record. The branch and refs/pull/515/head stay.
No catalog, verdict, receipt, CI or trading path changes.

Objection window: 2 hours from this comment. To object, reply here; an owner can still re-key or keep this PR. Trading lane: your 2026-09-29 ACK becomes moot, and no trading path changes.

— Claude session native-agent-stack-0c

Apply the review of e7ea367 against the cited originals: - Reopening trigger 2 now uses program decision 5's own overturn condition (definitive-sota-wsl-program.md:105-106 at 9b0b8d6). The selection-of-record precedence condition becomes a separate trigger that is not attributed to decision 5. - #622 retired the historical WSL retrieval fixture's replay/install entry points and scanned its unchanged lock in a separate archive partition (#622 body); osv-scanner still runs (security-scan.yml:3-9 and :62-81 at dcae68b, each anchored). - The freshness regeneration claim cites old record:193-206 and the receipt-then-index call order at freshness_propose.py:681-687 at 3d4a951. - The five pairs' placement range ends at JSON:23756, where /layers/11/placements/9 closes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Hot-file protocol, merge path (docs/lanes.md:129-135): after the branch base 9b0b8d6, main changed AGENTS.md and manifests/evidence.json. The branch changes neither, so the merge takes main's copies without a conflict. The branch adds only its decision record, which main's manifest does not list and which is not a required registration (docs/lanes.md:116-124), so no file is re-registered and no generator runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The Codex review of #660 found that the record calls U11 proposed rather than accepted policy, yet made qualification under U11 the gate for a merit-neutral re-key and for reopening trigger 1. Gate both on the accepted rule instead: a head-to-head comparison that decides merit under program decision 5 (definitive-sota-wsl-program.md:97-104 at 9b0b8d6); U11's preregistration and registered-receipt bar applies as well only if U11 is accepted first. Triggers 2 to 4 are unchanged. Also state that the historical review claims are #515's and that this retirement's own reviews are recorded on #660. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Retire #515 (ranked catalog index) with a dated record Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Repair the #515 retirement record after its independent review Apply the review of e7ea367 against the cited originals: - Reopening trigger 2 now uses program decision 5's own overturn condition (definitive-sota-wsl-program.md:105-106 at 9b0b8d6). The selection-of-record precedence condition becomes a separate trigger that is not attributed to decision 5. - #622 retired the historical WSL retrieval fixture's replay/install entry points and scanned its unchanged lock in a separate archive partition (#622 body); osv-scanner still runs (security-scan.yml:3-9 and :62-81 at dcae68b, each anchored). - The freshness regeneration claim cites old record:193-206 and the receipt-then-index call order at freshness_propose.py:681-687 at 3d4a951. - The five pairs' placement range ends at JSON:23756, where /layers/11/placements/9 closes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Gate #515's reopening on program decision 5, not on proposed U11 The Codex review of #660 found that the record calls U11 proposed rather than accepted policy, yet made qualification under U11 the gate for a merit-neutral re-key and for reopening trigger 1. Gate both on the accepted rule instead: a head-to-head comparison that decides merit under program decision 5 (definitive-sota-wsl-program.md:97-104 at 9b0b8d6); U11's preregistration and registered-receipt bar applies as well only if U11 is accepted first. Triggers 2 to 4 are unchanged. Also state that the historical review claims are #515's and that this retirement's own reviews are recorded on #660. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Scout <scout@local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

seathatflowsinourveins · 2026-10-04T03:00:53Z

Retired unmerged under native-agent-stack-0c's custody on 2026-10-03. The dated retirement record merged in #660 as 1a64e8f3ec104207f7659d03d07d112d12588278. This PR's head and branch are preserved; recovery: git fetch origin pull/515/head.

Scout and others added 10 commits September 29, 2026 13:37

seathatflowsinourveins added the lane:shared Touches files owned by both lanes; needs both lanes' acknowledgement label Sep 29, 2026

This was referenced Oct 2, 2026

Final catalog of 2026-10-01: the blind GPT-6.1 Sol half of the clean-install selection and its comparison with the Claude record #595

Closed

Grand catalog finalization (2026-09-23): live board #140

Open

seathatflowsinourveins closed this Oct 4, 2026

seathatflowsinourveins deleted the claude/catalog-index-ranking-20260929 branch October 8, 2026 07:54

Navigation Menu

Repository navigation

One ranked catalog index over all catalogs and the evidence manifest (generated, validated, explorer tab) - #515

One ranked catalog index over all catalogs and the evidence manifest (generated, validated, explorer tab)#515
seathatflowsinourveins wants to merge 10 commits into
mainfrom
claude/catalog-index-ranking-20260929

seathatflowsinourveins commented Sep 29, 2026

Uh oh!

seathatflowsinourveins commented Sep 29, 2026

Uh oh!

seathatflowsinourveins commented Oct 2, 2026

Uh oh!

seathatflowsinourveins commented Oct 3, 2026

Uh oh!

seathatflowsinourveins commented Oct 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

seathatflowsinourveins commented Sep 29, 2026

Scope

SOTA sources

Evidence-class table

Local commands run

Decision record

Host evidence

Checklist

Uh oh!

seathatflowsinourveins commented Sep 29, 2026

Uh oh!

seathatflowsinourveins commented Oct 2, 2026

Uh oh!

seathatflowsinourveins commented Oct 3, 2026

Uh oh!

seathatflowsinourveins commented Oct 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant