Repository navigation
Landscape sweep 2026-09-26 record: all 32 layers, 56 survivors, manifest-20260926 for the verdict wave (11 lane limits) - #357
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Trading-lane ack (native-agent-stack-f5) at head ea7e5b2, with one factual correction requested in the trading section. Checked:
Correction (execution-broker, the #5041 re-pin candidate): the text says "rc5 also stays blocked by open issue #4983", and its overturn condition says rc5 fails "the #4983 local order denial". The primary sources refute that at the pin:
This does not establish that rc5 accepts stock orders; no rc5 stock order has been observed. Please reword the claim to an unqualified rc5 execution boundary, or add a dated limitation, before merge. Otherwise OK to merge from the trading side. The trading convergence-20260926 record will carry the correction for the verdict wave either way. |
…ort deviations
Recording the 2026-09-26 sweep (wf_8397ada1-777, 201 agents) exposed three gaps
in the packaged harness for a run it legitimately produced:
- child-usage.mjs printed its summary and then called process.exit(), which
drops stdout writes still pending on a pipe. usage_record.py captures stdout
through a pipe, so any run whose summary exceeds 64 KiB was cut at exactly
65,536 bytes and failed to parse (184,831 bytes for this run). It now sets
process.exitCode instead, as the Node.js process.exit() documentation advises.
- At a Claude usage limit the Workflow paused and re-ran its 8 waiting agents
after the reset under the same journal key ("Re-running 8 waiting agents").
child-usage.mjs counted each earlier attempt as an incomplete child, so a run
in which every call returned read as incomplete, and convert.py measured
those votes from the failed attempt ("<synthetic>+claude-opus-5-5").
child-usage.mjs now lists an attempt that returned nothing and whose key
started again under superseded_attempts (with superseded_by), keeps its
issues and effort check, and still counts its usage in by_resolved_model;
convert.py takes the attempt that returned and never names <synthetic>.
- A worker measured at another effort than max (here a pinned skill with
`effort: low` frontmatter lowered the final turn of three workers) made
make_result.py refuse the whole record. convert.py --usage now records each
such worker as an effort_deviation retained failure of its layer (the
critic's of every layer), which reopens the layer; make_result.py accepts
the usage only when every deviation is recorded so.
Tests: new cases in test-child-usage.mjs and tests/test_landscape_sweep_harness.py
fail on 3822178 and pass here; SHA256SUMS updated for the two changed files.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…at its Hub commit The 2026-09-26 sweep (wf_8397ada1-777) kept a Hugging Face model repository in agents-models-workers (https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B), a legitimate survivor of a model layer. source_reviews.py read only GitHub through gh api, refused it as "not a GitHub repository URL", and make_result.py then refused the record for a survivor without a source review. source_reviews.py now reviews a Hugging Face model repository at the commit of its default revision. The Hub's model-info endpoint /api/models/<repo_id> (the endpoint huggingface_hub's HfApi.model_info calls, checked at huggingface_hub main 62a1f2fa8383) gives the commit, the card license and the repository state. The model card is read at that commit through the documented "Resolve a file" endpoint (https://huggingface.co/.well-known/openapi.json), anonymously, and its YAML metadata block is not excerpted. The review is named hf-<namespace>-<name>.json. A Hub dataset or Space URL is still reported and skipped. The GitHub path is unchanged apart from sharing the excerpt helper. Test: SourceReviewTests.test_a_hugging_face_model_survivor_is_reviewed_at_its_hub_commit (stubbed Hub fetch, no network) fails before this change and passes with it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…records test_no_seed_layer_counts_as_clean asserted that no layer of the whole committed ledger counts as clean. That held while the ledger held only the 2026-09-23 seed records; a later completed sweep with clean layers (landscape-sweep-20260926 leaves agent-sdks, backtesting-engine and token-efficiency at 1) is its own evidence and made the whole-ledger assertion false. The test now derives the state from the two seed records, which is what its name and the seed checks around it assert. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An independent headless review (claude -p, Opus, read-only) of the Hugging Face support found three low-severity defects. Each is now covered by SourceReviewTests.test_hub_reviews_survive_truncated_responses_trailing_slashes_and_indented_card_metadata, which fails on the previous code and passes here: - hub_get caught URLError, OSError and ValueError, but http.client.HTTPException (IncompleteRead, BadStatusLine) is not an OSError. One truncated Hub response therefore aborted the whole run with a traceback before any review was written. It is now a HubError and is reported and skipped like any other. - Survivors were grouped by sweep_common.slug(), which keeps a Hugging Face URL whole, so the same model with and without a trailing slash got two reviews. repository_key() now groups a model as hf:<namespace>/<name>. make_result.py matches a review to a survivor as the ledger compares repositories (no trailing slash, lowercased), so it finds the review either way. - The card's YAML metadata block was stripped only when the card began with "---". It now uses huggingface_hub's repocard.REGEX_YAML_BLOCK (checked at main 62a1f2fa8383), which allows leading whitespace. A model-info response that is not a JSON object is also refused as a HubError. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…value catalogs/sota-convergence/manifest-20260926.json (landscape-sweep-20260926) names the rule id "sourcegraph-access-token" once, in a secrets-credentials refuter's reasoning. The rule's keyword then makes every 40-hex value in the file a candidate. The lane's own free-text commit ids are shortened to 12 characters in the retained evidence. The only 40-hex values left are the 39 distinct git commit ids of the 60 pins that the generator copies from the catalog layer files, and those pins come in free-form shapes such as "v0.44.0; source <id>" and "<id> (blueprints/...)". The allowlist is rule-scoped and limited to that exact file. It exempts only a finding whose secret is exactly one of those reviewed ids, pinned by value like the ai-memory fingerprints. It does not exempt a line shape: an independent review of a first, line-shaped version found that any bare 40-hex value on a pin line, such as a legacy Sourcegraph token or an uppercase variant, would have passed. An unreviewed 40-hex value on a pin line or elsewhere, the uppercase form of a reviewed id, an sgp_-prefixed token, and the reviewed ids in any other file all stay detected. Tests: test_d3 and test_d4 in tests/test_gitleaks_config.py read the pinned ids from .gitleaks.toml, because a 40-hex literal in the test file would itself be a finding. Both fail without the allowlist and pass with it. The full suite passes with GITLEAKS_TESTS_REQUIRED=1 on the pinned gitleaks 8.30.1 (27 tests, none skipped). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rst attempt Records the 2026-09-26 saturation sweep of all 32 layers (Workflow run wf_8397ada1-777, 201 Claude children at effort max, 80 GPT-6-Astra jobs) with the packaged tools/sota-convergence/landscape-sweep/ harness: - usage_record.py: child-usage complete, with 201 children and 8 superseded attempts (re-run after a Claude usage-limit pause). Three effort deviations (the property-based-testing skill's effort: low) are recorded as retained failures. - convert.py: 32 layers, 287 proposals and 56 survivors (54 repositories), with no lost or excluded rounds. 7 layers have Claude-only discovery, 9 layers have no first-round GPT-6 fit votes, and 18 layers are reopened. GPT-6 jobs: ok 64, failed_exit_null 8, failed_exit_3 6, failed_exit_1 2; copy check match 64. Nine run-specific lane limits cover the prototype harness, the cut known-repository slugs, the GPT-6 failures and their causes (the job timeline is retained in gpt6-jobs.json), the seed mapping and the 12 unsurfaced seeds, effort, the Claude usage-limit re-run, call scope, the Codex review step and the earlier attempts. - Redactions: host strings with host_receipts.sanitize(); the manifest builder's credential-name marker; free-text 40-hex commit ids shortened to 12 characters, because the file names the sourcegraph-access-token rule. - source_reviews.py: one upstream-provenance review per surviving repository (53 GitHub, 1 Hugging Face model). - build_manifest.py: catalogs/sota-convergence/manifest-20260926.json. Its Codex review step is this run's GPT-6 lanes, not a separate review. - make_result.py and saturation_ledger.py --append: landscape-sweep-20260926 (completed) after landscape-sweep-20260926-attempt-1 (wf_a874897e-af1, stopped, lower-bound usage). Reopen entries also copy the saturation report's current pin_moved and selection_changed triggers. The one-layer smoke wf_1753e674-5dc is retained as an attempt without a ledger record. - The Tavily Research cross-check is kept as discovery leads only (no report text, hashed request ids). - independent-review.json records a separate headless Opus review (read-only): needs_changes with six low findings, all repaired in this branch (one repair round). Evidence class: proposals, refuter votes and upstream source reviews. Nothing was installed and no winner changed. manifests/evidence.json is not edited here: the coordinator registers the new and changed files. With a simulated registration, saturation_ledger.py --check --base origin/main, scripts/validate.py and the harness and ledger tests pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…by absence
Repairs two defects that the review of the 2026-09-26 record found in the packaged lane.
WebSearch session cap. Claude Code allows one session at most
CLAUDE_CODE_MAX_WEB_SEARCHES_PER_SESSION WebSearch calls (default 200 since
v2.1.212), counted across the main conversation and every subagent. A capped
call returns a notice that tells the worker to go on without searching
(tools-reference "Session search limit"; env-vars; the 2.1.283 client's
default is Q=200). The lane's budgets allow 480 discovery and 640 refuter
searches, so a full sweep can pass the cap unnoticed:
- child-usage.mjs counts each child's WebSearch calls and capped calls
(web_search, from tool_use/tool_result pairs; the notice must open the
result section) and totals them per run. Neither usage nor exit codes change.
- convert.py --usage makes every capped worker a web_search_capped retained
failure of its layer (the critic's of every layer), as with effort_deviation.
- make_result.py refuses a usage record without a measured web_search, or one
whose capped workers the returns do not list as that failure.
- The harness README (Coordination), the recipe and usage_record.py say how to
raise the cap before a full sweep.
Refuted by absence. A proposal refuted only because a vote did not return
(the GPT-6 fit vote in nine layers of 2026-09-26) counted downstream as
refuted on merit: as known in later sweeps and as previous_sweep.refuted in
the next discovery input.
- convert.py marks a missing facts vote {missing: true} (fit members already
were) and names such proposals in the layer's votes_note.
- saturation_ledger.py: refutes_on_merit and refuted_by_absence read the
marker; adjudicated_repos leaves those entries out in --check and --append,
so a later re-proposal stays new. Existing records are unaffected: none
before 2026-09-26 has such an entry.
- build_inputs.py lists them under previous_sweep.not_adjudicated with a note.
Tests: node test-child-usage.mjs (46), tests.test_landscape_sweep_harness and
tests.test_saturation_ledger (one expected failure until the coordinator
registers this branch's evidence), tests.test_verdict_lane_vendoring;
SHA256SUMS updated for the two changed example files.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…bsence The repair round for the record review's two findings, rebuilt with the tools of 1bd2416b. - Usage: the three usage records were re-measured with child-usage.mjs at 1bd2416b. Every usage figure is unchanged, and each record gains web_search. In wf_8397ada1-777, 200 WebSearch calls were made, 154 returned results and 46 were capped, in 30 workers. The first capped call was at 04:10:13Z and the last at 11:55:57Z. Run 1 made 25 calls and the smoke 1, none capped. The compact attempt records copy those counts. - convert.py: counts unchanged (32 layers, 287 proposals, 56 survivors, no lost or excluded round; copy check match 64). returns.json differs from the previous record only by 61 added web_search_capped failures: 29 for workers in 16 layers and one for the critic in each of the 32 layers. All 32 layers are reopened. No layer is clean. agent-sdks, backtesting-engine and token-efficiency were clean before this repair. - Lane limits: limit 3 now states the 27 proposals refuted by absence, limit 7 gives the measured WebSearch counts, and the new limit 10 covers the session cap. The manifest was rebuilt; only its lane limits changed, and its 39 pin ids are unchanged. - Ledger: both 2026-09-26 records were re-appended onto origin/main's ledger. The records gain nine votes_note entries, the new reopen entries and a note on the cap. known/new, survived and refuted are unchanged. - README: new sections on the WebSearch session cap and on refuted by absence, a regenerated layer table, and the second review round with the rejected alternative (new GPT-6 fit votes would come from a new model run, not from this run). independent-review.json records that second round. Verification, with a simulated registration of every new and changed file: saturation_ledger.py --check --base origin/main passes, and scripts/validate.py passes. The CI-style suites pass: test_saturation_ledger, test_landscape_sweep_harness, test_verdict_lane_vendoring and test_gitleaks_config (181 tests, 2 skipped for bash 3.2 and shellcheck). node test-child-usage.mjs passes (46). The coordinator registers the files in manifests/evidence.json. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…vily lead vetting Adds two facts to the record of wf_8397ada1-777 and regenerates it with the packaged tools, as the previous rounds did. - Lane limit 11: every GPT-6-Astra lane, discovery and fit, searched Codex's cached web index, not the live web. The prototype runner started each job as codex --search exec. The #332 Codex qualification (sota-refresh-20260926/codex/results/websearch-0.155.1.json and websearch-0.157.1.json, local stand-in provider) shows --search before exec (W1) and no flag (W2) both sending external_web_access: false, and -c web_search="live" (W3) sending true. The coordinator's observation of a real 0.157.1 live search is not retained. GPT-6 discoveries and votes rested on the cached index, and the release and activity facts GPT-6 cited may lag upstream; with limit 10, no lane after 04:10:13Z had live web search except through gh api and WebFetch. Future runs pass -c web_search="live"; the codex_job.py fix follows in a separate PR. - Regeneration: convert.py with all eleven --limit values (exit 3 for the same two notes; summary identical to the previous round), redaction, id shortening, the credential-name marker and build_manifest.py (pin ids unchanged: 60 occurrences, 39 distinct). returns.json and layers.json are byte-identical; lanes.json and manifest-20260926.json differ only by the added limit. - Ledger: both 2026-09-26 records were re-appended onto origin/main's ledger (unchanged since 3822178). Only manifest_sha256 (both records), one added note on limit 11 (completed record) and the chain hashes changed. - Addendum tavily-leads-vetting.json: the lead vetting of wf_6a6cb7b8-c22 (Sonnet screen, then the sweep's facts refuter and two-family fit refuters). 136 leads in 30 layers, 100 dismissed, 36 proposed, 29 kept, 0 survivors; the Claude fit refuter refuted all 29, GPT-6 21, the facts refuter 4; 8 split cases where GPT-6 did not refute but Claude did. Discovery-completeness evidence, not a verdict, with both search limits (Claude WebSearch capped: 2 calls, both capped; GPT-6 cached). Sanitized with host_receipts.sanitize(); one ref reads <checkout>. Its 23 40-hex ids are shortened to 12 characters, as in returns.json, because the file names sourcegraph/* leads (gitleaks sourcegraph-access-token rule). Its usage record child-usage-wf_6a6cb7b8-c22.json (usage_record.py): complete, 75 children at effort max. - README: GPT-6 cached web search and lead vetting sections, limit 11, files, redaction, verification; limit 11 and the addendum are not yet reviewed. Verification, with a simulated registration of every new and changed file: saturation_ledger.py --check --base origin/main passes (4 records, chain intact) and scripts/validate.py passes. tests.test_saturation_ledger and tests.test_landscape_sweep_harness: 141 run, OK, 2 skipped (no bash 3.2, no shellcheck). tests.test_gitleaks_config and tests.test_verdict_lane_vendoring: 40 run, OK, once another scan's per-user gitleaks lock was free. Without the registration, test_the_committed_ledger_checks fails only on unregistered lane files, as before. manifests/evidence.json is not edited; the coordinator registers the files. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…res; failure coverage per worker and layer Repairs the two P2 findings of a read-only GPT-6-Astra review (max effort, live search) of the 2026-09-26 record's checkout. Each repair comes with a regression test that failed before it. (A) child-usage.mjs: a superseded attempt whose transcript held an assistant message without provider usage left the run status "complete", although by_resolved_model could not count that usage, so downstream usage could be marked complete falsely. An attempt's usage-integrity failures (an assistant message without provider usage or without a resolved model, or no transcript file) are now its usage_issues; a superseded attempt with any makes the run incomplete (the reason names them) and the CLI exit 1. Its other issues (no result entry, the <synthetic> usage-limit row outside the family) stay expected, and an attempt that made no request still counts as zero usage. usage_record.py's summary names such attempts (superseded_usage_issues). test-child-usage.mjs: four new checks failed before the repair (missing usage, unresolved model, no transcript, CLI exit); a guard for an attempt without requests passes. SHA256SUMS updated for the two changed files. (B) make_result.py: failure coverage was checked over all layers at once, so removing one layer's critic-cap failure and its reopen entry still passed (another layer's "critic" failure satisfied the check) and that layer derived a clean count of 1. effort_deviation and web_search_capped failures are now checked per worker and per layer, with the mapping convert.py records them by (deviation_rounds, moved unchanged to sweep_common.py): a <role>:<layer>[:followup] worker's failure in that layer's round, the critic's in every layer of the record; a worker naming no layer of the record is refused as before. test_failure_coverage_is_checked_per_worker_and_per_layer drops the critic's cap failure, then its effort deviation, from one of two healthy layers, and moves a worker's failure to the other layer; all three subtests failed before the repair. Docs: the harness README (evidence contract, WebSearch cap, review repairs) and the workflows README. Tests: test-child-usage.mjs 51 passed, test-usage-receipts.mjs 18, test-envelope.mjs 214, test-contract-mutations.mjs 47; test-codex-envelope.mjs and check-syntax.mjs exit 0. tests.test_saturation_ledger, tests.test_landscape_sweep_harness, tests.test_adoption_docs_consistency and tests.test_verdict_lane_vendoring: 198 run, OK (3 skipped: no bash 3.2, no shellcheck, no profile coverage change since the pinned release). manifests/evidence.json is not edited; the coordinator registers the changed files. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…und, lane limit 12 Regenerated once with the pipeline of the earlier rounds, on the tools of 7760b1da, so the manifest sha changes once. - Usage: the three usage records were re-measured with child-usage.mjs at 7760b1da (sha256 85b95813...730d). Each child_usage is identical to the 1bd2416b measurement: wf_8397ada1-777 stays complete (none of its 8 superseded attempts holds uncounted usage) and run 1 stays incomplete, so lower_bound_usage and every ledger outcome are unchanged. Only tool_commit, tool_sha256 and measured_at_utc changed. - Lane limit 12, a dated correction from the trading lane's review of #357: the execution-broker proposal of wboayue/rust-ibapi (a re-pin through nautilus_trader PR #5041) attributes #4983 to rc5. Checked with gh api: #4983's body reports Version v1.227.0; 1b0a49d2 is identical to tag v2.0.0rc5, and there crates/adapters/interactive_brokers/src/execution/ core.rs:493-495 overrides handles_order_venue to return true, so the ClientVenueMismatch denial behind crates/execution/src/engine/mod.rs:2222 cannot fire for the IB client; the open PR #280 reclassifies #4983 in catalogs/us-equities/runtime-target.json. Read instead: rc5 IBKR execution is unqualified (no rc5 stock order observed). The proposal text is kept. - convert.py with limits 1-12: exit 3 on the same two notes, summary identical. returns.json and layers.json are byte-identical; lanes.json and manifest-20260926.json differ only by the added limit (pin ids unchanged: 60 occurrences, 39 distinct). - make_result.py with the per-worker, per-layer coverage check passes on this record and gives the same RESULT.json. - Ledger: both 2026-09-26 records were re-appended onto origin/main's ledger (803bc35, unchanged since 3822178). Only manifest_sha256, usage_sha256 and the chain hashes changed. - README and independent-review.json: the third review round (GPT-6-Astra, NOT_READY, two P2 findings, both reproduced and repaired in 7760b1da) and the trading lane's correction. Verification, with a simulated registration of the 18 changed registered files: saturation_ledger.py --check --base origin/main passes (4 records, chain intact) and scripts/validate.py passes. Tests: tests.test_saturation_ledger, tests.test_landscape_sweep_harness, tests.test_adoption_docs_consistency and tests.test_verdict_lane_vendoring 198 run, OK (3 skipped: no bash 3.2, no shellcheck, no profile coverage change); test-child-usage.mjs 51 passed; tests.test_gitleaks_config 27 OK. manifests/evidence.json is not edited; the coordinator registers the changed files. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ea7e5b2 to
4e4a7ef
Compare
Review round 3 and the trading-lane correction (head 4e4a7ef)GPT-6-Astra (max, read-only on the checkout, live web search) returned NOT_READY with two P2 findings. Both were confirmed and fixed in 7760b1da, each with a regression test that fails on the pre-fix code:
Lane limit 12 (dated correction, from the trading lane's review). The
Read instead: rc5 IBKR execution is unqualified (no rc5 stock order observed); the open PR #280 reclassifies it. The model's text is kept as written. Regeneration: 🤖 Generated with Claude Code |
lane:shared: it adds
catalogs/sota-convergence/manifest-20260926.jsonand updatescatalogs/saturation/ledger.json. It needs the trading lane's acknowledgement. It is the frozen input for the coordinator's verdict wave.This PR records the 2026-09-26 saturation sweep of all 32 layers (both catalogs), which the user requested: full SOTA coverage with winners chosen on merit. The run was Workflow
wf_8397ada1-777, with 201 agents, 0 errors and 0 empty results.Lane limits (11, all in the evidence README and the manifest)
The two that matter most:
CLAUDE_CODE_MAX_WEB_SEARCHES_PER_SESSION=1500for future sessions.codex --search exec. The Workstation tool refresh: MCP Inspector 2.8.0 and Prometheus 3.15.0 switched, Codex 0.157.1 staged (daemon-safe switch plan) #332 artifacts show that this sendsexternal_web_access: false; only-c web_search="live"sends true, and GPT-6 live web search, Claude WebSearch session budget, and GPT-6/Claude family names in the blind-adjudication leak checks #352 fixes the runner.The others:
gpt6-jobs.json.lowafter loading a skill whose frontmatter sets it.Also in this PR
wf_a874897e-af1) is recorded as stopped, and the smoke run (wf_1753e674-5dc) as an attempt of its own. Nothing was dropped.child-usage.mjsno longer truncates summaries over 64 KiB, and it counts agents re-run after a usage-limit pause correctly.convert.pyturns capped workers into retained failures.source_reviews.pyhandles Hugging Face repositories..gitleaks.tomlallowlist covers the 39 reviewed catalog pin commit ids.Reviews: Claude Opus found 1 high (the WebSearch cap, not recorded before) and 1 medium (refuted-by-absence counted as adjudicated). Both were fixed in the repair round. The GPT-6 review saw only a truncated diff and was inconclusive; a read-only GPT-6 review of the checkout (live search) runs on this PR before merge.
Checks:
saturation_ledger.py --check --base origin/main: 4 sweep records, chain intact, bindings verified.validate.pypassed.test_saturation_ledger,test_landscape_sweep_harness,test_gitleaks_configandtest_adoption_docs_consistencyran 211, OK.SOTA sources
evidence/artifacts/landscape-sweep-20260926/, each pinned to the default-branch commit, with licence and README excerpts. A Hugging Face model is reviewed through the Hub API.recipes/saturation-sweep.mdand the packagedtools/sota-convergence/landscape-sweep/(Landscape-sweep harness: two-family (Claude + GPT-6) saturation sweep as a reproducible repo tool #324).websearch-*.json; Claude CodeCLAUDE_CODE_MAX_WEB_SEARCHES_PER_SESSION.🤖 Generated with Claude Code