Repository navigation
Conversation
…kflow
A search hit names no revision: perform_workflow_source_read needs an exact
ref and canonicalize_hit takes the identity's ref from that request, so
neither acquires one. This adds the chain that does.
extdeps.github.repositories models GET /repos/{owner}/{repo} (FullRepository:
id, name, full_name, private, fork, default_branch). extdeps.github.branches
models GET /repos/{owner}/{repo}/branches/{branch} (BranchWithProtection with
commit.sha kept as the wire string, since CommitSha is an unvalidated alias).
gunbc.ci_workflow_source_read gains perform_hit_revision_resolution: hit ->
repository -> default-branch head -> exact-ref file read -> canonicalize_hit.
Each link is a total classifier the effectful func calls, with arms split by
repair: unaddressable hit, repository read refused, repository id differs from
the hit (a matching name never licenses the join), same id renamed since
indexing, branch read refused, head not an exact commit (refused before any
file is fetched at it). The file stage delegates to canonicalize_hit rather
than re-deriving its arms. The hit travels beside the identity as discovery
provenance; its indexed blob need not match the blob read and is not a refusal.
The page walk and ci_workflow_scan_producer_frontier_rows are untouched.
Witnesses: 21/21 PASS in ci_workflow_source_read_witness_test (7 new); a
three-point mutation (drop the id check, mint blob_sha from the hit, admit any
head) turns 4 claims red. dissolution_census_mechanism_witness_test 8/8 PASS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
…ealization std.object_store takes the realization-agnostic observations out of std.materialization_object (StoreFault, StoreObjectRead, StoreObjectPublish), which already described them as the store seam and never named an artifact. It adds a snapshot-named listing observation, which no materialization lookup can receive, and an immutable-record put: key-derived name, put-if-absent, and a read-back settle that answers Stored / AlreadyPresent / Conflict / OccupiedByEmptyObject / PublishRefused / ReadBackRefused. extdeps.realization.object_store_git realizes those observations on a named branch of a named remote: open by marker at a synced head, read by exact path, list by ls-tree at a commit, publish by create-new + commit + push with a bounded retry that re-reads the name after a rejected push. Founding uses `git worktree add --orphan`, so it needs no commit in the storage clone. Hermetic: test.claim.object_store_witness_test 18/18. Wet against local bare remotes with two clones: object_store_git_wet_witness_test 5/5. Regression: materialization_store_witness 22/22, materialization_store_local_wet 8/8. Mutation: disabling the byte comparison turns three conflict claims red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
…ad-discovery-prereqs
gunbc.public_workload_scan pages perform_code_search_page under an admitted WorkloadScanPolicy (queries, per_page, page bound, distinct-hit bound). Each query records its pages' delivered item counts and exactly why it stopped: pages exhausted, page limit, hit limit, truncated upstream, or page refused. Hits are distinct by (repository id, path) with sightings counted, and each distinct hit is resolved through perform_hit_revision_resolution to the default-branch head and read at that exact commit. This is the production page walk ci_workflow_scan_producer_frontier_rows named as its consumer, so the row and its census_closure_frontier entry are deleted. Hermetic: public_workload_scan_witness_test 9/9 (every stop arm plus a whole scan over the published mocks). ci_workflow_source_read_witness_test 21/21, dissolution_census_mechanism_witness_test 8/8, cloudflare_r2_token 17/17, annotation_carrier 7/7, ebay_browse 13/13. Mutation: disabling the exhaustion rule turns a_short_page_exhausts_the_query red. Not run locally (host memory): test/claim/long/dissolution_census_witness_test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
…d fail-closed The Actions model could be emitted as YAML but never read. This adds the reverse direction over the same authorities, so a public workflow's jobs can be inspected as typed evidence: - extdeps.github.actions: Job.strategy (JobStrategy / MatrixSpec with ordered axes, include, exclude), matrix_combinations implementing GitHub's product, partial-match exclude, and include-extends-or-creates semantics with the 256-combination ceiling; RunnerSpec gains RunsOnLabel for a literal label that is neither GitHub's catalog nor an expression. - extdeps.languages.yaml.gha_workflow: every new field is projected (strategy, fail-fast false, RunsOnLabel); no generated workflow changes. - extdeps.languages.yaml.gha_workflow_read (new): per-job readings with located, typed unresolved arms (dynamic matrix, expression axis, group mapping, label list without self-hosted, reusable workflow call, local or docker actions, unmodeled keys). job_read_as_job mints a Job only when nothing is short. runs-on expressions evaluate per matrix instance. - extdeps.github.expressions: one evaluator with a context resolver (evaluate_expression_in_contexts, evaluate_template_in_contexts); evaluate_expression keeps its github-only contract. - extdeps.languages.yaml.ingest: lexical comment separation (retires the inline-comment refusal), blank lines, comments at any indent, any indentation step, compact sequences, single-quoted scalars, quoted-comma-safe flow sequences and flow mappings, `|-` and `>-` block scalars, `---`, CRLF; anchors/aliases/tags, other block headers and unknown escapes refuse. - Consumers of RunnerSpec (toolchain_home_standing, runner_unit_file, runner_deploy_emit) decide the new arm explicitly, with controls. Witness: test.claim.gha_workflow_reverse_read_witness_test, including pinned real bytes from biomejs/biome, openobserve/openobserve and PowerDNS/pdns (excerpt, ranges stated in the module). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
The repository and branch reads depend only on a hit's repository, so every hit in one repository now shares one RepositoryHeadResolution: two extra requests saved per additional workflow file, and no two files of one repository read at two different heads. perform_hit_revision_resolution is kept as the composition of perform_repository_head_resolution and perform_hit_read_at_repository_head. Hermetic: public_workload_scan 9/9, ci_workflow_source_read 21/21. Wet (real GitHub, one query, per_page 5, 3 distinct hits): search, then llvm repo+branch+file, biome repo+branch once and two files; 8 requests, down from 10. Head and blob values cross-checked against gh api. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
…d-discovery-prereqs
…overy-prereqs # Conflicts: # dag/gunbc/census_closure_frontier.dag
gunbc.public_workload_evidence turns a scan's canonicalized reads into WorkflowRevisionEvidence: repository id, workflow path, exact ref and blob as the key; the repository name at observation; and the reader's per-job reading (runs-on as declared, strategy, per-matrix-instance runner, steps, unread keys). The record holds the observation, not its interpretation: architecture and provider are derived from vendor catalogs at projection time, so a catalog refresh never changes stored bytes. The reader version is in the key. Collection time is not in the record. Canonical line format, free text escaped with percent_encode_payload (moved to std.object_store so both line formats share one authority); runners and matrix selectors are rendered by the existing YAML emitter. Harvest partitions every distinct hit into stored / text unreadable / not reached. extdeps.github.repository_contents repository_file_text decodes the endpoint's line-wrapped base64 and refuses when the decoded octet count disagrees with the declared size. HitCanonicalized now carries the file it canonicalized, so interpretation never needs a second read. Hermetic: public_workload_evidence_witness_test 8/8 over the pinned Biome and OpenObserve bytes. Regression on a binary rebuilt from the merged tree: ci_workflow_source_read 21/21, materialization_store 23/23, object_store 18/18, public_workload_scan 9/9. Mutation: removing field escaping turns the line-forgery and instance-line claims red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
…hortlist gunbc.public_workload_discovery runs one bounded discovery: - queries are the distinct Aarch64 labels on the runs-on surface of gunbc.runner_provider_survey (29 today), windowed by offset/count because code search is rate limited far below the roster; - every hit is read at its repository's exact head and stored as a WorkflowRevisionEvidence record in a caller-named git object store; - a report names each repository with a catalog-confirmed ARM job instance (job, matrix combination, label, provider, commit, blob, storage outcome), its unknowns, everything examined without one, and the scan's per-query stops. Architecture is read from the catalogs at run time, never stored. initialize_public_workload_evidence_store is the only thing that founds a store; discover_public_workloads refuses an unfounded one. The run exits non-zero if any evidence record was not stored. Hermetic: public_workload_discovery_witness_test 4/4 (roster, window, which runners are looked up, Biome main tallied to its five Depot ARM instances); public_workload_evidence_witness_test 8/8. Mutation: dropping confirmed ARM instances turns the Biome tally red. Wet, real GitHub into a local bare remote, queries 0..3: 116 REST reads, 50 workflow revisions stored, 12 repositories with catalog-confirmed ARM jobs. Rerun of the same window: exit 0, all 50 already present, store still 50 records and 51 commits, shortlist identical apart from the timestamp. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
Review of the stored format found it was a lossy interpretation presented as the observation (matrix axes, step fields, unread key values and workflow-level fields were dropped, and the reader version was in the key). Amended: - workflow-source-v1 records hold the exact file bytes (canonical base64), keyed by repository id, commit, path and blob, with no reader in the key. Interpretation is not persisted: it is a pure function of stored bytes and the current reader, recomputed at projection time. - scan-attempt-v1 records hold when the scan ran, each query's bounds, pages and stop, every individual sighting (query, page, position, indexed blob, admitted or declined by bound), and each distinct file's outcome naming the source record it produced. - Each query has its own page and distinct-hit bounds; a shared budget no longer starves later queries. Files found by several queries are read once. - Non-UTF-8 bytes are stored; std.encoding utf8_octets_well_formed (RFC 3629) makes text interpretation refuse per file instead of aborting the run. - Record names carry a family prefix so a listing separates record kinds. The live label run exposed std.encoding base64 decode recursing once per character: a ~75 KB workflow file hit the 100,000-frame call-depth bound and aborted the scan. Encode and decode were also quadratic (a 140 KB round trip took 855 s CPU). Both now index with get and scale linearly (4.6 s for 140 KB); RFC 4648 vectors unchanged. Witnesses: public_workload_scan 10/10, public_workload_evidence 9/9, public_workload_discovery 4/4, object_store 18/18, base64_rfc4648 11/11, npm_sri 13/13, random_bytes_csprng 4/4 (wet; hermetic route has no ReadBytes mock), long/base64_large_payload 1/1 (4.9 s, hence long/). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
…e private The bounded scan walk, the workflow-source and scan-attempt record families, and the discovery entry point are acquisition policy, so they leave the public tree for the private overlay (strategy.ci_workflow_scan, strategy.ci_workflow_corpus, strategy.ci_workload_collection). This PR keeps only reusable substrate: the immutable object store and its git realization, UTF-8 and linear base64, the GitHub repository/branch/contents reads, the Actions matrix model and reverse workflow reader, and the exact-revision join. ci_workflow_scan_producer_frontier_rows is restored and re-enrolled in the frontier census: the public tree cannot see a private consumer, so no public production caller walks these reads. The row now names the repository-head split functions and says a private-overlay consumer does not dissolve it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
The merge from main brought #10850, which removed func as an item keyword; the object store git realization, its wet witness and the exact-revision reads still declared func items and no longer parsed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
…lane Every function in object_store_git_wet_witness_test builds a THROWAWAY LOCAL REPOSITORY and runs git against it, which the hermetic envelope refuses - so under the floor each one is planned and never reaches its subject, reported as an unenrolled route gap rather than a verdict. The file is excluded from the discovery corpus and its five identities are scheduled on the local-repo wet lane, where the floor runs each member wet and joins the observed terminals back at identity grain, so a member that stops executing refuses rather than reading as covered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…overy-prereqs # Conflicts: # src/v2/workflow/local_repo_wet_terminal.dag
|
CI state at Fixed here: the five Two gates still refuse:
🤖 Generated with Claude Code |
|
Closing unmerged to wind down the session. Tracked in #11400, which carries the branch and head ( |
Shared, reusable substrate for reading GitHub workflows at exact revisions and storing immutable records. The acquisition scan, corpus and collector that consume this live in the private overlay (companion
gunbc-privatePR). This PR has no query roster, scan policy, record family, shortlist or collection entry point.What lands
std.object_store+extdeps.realization.object_store_gitStoreFault,StoreObjectReadandStoreObjectPublishmove here fromstd.materialization_object; they never named an artifact.StoreObjectListingnames the snapshot it lists; no materialization lookup can receive it.ls-treeat a commit, where a failure is unavailable, never an empty store;git worktree add --orphan.std.encodingutf8_octets_well_formed(RFC 3629) decides UTF-8 before the host decode, whose refusal stops a run.GitHub reads
extdeps.github.repositoriesandextdeps.github.branchesare new.repository_contentsgetsrepository_file_bytes, which handles the endpoint's line-wrapped base64 and refuses a size mismatch, andrepository_octets_text.Actions model and reverse reader (
extdeps.github.actions,extdeps.languages.yaml.gha_workflow_read,ingest.dag)Job.strategywith static matrices and GitHub's include/exclude semantics, plusRunnerSpec.RunsOnLabel.gunbc.ci_workflow_source_readHitCanonicalizedcarries the bytes it canonicalized.ci_workflow_scan_producer_frontier_rowsstays enrolled: no public production caller walks these reads, and a private-overlay consumer does not dissolve it.Verification (local, by execution, on this head merged with main)
ci_workflow_source_read21/21,dissolution_census_mechanism8/8,object_store18/18,base64_rfc464811/11,gha_workflow_reverse_read19/19 (real Biome / OpenObserve / PowerDNS bytes),materialization_store23/23.object_store_git_wet5/5 (local bare remote, two clones),materialization_store_local_wet8/8,random_bytes_csprng4/4 (its hermetic route has no ReadBytes mock).long/base64_large_payload1/1 (4.9 s).gh api). That is one live observation, not the capture-and-replay admissionrest_success_and_decode_are_separately_trueasks for.d7b7ab96c1f.Not run locally
test/claim/long/dissolution_census_witness_test(host memory).🤖 Generated with Claude Code
https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk