Skip to content

Workflow-reading and immutable-storage substrate: object store, exact-revision GitHub reads, Actions reverse reader, linear base64 - #11270

Closed
briansrls wants to merge 16 commits into
mainfrom
session/workload-discovery-prereqs
Closed

briansrls wants to merge 16 commits into
mainfrom
session/workload-discovery-prereqs

Conversation

@briansrls

@briansrls briansrls commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Shared, reusable substrate for reading GitHub workflows at exact revisions and storing immutable records. The acquisition scan, corpus and collector that consume this live in the private overlay (companion gunbc-private PR). This PR has no query roster, scan policy, record family, shortlist or collection entry point.

What lands

std.object_store + extdeps.realization.object_store_git

  • Shared store observations. StoreFault, StoreObjectRead and StoreObjectPublish move here from std.materialization_object; they never named an artifact.
  • New observations and helpers:
    • StoreObjectListing names the snapshot it lists; no materialization lookup can receive it.
    • Immutable-record put: the name comes from family and key, the write is put-if-absent, and a read-back settles Stored / AlreadyPresent / Conflict / OccupiedByEmptyObject / PublishRefused / ReadBackRefused.
    • Line-field percent-encoding, moved here as one authority.
  • Git-branch realization on a caller-named remote and branch:
    • opens on a marker read at a synced head;
    • lists via ls-tree at a commit, where a failure is unavailable, never an empty store;
    • publishes by create-new, commit and push, with a bounded retry that re-reads the name;
    • founds the branch with git worktree add --orphan.

std.encoding

  • utf8_octets_well_formed (RFC 3629) decides UTF-8 before the host decode, whose refusal stops a run.
  • Base64 encode and decode are now indexed rather than recursive or accumulating. The decoder recursed once per character, so a real 75 KB workflow file hit the 100,000-frame call-depth limit. Both were also quadratic: a 140 KB round trip took 855 s of CPU and now takes 4.6 s. RFC 4648 vectors unchanged.

GitHub reads

  • extdeps.github.repositories and extdeps.github.branches are new.
  • repository_contents gets repository_file_bytes, which handles the endpoint's line-wrapped base64 and refuses a size mismatch, and repository_octets_text.

Actions model and reverse reader (extdeps.github.actions, extdeps.languages.yaml.gha_workflow_read, ingest.dag)

  • Job.strategy with static matrices and GitHub's include/exclude semantics, plus RunnerSpec.RunsOnLabel.
  • runs-on is evaluated per matrix instance.
  • Every unsupported shape is a located, unresolved outcome.
  • The YAML ingest handles the syntax real workflows use.

gunbc.ci_workflow_source_read

  • A search hint joins to the repository, its default-branch head, and the file read at that exact commit, refusing on any repository-id disagreement.
  • Each repository's head is resolved once.
  • HitCanonicalized carries the bytes it canonicalized.
  • ci_workflow_scan_producer_frontier_rows stays enrolled: no public production caller walks these reads, and a private-overlay consumer does not dissolve it.

Verification (local, by execution, on this head merged with main)

  • Hermetic: ci_workflow_source_read 21/21, dissolution_census_mechanism 8/8, object_store 18/18, base64_rfc4648 11/11, gha_workflow_reverse_read 19/19 (real Biome / OpenObserve / PowerDNS bytes), materialization_store 23/23.
  • Wet: object_store_git_wet 5/5 (local bare remote, two clones), materialization_store_local_wet 8/8, random_bytes_csprng 4/4 (its hermetic route has no ReadBytes mock).
  • Long: long/base64_large_payload 1/1 (4.9 s).
  • Mutation checks: byte comparison, repository-id join and exact-head check each turn their claims red.
  • Live GitHub: search, repository, branch and contents reads decoded to real values (head and blob matched gh api). That is one live observation, not the capture-and-replay admission rest_success_and_decode_are_separately_true asks for.
  • Direct importers of the reader's modules: the 60 failures also fail, by name, on base d7b7ab96c1f.

Not run locally

  • test/claim/long/dissolution_census_witness_test (host memory).
  • The full transitive consumer closure.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk

gunbc-ci-auto-heal and others added 8 commits September 13, 2026 04:28
…kflow

A search hit names no revision: perform_workflow_source_read needs an exact
ref and canonicalize_hit takes the identity's ref from that request, so
neither acquires one. This adds the chain that does.

extdeps.github.repositories models GET /repos/{owner}/{repo} (FullRepository:
id, name, full_name, private, fork, default_branch). extdeps.github.branches
models GET /repos/{owner}/{repo}/branches/{branch} (BranchWithProtection with
commit.sha kept as the wire string, since CommitSha is an unvalidated alias).

gunbc.ci_workflow_source_read gains perform_hit_revision_resolution: hit ->
repository -> default-branch head -> exact-ref file read -> canonicalize_hit.
Each link is a total classifier the effectful func calls, with arms split by
repair: unaddressable hit, repository read refused, repository id differs from
the hit (a matching name never licenses the join), same id renamed since
indexing, branch read refused, head not an exact commit (refused before any
file is fetched at it). The file stage delegates to canonicalize_hit rather
than re-deriving its arms. The hit travels beside the identity as discovery
provenance; its indexed blob need not match the blob read and is not a refusal.

The page walk and ci_workflow_scan_producer_frontier_rows are untouched.

Witnesses: 21/21 PASS in ci_workflow_source_read_witness_test (7 new); a
three-point mutation (drop the id check, mint blob_sha from the hit, admit any
head) turns 4 claims red. dissolution_census_mechanism_witness_test 8/8 PASS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
…ealization

std.object_store takes the realization-agnostic observations out of
std.materialization_object (StoreFault, StoreObjectRead, StoreObjectPublish),
which already described them as the store seam and never named an artifact.
It adds a snapshot-named listing observation, which no materialization lookup
can receive, and an immutable-record put: key-derived name, put-if-absent, and
a read-back settle that answers Stored / AlreadyPresent / Conflict /
OccupiedByEmptyObject / PublishRefused / ReadBackRefused.

extdeps.realization.object_store_git realizes those observations on a named
branch of a named remote: open by marker at a synced head, read by exact path,
list by ls-tree at a commit, publish by create-new + commit + push with a
bounded retry that re-reads the name after a rejected push. Founding uses
`git worktree add --orphan`, so it needs no commit in the storage clone.

Hermetic: test.claim.object_store_witness_test 18/18. Wet against local bare
remotes with two clones: object_store_git_wet_witness_test 5/5. Regression:
materialization_store_witness 22/22, materialization_store_local_wet 8/8.
Mutation: disabling the byte comparison turns three conflict claims red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
gunbc.public_workload_scan pages perform_code_search_page under an admitted
WorkloadScanPolicy (queries, per_page, page bound, distinct-hit bound). Each
query records its pages' delivered item counts and exactly why it stopped:
pages exhausted, page limit, hit limit, truncated upstream, or page refused.
Hits are distinct by (repository id, path) with sightings counted, and each
distinct hit is resolved through perform_hit_revision_resolution to the
default-branch head and read at that exact commit.

This is the production page walk ci_workflow_scan_producer_frontier_rows named
as its consumer, so the row and its census_closure_frontier entry are deleted.

Hermetic: public_workload_scan_witness_test 9/9 (every stop arm plus a whole
scan over the published mocks). ci_workflow_source_read_witness_test 21/21,
dissolution_census_mechanism_witness_test 8/8, cloudflare_r2_token 17/17,
annotation_carrier 7/7, ebay_browse 13/13. Mutation: disabling the exhaustion
rule turns a_short_page_exhausts_the_query red. Not run locally (host memory):
test/claim/long/dissolution_census_witness_test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
…d fail-closed

The Actions model could be emitted as YAML but never read. This adds the
reverse direction over the same authorities, so a public workflow's jobs can
be inspected as typed evidence:

- extdeps.github.actions: Job.strategy (JobStrategy / MatrixSpec with ordered
  axes, include, exclude), matrix_combinations implementing GitHub's product,
  partial-match exclude, and include-extends-or-creates semantics with the
  256-combination ceiling; RunnerSpec gains RunsOnLabel for a literal label
  that is neither GitHub's catalog nor an expression.
- extdeps.languages.yaml.gha_workflow: every new field is projected (strategy,
  fail-fast false, RunsOnLabel); no generated workflow changes.
- extdeps.languages.yaml.gha_workflow_read (new): per-job readings with
  located, typed unresolved arms (dynamic matrix, expression axis, group
  mapping, label list without self-hosted, reusable workflow call, local or
  docker actions, unmodeled keys). job_read_as_job mints a Job only when
  nothing is short. runs-on expressions evaluate per matrix instance.
- extdeps.github.expressions: one evaluator with a context resolver
  (evaluate_expression_in_contexts, evaluate_template_in_contexts);
  evaluate_expression keeps its github-only contract.
- extdeps.languages.yaml.ingest: lexical comment separation (retires the
  inline-comment refusal), blank lines, comments at any indent, any
  indentation step, compact sequences, single-quoted scalars, quoted-comma-safe
  flow sequences and flow mappings, `|-` and `>-` block scalars, `---`, CRLF;
  anchors/aliases/tags, other block headers and unknown escapes refuse.
- Consumers of RunnerSpec (toolchain_home_standing, runner_unit_file,
  runner_deploy_emit) decide the new arm explicitly, with controls.

Witness: test.claim.gha_workflow_reverse_read_witness_test, including pinned
real bytes from biomejs/biome, openobserve/openobserve and PowerDNS/pdns
(excerpt, ranges stated in the module).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
The repository and branch reads depend only on a hit's repository, so every
hit in one repository now shares one RepositoryHeadResolution: two extra
requests saved per additional workflow file, and no two files of one
repository read at two different heads. perform_hit_revision_resolution is
kept as the composition of perform_repository_head_resolution and
perform_hit_read_at_repository_head.

Hermetic: public_workload_scan 9/9, ci_workflow_source_read 21/21.
Wet (real GitHub, one query, per_page 5, 3 distinct hits): search, then
llvm repo+branch+file, biome repo+branch once and two files; 8 requests, down
from 10. Head and blob values cross-checked against gh api.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
…overy-prereqs

# Conflicts:
#	dag/gunbc/census_closure_frontier.dag
gunbc-ci-auto-heal and others added 6 commits September 13, 2026 12:32
gunbc.public_workload_evidence turns a scan's canonicalized reads into
WorkflowRevisionEvidence: repository id, workflow path, exact ref and blob as
the key; the repository name at observation; and the reader's per-job reading
(runs-on as declared, strategy, per-matrix-instance runner, steps, unread
keys). The record holds the observation, not its interpretation: architecture
and provider are derived from vendor catalogs at projection time, so a catalog
refresh never changes stored bytes. The reader version is in the key.
Collection time is not in the record.

Canonical line format, free text escaped with percent_encode_payload (moved to
std.object_store so both line formats share one authority); runners and
matrix selectors are rendered by the existing YAML emitter. Harvest partitions
every distinct hit into stored / text unreadable / not reached.

extdeps.github.repository_contents repository_file_text decodes the endpoint's
line-wrapped base64 and refuses when the decoded octet count disagrees with
the declared size. HitCanonicalized now carries the file it canonicalized, so
interpretation never needs a second read.

Hermetic: public_workload_evidence_witness_test 8/8 over the pinned Biome and
OpenObserve bytes. Regression on a binary rebuilt from the merged tree:
ci_workflow_source_read 21/21, materialization_store 23/23, object_store 18/18,
public_workload_scan 9/9. Mutation: removing field escaping turns the
line-forgery and instance-line claims red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
…hortlist

gunbc.public_workload_discovery runs one bounded discovery:
- queries are the distinct Aarch64 labels on the runs-on surface of
  gunbc.runner_provider_survey (29 today), windowed by offset/count because
  code search is rate limited far below the roster;
- every hit is read at its repository's exact head and stored as a
  WorkflowRevisionEvidence record in a caller-named git object store;
- a report names each repository with a catalog-confirmed ARM job instance
  (job, matrix combination, label, provider, commit, blob, storage outcome),
  its unknowns, everything examined without one, and the scan's per-query
  stops. Architecture is read from the catalogs at run time, never stored.

initialize_public_workload_evidence_store is the only thing that founds a
store; discover_public_workloads refuses an unfounded one. The run exits
non-zero if any evidence record was not stored.

Hermetic: public_workload_discovery_witness_test 4/4 (roster, window, which
runners are looked up, Biome main tallied to its five Depot ARM instances);
public_workload_evidence_witness_test 8/8. Mutation: dropping confirmed ARM
instances turns the Biome tally red.

Wet, real GitHub into a local bare remote, queries 0..3: 116 REST reads, 50
workflow revisions stored, 12 repositories with catalog-confirmed ARM jobs.
Rerun of the same window: exit 0, all 50 already present, store still 50
records and 51 commits, shortlist identical apart from the timestamp.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
Review of the stored format found it was a lossy interpretation presented as
the observation (matrix axes, step fields, unread key values and workflow-level
fields were dropped, and the reader version was in the key). Amended:

- workflow-source-v1 records hold the exact file bytes (canonical base64),
  keyed by repository id, commit, path and blob, with no reader in the key.
  Interpretation is not persisted: it is a pure function of stored bytes and
  the current reader, recomputed at projection time.
- scan-attempt-v1 records hold when the scan ran, each query's bounds, pages
  and stop, every individual sighting (query, page, position, indexed blob,
  admitted or declined by bound), and each distinct file's outcome naming the
  source record it produced.
- Each query has its own page and distinct-hit bounds; a shared budget no
  longer starves later queries. Files found by several queries are read once.
- Non-UTF-8 bytes are stored; std.encoding utf8_octets_well_formed (RFC 3629)
  makes text interpretation refuse per file instead of aborting the run.
- Record names carry a family prefix so a listing separates record kinds.

The live label run exposed std.encoding base64 decode recursing once per
character: a ~75 KB workflow file hit the 100,000-frame call-depth bound and
aborted the scan. Encode and decode were also quadratic (a 140 KB round trip
took 855 s CPU). Both now index with get and scale linearly (4.6 s for 140 KB);
RFC 4648 vectors unchanged.

Witnesses: public_workload_scan 10/10, public_workload_evidence 9/9,
public_workload_discovery 4/4, object_store 18/18, base64_rfc4648 11/11,
npm_sri 13/13, random_bytes_csprng 4/4 (wet; hermetic route has no ReadBytes
mock), long/base64_large_payload 1/1 (4.9 s, hence long/).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
…e private

The bounded scan walk, the workflow-source and scan-attempt record families,
and the discovery entry point are acquisition policy, so they leave the public
tree for the private overlay (strategy.ci_workflow_scan,
strategy.ci_workflow_corpus, strategy.ci_workload_collection). This PR keeps
only reusable substrate: the immutable object store and its git realization,
UTF-8 and linear base64, the GitHub repository/branch/contents reads, the
Actions matrix model and reverse workflow reader, and the exact-revision join.

ci_workflow_scan_producer_frontier_rows is restored and re-enrolled in the
frontier census: the public tree cannot see a private consumer, so no public
production caller walks these reads. The row now names the repository-head
split functions and says a private-overlay consumer does not dissolve it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
The merge from main brought #10850, which removed func as an item keyword;
the object store git realization, its wet witness and the exact-revision
reads still declared func items and no longer parsed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yb9agVcWYWzjhPsJ42XQAk
@briansrls briansrls changed the title Public workload discovery prerequisites: immutable object store, exact-revision GitHub reads, bounded scan, workflow reader Workflow-reading and immutable-storage substrate: object store, exact-revision GitHub reads, Actions reverse reader, linear base64 Sep 13, 2026
gunbc-ci-auto-heal and others added 2 commits September 14, 2026 16:36
…lane

Every function in object_store_git_wet_witness_test builds a THROWAWAY LOCAL REPOSITORY and runs git
against it, which the hermetic envelope refuses - so under the floor each one is planned and never
reaches its subject, reported as an unenrolled route gap rather than a verdict. The file is excluded
from the discovery corpus and its five identities are scheduled on the local-repo wet lane, where the
floor runs each member wet and joins the observed terminals back at identity grain, so a member that
stops executing refuses rather than reading as covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…overy-prereqs

# Conflicts:
#	src/v2/workflow/local_repo_wet_terminal.dag
@briansrls

Copy link
Copy Markdown
Contributor Author

CI state at 3e8ce5c4217 (wet-lane enrolment + current main merged).

Fixed here: the five object_store_git_wet_witness_test identities build a throwaway local repository, which the hermetic envelope refuses, so the floor planned them and they never reached their subject — reported as unenrolled route gaps rather than verdicts. They are now excluded from the discovery corpus and scheduled on the local-repo wet lane at identity grain.

Two gates still refuse:

  1. namespace-wave-admission — 55 unadjudicated deltas, largely TargetChanged bindings moving from std.materialization_object to std.object_store. That is this PR's own move, and it needs authored admission rows (roster const in src/v1/stage0/src/namespace_wave_admission.rs), each naming the exact expected head candidate set.
  2. floor — refused with subject=fe461f61ee993b00 modules_resolved=2970 modules_excluded=2; I did not isolate the cause.

heal-generated-artifacts and required-witnesses-build pass.

🤖 Generated with Claude Code

@briansrls

Copy link
Copy Markdown
Contributor Author

Closing unmerged to wind down the session. Tracked in #11400, which carries the branch and head (session/workload-discovery-prereqs at 3e8ce5c4217), what the substrate delivers, and the two gates that refuse — the 55 wave deltas this branch genuinely owes, and an unisolated floor refusal. The branch is left in place.

@briansrls briansrls closed this Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant