Skip to content

v4_lens_gate root cause: batch discovery resolves (resolve-once), restore 20m timeout - #4719

Merged
briansrls merged 5 commits into
mainfrom
session/neat-crab-796
Jun 12, 2026
Merged

briansrls merged 5 commits into
mainfrom
session/neat-crab-796

Conversation

@briansrls

@briansrls briansrls commented Jun 11, 2026 •

Copy link
Copy Markdown
Contributor

Root cause

v4_lens_gate spends ~14.5 of its ~18 green minutes inside a single discover_owned_data invocation (CI log: 14m39s silent gap, e.g. run 27376554922). discover_owned_data_decls ran a full compile_to_resolved per candidate claim file — 44 files × ~19s ≈ 14 min — re-resolving nearly identical src/v4 import closures with no sharing. Cost grows linearly with every new unified_claim_* file. PR #4633 introduced this shape and bumped the job timeout 20→35 in the same commit; under runner contention or a binary-cache miss the job blows through 35 minutes, fails, and ci spot rerun re-attempts it up to 3 more times — a load feedback loop.

Fix: resolve-once batched discovery (the #4614 "resolve-once-run-many" pattern)

discover_owned_data_decls now groups candidate entries into collision-free closure groups and runs ONE compile_to_resolved over each group's merged source union. Facts are still read per entry from the typed graph via the unchanged owned_data_decls_for_entry; grouping is pure orchestration over the same resolve + extraction primitives (single-authority preserved — resolve_entry_graph_with_index and the batched path share the extracted resolved_graph_from_sources).

Safety (fail-closed at every step):

  • Grouping is conservative: entries whose closures would put two same-named top-level decls (any item kind, scanned at column 0) into one merged resolve are split into separate groups. The name scan is used ONLY for the grouping decision, never as a fact source — facts still come exclusively from the resolved graph. A false negative in the scan still trips the resolver's own registry handling; a false positive only forces a loud extra group.
  • Marker-count guard: on the merged path, each entry's resolved unified_claim_* record count must equal its column-0 data unified_claim_ marker count; a shortfall (e.g. a registry collision swallowing a decl) is a hard error instead of a silent miss.
  • --max-resolves latency ratchet (new bin flag, wired into scripts/v4-discover-owned-data.sh): if a future decl-name collision forces an extra group split, CI fails with an actionable message — the binary now prints the exact colliding decl name and the two declaring files — instead of silently re-inflating wall-time. Declared-budget-is-a-gate-error per the S3 §3.1 amendment idiom.

Collisions found and resolved

Running the ratchet at 1 surfaced two real bare-name collisions in the corpus:

  • mvp1_forbidden_authority_source_text was declared (same name, different files) in each of the five mvp1_*_add_translate.dag manual claim files. Renamed per-language (mvp1_go_…, mvp1_python_…, …) — each decl is file-local, no cross-references, no unified_claim_* records affected.
  • type OverflowDisposition is declared in BOTH src/v4/std/integer.dag and src/v4/extdeps/languages/rust.dag. Renaming/unifying that pair is a substrate modeling decision (MODELING.md M9) out of scope for a CI-latency PR, so the ratchet is set to --max-resolves 2 with a comment naming the collision; entries pulling in rust.dag resolve in their own group.

Measured (local, by execution)

  • baseline (per-entry, old binary built from the refactor-only commit): 29m22s, 113 records
  • batched: 3m11s, 2 graph resolves covering 44 entry files, 113 records — manifest + transport TSV byte-identical to baseline
  • full scripts/v4-substrate-equivalence-gate.sh --perturb-check green with the new binaries in 5m54s end-to-end (all three perturb checks still fire)

CI

Follow-ups (separate PR): timeout-raise diff gate, spot-rerun timeout-awareness, per-phase gate latency notices.

🤖 Generated with Claude Code

@briansrls
briansrls marked this pull request as ready for review June 11, 2026 23:35
briansrls pushed a commit that referenced this pull request Jun 12, 2026
Root-cause fix lands in #4719; keep #4717 design-doc only per manager.

Co-authored-by: Cursor <cursoragent@cursor.com>
@briansrls briansrls changed the title ci root cause v4_lens_gate root cause: batch discovery resolves (resolve-once), restore 20m timeout Jun 12, 2026
@briansrls
briansrls merged commit f6df210 into main Jun 12, 2026
11 checks passed
@briansrls
briansrls deleted the session/neat-crab-796 branch June 12, 2026 00:35
briansrls pushed a commit that referenced this pull request Jun 12, 2026
Root-cause batched discovery landed on main (#4719); warmth persistence
stays as the durable complement so warm reruns stay inside the 20m budget.

Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls pushed a commit that referenced this pull request Jun 12, 2026
Branch already contains main @ 1da757f (#4719 f6df210). One CI run
with real v4_lens_gate merge gate (20m, batched discovery).

Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls pushed a commit that referenced this pull request Jun 12, 2026
Supersedes branch 30m bump — #4722 timeout_budgets gate on main
declares 20m ceiling. Post-#4719 root-cause fix keeps lens CI under
budget; do not absorb via timeout raise.

Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls pushed a commit that referenced this pull request Jun 12, 2026
Cross-job gunbc-bin restore-key fallback, save-always on cargo/binary caches,
and skip full v2-compiler rebuild when gunbc is already warm from partial restore.
Complements #4719 batched-discovery root cause (20m timeout on main).

Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls pushed a commit that referenced this pull request Jun 12, 2026
Supersedes branch 30m bump — #4722 timeout_budgets gate on main
declares 20m ceiling. Post-#4719 root-cause fix keeps lens CI under
budget; do not absorb via timeout raise.

Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls added a commit that referenced this pull request Jun 12, 2026
…4691)

* WIP: Proof §1.2: dissolve 03_normalize.dag onto fold_node(normalize_algebra)

* fix(normalize): tighten fold_node dissolution — drop bloated sugar rows

The prior WIP landed data-row sugar scaffolding that grew the file; this
replaces hand-rolled child recursion with normalize_algebra + fold_node
while keeping the original sugar/classify logic and shrinking the stage.

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(normalize): thread occurrence_id via node_rebuild shell in fold

occurrence_id rides the Outcome<Node> accumulator through node_rebuild
init/step instead of explicit re-specification each rebuild. Net 223 lines
(-1 vs main baseline).

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: Proof §1.2: dissolve 03_normalize.dag onto fold_node(normalize_algebra)

* fix(normalize): dissolve sugar at every depth in fold child slot

fold_node's recursive child slot only carries structural shells; sugar
finish must run per node via normalize_node in normalize_fold_step. Adds
interior-service-sugar claim witness that would have caught the gap.

Addresses cursor REQUEST_CHANGES on PR #4691.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore(ci): retrigger after sccache infra flake on ci_floor_parity

Prior run 27377586986 failed with sccache transport failure / rustc
ctrlc handler resource exhaustion on self-hosted runner — not a PR defect.

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: Proof §1.2: dissolve 03_normalize.dag onto fold_node(normalize_algebra)

* fix(normalize): drop dead normalize_algebra; mark Q4 traverse interim

Remove unused NodeFold/normalize_algebra wrapper — init/step wire the
manual fold whose child slot calls normalize_node for per-depth sugar.
Add traverse-dissolution disposition tag matching qualified_name.dag.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore(ci): retrigger after v4_lens_gate 35m timeout on C9 step

Run 27380370110: lens witnesses passed; job cancelled at 35m21s during
v4-substrate-equivalence-gate.sh — infra wall-clock, not PR code.

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: Proof §1.2: dissolve 03_normalize.dag onto fold_node(normalize_algebra)

* fix: resolve rebase conflict in ci.yml (take main v4_lens_gate timeout)

Rebase onto main: drop superseded 276a811 50m bump; align with main 20m
timeout. Removes conflict markers from WIP commit stack.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore(ci): fresh run post-#4719 v4_lens_gate waiver lift

Branch already contains main @ 1da757f (#4719 f6df210). One CI run
with real v4_lens_gate merge gate (20m, batched discovery).

Co-authored-by: Cursor <cursoragent@cursor.com>

* Proof §1.2: extract desugar model (dissolve SurfaceSugarKind) (#4694)

* fix(#4694): rebase model extraction onto session/merry-bee-671 @ 21f366f

Stack-sync onto rebased #4691 base: extract sugar model to std/sugar.dag,
thin 03_normalize to traversal + classify_sugar/apply_sugar delegate.
Dropped stale merge commits that pulled unrelated main drift.

03_normalize: 221→124L; std/sugar.dag 163L new. Claims green.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sugar): inline edge validation in apply_sugar; drop sugar_edges_ok

Address cursor REQUEST_CHANGES: remove undispositioned Bool predicate
(practice 10 predicate dissolution). Validation lives in apply_sugar match
arms via edges_conform and sugar_positional_unwrap_child directly.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(std): rehome sugar model under std/compilers/ (#4725)

Move surface-sugar transformation authority from std/sugar.dag to
std/compilers/sugar.dag (module v4.std.compilers.sugar). Normalize
stage import updated; no semantic change.

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(#4691): restore bind clause on CP-1b-surface-sugar 🟡 mark

Cursor review 28611: relocated gate in std/compilers/sugar.dag must pair
feature: with bind TASKS.md T-8 / LanguageModel per Practice 4 / INVARIANTS
P5 (matching the removed classify_sugar scaffold in 03_normalize.dag).

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls added a commit that referenced this pull request Jun 12, 2026
…ep-graph-2026-06-11 §1) (#4699)

* WIP: W2-model-extract: fold per-connective realization ROWS from translate_al

* fix: complete W2 model-extract — drop duplicate translate helpers now in target_model

Remove the last locally-defined lex/spelling/serialize helpers from 06_translate
(imported from target_model since the prior commit) and wire target_collection_fold_list_from_node.

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: W2-model-extract: fold per-connective realization ROWS from translate_al

* WIP: W1-Conj: translate_algebra Conj arm — repoint+delete Conj traversal/fuel

* WIP: W1-Conj: translate_algebra Conj arm — repoint+delete Conj traversal/fuel

* fix: tag TypeExprTranslateAcc coproduct per modeling-discipline Practice 4

Review #4687 requested disposition mark on the new translate_algebra fold
accumulator; Slots/Done are terminal internal NodeFold states for the Conj arm.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore: retrigger CI after superseded-run cancel on shared SHA

Prior ci failure on 6ede3eb was fail-closed aggregator seeing cancelled
deps (duplicate PR #4697 on same SHA triggered superseding run); ci_floor_parity
also hit sccache EAGAIN on shared runner. No code change — local ratchet green.

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: Mgr-§1: dissolve 06_translate.dag onto fold_node (keystone cleanup, dep-

* fix(PR-T-FINISH): homing correction — fold 6 node-walkers out of target_model

M9 audit: move type_expr_spelling, serialize field-label dispatch, and
collection list-walk helpers to 06_translate (translate_algebra traversal).
Delete mis-homed copies from target_model; unify on local
type_expr_atom_binding. Add structural mark binding W2-proper row-DATA
follow-on (node://adhoc-3bb63121-471).

Pair-net probe: translate -522 / target_model +619 / sum +97 (was +127).

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore: retrigger CI after sccache EAGAIN on shared runner (aca2003)

ci_floor + ci_floor_parity failed with sccache/os error 11 (Resource
temporarily unavailable) — infra transient on srv1, not a code defect.
Local: regen_stage0 --verify OK, parity ceiling test OK.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore: retrigger CI after v4_lens_ci 20m timeout (all steps green)

v4_lens_ci run 27377877188 completed every gate step (ALL OK) then hit
the 20-minute job timeout during post-job cleanup on a cold binary cache.
No code failure — warm-cache retrigger only.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(ci): bump v4_lens_ci timeout 20→30m (cold-cache perturb runs)

Run 27379345938 on #4699 passed all node-frontier witnesses then was
cancelled at 20m during batch-executor green pass — no code failure.
Prior retrigger (c6a907c) hit the same cap after main merge slowed
perturb checks. 30m matches observed cold-cache headroom on srv1.

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: Mgr-§1: dissolve 06_translate.dag onto fold_node (keystone cleanup, dep-

* fix(ci): rename ModuleRootCandidateFold to clear C9 decl collision

discover_owned_data exceeded --max-resolves 2: PassingCandidateFold
collided between find_witness.dag and 03_name_resolve.dag once
target_model growth on #4699 expanded the discovery graph. Rename the
name_resolve-local fold (and variants) per the latency ratchet message.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(ci): restore v4_lens_ci timeout 20m (timeout_budgets gate)

Supersedes branch 30m bump — #4722 timeout_budgets gate on main
declares 20m ceiling. Post-#4719 root-cause fix keeps lens CI under
budget; do not absorb via timeout raise.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(rebase): take main 03_name_resolve.dag (#4724 supersedes rename)

#4724 collapsed PassingCandidateFold into v4.std.passing_candidate_fold.
Drop superseded ModuleRootCandidateFold rename and conflict markers from
keystone branch rebase onto main.

Co-authored-by: Cursor <cursoragent@cursor.com>

* W1: drop 64 unused imports + 2 duplicates from 06_translate.dag (residue of W2 moves to std carriers)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* WIP: W1 delete-wave executor for #4699: shrink 06_translate ~97L more (pair-s

* WIP: W1 delete-wave executor for #4699: shrink 06_translate ~97L more (pair-s

* WIP: W1 delete-wave executor for #4699: shrink 06_translate ~97L more (pair-s

* WIP: Mgr-§1: dissolve 06_translate.dag onto fold_node (keystone cleanup, dep-

* WIP: Mgr-§1: dissolve 06_translate.dag onto fold_node (keystone cleanup, dep-

* chore: retrigger CI after cancelled ready-flip run (stale ci failure on a207653)

No code change — prior run 27394327412 cancelled mid-flight; run 27394364336
was green but duplicate failing ci check on same SHA blocked dashboard.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
briansrls pushed a commit that referenced this pull request Jun 12, 2026
briansrls pushed a commit that referenced this pull request Jun 12, 2026
briansrls pushed a commit that referenced this pull request Jun 12, 2026
briansrls pushed a commit that referenced this pull request Jun 13, 2026
briansrls pushed a commit that referenced this pull request Jun 13, 2026
briansrls pushed a commit that referenced this pull request Jun 13, 2026
briansrls added a commit that referenced this pull request Jun 13, 2026
… src/v2/stage0/src/bin/claim_batch.rs currently resolves ONE --entry closure and loops functions; the v4 lens gate (scripts/v4-lens-ci-gate.sh:87) still does a cold 'gunbc run --claim-run' per row because each lens row has its own (#4783)

* WIP: Multi-entry claim_batch + lens-gate rewiring (extends #4719 pattern): sr

* WIP: Multi-entry claim_batch + lens-gate rewiring (extends #4719 pattern): sr

* fix(ci): add claim_batch to v4_lens_gate binary cache and ensure step

v4-lens-ci-gate.sh now requires claim_batch (multi-entry green pass
added in this PR), but the v4_lens_gate CI job only cached gunbc +
discover_owned_data. On a cache hit, claim_batch would be missing and
the gate would exit 2 (E-10 fail-closed regression).

Fix: add claim_batch to the cache path, bump the key namespace to
gunbc-lens-gate2-bin- (so pre-PR caches without claim_batch are never
restored), and add an Ensure claim_batch fallback step matching the
existing discover_owned_data ensure pattern.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* WIP: Multi-entry claim_batch + lens-gate rewiring (extends #4719 pattern): sr

* WIP: Multi-entry claim_batch + lens-gate rewiring (extends #4719 pattern): sr

* WIP: Multi-entry claim_batch + lens-gate rewiring (extends #4719 pattern): sr

* WIP: Multi-entry claim_batch + lens-gate rewiring (extends #4719 pattern): sr

* perf(claim_batch): lazy parse-tree cache in MultiEntryIndex — 3m11s vs 5m36s baseline

Replace eager-parse-all approach (pre-parses all 475 .dag files at index
build time — regressed to 6m33s) with demand-driven caching via
RefCell<HashMap>. Each file is tokenised and parsed at most once per
claim_batch session; the accumulated intern table grows monotonically so
cached ParseResult IDs remain valid for all subsequent entries.

Lens gate GREEN pass BEFORE/AFTER:
  baseline (no cache, cold gunbc per row): ~5m36s
  eager-parse-all (all 475 files upfront):  6m33s  [regression, not shipped]
  lazy-cache (parse on first access):        3m11s  [this commit]

~43% wall-clock reduction (2m25s off) for the 8-entry lens GREEN pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* ci: save v4 lens gate binary cache even on job failure

The binary cache key changes whenever src/v2/stage0 source files change.
Previously, if the v4_lens_gate job timed out (before cache-save ran),
the next run would get another cold build — perpetuating the timeout.

save-always: true ensures the compiled binaries are cached regardless of
job outcome, so any subsequent run on the same source tree gets a cache
hit and skips the ~4-minute cold build.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* ci: save v4 lens gate binary cache immediately after build

Replace actions/cache@v5 (restore+save in post-job phase) with an explicit
actions/cache/restore@v5 at the top and actions/cache/save@v5 immediately
after the build step. This ensures binaries are cached before the long-running
lens tests begin — so if the job is cancelled by timeout, the next run still
gets a cache hit and skips the ~4-minute cold build.

The save step is guarded by cache-hit != 'true' to skip when binaries were
already restored.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* WIP: Multi-entry claim_batch + lens-gate rewiring (extends #4719 pattern): sr

---------

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant