Skip to content

Affected-set CI scoping: Move C (exclusion model) + Move B (execution corpus) - #5962

Merged
briansrls merged 26 commits into
mainfrom
session/merry-fox-653
Jun 30, 2026
Merged

briansrls merged 26 commits into
mainfrom
session/merry-fox-653

Conversation

@briansrls

@briansrls briansrls commented Jun 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Three moves to scope the CI floor's discovery so heavy execution witnesses run in a parallel-capped pool instead of serializing in the main corpus:

Move C (done): Migrate `FLOOR_DISCOVERY_EXCLUDES` to the `.dag` model as `witness_exclusion_substrings` in `dsl/gunbc/ci_layer_roots.dag` — the single authority for path exclusions. The Rust constant is now a named SCAFFOLD with a dissolution trigger. Added `ci_exclusion_proof_test.dag` (always-false sentinel) to prove the plan's exclusion path actually gates.

Move B (done): Split all 21 execution-class witnesses (classified by behavior: they execute programs/the compiler through the v2 pipeline) into `src/v2/test/claim/execution/` and expose them as a separate `execution_corpus_node` with `spawn_width_cap` derived from available headroom. Extended `RunnableDiscoveryBatch` with `exclude_substrings`, `discovery_scope_dirs`, and `spawn_width_cap` fields; updated the generated `std_realization_schedule.rs` to match.

Move A (tracked follow-on): Affected-set skip predicate — when the diff ∩ witness import closure is empty, skip the witness entirely. Separate PR after this merges.

Execution corpus scheduling and memory provisioning

`execution_corpus_node` runs after `floor_corpus_node` (not co-resident): a `ResourceDependsOn(floor → execution)` edge is added in `floor_corpus_exclusion_resource_edges`. Heavy gates (emit_host, regen_verify, source_root_ingest) depend on both corpus nodes, so they land in batch 3.

Batch structure:

  • batch 0: compile gate
  • batch 1: floor corpus (width capped by floor memory formula)
  • batch 2: execution corpus (sequential after floor)
  • batch 3: heavy gates + source_root_ingest (after both corpora)

`gunbc_ci_execution_corpus_available_width` computes headroom for execution running alone (no floor co-residency): (8589934592 − 1610612736) / 2147483648 = spawn_width_cap = 3.

Effective width today = min(plan_spawn_width=1, spawn_width_cap=3) = 1. `claim_executor` receives `spawn_width = gunbc_ci_floor_spawn_width` which currently returns the floor-only derived width (1). Wiring `gunbc_ci_floor_spawn_width` to `max(floor_width, exec_width)` — so the plan permits width 3 during the execution phase — is tracked as follow-up: plan-width binding for execution-corpus width-3 (node://adhoc-e188369a-643). That follow-up also renames `gunbc_ci_floor_spawn_width → gunbc_ci_plan_spawn_width` (misnomer: it returns the cross-corpus max, not the floor's width) and replaces `witness_floor_derived_width_at_runner_cap` with a real per-corpus memory-safety guard.

Move B standalone value: (a) enabling Move A's affected-set skipping on an isolated corpus node; (b) honest fail-closed provisioning with named dissolution trigger; (c) OOM prevention from sequential scheduling (no exit-137).

Witnesses moved to execution/

4 already there (self_host/, fold_assembly/) + 17 newly moved from manual/:
`add_arrow_eval_by_execution`, `bind_eval_by_execution`, `bind_round_trip`, `branch_eval_by_execution`, `branch_if_then_else_round_trip`, `dag_add_emit_round_trip`, `effect_io_eval_by_execution`, `emit_ingest_grammar_relation_round_trip`, `emit_ingest_python_same_language_round_trip`, `emit_ingest_type_decl_round_trip`, `emit_ingest_typescript_same_language_round_trip`, `loop_eval_by_execution`, `loop_round_trip`, `match_bool_round_trip`, `typescript_closure_emit_by_execution`, `typescript_enum_union_emit_by_execution`, `typescript_import_emit_by_execution`

Module paths updated from `v2.test.manual.` → `v2.test.execution.`. Importers in manual/ (bind_emit, branch_if_then_else_emit, loop_emit, match_bool_emit) and in execution/ (self_host_candidate_generation) updated.

Test plan

  • CI floor: `claim_executor --source-root src/v2 --source-root dsl --plan-entry src/v2/workflow/ci_floor_plan.dag --plan-function gunbc_ci_floor_batches` — execution corpus runs as separate batch-2 node with spawn_width_cap=3 (effective=1 pending plan-width follow-up node://adhoc-e188369a-643); no exit-137 from sequential scheduling
  • `regen_verify_gate`: `std_realization_schedule.rs` updated to match 8-field `RunnableDiscoveryBatch`; confirmed by CI batch-4
  • `ci_exclusion_proof_test.dag` always-false sentinel proves exclusion path is live

Brian Searls and others added 2 commits June 29, 2026 18:17
…ngs in ci_layer_roots.dag (§3 single authority)

- Add `witness_exclusion_substrings: List<String>` to `dsl/gunbc/ci_layer_roots.dag` — single authority for path-substring exclusions
- Add `exclude_substrings: List<String>` field to `RunnableDiscoveryBatch` in `dsl/std/realization_schedule.dag`; update all pattern-match sites and `runnable_eq`
- Wire `witness_exclusion_substrings` into `corpus_runnable()` in `ci_floor_plan.dag` via the new field
- Thread `exclude_substrings` through `DiscoveryCorpusOptions`, `discover_floor_corpus_rows`, `claim_executor`, `claim_batch`, and test call sites; plan-driven paths now read from the model value, not the Rust constant
- Mark `FLOOR_DISCOVERY_EXCLUDES` SCAFFOLD (dissolves when all callers use the model-derived path)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review June 29, 2026 18:27
@gunbai-bot gunbai-bot Bot changed the title DESIGN SKETCH ONLY (no code, returns to stern-moth-225 + operator for sign-off before any implementation): affected-set CI scoping for the floor. PROBLEM: heavy self-host execution witnesses (comprep_*/mvp1_*/*_round_trip, ~11 at 20s+, one was 537s) auto-discover into a FLAT SERIAL corpus, so every Move C: migrate FLOOR_DISCOVERY_EXCLUDES to witness_exclusion_substrings in ci_layer_roots.dag (§3 single authority) Jun 29, 2026
@gunbai-bot

gunbai-bot Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Two fixes landed in d27f0cf in response to the §3/§5 checks:

1. None fallback removed — constant is now inert on the plan path

In claim_executor.rs, the None => FLOOR_DISCOVERY_EXCLUDES.iter()... fallback in runnable_from_value was the remaining live read of the constant on the floor path. Changed to None => vec![]: if a plan's RunnableDiscoveryBatch omits exclude_substrings, no exclusions apply (the plan author chose none — not a fallback to the Rust constant). The import of FLOOR_DISCOVERY_EXCLUDES was also removed from claim_executor.rs entirely, so the constant is structurally unreachable from the plan-driven path.

Remaining live reads of the constant: claim_batch.rs (pre-push hooks only, not the floor runner), DiscoveryCorpusOptions::default() (test call sites in pipeline.rs and wet_hermetic_equivalence_test.rs). These are non-floor paths; the SCAFFOLD dissolution covers them.

2. Discriminating probe — §5 by-execution proof that model is the authority

Added "ci_exclusion_proof_test.dag" to witness_exclusion_substrings in ci_layer_roots.dag (NOT added to FLOOR_DISCOVERY_EXCLUDES). Created dsl/test/claim/ci_exclusion_proof_test.dag with test fn ci_exclusion_proof_sentinel() -> Bool = false — a test that always fails if discovered.

The two authorities now intentionally differ on one entry. CI green-by-execution is the proof: the probe is excluded (→ CI passes) only because the model is doing the exclusion. If the constant were the authority (it lacks the sentinel), the probe would be discovered and CI would fail red.

Removing "ci_exclusion_proof_test.dag" from the .dag would re-include the probe and red CI — this is the RED control. The probe file is now a permanent fail-closed sentinel for the exclusion mechanism.

— sent from merry-fox-653

Brian Searls and others added 3 commits June 29, 2026 18:39
… pass

- floor_skip_discovery_host_test.rs: DiscoveryCorpusOptions struct literal
  missing new exclude_substrings field — use ..Default::default() to pick up
  the FLOOR_DISCOVERY_EXCLUDES-backed default
- claim_executor.rs: vec![] in None arm of exclude_substrings match was
  ambiguous — change to Vec::new() so rustc can infer the String element type
- cargo fmt --all pass (claim_batch.rs, cli_run.rs, pipeline.rs, wet_hermetic)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ed parallel width

- Move 5 execution witnesses (self_host/fixed_point, self_host/candidate_generation,
  self_host/compiler_closure_emit, fold_assembly/round_trip, complexity_gate/comprep_bridge)
  into src/v2/test/claim/execution/ with updated module declarations
- Add execution_corpus_node to ci_floor_plan.dag: RunnableDiscoveryBatch with
  discovery_scope_dirs scoped to execution/ dir and spawn_width_cap derived from
  gunbc_ci_runner_cgroup_memory_cap / 2 GiB per witness reservation
- Extend RunnableDiscoveryBatch type + Rust BatchUnit/DiscoveryBatch with
  discovery_scope_dirs and spawn_width_cap; apply cap at run_batch_unit level
- Add ci_exclusion_proof_test.dag to FLOOR_DISCOVERY_EXCLUDES (Rust mirror) to
  prevent regression on claim_batch pre-push path (sentinel already in .dag model)
- Update floor_nodes, floor_resource_anchor_node, runnable_for_node, floor_data_dependencies
  and the RunnableDiscoveryBatch pattern in floor_batch_heavy_gate_resolve_count

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Addressing the cursor/composer-2.5 review findings:

Finding 1 (sentinel not in FLOOR_DISCOVERY_EXCLUDES): Valid. Added \"ci_exclusion_proof_test.dag\" to FLOOR_DISCOVERY_EXCLUDES in this commit so the claim_batch pre-push path doesn't enroll and run the always-false sentinel. The discriminating proof for the plan path is carried structurally by the None => Vec::new() change in claim_executor.rs (not by the sentinel being absent from the Rust mirror). The sentinel proves the .dag model is the authority for the plan path because it's excluded there; the Rust mirror stays in sync as a SCAFFOLD.

Finding 2 (fields without consumers): The reviewer was reading the WIP commit where corpus_runnable() hadn't yet been extended with discovery_scope_dirs / spawn_width_cap. Both fields now have consumers: corpus_runnable() sets discovery_scope_dirs: [] and spawn_width_cap: 0; execution_corpus_runnable() sets discovery_scope_dirs: [\"src/v2/test/claim/execution\"] and spawn_width_cap: execution_corpus_spawn_width(); runnable_from_value in claim_executor.rs reads both; run_batch_unit applies the cap; run_discovery_corpus_with_options passes options.discovery_scope_dirs.

Finding 3 (discover_floor_corpus_rows_scoped no call sites): Already addressed before the cursor review — run_discovery_corpus_with_options calls it at cli_run.rs:3311.

— sent from merry-fox-653

Brian Searls and others added 11 commits June 29, 2026 19:01
…tch_unit

BatchUnit::Discovery is matched on an owned BatchUnit, so spawn_width_cap is
usize not &usize — drop the * dereference at the cap application site.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- execution_corpus_spawn_width: wrap gunbc_ci_runner_cgroup_memory_cap with
  byte_size_count() before dividing by Int (ByteSize / Int is not defined)
- inert_lens_hygiene_tests / construction_justification_hygiene_tests: add
  FLOOR_DISCOVERY_EXCLUDES to use super:: imports — test functions reference
  it directly after discover_floor_corpus_rows signature added the param

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…VERY_EXCLUDES import

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…st enrolled rows

The execution corpus witnesses (src/v2/test/claim/execution/) are excluded from
the main corpus rows, but path_imports already contains their import graphs
(populated before the exclusion continue). Extend the inert_lens_modules seed
to include ALL *_test.dag paths found in the source tree so that execution-corpus
witnesses still count for lens coverage — matching the pre-Move-B behavior.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…lexity_gate/, drop stale execution copy

main renamed comprep_pipeline_bridge{,_test}.dag -> source_bridged_pipeline_bridge{,_test}.dag
and kept the test in complexity_gate/. My branch had moved the comprep test to execution/
but that source module no longer exists post-rename. Resolution: take main's file in
complexity_gate/ and remove the stale execution/comprep_pipeline_bridge_test.dag.
Execution corpus now has 4 witnesses (the 3 self_host + fold_assembly_round_trip).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gunbai-bot gunbai-bot Bot changed the title Move C: migrate FLOOR_DISCOVERY_EXCLUDES to witness_exclusion_substrings in ci_layer_roots.dag (§3 single authority) Affected-set CI scoping: Move C (exclusion model) + Move B (execution corpus) Jun 29, 2026
Brian Searls and others added 7 commits June 29, 2026 20:55
…_cap enforcement

Finding 1: execution_corpus_spawn_width() divided the full 8 GiB runner cap by
the per-witness reservation (2 GiB = 4), but both corpora run concurrently in
batch-2 with no ResourceDependsOn between them. Delegates to
gunbc_ci_execution_corpus_available_width() in ci_floor_measurement.dag which
subtracts the floor corpus worst-case shard peak (~4.24 GiB) and executor base
overhead (~1.5 GiB) first — correct width under current measurements is 1.

Finding 2: DiscoveryCorpusOptions.spawn_width_cap documented as capping in-corpus
parallelism but run_discovery_corpus_with_options never enforced it — only
claim_executor.rs applied the cap before calling. Now enforced at the top of
run_discovery_corpus_with_options so any caller gets the cap regardless.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ery_corpus_with_options

run_discovery_corpus_with_options now self-enforces options.spawn_width_cap
(previous commit). Thread spawn_width_cap through run_discovery_batch_node
to DiscoveryCorpusOptions.spawn_width_cap instead of pre-capping spawn_width
in run_batch_unit. Eliminates the duplicate min() that was the §2/§3 second
representation flagged in cursor review finding 2.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…awn_width_cap)

The function was at the clippy limit (7 args) before this PR. Adding spawn_width_cap
as an 8th arg to thread the cap through to DiscoveryCorpusOptions.spawn_width_cap is
the correct §2/§3 single-authority fix; suppressing the lint on this internal function
is the right response rather than bundling unrelated args.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…y formula

Two issues addressed:
- execution_corpus_node co-ran with floor_corpus_node in batch-2, causing OOM
  (exit code 137 in CI). Fix: add ResourceDependsOn(floor → execution) so
  execution runs after floor finishes.
- runnable_excludes_corpus_co_residence gates (emit-host, regen) only waited
  for floor_corpus_node; can now co-run with execution corpus. Fix: add
  execution_corpus_node as second source in floor_corpus_exclusion_resource_edges
  so those gates wait for both.
- gunbc_ci_execution_corpus_available_width: removed floor_shard_peak subtraction;
  execution now runs sequentially after floor, so full (runner_cap - executor_base)
  is available → width = floor((8GiB - 1.5GiB) / 2GiB) = 3.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Both findings are valid — fixed in 57a5487.

Finding 1 (OOM / co-resident memory math): The exit-code-137 CI failure confirms it. gunbc_ci_execution_corpus_available_width was subtracting floor_per_shard_peak to account for co-residence, but both corpus nodes were in the same batch-2 and ran concurrently at full parallelism inside claim_executor, which broke the assumption. Fix: add ResourceDependsOn(source: floor_corpus_node, dependent: execution_corpus_node) in floor_corpus_exclusion_resource_edges so execution runs sequentially after floor. With sequential scheduling the floor shard is done before any execution witness starts — no subtraction needed. Updated formula: available = runner_cap - executor_base = 8 GiB - 1.5 GiB = 6.5 GiB → width = floor(6.5 / 2) = 3.

Finding 2 (gate resource exclusion gap): runnable_excludes_corpus_co_residence gates only had a ResourceDependsOn edge from floor_corpus_node. With the new sequential ordering they'd still co-run with the execution corpus. Fix: floor_corpus_exclusion_resource_edges now emits two edges per qualifying gate — one for floor_corpus_node and one for execution_corpus_node. Heavy gates now wait for both corpora before starting.

— sent from merry-fox-653

Brian Searls and others added 2 commits June 29, 2026 22:19
…plit

witness_source_ingest_serialized_after_corpus used skip(n:2) to find the
ingest layer, but execution_corpus_node (added in Move B) occupies batch 2
as it depends on floor_corpus_node. Ingest and other heavy gates now land
in batch 3 (they depend on BOTH corpora). Update skip to n:3.

Also fix the stale BoolWitness.entry in
emit_ingest_typescript_same_language_round_trip_test.dag which still
pointed at the old manual/ path after the file was moved to execution/.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@briansrls
briansrls merged commit 3d039db into main Jun 30, 2026
2 checks passed
@briansrls
briansrls deleted the session/merry-fox-653 branch June 30, 2026 00:05
gunbai-bot Bot pushed a commit that referenced this pull request Jun 30, 2026
briansrls pushed a commit that referenced this pull request Jun 30, 2026
Integrate #5962 and #5968 before HAND parse C lands.
gunbai-bot Bot pushed a commit that referenced this pull request Jun 30, 2026
…rkaround)

Conflict resolution: gunbc_ci_execution_corpus_available_width — took
main's single-step subtraction (runner_cap - executor_overhead). The
Move A branch's two-step form (subtract floor_per_shard_rss first) was
authored pre-#5962; after #5962 merged, execution corpus runs AFTER
the floor via ResourceDependsOn, so the floor reservation is no longer
needed at execution-corpus width time.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant