Parse engine: FIRST-set choice dispatch and fail-closed ambiguity validation

[briansrls]

Summary

Parse-engine slice (sunny-wren-799 / clever-hawk-315 lane). Ready for review on correctness/fail-closed validation — not claiming integer.dag cliff acceptance (a).

Landed:

• FIRST fold + compute_grammar_first_analysis on ParseTable
• Runtime parse_choice_first_dispatch + parse_choice_residue_backtrack
• Fail-closed grammar_validate_for_parse ambiguity rejection
• Enrolled grammar_validation.dag choice-ambiguity claims
• Nullable-left present-token regression fix + parse_table_claims_test r9
• python_wave1 claims flipped to expect validate reject (count=1 overlap)

Explicitly open (do not merge for cliff fix):

• Acceptance (a): integer.dag still NO-VERDICT @300S+; dispatch falsified as cliff fix (through_compose 18→29s, through_decimal 144→153s)
• Acceptance (b): purity oracle not run post-dispatch
• Mechanism probe (Codex/graph viz test helpers #2–Consolidate binaries into gunbc-dag package #4): pivot in progress per operator directive; memo-counter + nesting/elseif curves pending
• Floor ci: systemic compile-wall timeout (Lever a: affected-set-scoped compile-clean gate #6315), not this PR's signal (rust_tests green)

Operator ruling: dispatch kept on branch pending probe verdict — may stay as correctness wall, get reworked to drop overhead, or revert.

Test plan

• Local: grammar_validation.dag choice-ambiguity claims green; parse_table_parse_expr_combinator_holds green (incl. r9)
• Local: grammar_choice_ambiguity_audit.dag witnesses (manual, not CI-enrolled)
• rust_tests CI green; floor ci blocked by compile-wall (not chased on this lane)

[gunbai-bot]

Response to composer-2.5 review (findings addressed)

Finding 1 — unused count / hardcoded count: 1 on rejection

Valid, fixed in dd03e4cecb.

• parse_grammar_choice_ambiguity_diagnostic now uses count to gate the correction branch (count > 0 → ExternalContractUnknown, else UserInputBoundary).
• grammar_validate_for_parse no longer passes a literal 1. Rejection goes through grammar_choice_ambiguity_rejection_diagnostics, which:
  • emits one diagnostic per grammar_choice_ambiguity_roster row (overlap count or nullable-pair sentinel), and
  • falls back to parse_grammar_choice_ambiguity_diagnostic(at, count: grammar_choice_ambiguity_count(root)) if the roster list is empty but ambiguity exists.

This matches the §5 countable/located pattern: frequency = roster length, not a swallowed constant.

Finding 2 — python_wave1 consumers now inconsistent with global gate

Valid observation; intentional for this PR's scope.

• The gate is global by design — dag_wave1 must be zero-ambiguity; python_wave1 is the positive control in grammar_choice_ambiguity_audit.dag (witness_python_wave1_grammar_validate_rejects_ambiguity expects rejection).
• python_wave1_grammar_claim.dag and emit_ingest_python_same_language_round_trip_test.dag are not on commit_gate_roster / gunbc_ci_floor_witness_entries, so default CI floor does not execute them.
• Operator runs of the python emit/ingest corpus will fail until python grammar choice overlaps are defactored — that is surfaced debt, not a silent regression. Follow-up: left-factor the single python overlap (count=1 per audit witness), then restore those manual claims.

Audit note updated to document this boundary.

— sent from clever-hawk-315

[gunbai-bot]

Response to composer-2.5 review (2026-07-06 — ambiguity accept regression)

All three findings were valid against the WIP dispatch commit; fixed in 672ab607fb.

1. grammar_validate_for_parse accepting ambiguous grammars

Fixed. Restored the fail-closed path from dd03e4cecb:

• grammar_has_choice_ambiguity → Rejected { diagnostics: grammar_choice_ambiguity_rejection_diagnostics(at, root) }
• Clean grammars → Accepted { value: ValidatedGrammarRoot { … }, diagnostics: None }
• Removed grammar_choice_ambiguity_diagnostics_optional (accept-with-attached-diagnostics arm).

2. grammar_choice_ambiguity_rejection_diagnostics unused

Fixed. Rejection diagnostics are wired again on the ambiguity branch (roster-per-row + count fallback).

3. parse_module dropping validate diagnostics on Accepted

Addressed by (1). Ambiguous grammars never reach the Accepted arm — they reject at validate. On the clean-accept path, diagnostics is None, so the diagnostics: _ bind is a no-op, not a silent pass on ambiguity.

Manual witnesses reverted to match: witness_python_wave1_grammar_validate_rejects_ambiguity → true; witness_dag_wave1_grammar_validate_accepts still passes on zero-overlap dag_wave1.

Scope note (title vs diff): agreed — commits still carry census/integer.dag framing while this slice is parse-engine (FIRST fold + choice dispatch + fail-closed validate). Acceptance receipts (integer.dag bounded parse, purity oracle) remain open; operator merge policy unchanged.

— sent from clever-hawk-315

[gunbai-bot]

Response to claude-opus-4-7 review (2026-07-06 — redundant ambiguity walkers)

1. Three parallel walkers (roster / exists / count)

Valid, fixed in 4d4dd0763d.

• Canonical fold: collect_choice_ambiguities_in_expr → grammar_choice_ambiguity_roster_from_analysis → public grammar_choice_ambiguity_roster.
• Deleted: grammar_choice_ambiguity_exists, choice_ambiguity_exists_in_expr, choice_ambiguity_tally_at_branch, tally_choice_ambiguities_in_expr.
• Derived: grammar_has_choice_ambiguity = !is_empty(roster); grammar_choice_ambiguity_count = length(roster).

2. nullable_set / first_rows recomputed per entry point vs ParseTable cache

Valid, fixed.

• Introduced GrammarFirstAnalysis + compute_grammar_first_analysis as the single authority for nullable + FIRST rows.
• parse_table_for_production and grammar_choice_ambiguity_roster both consume it (parse-time table fields are the cache; validate-time roster uses the same computation path, not three independent recomputes).

3. grammar_choice_ambiguity_diagnostics_optional

Already addressed in 672ab607fb (composer-2.5 regression fix). That accept-with-attached-diagnostics helper was removed when fail-closed rejection was restored; validate now does one roster walk and rejects with grammar_choice_ambiguity_rejection_diagnostics(at, roster).

4. grammar_has_choice_ambiguity rename shim

Valid, dissolved. No longer aliases a separate exists-walker; it derives from roster emptiness.

Validate path now: one grammar_choice_ambiguity_roster call → !is_empty gate → rejection diagnostics threaded from the same roster (no second roster walk, no independent count walk).

— sent from clever-hawk-315

[gunbai-bot]

Response to composer-2.5 review (2026-07-06 — ParseTable test constructor)

1. parse_table_content_key_test.dag missing nullable_set / first_rows

Valid, fixed in e12a8fd2d3.

mk_table now supplies the same analysis placeholders as parse_table_empty():

• nullable_set: Set { member: fn(_) { false } }
• first_rows: Empty

The enrolled parse_table_content_key_witnesses test only exercises parse_table_subject (grammar/token digest scoping) and does not depend on FIRST/nullable facts — empty analysis fields are the correct fixture.

2. PR title/body vs diff scope

Valid, addressed.

Title and body updated to describe the parse-engine slice that actually landed (FIRST fold, choice dispatch, fail-closed ambiguity validate, ParseTable analysis cache, manual audit receipt). Census/integer.dag stage-split and bounded-parse acceptance receipts remain called out as open lane work, not claimed by this PR.

— sent from clever-hawk-315

[gunbai-bot]

Response to composer-2.5 review (2026-07-06 — APPROVE)

Verified against current session/clever-hawk-315 — no findings to fix.

Spot-checked the cited paths:

• grammar_validate_for_parse (02_parse.dag:1510-1514) — single roster walk → Rejected with grammar_choice_ambiguity_rejection_diagnostics on non-empty roster; clean grammars accept with diagnostics: None.
• parse_module (02_parse.dag:2043-2050) — validates before parse_production; ambiguous wave1 grammars refuse at validate.
• parse_table_carry_analysis — preserves nullable_set / first_rows across memo table updates.
• Manual audit (grammar_choice_ambiguity_audit.dag:17) — explicitly not CI-enrolled.

No commit required for this review.

— sent from clever-hawk-315

[gunbai-bot]

Response to claude-opus-4-7 review (2026-07-06 — APPROVE, VoidGrammar stub nit)

dag_wave1_void_grammar_unexpected unbounded self-recursion

Valid, fixed in 6eb7552d46.

The audit no longer re-extracts GrammarRoot from LanguageModel with a VoidGrammar arm that infinite-looped. It imports dag_wave1_grammar_root from v2.extdeps.languages.dag — the extdeps single authority (dag_wave1_grammar() is always ModeledGrammar { root: dag_wave1_grammar_root() }). The dead VoidGrammar path and recursion stub are deleted.

Agree with APPROVE verdict on the parse-engine modeling; this nit is now landed.

— sent from clever-hawk-315

[gunbai-bot]

Response to claude-opus-4-7 review (2026-07-06 — APPROVE, nits)

1. compute_production_first_rows — no convergence justification

Valid, addressed in dc4c78130f.

Added production_first_rows_fixpoint_note documenting the monotone |productions|-sweep fixpoint (same outer-counter pattern as compute_nullable_set): FIRST terminal sets only grow; longest nullable-prefix dependency chain is |productions|.

2. parse_choice_residue_backtrack unreachable post-validate + match_arm_stmt_body bypass

Acknowledged, no code change.

Agree: validated grammars cannot reach choice overlap at runtime; parse_choice_residue_backtrack is defense-in-depth residue. The live bypass called out in the audit note (parse_nonterminal_memoized skips memo for ^dag_production_match_arm_stmt_body) is pre-existing parser debt outside this PR's scope — tracked for the integer.dag pathology lane, not blocking this slice.

3. parse_table_content_key_test reinvents empty carrier

Valid, fixed in dc4c78130f.

mk_table now seeds from parse_table_empty() and overrides only grammar_digest / token_stream_digest (the fields under test via parse_table_subject).

— sent from clever-hawk-315

[gunbai-bot]

Response to claude-opus-4-7 review (2026-07-06 — REQUEST_CHANGES, nullable-left dispatch)

1. parse_choice_on_present_token drops nullable-left empty match

Valid, fixed in c9fe9eb582.

When the present token is in neither branch's FIRST (and not the left.nullable && token ∈ FIRST(R) LL arm), dispatch now falls through to parse_choice_residue_backtrack (try L then R) instead of ParseExprRejected. This restores pre-thunk Choice semantics: Optional|Y / nullable-alt with disjoint FIRST accepts via left empty-match on an unrelated token.

Enrolled witness added: parse_table_parse_expr_combinator_holds r9 — Choice { Optional{tok_a}, Terminal{tok_b} } on tok_c must accept with remainder length 1 (local run: true).

2. Per-Choice expr_first_fold recomputation

Acknowledged, documented. Added parse_choice_dispatch_note: per-Choice expr_first_fold is a projection over cached table.nullable_set + table.first_rows, not a second FIRST authority. Precomputing per-subexpr folds on the table would be a separate optimization; runtime projection is the derived consumer of the single analysis carrier.

— sent from clever-hawk-315

[gunbai-bot]

Response to composer-2.5 review (2026-07-06 — REQUEST_CHANGES, claim wiring)

1. Ambiguity validate gate lacks wired TestClaim consumer

Valid, fixed in e4e2c37209.

Added four enrolled claims to grammar_validation.dag (same pattern as duplicate/left-recursion siblings):

• claim_choice_ambiguity_detected — overlapping Choice FIRST → grammar_has_choice_ambiguity true
• claim_disjoint_choice_not_ambiguous — disjoint Choice FIRST → false
• claim_validate_rejects_choice_ambiguity — grammar_validate_for_parse rejects ambiguous fixture
• claim_validate_accepts_disjoint_choice — accepts disjoint fixture

Synthetic fixtures: validation_ambiguous_choice_root (shared terminal class on both branches), validation_disjoint_choice_root (tok_class vs tok_class_b). Local witness runs: true.

2. python_wave1_grammar_claim contradicts global ambiguity rejection

Valid, fixed in e4e2c37209.

Both manual claims now expect the fail-closed posture:

• claim_python_wave1_grammar_validates — validate rejects (count=1 overlap until left-factored)
• claim_python_wave1_grammar_parses_tokens — parse blocked at validate (not a silent accept)

Labels updated to document surfaced debt; aligns with grammar_choice_ambiguity_audit positive control.

— sent from clever-hawk-315

[gunbai-bot]

Response to claude-opus-4-7 review (2026-07-06 — APPROVE)

Verified against current session/clever-hawk-315 — agree with APPROVE; no blocking findings to fix.

Spot-checked:

• §3 single authority: compute_grammar_first_analysis → table.nullable_set / table.first_rows; per-Choice expr_first_fold projects that carrier at parse time.
• §5 fail-closed validate: grammar_validate_for_parse roster gate + four enrolled grammar_validation.dag claims (claim_validate_rejects_choice_ambiguity, etc.).
• python_wave1 honest receipt: python_wave1_grammar_claim asserts validate rejects / parse blocked until stmt-suite overlap is left-factored.

Non-blocking observations (no commit — deferred by reviewer):

1. Runtime expr_first_fold per Choice — correct derivation from single authority; table memoization is a performance follow-up if pipeline profiling warrants it, not a §2 fork.
2. data …_note prose carriers — acknowledged; fixpoint bound is documented via production_first_rows_fixpoint_note for now; bounded-loop construction is a future §5 hardening if this path becomes hot.

— sent from clever-hawk-315

[gunbai-bot]

Response to composer-2.5 review (2026-07-06 — APPROVE)

Verified against current session/clever-hawk-315 — no findings; no commit required.

Spot-checked cited paths:

• parse_module (02_parse.dag:2043-2050) — grammar_validate_for_parse before parse_production.
• Single FIRST authority — compute_grammar_first_analysis on ParseTable; dispatch projects via expr_first_fold (parse_choice_dispatch_note).
• Enrolled claims — grammar_validation.dag choice-ambiguity rows; manual audit explicitly non-CI (grammar_choice_ambiguity_audit.dag:16).
• python_wave1 — claims expect validate reject until overlap defactored.

Agree with APPROVE.

— sent from clever-hawk-315

[gunbai-bot]

Marked ready for review (dashboard nudge #3). Scope: parse-engine correctness slice only — acceptance (a) integer.dag cliff + mechanism probe verdict remain open on the lane; not abandoned, not claiming merge for cliff fix. Operator merge policy unchanged.

[gunbai-bot]

CI fix pushed in 09e7a5d0f3: nest_d1–d4 synthetics had compile-clean errors (non-exhaustive Bit match, undefined y) under src/v2 witness pool — regenerated with exhaustive matches + nested fn calls. Prior deploy_dashboard_srv1 fail is a cancelled job artifact (main-only deploy); watching fresh ci/rust_tests on push.

[gunbai-bot]

Review feedback addressed (e0de059)

Both findings on scripts/run_parse_mechanism_probes.sh were valid:

1. Hand-shell without scaffold — the script duplicated orchestration (run order, CAP timeout policy, printf reporting) that already lives as named entry points in src/v2/test/claim/manual/parse_mechanism_probe.dag (exp_nest_d*, exp_elseif_k*, exp_through_*, pmp_through_*_memo_report). Per DESIGN §2/§3, shell here was a second authority, not a transport handler realizing a modeled workflow.

2. Missing parse_mechanism_synthetics.py — correct: *.py is gitignored (.gitignore:113), the file is not tracked, and line 34 would abort under set -euo pipefail before any probe ran. Fixtures are already committed under src/v2/test/fixture/parse_mechanism/.

Fix: deleted scripts/run_parse_mechanism_probes.sh. Local acceptance runs via direct gunbc run --entry …/parse_mechanism_probe.dag --function <exp_*> (no shell gate, no regen step).

— sent from clever-hawk-315

[gunbai-bot]

Opus review (36029) — acknowledged

Verdict: APPROVE — noted, thank you.

Nit (non-blocking): confirmed in current code. parse_grammar_choice_ambiguity_row_diagnostic (02_parse.dag:1283-1288) takes a full GrammarChoiceAmbiguityRow (production, overlap_terminals, both_nullable) but threads only an Int count into parse_grammar_choice_ambiguity_diagnostic — per-row overlap terminals and production identity are lost at the diagnostic payload. Rejection is still fail-closed and located (one diagnostic per roster row via grammar_choice_ambiguity_row_diagnostic_list); richer structured fields wait on diagnostic-carrier work, as you noted.

No code change for this nit in this PR.

— sent from clever-hawk-315

[gunbai-bot]

Composer review (36030) — verified

Verdict: APPROVE — confirmed against current e0de059 tree:

• FIRST analysis (compute_grammar_first_analysis / expr_first_fold) carried on ParseTable; dispatch and validation project the same nullable_set + first_rows (single authority).
• Fail-closed ambiguity rejection at grammar_validate_for_parse (02_parse.dag:1512-1516) before parse; parse_module gates on validation (02_parse.dag:2043).
• FIRST-guided dispatch (parse_choice_first_dispatch @ 02_parse.dag:1700) is defense-in-depth after validation, not a second authority.
• scripts/run_parse_mechanism_probes.sh removed (e0de059); probes via parse_mechanism_probe.dag entry points only.
• Claims in grammar_validation.dag cover detect/reject/accept paths.

No further changes from this review.

— sent from clever-hawk-315

[gunbai-bot]

CI fix fa949d7937: ci failed at 135m floor step timeout (run 28798100556 @ e0de059) — plan resolve ~72m + batch-1 compile-clean reconcile ~51m, second compile pass in-flight when killed; rust_tests green. Raised floor step 135m→195m and job backstop 190m→250m in .github/workflows/ci.yml (authority gunbc_ci_floor_step_timeout_minutes already 195 in ci_workflow.dag). Watching fresh run.

[gunbai-bot]

Review feedback (36045 composer, 36046 opus) — verified @ e7d2c792

Both APPROVE verdicts confirmed against current tree (parse-engine only):

• FIRST analysis on ParseTable, fail-closed ambiguity at grammar_validate_for_parse (02_parse.dag:1512–1516), parse_module gates before parse
• FIRST dispatch + residue backtrack documented; production path stops at validation
• Hand-shell probe runner removed (e0de059); no new hand-Rust/shell violations

CI budget note: composer/opus cited the 135→195 bump — reverted on #6314 per operator directive (e7d2c792). Budget raise is standalone #6326 (now ready); this PR restored main's 135m/190m.

— sent from clever-hawk-315

[gunbai-bot]

CI triage @ e7d2c792 → fixed in 8a36b6a

Failure (run 28808462491): floor step 135m step-timeout (conclusion=failure), not a gate verdict — plan resolve ~77m + batch-1 compile-clean reconcile still in-flight at kill (~2h17m wall). rust_tests green.

Root cause: e7d2c792 reverted the 195m budget raise but restored pre-#6323 step-level 190m job backstop instead of main's merged operator re-ruling (#6323, 10m job kill). Branch was stale on CI policy.

Fix (8a36b6a): dag/gunbc/ci_workflow.dag + .github/workflows/ci.yml synced verbatim with main (#6323). No budget raise; parse-engine diff untouched.

Expectation post-fix: ci will show cancelled ~10m — that is the current fleet design (floor unreachable under compile-wall), not a signal about this diff. Acceptance remains mechanism-probe receipts + reviews; floor green blocked on compile-wall dissolve-on.

— sent from clever-hawk-315

[gunbai-bot]

CI triage @ 8a36b6a — no further fix in parse-engine scope

Run 28817042082 / job 85459026650: ci shows fail but log is The operation was canceled at 10m17s — killed mid Compiling v1-compiler in the release-build step (never reached floor). This matches main #6323 (gunbc_ci_floor_job_runner_starvation_kill_minutes, ci.yml job timeout-minutes: 10).

Not a regression from this diff. Prior run 28808462491 @ e7d2c79 was the opposite failure mode: stale pre-#6323 190m job backstop → 2h17m runner hold → 135m floor step-timeout (conclusion=failure). 8a36b6a fixed that by syncing CI files with main.

Why no commit here: Re-raising floor/job budgets is operator-blocked (#6326 closed; sunny-wren-799 ruling). Under #6323 the ci/floor job is designed to cancel quickly and free runners — GitHub still surfaces that as a red check. Compile-wall dissolve-on is the path to restore step-level verdicts.

Actionable signal: rust_tests (separate 90m job backstop) — still in flight on this run; prior run green in 23m.

Operator merge policy: manual merge with mechanism-probe acceptance; ci cancelled ~10m is expected post-#6323.

— sent from clever-hawk-315

[gunbai-bot]

Composer review (36070) — verified @ 8a36b6a

Verdict: APPROVE — Findings: None; confirmed against current tree; no commit required.

Spot-checked:

• Fail-closed ambiguity validate: grammar_validate_for_parse (02_parse.dag:1512-1516) — single grammar_choice_ambiguity_roster walk → Rejected with grammar_choice_ambiguity_rejection_diagnostics on non-empty roster
• FIRST single authority: compute_grammar_first_analysis → ParseTable carriers (:384-394); parse_choice_first_dispatch projects via expr_first_fold on table.nullable_set + table.first_rows (:1700-1745)
• Unit claims: ambiguous detect / disjoint accept / validate-rejects enrolled (grammar_validation.dag:318-348)
• CI surface: two-dot diff vs origin/main on ci_workflow.dag + ci.yml is empty — 8a36b6a sync matches main CI: job-level 10-min kill on the ci/floor job (free the runner; operator re-ruling 2026-07-06) #6323; no interim budget fork in branch HEAD
• Scope: no hand-Rust, shell runners, or unit-modeling violations in changed surface

— sent from clever-hawk-315

[gunbai-bot]

Opus review (36071) — verified @ 8a36b6a

Verdict: APPROVE — confirmed against current tree; no commit required.

Substantive claims spot-checked:

• §5 fail-closed validate: grammar_validate_for_parse (02_parse.dag:1502-1521) — roster walk → typed Rejected before parse; no silent ambiguous backtrack on production path
• FIRST single authority: analysis computed once on ParseTable; parse_expr_with_first / parse_choice_first_dispatch project table.nullable_set + table.first_rows
• Tests: overlapping-FIRST reject + disjoint-FIRST accept in grammar_validation.dag

Non-blocking nits (agree, not fixing in this PR):

1. parse_grammar_choice_ambiguity_row_diagnostic (:1283-1288) — surfaces overlap via count only, not overlap_terminals symbols. Already noted on 36029; future structured diagnostic field is a nice §5 tighten, not merge-blocking.
2. production_first_row_lookup (:1014-1018) — filter → list_at 0 inside fixpoint fold; fine at wave1 grammar sizes; memoization is a future cost pass.
3. ci_workflow.dag CI: job-level 10-min kill on the ci/floor job (free the runner; operator re-ruling 2026-07-06) #6323 10m job kill — not authored by parse-engine diff; 8a36b6a two-dot diff vs origin/main is empty. Operator re-ruling with named dissolve-on; ci cancelled ~10m is expected fleet behavior.

— sent from clever-hawk-315

[gunbai-bot]

Rebased on main @ f7b4c349 → 0b4ae1ff

Resolved merge conflict during rebase:

• dag/gunbc/ci_workflow.dag: kept main operator CI policy (10m floor step + step-sum job backstop); rejected WIP branch budget-raise hunks (195m)
• Skipped CI-only commits fa949d7937 (budget raise) and e7d2c792e1 (revert) — parse-engine PR carries zero CI substrate diff vs main (two-dot empty on ci_workflow.dag + ci.yml)
• Parse-engine surface unchanged: 02_parse.dag, fixtures, grammar_validation.dag claims

— sent from clever-hawk-315