Repository navigation
parse: nullable-only ambiguity rejection; non-nullable FIRST overlap becomes counted residue (fixes main parse refusing all input) - #6337
Conversation
…s counted overlap residue grammar_validate_for_parse rejected ANY choice-FIRST overlap, but dag_wave1 carries 5 non-nullable overlap rows (expr, arg, match_arm, field_pattern, if_expr) - canonical non-LL(1) choice points its own runtime handles via parse_choice_residue_backtrack. Result on main: parse_module refused every input (receipt: exp_nest_d1_parse_holds false, probe module parse rejected with 5x parse_grammar_choice_ambiguity), invisible under dark CI, while the backtrack arm sat dead. Split the states instead of conflating them (state-space conflation): - both_nullable=true rows: genuine ambiguity, still Rejected (fail-closed); new red control claim_validate_rejects_nullable_choice_ambiguity. - both_nullable=false rows: Accepted with counted typed residue diagnostics (reason parse_grammar_choice_overlap_residue, one per row), threaded through parse_module via diagnostics_merge / rejected_with_pending - no silent drop. - disjoint choice: residue None (claim_disjoint_choice_validate_residue_none discriminates residue from clean). Claims re-derived by execution: dag_wave1 validate accepts, zero nullable ambiguity, residue pinned at 5; python_wave1 validates and its MVP-1 fixture now parses end-to-end (residue 1); exp_nest_d1/exp_elseif_k4 true again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Opus review (36099) — acknowledgedVerdict APPROVE — thank you. On the non-blocking nit (residue diagnostics indistinguishable per row, No code change in this PR; not resetting 2× APPROVE for a non-blocking nit whose proper home is the shared carrier. — sent from sunny-wren-799 |
…ust gate/warm values, re-derive backstops The #6246 rebase restored OLD budget values while keeping the NEW notes that document larger receipt-backed values: - gunbc_ci_rust_gate_step_timeout_minutes: 10 -> 45. Its own note: nextest killed mid-run at the 30m cap (receipt job 85319650284: warm 14m38s, reconcile 1339s before main, nextest in-flight at kill); prior 30m receipt: clippy in-flight at 15m kill. - gunbc_ci_rust_sccache_warm_step_timeout_minutes: 15 -> 45. Its own note: PRs 6286/6274/6290 timed out at 15m under cold-cache fallback. Live consequence repaired: every rust_tests run false-killed at the 10m gate step (receipt: PR #6337 job 85487003660, step timeout at exactly 10m, zero test output). Values were one config edit below their own receipts. ci.yml + falsifier.yml regenerated via generated_artifact_gate::main_wet - derived values move together (rust_tests job backstop 40 -> 105, falsifier job 155 -> 170 + its stale build-command drift repaired by the same regen). Deliberately NOT swept in from the same regen (pre-existing drift, reported separately): DESIGN.md - regenerating it DROPS the operator's 2026-07-05 section-5 ruling text (escape hatches / factory model / review bar) because design_document.dag is stale behind the hand-authored authority doc; and a .gitignore exception line. Both left at their committed state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
CI disposition post branch-update: the failing — sent from sunny-wren-799 |
…15m->45m, derived backstops) (#6338) * CI budgets: repair #6246 rebase regression - restore receipt-backed rust gate/warm values, re-derive backstops The #6246 rebase restored OLD budget values while keeping the NEW notes that document larger receipt-backed values: - gunbc_ci_rust_gate_step_timeout_minutes: 10 -> 45. Its own note: nextest killed mid-run at the 30m cap (receipt job 85319650284: warm 14m38s, reconcile 1339s before main, nextest in-flight at kill); prior 30m receipt: clippy in-flight at 15m kill. - gunbc_ci_rust_sccache_warm_step_timeout_minutes: 15 -> 45. Its own note: PRs 6286/6274/6290 timed out at 15m under cold-cache fallback. Live consequence repaired: every rust_tests run false-killed at the 10m gate step (receipt: PR #6337 job 85487003660, step timeout at exactly 10m, zero test output). Values were one config edit below their own receipts. ci.yml + falsifier.yml regenerated via generated_artifact_gate::main_wet - derived values move together (rust_tests job backstop 40 -> 105, falsifier job 155 -> 170 + its stale build-command drift repaired by the same regen). Deliberately NOT swept in from the same regen (pre-existing drift, reported separately): DESIGN.md - regenerating it DROPS the operator's 2026-07-05 section-5 ruling text (escape hatches / factory model / review bar) because design_document.dag is stale behind the hand-authored authority doc; and a .gitignore exception line. Both left at their committed state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: warm=46 (serializer witness pins warm>gate + emitted 46), re-derive disposition prose to 106m sum Both cursor findings verified real and fixed: - warm 45->46: witness_warm_step_budgeted_separately_from_gate asserts warm strictly > gate; witness_rust_tests_yaml_has_observable_warm_step_before_gate pins the emitted 'timeout-minutes: 46' before the gate step. Both now green by execution. git -S receipt: #6246's own branch carried 'fix(ci): restore rust gate step budgets that regressed to 10m' (303d379, warm=46/gate=45) - the squash dropped it; this restores the intended state. - disposition prose 40m/15+10+5+5+5 -> 106m/46+45+5+5+5 (single authority: prose re-derived with the carriers, warm>gate separation named). ci.yml regenerated via main_wet (warm 46, derived rust_tests backstop 106). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * resolve #6339 merge: warm=46 serializer-pinned, backstop 106m re-derived, keep 6339 gate-note drift history Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Brian Searls <briansrls@gunb.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…+ empty-list semantics pin (fix 2) Manager-mandated merge-bar demands on #6362, addressed with durable, re-runnable witnesses rather than a one-off manual diff: - parse_tree_content_hash_witness.dag: content_hash(ParseTree) over the tiny/small corpus sample, computed once against the true PR base (0f98a26, built as its own release binary in an isolated worktree) and once against HEAD. Both hashes are byte-identical (witness.dag=9afb740ee43f9e61, execution_mode.dag=ed071cb2baeaeb61), proving fix 2 (list_at_optional -> list_head) and fix 3 (fold_list -> parse_match_arm_stmt_step) preserve parse semantics exactly, not just "no test regressions." - parse_token_first_empty_semantics_test.dag: two `test fn` claims pinning parse_token_first (post-fix, list_head-based) equal to list_at_optional(xs, index: 0) (pre-fix arm) on both the empty-list boundary case that motivated the rewrite and the ordinary present case. Both PASS via claim_batch. Note: the local `main` ref was stale (21fb5d1); the true PR base is origin/main's 0f98a26 (merge-base origin/main HEAD) — using the stale ref for the pre-fix binary produced a false "grammar_invalid" rejection unrelated to this PR (fixed upstream by #6337, landed after 21fb5d1 but before 0f98a26).
…(n)/O(n^2) drivers, name honest residual (#6362) * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * 02_parse verification receipts: byte-equal parse-tree hash (fix 2+3) + empty-list semantics pin (fix 2) Manager-mandated merge-bar demands on #6362, addressed with durable, re-runnable witnesses rather than a one-off manual diff: - parse_tree_content_hash_witness.dag: content_hash(ParseTree) over the tiny/small corpus sample, computed once against the true PR base (0f98a26, built as its own release binary in an isolated worktree) and once against HEAD. Both hashes are byte-identical (witness.dag=9afb740ee43f9e61, execution_mode.dag=ed071cb2baeaeb61), proving fix 2 (list_at_optional -> list_head) and fix 3 (fold_list -> parse_match_arm_stmt_step) preserve parse semantics exactly, not just "no test regressions." - parse_token_first_empty_semantics_test.dag: two `test fn` claims pinning parse_token_first (post-fix, list_head-based) equal to list_at_optional(xs, index: 0) (pre-fix arm) on both the empty-list boundary case that motivated the rewrite and the ordinary present case. Both PASS via claim_batch. Note: the local `main` ref was stale (21fb5d1); the true PR base is origin/main's 0f98a26 (merge-base origin/main HEAD) — using the stale ref for the pre-fix binary produced a false "grammar_invalid" rejection unrelated to this PR (fixed upstream by #6337, landed after 21fb5d1 but before 0f98a26). --------- Co-authored-by: Brian Searls <briansrls@gunb.ai>
…algorithmic residue in v2 02_parse under the v1 interpreter - same-file compiled-vs-interpreted timing, packrat memo effectiveness + table-carry cost interpreted, verdict = engine fix / interpreter fix / honest no-op-until-S2 (upda (#6365) * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * 02_parse verification receipts: byte-equal parse-tree hash (fix 2+3) + empty-list semantics pin (fix 2) Manager-mandated merge-bar demands on #6362, addressed with durable, re-runnable witnesses rather than a one-off manual diff: - parse_tree_content_hash_witness.dag: content_hash(ParseTree) over the tiny/small corpus sample, computed once against the true PR base (0f98a26, built as its own release binary in an isolated worktree) and once against HEAD. Both hashes are byte-identical (witness.dag=9afb740ee43f9e61, execution_mode.dag=ed071cb2baeaeb61), proving fix 2 (list_at_optional -> list_head) and fix 3 (fold_list -> parse_match_arm_stmt_step) preserve parse semantics exactly, not just "no test regressions." - parse_token_first_empty_semantics_test.dag: two `test fn` claims pinning parse_token_first (post-fix, list_head-based) equal to list_at_optional(xs, index: 0) (pre-fix arm) on both the empty-list boundary case that motivated the rewrite and the ordinary present case. Both PASS via claim_batch. Note: the local `main` ref was stale (21fb5d1); the true PR base is origin/main's 0f98a26 (merge-base origin/main HEAD) — using the stale ref for the pre-fix binary produced a false "grammar_invalid" rejection unrelated to this PR (fixed upstream by #6337, landed after 21fb5d1 but before 0f98a26). * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * adhoc-c328b166-bca: 01_tokenize O(n^2) fix + full parse-pipeline census + token-stream witness lex_walk_step accumulated tokens via list_snoc_item every token (O(n^2) before parsing even starts) -- 4th instance of the same shape, this time in a different pipeline stage than the three already-landed 02_parse.dag fixes. Rewritten as reverse-cons + single .reverse() at tokenize()'s exit. Verified byte-equal via a new order-sensitive token-stream content hash (class+lexeme, in stream order) run against independently-built pre-fix and post-fix binaries on tiny/small corpus samples -- identical both sides, confirming a pure algorithmic rewrite. Full read-based census of 02_parse.dag's 9 list_snoc_item call sites, each dispositioned (dead/test-only, bounded grammar-analysis cost, or already-fixed) -- none explain the file-size-scaling growth still seen deeper in live parsing on datetime.dag. That residual is not resolved by this PR; grammar_coverage_p8_wall_clock_blocker records the honest state (tokenize confirmed non-bottleneck, DNF persists past grammar-validation, driver not yet pinned) so the marker does not claim a false completion. Added parse_table_record_hit/miss/lookup_call and parse_choice_residue_backtrack to the interpreter's call-frequency watchlist for the continuing in-session hunt. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * fixup: complete the auto-committed partial ProductionFirstRow removal The auto-committer snapshotted a mid-edit state (b16a357) that deleted the ProductionFirstRow type but left its consumers referencing it, breaking typecheck. This completes that edit -- not a new change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Revert "fixup: complete the auto-committed partial ProductionFirstRow removal" This reverts commit dae6602. * Revert "Revert "fixup: complete the auto-committed partial ProductionFirstRow removal"" This reverts commit e2a24c3. * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * adhoc-c328b166-bca: 2 more diag-accumulator sites fixed, parse-memo distinct-key discriminator parse_match_arm_stmt_step and parse_expr_repeat_step both accumulated per-iteration diagnostics via list_append(left: diags.reverse(), right: acc.acc_diags) -- the acc_nodes field in the same accumulators was already reverse-cons fixed, but acc_diags was missed by the prior list_snoc_item-scoped census (this is a different primitive). Converted to List<List<Diagnostic>> batches (Cons-prepended, O(1) per step) plus a single flatten pass at the end (parse_flatten_diag_batches). Verified byte-equal via parse_tree_content_hash_witness.dag on tiny/small (9afb740ee43f9e61 / ed071cb2baeaeb61, unchanged). Honest caveat: list_append(left: small, right: large) is O(len(left)) per the native fold_list_right implementation (it walks `left`, not `right`), so this may not be the O(n^2) site the growing list_append call count suggested -- kept as a correctness-neutral cleanup bringing acc_diags in line with the already-safe acc_nodes pattern, not claimed as a proven complexity fix. Removed the temporary witness_forensics_scale_datetime_half_report (scratch-path fixture, not for the repo). Added a cross-thread parse-memo effectiveness discriminator (PARSE_MEMO_LOOKUPS/HITS/DISTINCT_KEYS globals + record_parse_memo_lookup, wired into try_parse_table_memo_dispatch's parse_table_lookup arm) per the manager's decisive-discriminator request: lookups >> distinct with hits == 0 would prove the native memo cache never serves a re-attempted span. Result on the datetime.dag half-truncation: lookups=0 hits=0 distinct_keys=0 -- the native fast-path memo cache never engages at all on this run, a separate real finding (not proven to be the DNF driver, since the .dag-level table's own stats are never reached either, given the run still doesn't finish). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * adhoc-c328b166-bca: gate residual-hunt recorders behind GUNBC_FLATTEN_SITE_DUMP_SECS, drop debug eprintln Addresses cursor/composer-2.5's REQUEST_CHANGES on #6365: record_call_frequency, record_flatten_site, record_list_cons_tail_split, and record_parse_memo_lookup ran unconditionally on hot paths (eval_call, every free_monoid_to_vec materialization, every native List Cons match, every packrat memo lookup) -- only the stderr dump was env-gated, not the recording itself. Added a residual_hunt_forensics_enabled() OnceLock check (same env var the dump already uses) and an early return at the top of each recorder, so the default/production path pays one relaxed load per call site instead of a mutex lock + HashMap/HashSet write. Added a SCAFFOLD/dissolve-on marker per DESIGN Sec6 (dissolve when adhoc-c328b166-bca closes). Also removed the ADHOC-DEBUG eprintln arms in parse_table_memo_scope_and_key -- a local one-off diagnostic (auto-committer captured a mid-investigation snapshot before I could revert it) that already served its purpose: it proved the extraction logic is sound (zero branches fire on a working file, lookups/hits/distinct_keys populate correctly), so it's not needed going forward and was correctly flagged as ambient unconditional stderr noise on a production code path. Re-verified byte-equal (9afb740ee43f9e61) and forensics gating with/without GUNBC_FLATTEN_SITE_DUMP_SECS set. cargo fmt --all applied (pre-existing formatting drift on this file, unrelated to this change, caught by the --check gate). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Brian Searls <briansrls@gunb.ai> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * 02_parse verification receipts: byte-equal parse-tree hash (fix 2+3) + empty-list semantics pin (fix 2) Manager-mandated merge-bar demands on #6362, addressed with durable, re-runnable witnesses rather than a one-off manual diff: - parse_tree_content_hash_witness.dag: content_hash(ParseTree) over the tiny/small corpus sample, computed once against the true PR base (0f98a26, built as its own release binary in an isolated worktree) and once against HEAD. Both hashes are byte-identical (witness.dag=9afb740ee43f9e61, execution_mode.dag=ed071cb2baeaeb61), proving fix 2 (list_at_optional -> list_head) and fix 3 (fold_list -> parse_match_arm_stmt_step) preserve parse semantics exactly, not just "no test regressions." - parse_token_first_empty_semantics_test.dag: two `test fn` claims pinning parse_token_first (post-fix, list_head-based) equal to list_at_optional(xs, index: 0) (pre-fix arm) on both the empty-list boundary case that motivated the rewrite and the ordinary present case. Both PASS via claim_batch. Note: the local `main` ref was stale (21fb5d1); the true PR base is origin/main's 0f98a26 (merge-base origin/main HEAD) — using the stale ref for the pre-fix binary produced a false "grammar_invalid" rejection unrelated to this PR (fixed upstream by #6337, landed after 21fb5d1 but before 0f98a26). * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * adhoc-c328b166-bca: 01_tokenize O(n^2) fix + full parse-pipeline census + token-stream witness lex_walk_step accumulated tokens via list_snoc_item every token (O(n^2) before parsing even starts) -- 4th instance of the same shape, this time in a different pipeline stage than the three already-landed 02_parse.dag fixes. Rewritten as reverse-cons + single .reverse() at tokenize()'s exit. Verified byte-equal via a new order-sensitive token-stream content hash (class+lexeme, in stream order) run against independently-built pre-fix and post-fix binaries on tiny/small corpus samples -- identical both sides, confirming a pure algorithmic rewrite. Full read-based census of 02_parse.dag's 9 list_snoc_item call sites, each dispositioned (dead/test-only, bounded grammar-analysis cost, or already-fixed) -- none explain the file-size-scaling growth still seen deeper in live parsing on datetime.dag. That residual is not resolved by this PR; grammar_coverage_p8_wall_clock_blocker records the honest state (tokenize confirmed non-bottleneck, DNF persists past grammar-validation, driver not yet pinned) so the marker does not claim a false completion. Added parse_table_record_hit/miss/lookup_call and parse_choice_residue_backtrack to the interpreter's call-frequency watchlist for the continuing in-session hunt. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * fixup: complete the auto-committed partial ProductionFirstRow removal The auto-committer snapshotted a mid-edit state (b16a357) that deleted the ProductionFirstRow type but left its consumers referencing it, breaking typecheck. This completes that edit -- not a new change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Revert "fixup: complete the auto-committed partial ProductionFirstRow removal" This reverts commit dae6602. * Revert "Revert "fixup: complete the auto-committed partial ProductionFirstRow removal"" This reverts commit e2a24c3. * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * adhoc-c328b166-bca: 2 more diag-accumulator sites fixed, parse-memo distinct-key discriminator parse_match_arm_stmt_step and parse_expr_repeat_step both accumulated per-iteration diagnostics via list_append(left: diags.reverse(), right: acc.acc_diags) -- the acc_nodes field in the same accumulators was already reverse-cons fixed, but acc_diags was missed by the prior list_snoc_item-scoped census (this is a different primitive). Converted to List<List<Diagnostic>> batches (Cons-prepended, O(1) per step) plus a single flatten pass at the end (parse_flatten_diag_batches). Verified byte-equal via parse_tree_content_hash_witness.dag on tiny/small (9afb740ee43f9e61 / ed071cb2baeaeb61, unchanged). Honest caveat: list_append(left: small, right: large) is O(len(left)) per the native fold_list_right implementation (it walks `left`, not `right`), so this may not be the O(n^2) site the growing list_append call count suggested -- kept as a correctness-neutral cleanup bringing acc_diags in line with the already-safe acc_nodes pattern, not claimed as a proven complexity fix. Removed the temporary witness_forensics_scale_datetime_half_report (scratch-path fixture, not for the repo). Added a cross-thread parse-memo effectiveness discriminator (PARSE_MEMO_LOOKUPS/HITS/DISTINCT_KEYS globals + record_parse_memo_lookup, wired into try_parse_table_memo_dispatch's parse_table_lookup arm) per the manager's decisive-discriminator request: lookups >> distinct with hits == 0 would prove the native memo cache never serves a re-attempted span. Result on the datetime.dag half-truncation: lookups=0 hits=0 distinct_keys=0 -- the native fast-path memo cache never engages at all on this run, a separate real finding (not proven to be the DNF driver, since the .dag-level table's own stats are never reached either, given the run still doesn't finish). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * adhoc-c328b166-bca: gate residual-hunt recorders behind GUNBC_FLATTEN_SITE_DUMP_SECS, drop debug eprintln Addresses cursor/composer-2.5's REQUEST_CHANGES on #6365: record_call_frequency, record_flatten_site, record_list_cons_tail_split, and record_parse_memo_lookup ran unconditionally on hot paths (eval_call, every free_monoid_to_vec materialization, every native List Cons match, every packrat memo lookup) -- only the stderr dump was env-gated, not the recording itself. Added a residual_hunt_forensics_enabled() OnceLock check (same env var the dump already uses) and an early return at the top of each recorder, so the default/production path pays one relaxed load per call site instead of a mutex lock + HashMap/HashSet write. Added a SCAFFOLD/dissolve-on marker per DESIGN Sec6 (dissolve when adhoc-c328b166-bca closes). Also removed the ADHOC-DEBUG eprintln arms in parse_table_memo_scope_and_key -- a local one-off diagnostic (auto-committer captured a mid-investigation snapshot before I could revert it) that already served its purpose: it proved the extraction logic is sound (zero branches fire on a working file, lookups/hits/distinct_keys populate correctly), so it's not needed going forward and was correctly flagged as ambient unconditional stderr noise on a production code path. Re-verified byte-equal (9afb740ee43f9e61) and forensics gating with/without GUNBC_FLATTEN_SITE_DUMP_SECS set. cargo fmt --all applied (pre-existing formatting drift on this file, unrelated to this change, caught by the --check gate). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg --------- Co-authored-by: Brian Searls <briansrls@gunb.ai> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * 02_parse verification receipts: byte-equal parse-tree hash (fix 2+3) + empty-list semantics pin (fix 2) Manager-mandated merge-bar demands on #6362, addressed with durable, re-runnable witnesses rather than a one-off manual diff: - parse_tree_content_hash_witness.dag: content_hash(ParseTree) over the tiny/small corpus sample, computed once against the true PR base (0f98a26, built as its own release binary in an isolated worktree) and once against HEAD. Both hashes are byte-identical (witness.dag=9afb740ee43f9e61, execution_mode.dag=ed071cb2baeaeb61), proving fix 2 (list_at_optional -> list_head) and fix 3 (fold_list -> parse_match_arm_stmt_step) preserve parse semantics exactly, not just "no test regressions." - parse_token_first_empty_semantics_test.dag: two `test fn` claims pinning parse_token_first (post-fix, list_head-based) equal to list_at_optional(xs, index: 0) (pre-fix arm) on both the empty-list boundary case that motivated the rewrite and the ordinary present case. Both PASS via claim_batch. Note: the local `main` ref was stale (21fb5d1); the true PR base is origin/main's 0f98a26 (merge-base origin/main HEAD) — using the stale ref for the pre-fix binary produced a false "grammar_invalid" rejection unrelated to this PR (fixed upstream by #6337, landed after 21fb5d1 but before 0f98a26). * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * adhoc-c328b166-bca: 01_tokenize O(n^2) fix + full parse-pipeline census + token-stream witness lex_walk_step accumulated tokens via list_snoc_item every token (O(n^2) before parsing even starts) -- 4th instance of the same shape, this time in a different pipeline stage than the three already-landed 02_parse.dag fixes. Rewritten as reverse-cons + single .reverse() at tokenize()'s exit. Verified byte-equal via a new order-sensitive token-stream content hash (class+lexeme, in stream order) run against independently-built pre-fix and post-fix binaries on tiny/small corpus samples -- identical both sides, confirming a pure algorithmic rewrite. Full read-based census of 02_parse.dag's 9 list_snoc_item call sites, each dispositioned (dead/test-only, bounded grammar-analysis cost, or already-fixed) -- none explain the file-size-scaling growth still seen deeper in live parsing on datetime.dag. That residual is not resolved by this PR; grammar_coverage_p8_wall_clock_blocker records the honest state (tokenize confirmed non-bottleneck, DNF persists past grammar-validation, driver not yet pinned) so the marker does not claim a false completion. Added parse_table_record_hit/miss/lookup_call and parse_choice_residue_backtrack to the interpreter's call-frequency watchlist for the continuing in-session hunt. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * fixup: complete the auto-committed partial ProductionFirstRow removal The auto-committer snapshotted a mid-edit state (b16a357) that deleted the ProductionFirstRow type but left its consumers referencing it, breaking typecheck. This completes that edit -- not a new change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Revert "fixup: complete the auto-committed partial ProductionFirstRow removal" This reverts commit dae6602. * Revert "Revert "fixup: complete the auto-committed partial ProductionFirstRow removal"" This reverts commit e2a24c3. * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * adhoc-c328b166-bca: 2 more diag-accumulator sites fixed, parse-memo distinct-key discriminator parse_match_arm_stmt_step and parse_expr_repeat_step both accumulated per-iteration diagnostics via list_append(left: diags.reverse(), right: acc.acc_diags) -- the acc_nodes field in the same accumulators was already reverse-cons fixed, but acc_diags was missed by the prior list_snoc_item-scoped census (this is a different primitive). Converted to List<List<Diagnostic>> batches (Cons-prepended, O(1) per step) plus a single flatten pass at the end (parse_flatten_diag_batches). Verified byte-equal via parse_tree_content_hash_witness.dag on tiny/small (9afb740ee43f9e61 / ed071cb2baeaeb61, unchanged). Honest caveat: list_append(left: small, right: large) is O(len(left)) per the native fold_list_right implementation (it walks `left`, not `right`), so this may not be the O(n^2) site the growing list_append call count suggested -- kept as a correctness-neutral cleanup bringing acc_diags in line with the already-safe acc_nodes pattern, not claimed as a proven complexity fix. Removed the temporary witness_forensics_scale_datetime_half_report (scratch-path fixture, not for the repo). Added a cross-thread parse-memo effectiveness discriminator (PARSE_MEMO_LOOKUPS/HITS/DISTINCT_KEYS globals + record_parse_memo_lookup, wired into try_parse_table_memo_dispatch's parse_table_lookup arm) per the manager's decisive-discriminator request: lookups >> distinct with hits == 0 would prove the native memo cache never serves a re-attempted span. Result on the datetime.dag half-truncation: lookups=0 hits=0 distinct_keys=0 -- the native fast-path memo cache never engages at all on this run, a separate real finding (not proven to be the DNF driver, since the .dag-level table's own stats are never reached either, given the run still doesn't finish). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * adhoc-c328b166-bca: gate residual-hunt recorders behind GUNBC_FLATTEN_SITE_DUMP_SECS, drop debug eprintln Addresses cursor/composer-2.5's REQUEST_CHANGES on #6365: record_call_frequency, record_flatten_site, record_list_cons_tail_split, and record_parse_memo_lookup ran unconditionally on hot paths (eval_call, every free_monoid_to_vec materialization, every native List Cons match, every packrat memo lookup) -- only the stderr dump was env-gated, not the recording itself. Added a residual_hunt_forensics_enabled() OnceLock check (same env var the dump already uses) and an early return at the top of each recorder, so the default/production path pays one relaxed load per call site instead of a mutex lock + HashMap/HashSet write. Added a SCAFFOLD/dissolve-on marker per DESIGN Sec6 (dissolve when adhoc-c328b166-bca closes). Also removed the ADHOC-DEBUG eprintln arms in parse_table_memo_scope_and_key -- a local one-off diagnostic (auto-committer captured a mid-investigation snapshot before I could revert it) that already served its purpose: it proved the extraction logic is sound (zero branches fire on a working file, lookups/hits/distinct_keys populate correctly), so it's not needed going forward and was correctly flagged as ambient unconditional stderr noise on a production code path. Re-verified byte-equal (9afb740ee43f9e61) and forensics gating with/without GUNBC_FLATTEN_SITE_DUMP_SECS set. cargo fmt --all applied (pre-existing formatting drift on this file, unrelated to this change, caught by the --check gate). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * parse forensics: pin datetime DNF root cause (lex_repeat_loop O(n²)) + measured-zero on interpreter fold-site cut Continuation of adhoc-c328b166-bca residual-driver hunt (interpreter-side free_monoid_to_vec fold chokepoint; .dag-side fixes stay in #6365). Root cause PINNED via a new caller-attribution probe: the datetime interpreted- parse DNF is lex_repeat_loop (src/v2/compiler/01_tokenize.dag:158) folding the whole remaining source char-list (max_len=19021=file char count, elem=Int) once per repeat-token => O(n²) in source length. A .dag tokenizer dataflow bug, routed to the tokenize lane; this receipt is its acceptance baseline. Interpreter fold-site cut = MEASURED CLEAN ZERO, dropped: a streaming left-fold (no intermediate Vec) was built, proven byte-identical (parse-tree hash equal, tiny+small, two independently-built binaries), and moved neither wall-clock (~20s both) nor peak RSS (~168 MiB both) on small — elem=Int makes the removed alloc churn negligible and the Vec is transient (no peak-RSS contribution). Kept free_monoid_to_vec + a permanent note (DESIGN §6: a no-op displaces nothing). Lands: caller-attribution probe (DagFnGuard + record_fold_caller, gated behind GUNBC_FLATTEN_SITE_DUMP_SECS, dissolve-on the hunt) extending neat-stag's flatten-site instrument with the .dag-caller axis; + the measured-zero note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Merge origin/main (post-#6365 squash) into parse-forensics probe branch Reconciles the caller-attribution probe + measured-zero note onto the squash-merged #6365 tokenizer/parse fixes. Conflicts were purely additive (my probe blocks sit adjacent to neat-stag's flatten-site instrument); kept both sides. Diff vs main is now only v1_interpreter.rs + cli_run.rs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Brian Searls <briansearls1@gmail.com>
…arness (#6371) * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * 02_parse verification receipts: byte-equal parse-tree hash (fix 2+3) + empty-list semantics pin (fix 2) Manager-mandated merge-bar demands on #6362, addressed with durable, re-runnable witnesses rather than a one-off manual diff: - parse_tree_content_hash_witness.dag: content_hash(ParseTree) over the tiny/small corpus sample, computed once against the true PR base (0f98a26, built as its own release binary in an isolated worktree) and once against HEAD. Both hashes are byte-identical (witness.dag=9afb740ee43f9e61, execution_mode.dag=ed071cb2baeaeb61), proving fix 2 (list_at_optional -> list_head) and fix 3 (fold_list -> parse_match_arm_stmt_step) preserve parse semantics exactly, not just "no test regressions." - parse_token_first_empty_semantics_test.dag: two `test fn` claims pinning parse_token_first (post-fix, list_head-based) equal to list_at_optional(xs, index: 0) (pre-fix arm) on both the empty-list boundary case that motivated the rewrite and the ordinary present case. Both PASS via claim_batch. Note: the local `main` ref was stale (21fb5d1); the true PR base is origin/main's 0f98a26 (merge-base origin/main HEAD) — using the stale ref for the pre-fix binary produced a false "grammar_invalid" rejection unrelated to this PR (fixed upstream by #6337, landed after 21fb5d1 but before 0f98a26). * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * WIP: interpreted-parse cost forensics: discriminate interpretation-tax vs alg * Interpreted-parse superlinear drivers closed: medium fixture DNF(1200s+) -> 88s Continuation of neat-stag's adhoc-c328b166-bca forensics (fixes 1-3 merged from session/neat-stag-780). Three new env-gated instruments in the v1 interpreter named each residual driver; each was then killed at its single authority in v2 source: - 02_parse: memo position key O(1) via head-token start+1 (was length(all) - length(remaining) per nonterminal attempt; v2's length() is a .dag fold_list body, so the seed's native_len fast path never fired) - 01_tokenize: lex_repeat_loop + delimited combinator short-circuit recursion (were done-flag no-op folds over the entire remaining source per attempt); lex rule matcher thunks compiled once per tokenize (CompiledLexRule) instead of fold_lex_pattern per (rule, position) - std/nat: nat_compare grounded on the realization's native < / == (ordering side of the #5428 numeric-tower grounding; the Peano walk cost ~66 frames per char-class range check, 5.2M frames/min) - std/provenance: SpanIndex assoc list -> Map<OccurrenceId, OriginEvent> (ParseTable.entries carrier pattern; lookup was 13.4M frames / 242s self-time on one medium parse; merge stays base-wins) - v1 interpreter instruments (recording gated on GUNBC_FLATTEN_SITE_DUMP_SECS, off = one atomic load): big-fold .dag-site attribution, per-builtin inclusive wall time, per-.dag-fn self-time profile Receipts: datetime.dag DNF>1200s/4.3GB RSS -> 88s; node.dag 116s; grammar.dag 201s (class break closed, ~linear in content); tiny/small packrat stats byte-identical (0/43/43, 0/200/200); 13 targeted claims green; compile --entry 0 diagnostics; regen divergence 0; fmt/clippy green. P8 wall-clock blocker ledger updated with the closure and the two named, priced residuals (delimited lexeme O(k^2) accumulation, per-file grammar-analysis base). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018qZSaWfysRfg28bNVS7ueX * S1 groundwork: closure receipt harness + dissolve the one v1-only construct in the parser's closure - dag/std/algebra.dag: kernel_algebra_profile's map literal (the only string-keyed brace literal in the parse pipeline's transitive closure) becomes an ordinary builder fn over empty_map/map_insert -- same map value, one dialect construct fewer; std_algebra.rs regenerated (regen_stage0, divergence 0 after) - s1_closure_receipt_test.dag: S1 self-hosting receipt harness. Subject = the parse pipeline's transitive import closure (40 files rooted at 01_tokenize/02_parse/03_normalize/extdeps-languages-dag), which is "v2's parser parsing itself"; the harness's own filesystem plumbing is test infrastructure and deliberately outside the subject (note in file). Grammar validated once per run; per-file tokenize -> parse_production -> normalize; failures collected by path, claim = failures == []. Full-closure census run in flight; receipt lands with its result. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018qZSaWfysRfg28bNVS7ueX * S2 strategic-lane worker brief + target_model parse probe - docs/plans/s2-v2-self-emit-brief.md: hand-off brief for the parallel v2-emits-v2 lane (receipt ladder, measured landmines incl. the Symbol carrier seam = 1,790 of 3,560 E0308s on fresh v1-emit of the closure, file-ownership split vs the tactical lane) - tm_probe.dag: profiled single-file parse probe for target_model.dag (census residency diagnosis; forensics-enabled run) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018qZSaWfysRfg28bNVS7ueX * S1: dissolve target_model's two pipeline-operator sites + stall-offset diagnosis probe - target_model.dag: the closure's only live |> uses migrate to the v2 surface (length(xs:), any(xs:, predicate: fn)) -- v1 frontend 0 diags; these were two of the census-named parse failures' construct class - s1_stall_probe.dag: parses a file at the start production via parse_expr directly and reports the byte offset of the first token the parse could not consume (greedy repeat => Accepted-with-remaining pinpoints the first uncovered construct); batch fns for the remaining census failures (runtime, refinement, node, grammar, languages/dag, algebra post-migration re-verify) Census receipt (interpreted, 68 min, pre-fix tree): 35/45 parsed clean; subject-relevant failures under diagnosis: runtime, refinement, node, grammar, target_model (pipes, fixed here), languages/dag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018qZSaWfysRfg28bNVS7ueX * S1: keyword-after-dot grammar row + dissolve negative-literal sites - languages/dag.dag: the postfix projection name after '.' uses dag_grammar_binding_name_terminal() (the soft-keyword choice already used for field/param names), so interpretation.loop / .match parse -- the runtime.dag census failure's construct class - refinement.dag, grammar.dag: the subject's three negative int literals (-1/-2) become parenthesized binary minus (0 - 1); the wave1 grammar has no negative-literal lexeme and three sites do not justify the lexer/normalize lever yet -- class noted for the tree-wide census All three v1-frontend green (0 diagnostics). Stall probes re-verifying under the new grammar; node.dag + languages/dag.dag offsets pending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018qZSaWfysRfg28bNVS7ueX * S1: dissolve node.dag's nine arrow-lambda sites to fn-lambdas The content-hash canonicalizer block used v1 arrow lambdas (e => match ...) with comma-separated inline match arms at two sites -- node.dag's census failure class (stall byte 13535 = the first such fn). All nine sites become fn(e) { ... } with newline arms; pure syntax migration, identical behavior, v1 frontend 0 diagnostics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018qZSaWfysRfg28bNVS7ueX * S1: dquote lex rule via CharPattern + node.dag's last arrow lambda - languages/dag.dag: the string-literal rule's own delimiters were LiteralPattern { text: "\"" } -- an escaped-quote string the wave1 lexer cannot tokenize when reading this file itself (the stream garbles silently and the parse dies ~2,400 lines later; stall probe receipt). Now CharPattern { char: 34 }, the exact idiom the string-BODY escape element two rules up already uses. - node.dag: sort_by(xs, h => h) -> fn(h) { h } (the census's ninth-plus arrow site, hidden earlier by a head-truncated grep). Both v1-frontend 0 diagnostics. Full 40-file receipt run in flight. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018qZSaWfysRfg28bNVS7ueX * S1: add target_model stall probe fn Merged-tree receipt: 38/40 subject files parse clean; the two giants (target_model.dag, languages/dag.dag) each have a residual construct behind the first fix -- stall probes running to name them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018qZSaWfysRfg28bNVS7ueX * S1: rename dag.dag's keyword let-binding + explicit field puns in target_model - languages/dag.dag: `let module = ...` used the bare `module` keyword as a let-binding name (lexes as kw_module, so the whole dag_wave1_grammar_root decl failed); renamed to module_prod -- a local binding, so a rename beats extending keywords into let-name/expr positions where they would collide with the match/if productions - target_model.dag: NamedMetadataFoldOk field puns made explicit (name: name); NOTE the sweep shows shorthand patterns parse fine in green files (integer.dag's Cons { head, tail }), so this may be hygiene rather than the poison -- fresh stall probes running to arbitrate Both v1-frontend 0 diagnostics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018qZSaWfysRfg28bNVS7ueX --------- Co-authored-by: Claude <noreply@anthropic.com>
What
On main,
grammar_validate_for_parserejects any choice-FIRST overlap, andparse_modulegates on validation — butdag_wave1itself carries 5 non-nullable overlap rows (expr,arg,match_arm,field_pattern,if_expr). Net effect:parse_modulerefuses every input on main (receipts:exp_nest_d1_parse_holds= false on fresh main; any module parse rejects with 5×parse_grammar_choice_ambiguity), while the runtime'sparse_choice_residue_backtrackarm — built precisely for overlap — sits dead. Dark CI masked the red; the zero-ambiguity audit witnesses were enrolled green early and not re-executed as the FIRST fold evolved (5 on the source branch tip too).Fix (state split, not a relaxation)
both_nullable = trueoverlap = genuine ambiguity → still Rejected (fail-closed). New red control:claim_validate_rejects_nullable_choice_ambiguity(Choice of two nullable arms sharing a FIRST terminal).both_nullable = falseoverlap = non-LL(1) choice point the runtime handles by construction → Accepted with counted typed residue (parse_grammar_choice_overlap_residue, one diagnostic per row), threaded throughparse_moduleviadiagnostics_merge/rejected_with_pending— never silently dropped.claim_disjoint_choice_validate_residue_none), so the residue channel discriminates overlap from clean.Verified by execution (fresh build, this branch)
witness_dag_wave1_grammar_validate_acceptswitness_dag_wave1_zero_nullable_choice_ambiguitywitness_dag_wave1_overlap_residue_row_countvalidation_nullable_ambiguity_validate_rejects(red control)validation_overlap_choice_validate_accepts_with_residuevalidation_disjoint_choice_residue_is_nonepython_wave1_grammar_validates_fixture/python_wave1_grammar_parse_acceptsexp_nest_d1_parse_holds/exp_elseif_k4_parse_holdsCounts are pins: residue growth is a loud diff, never silence. No Rust changes.
Found while building the σ-family lens demo (its corpus witness needs a working parser); that lane continues on
sigma-family-lensand depends on this.🤖 Generated with Claude Code