v2 parse: layout derived from the stream's source; RefuseOnMatch and AfterLineBreak grammar arms (PR1 of 2)

[gunbai-bot]

PR1 of 2 for the block-headed left operand work (quiet-gull-780's lane), approved as design (c). It supplies the layout model that the #12769 trigger needs. PR2 uses it.

Representation, and why

A token's start/end say that two tokens have a gap, not whether it holds a newline: x - 1 and x⏎- 1 lex to identical tokens at identical octets. Before this change the source stopped at the lexer, so no production could tell the two apart. Of the three options:

• a line_break_before flag on Token: rejected. It stores a second representation of a source fact, and 1,179 Token literals would carry it.
• a newline token in the stream: rejected. It is general trivia, and every production would have to skip it.
• chosen: the stream carries its layout, derived from the source once at lexing, and a layout question is a named lookup at the point of use. The source stays the one authority: nothing is stored per token, and the index has one constructor.

v2.std.compilers.lexing TokenStream gains layout: Optional<StreamLayout>: the line-break facts derived from the source once, by the tokenizer (the only place the source is read):

• token_stream_lexed is the tokenizer's constructor: it passes the source through unchanged (v2.compiler.tokenize lex_walk_artifact).
• token_stream_new (fixtures) sets Absent.
• StreamLayout is the starts of the tokens that have a line break before them, derived in ONE walk over source and tokens by its only constructor stream_layout_of; token_stream_line_break_before(layout, token_start) answers LineBreakInGap / NoLineBreakInGap / LineBreakUnknownWithoutSource by lookup (review 73189: no per-question rescan). A line feed inside a token is not layout.

Nothing is added to Token, so occurrence identity, the semantic graph and semantic hashes don't move. A control proves it.

Grammar: two general arms (v2.std.grammar)

• RefuseOnMatch { element, reason }: a production whose match is a refusal. Where element matches, the whole parse refuses with the typed ParseRefusalReason, located at the match's first token. It exists for source the seed refuses although another reading succeeds, which an Expect relabel can't express. It is named for what it is, so the next seed-refused success can reuse it.
• AfterLineBreak { element }: matches element only when the source holds a line break before it. On a sourceless stream it refuses with the new LayoutUnknownWithoutSource, never assuming one line.

Parse semantics (02_parse)

• A refusal is recorded on the parse table. From then on, parse_expr_with_first rejects every attempt at once, so no ordered-choice alternative can succeed around it, and the production door reports the refusal as the outcome even if the rest of the input parsed.
• Memo entries carry the refusal, so a warm hit replays it exactly as a cold parse would.
• The stream digest includes the layout (parse_token_stream_layout_digest, over the line-break-preceded token starts). Two layouts can lex to byte-identical tokens, and a memo entry must never serve one's answer to the other.
• The new arms are listed explicitly in every enumerated GrammarExpr/PreparedGrammarExpr match and every GrammarExprFold algebra; no wildcard absorbs them.

Emit, and both directions (DESIGN §4)

Emit and coercion read the formal productions and relation rows, never GrammarExpr, so they don't see these arms. RefuseOnMatch derives no tree for emit to select. In this PR no language grammar uses the arms, so an emit-side control here would be vacuous. The structural emit guarantee — the dag emitter never produces a line-break-preceded - at a statement boundary — lands in PR2, together with the dag grammar's first use of the arms and its emit-side control.

Controls (v2.test.parse.layout_line_break), all green

• x - 1 and x⏎- 1 lex to the same tokens, and the derivation tells them apart.
• A sourceless stream answers LineBreakUnknownWithoutSource.
• Carrying the source changes no parse tree (content hash with and without the source).
• Same tokens, different layout, different stream digests (the memo hazard).
• Hand grammar using both arms: a same-line - continues; a next-line - refuses at the - (bytes 2..3) although subtraction would have parsed; with no source the parse refuses LayoutUnknownWithoutSource at the - instead of accepting.

The existing parse modules I touched stay green: grammar_expect, expression_bodied_fn_decl_parse, parse_memo_frame_replay, parse_table_content_key, choice_plan.

The stream digest is a memo key and nothing else

parse_token_stream_layout_digest (the token digest combined with the stream's layout, when present: the starts of the line-break-preceded tokens, the one fact two identically-lexed sources can differ in) is stored as ParseTable.token_stream_digest. Its consumers, named:

• v2.compiler.parse parse_table_subject: the memo key (grammar digest + stream digest + position + production) that parse_table_lookup / parse_table_insert read.
• The seed's native memo intercept, v1_interpreter parse_table_memo_scope_and_key, which reads the table's token_stream_digest field as part of the same memo scope.

Nothing else reads it. Occurrence identity comes from the provenance allocator (parse_prov_seeded / OccurrenceIdAllocator), and semantic hashes are content_hash over the tree. Neither reads the stream digest, and the identity control shows the tree hash is unchanged with the source attached.

Consumption (§3c): a declared frontier

In production, RefuseOnMatch, AfterLineBreak, LayoutUnknownWithoutSource and the stream's source (as read by a layout question) are unconsumed by this PR: only the controls exercise them. They are a declared frontier with a named consumer.

• Consumer: PR2, the block-headed left-operand change on session/calm-fox-43-block-left-operand-v2, stacked on this branch. There the dag grammar's additive continuation carries RefuseOnMatch { AfterLineBreak { - }, NewlineBeforeDualRoleOperator }, refusing a newline-preceded dual-role - exactly where the seed does, for every statement form.
• Trigger that retires the frontier: PR2 merges, and with it the RFM: v2 lexer carries no layout, so a newline before '-' merges two statements silently #12769 row's trigger ("the v2 token stream carries layout and the dag grammar refuses a dual-role operator that follows a newline at an operand boundary with a typed ParseRefusalReason").
• Landing order: PR2 is meant to enqueue directly behind this one. If PR2 does not land, this PR's arms are dangling and should not be left standing.

🤖 Generated with Claude Code

[gunbai-bot]

Re review 73189: fixed in ef567ab. The layout question no longer rescans the source. v2.std.compilers.lexing StreamLayout pairs the source with a line-break-before answer for every token, keyed by the token's start octet. It is derived in ONE walk over the source and the tokens by its only constructor, stream_layout_of, called from token_stream_lexed. So the source stays the only authority, the answers can't drift from it, and token_stream_line_break_before(layout, token_start) is a lookup. The parser's parse_line_break_before_first also no longer computes length(all_tokens). A line feed inside a token (a multi-line string) is not layout, because the walk counts line feeds only at or after the previous token's end. New control: a_line_feed_inside_a_token_is_not_layout_holds. The stream digest still includes the source, read through the layout.

Separately, CI on 8371ea0 refused carrying_the_source_changes_no_parse_tree_holds on the new-claim budget: 96,874 eval steps against 72,300, because it prepared the dag grammar twice. 22131ce parses against the floor-warmed dag_prepared_grammar instead. The assertion is unchanged.

— sent from calm-fox-43

[gunbai-bot]

Re review 73282, fixed in f7888e6:

• Two representations: agreed, fixed. StreamLayout is now one field, line_break_before: Map<Int, Int>. It maps each line-break-preceded token's start to the 1-based line it begins, so there is one representation of the set, and the value is a fact about the token rather than a placeholder Bool. The ordered list is gone. parse_token_stream_layout_digest folds over the same tokens the token digest folds and reads layout by lookup.
• Accumulator cost: the list is gone, so the question no longer arises. For the record, list_snoc_item is the primitive persistent push, O(log n) per the note at v2.compiler.parse parse_minted_id_snoc, not a copying append. The walk stays one pass over the source.
• Arms not yet used by a production grammar: declared frontier, consumed by PR2 (named in the PR body with its trigger).

— sent from calm-fox-43