Repository navigation
can we make modeled lex rules more elegant somehow? ModeledLexRules { root: LexRuleSet { - it would be nice to have a section for the resulting phrase (in text), and then associate them in a more cosmetic way - #5327
Merged
Conversation
A fixed-token phrase was authored three ways that could silently drift: a LexRuleSet of (token_class, literal-text) rules, a parallel ConcreteSyntaxToken sequence repeating the same classes, and a whole-phrase source-text string equal to the rule texts concatenated. DESIGN.md §2 horizontal: one concept, derive every use. std/compilers/lexing.dag: new LexicalPhrase carrier (ordered (token_class, fragment) spans) + span/phrase constructors + phrase_to_lex_rules / phrase_to_source_text projections. target_model.dag (which owns ConcreteSyntaxToken; import arrow runs target_model -> lexing) gets phrase_to_fixed_concrete_tokens. bash.dag: the 7 pure-FixedToken phrases are each authored ONCE as a LexicalPhrase whose fragment column reads as the shell text; lex/concrete_tokens/source_text all project from it. Removes the bash_lex_rule helper and two LexRuleSet builders (-463 lines in bash.dag). Bound-word phrases (bound_words/embedded_quote) stay literal pending a binding-carrying span (T-6). Verified by execution: all 11 bash_emit_command_test witnesses green (full lex -> grammar-row -> byte-identical emit pipeline); a scratch oracle confirmed each derived source_text is byte-identical to the literal it replaced and the token classes project (with a discriminating non-member -> false). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#5286 ("NanosecondDuration decomposition") relocated the nanosecond scale CHOICE out of std.measure to std.verification and made the cost ledger scale-generic (CostAccount<S>, RealizationPlan<S>, cost_account_time_count<S> replacing cost_account_time_nanos). #5225's realization_width.dag (landed concurrently) still imported the old names — a semantic merge-skew that left main's CI floor resolve RED (blocking every branch, since the floor scans the whole dsl tree). - nanosecond_duration now imported from std.verification (dropped the unused NanosecondDuration type import). - cost_account_time_nanos -> cost_account_time_count (scale-agnostic Nat count). - thread <S> through shard_balance_slot_for_cost / WidthBoundedRealization / width_bounded_realization / witness_width_fold_preserves_schedule_topology so CostAccount<S> / RealizationPlan<S> are well-formed; width never reads the ledger scale, so S only rides through. Verified: realization_width_witnesses() green by execution. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per reviewer/operator: author each phrase so the resulting shell text reads left-to-right like a sentence. phrase_text(text: [...], classes: [...]) — the `text:` row reads L→R as the phrase, the `classes:` row names each fragment's token class in the same order. Zipped into TokenSpans by zip_spans (typed over FreeMonoid so match resolves Cons/Empty; List is an opaque alias to the matcher). A text/classes desync surfaces RED in the emit witnesses, not a silent zip. Projections (lex/concrete/source_text) unchanged — byte-identical output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # dsl/std/realization_width.dag
The phrase_text authoring form builds spans via zip_spans directly, so the bare span()/phrase() constructors had no callers. Drop them — authoring is the single phrase_text(text:, classes:) form, paired by list order. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto-opened by session-dashboard for session
quick-dove-702.Pushing to
session/quick-dove-702advances this PR.Worker attestation
Before flipping this PR to ready for review, confirm each item:
npm test,cargo test) and the result.Closes #Ndirective.Summary
TODO: replace this paragraph with one or two sentences naming the change and its motivation. Reviewers read this first.
Test plan