Skip to content

Claims tokenize .dag text through tokenize_prepared(dag_prepared_lex()) - #13406

Open
gunbai-bot[bot] wants to merge 6 commits into
mainfrom
session/vivid-fox-372
Open

gunbai-bot[bot] wants to merge 6 commits into
mainfrom
session/vivid-fox-372

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #13294. 40 claim/support modules that tokenized .dag text with the dag rules (tokenize(rules: dag_lex()) or rules: lm.lex over dag_language_model()) now call tokenize_prepared(prepared: dag_prepared_lex()), so lex_compile_rules stops running per claim (#13294 floor: producer=lex_compile_rules claims=26 evals=26).

Left as-is: lexer-subject claims (test/claim/tokenize/*, self_host_contract/tokenize_emitted_contract), non-dag language claims, and production callers. Assertions unchanged.

Receipt (eval-step / CPU delta, cross-claim-demand line): to be posted from this PR's floor run.

🤖 Generated with Claude Code

gunbc-ci-auto-heal and others added 3 commits October 5, 2026 13:58
Test claims that tokenized .dag text with the dag rules called the one-text
door tokenize(rules: dag_lex() | lm.lex), re-running prepare_lex_rules /
lex_compile_rules per claim. They now share the floor's derived
dag_prepared_lex value. Assertions unchanged; lexer-subject claims
(tokenize/, self_host_contract) and non-dag languages keep their own rules.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both claims re-prepared the dag lex rules from dag_language_model().lex,
an argument the floor cannot share (ArgumentNotClosedConstant).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

On review 76613's one open point (fold_lowering_test.dag and the ambient name pool): the floor on run 37321193353 compiled and passed with that file switched to tokenize_prepared(..., prepared: dag_prepared_lex()), so both names resolve there. The current head is being re-checked by run 37328482005.

— sent from vivid-fox-372

@gunbai-bot

gunbai-bot Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Receipt from floor run 37340049841 on head cff2f6f, compared with #13294 floor run 37268749188:

  • [cross-claim-demand] lines for prepare_lex_rules / prepare_lex_rule_set / lex_compile_rules (claims=26, 517 / 510 / 365 ms total, ~497 ms cross-claim): gone. None of the three appears in the new run.
  • dag_prepared_lex shared by declared demand: 19 claims / 2 sites -> 265 claims / 47 sites, one fill (7 ms fill, 30 ms inclusive).
  • prepare_lex_rules unadmissible (ArgumentNotClosedConstant): 39 claims / 4 sites -> 35 / 4. These four sites are production folds that prepare lm.lex (reference_conservation, reference_conservation_census, 00_compile, program_assembly / closure_emission), reached by claims. The brief excludes production callers, so I left them. They are no longer a cross-claim-demand row.
  • CPU delta: about -0.5 s of per-claim lex preparation (the inclusive prepare_lex_rules cross-claim figure). Eval steps: the floor log does not attribute eval steps per producer, and the two runs select different claim sets (698 vs 774), so I have no clean whole-run step delta. The 14 touched claims that report shared-fill cost in both runs went from 17,835 to 14,529 measured CPU ms and from 4,437,775 to 4,349,149 measured eval steps.

Also switched, beyond the original 26 call sites: two claim sites, plus one new claim added on main, that re-prepared dag_language_model().lex inline. Assertions are unchanged throughout.

— sent from vivid-fox-372

@gunbai-bot
gunbai-bot Bot added this pull request to the merge queue Oct 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 6, 2026
@gunbai-bot
gunbai-bot Bot added this pull request to the merge queue Oct 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 7, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants