Repository navigation

feat(core): add arg_tokenizer and migrate git/grep/dotnet/golangci arg parsing onto it - #3681

Merged

KuSh merged 87 commits into

rtk-ai:developfrom

KuSh:feat/arg-tokenizer

Sep 12, 2026

Merged

feat(core): add arg_tokenizer and migrate git/grep/dotnet/golangci arg parsing onto it#3681
KuSh merged 87 commits into
rtk-ai:developfrom
KuSh:feat/arg-tokenizer

Commits on Sep 7, 2026
- refactor(core): add shared arg_tokenizer, migrate git.rs onto it
  
  KuSh
  and
  claude
  committed
  86c10c8
- refactor(search): migrate grep/rg arg parsing onto arg_tokenizer
  
  KuSh
  and
  claude
  committed
  b11e0d0
- fix(dotnet): restore -- before build/test/restore/format arg handling
  
  KuSh
  and
  claude
  committed
  978a575
- refactor(arg_tokenizer): add Msbuild dialect, migrate dotnet_cmd.rs flag checks
  
  KuSh
  and
  claude
  committed
  637a013
- fix(arg_tokenizer): Msbuild's -- is a forwarding boundary, not end-of-options
  
  KuSh
  and
  claude
  committed
  84f8516
- refactor(go): migrate golangci_cmd.rs subcommand detection onto arg_tokenizer
  
  KuSh
  and
  claude
  committed
  8311fa3
- style: alphabetize match arms/enums/lists, prefer _ over spelled-out catch-alls
  
  KuSh
  and
  claude
  committed
  396ef17
- refactor(arg_tokenizer): make case-insensitive flag lookup a Dialect property
  
  KuSh
  and
  claude
  committed
  6988a45
- fix(git): stop tokenizing run_log's args twice
  
  KuSh
  and
  claude
  committed
  d976d68
- fix: address second /code-review high pass (4 verified findings)
  
  KuSh
  and
  claude
  committed
  3698b94
- refactor(arg_tokenizer): genericize over AsRef<str>, simplify search.rs
  
  KuSh
  and
  claude
  committed
  70874e4
- refactor(dotnet): tokenize args once per invocation, not per helper call
  
  KuSh
  and
  claude
  committed
  e8caef4
- fix(go): give has_output_flag -- boundary awareness
  
  KuSh
  and
  claude
  committed
  9457a7f
- fix(dotnet): most dotnet_cmd.rs flags are double-dash-only, not -/--// interchangeable
  
  KuSh
  and
  claude
  committed
  8fd8b14
- fix(arg_tokenizer): never let a value-taking flag swallow the boundary --
  
  KuSh
  and
  claude
  committed
  8d37891
- fix: address lower-severity findings from third /code-review high pass
  
  KuSh
  and
  claude
  committed
  49041f4
- fix(golangci): restore -- boundary before classifying invocation args
  
  KuSh
  and
  claude
  committed
  b7eec87
- fix(git): restore -- boundary in run_add and migrate run_show to arg_tokenizer
  
  KuSh
  and
  claude
  committed
  1bec7f5
- test(git): add real-process regression for run_show's -- boundary fix
  
  KuSh
  and
  claude
  committed
  5f570c5
- fix(dotnet): has_nologo_arg must not match an attached-value spelling
  
  KuSh
  and
  claude
  committed
  f7720d8
- fix(git): restore -- boundary and migrate run_branch to arg_tokenizer
  
  KuSh
  and
  claude
  committed
  796c974
- refactor(dotnet): reuse the already-tokenized -- position in inject_report_trx_into_args
  
  KuSh
  and
  claude
  committed
  fd32fc2
- fix(search): migrate has_format_flag off its own inline mini-tokenizer
  
  KuSh
  and
  claude
  committed
  831fbc4
- refactor(dotnet): swap has_flag naming so the safe strict lookup is the short name
  
  KuSh
  and
  claude
  committed
  7ddaabf
- fix(arg_tokenizer): a multi-segment '/'-prefixed token is positional, not a flag
  
  KuSh
  and
  claude
  committed
  8f20216
- test(git): pin checkout -b/-B's glued short-flag capability against real git
  
  KuSh
  and
  claude
  committed
  9ecd50c
- refactor(git): centralize restore_double_dash at the run() dispatcher
  
  KuSh
  and
  claude
  committed
  9c314a4
- refactor(arg_tokenizer,git): dedupe the digit-run predicate
  
  KuSh
  and
  claude
  committed
  f17ffbd
- refactor(search): compute has_format_flag in extract_pattern_path's own token pass
  
  KuSh
  and
  claude
  committed
  74be125
- fix(git): restore -- boundary awareness in run_diff's wants_stat/wants_compact
  
  KuSh
  and
  claude
  committed
  413eb1b
- fix(search): make format-flag detection engine-aware for -L/-z
  
  KuSh
  and
  claude
  committed
  f83be19
- fix(dotnet): detect_test_runner_mode must not scan flag values as project paths
  
  KuSh
  and
  claude
  committed
  f9a53d0
- docs(search): document has_short_flag's reconstructed-string ambiguity
  
  KuSh
  and
  claude
  committed
  e55b415
- fix(dotnet): explicit_projects must stop at the user's own -- boundary
  
  KuSh
  and
  claude
  committed
  fcda021
- fix(git): run_branch's -u short flag must link its upstream value
  
  KuSh
  and
  claude
  committed
  a5879d5
- docs(golangci): document has_output_flag's global-vs-run-level scope gap
  
  KuSh
  and
  claude
  committed
  e901af4
- fix(golangci): scope has_output_flag's value predicate to run-level flags
  
  KuSh
  and
  claude
  committed
  a62bd30
- fix(arg_tokenizer,git): git's short-flag clustering doesn't match grep's
  
  KuSh
  and
  claude
  committed
  0701098
- fix(git): run_diff/run_show now share run_log's full raw-diff-shape flag list
  
  KuSh
  and
  claude
  committed
  23dd5ff
- docs(arg_tokenizer): restore flag_name_matches's own doc comment
  
  KuSh
  and
  claude
  committed
  1ae287b
- fix(search): give grep and rg each their own value-taking flag table
  
  KuSh
  and
  claude
  committed
  361057e
- fix(git): has_limit_flag must require -n to have actually captured a value
  
  KuSh
  and
  claude
  committed
  ac8e544
- fix(search): make -r value-taking detection engine-aware
  
  KuSh
  and
  claude
  committed
  2bdea45
- fix(git): run_stash's patch-mode check needs -- boundary awareness too
  
  KuSh
  and
  claude
  committed
  f7bd374
- fix(dotnet): expand dotnet_takes_value, and keep `/flag` from eating a value
  
  KuSh
  and
  claude
  committed
  08ec6eb
- fix(golangci): add missing out-format to golangci_run_takes_value
  
  KuSh
  and
  claude
  committed
  c108f24
- refactor(arg_tokenizer): dedupe double-dash-flag predicate; derive seen_dash_dash
  
  KuSh
  and
  claude
  committed
  7cacddb
- refactor(arg_tokenizer): add Token::is_free_positional(), dedupe 4 call sites
  
  KuSh
  and
  claude
  committed
  d05a705
- refactor(arg_tokenizer): add dashdash_index()/has_dashdash(), dedupe 4 call sites
  
  KuSh
  and
  claude
  committed
  0008b73
- refactor(git): add git_log_tokens() helper, dedupe 9 tokenize_git call sites
  
  KuSh
  and
  claude
  committed
  35305f1
- refactor(dotnet): dedupe has_nologo_arg/has_write_mode_override's shared predicate
  
  KuSh
  and
  claude
  committed
  ead95f1
- refactor(golangci): golangci_run_takes_value composes golangci_takes_value
  
  KuSh
  and
  claude
  committed
  95620f9
- refactor(arg_tokenizer): replace push_atomic_flag's 8 params with a Scanner struct
  
  KuSh
  and
  claude
  committed
  f36eff2
- docs(dotnet): document explicit_projects' filesystem-backstop limitation
  
  KuSh
  and
  claude
  committed
  234e88b
- fix(search): compute context/show_file/show_line from tokens, not reconstructed strings
  
  KuSh
  and
  claude
  committed
  15b52c7
- fix(git): stop git diff -p / git show -p from bypassing RTK's compaction
  
  KuSh
  and
  claude
  committed
  71fe764
- refactor(arg_tokenizer): share one token() constructor across construction sites
  
  KuSh
  and
  claude
  committed
  25d0de6
- fix(dotnet): fix Windows CI unused-import/dead-code in dotnet_double_dash_test.rs
  
  KuSh
  and
  claude
  committed
  8b2b18a
- refactor(arg_tokenizer): rename tokenize_git to tokenize_with_separate_value
  
  KuSh
  and
  claude
  committed
  9b40423
- fix(dotnet): reject attached-value spellings in has_report_trx_arg
  
  KuSh
  and
  claude
  committed
  210d67f
- refactor(git): delegate requests_diff_show_raw_shape to requests_raw_diff_shape
  
  KuSh
  and
  claude
  committed
  0a32e22
- fix(git): stop stash_show_wants_patch from swallowing -p as a log-only option's value
  
  KuSh
  and
  claude
  committed
  79f71fa
- fix(git): stop stash_show_wants_patch from treating -u as patch mode
  
  KuSh
  and
  claude
  committed
  f29760b
- fix(search): let grep/rg value-taking flags claim a literal -- as their value
  
  KuSh
  and
  claude
  committed
  2c21662
- fix(search): route on whether the engine reads stdin, not on a terminal check
  
  KuSh
  and
  claude
  committed
  a0e55f5
- docs(core): document arg_tokenizer.rs and the per-subcommand grammar rule
  
  KuSh
  and
  claude
  committed
  88d6785
- refactor: trim comment bloat across arg_tokenizer.rs and its callers
  
  KuSh
  and
  claude
  committed
  ddff880
- fix(git): tokenize emits_word_diff instead of scanning raw args
  
  KuSh
  and
  claude
  committed
  92c6755
- fix: scope every flag lookup to the region the tool actually parses
  
  KuSh
  and
  claude
  committed
  f897c7b
- docs: require arg_tokenizer for new commands, flags and arg fixes
  
  KuSh
  and
  claude
  committed
  6690142
- fix(git,search): repair the regressions the last round introduced
  
  KuSh
  and
  claude
  committed
  056c391
- fix(git,search): flags whose value or meaning the old scans got wrong
  
  KuSh
  and
  claude
  committed
  702d3d7
- fix(go,git,arg_tokenizer): close the last three from the review round
  
  KuSh
  and
  claude
  committed
  3606e63
- fix(git,dotnet,search): scope each rule to the grammar that owns it
  
  KuSh
  and
  claude
  committed
  6fcd489
- fix(git,search,go): drop the header's rival flags instead of outranking them
  
  KuSh
  and
  claude
  committed
  0907594
- refactor(search): one tokenizer entry point for the module
  
  KuSh
  and
  claude
  committed
  e507aa5
- fix(search,git,go): honour the engines' own flag arbitration
  
  KuSh
  and
  claude
  committed
  04001e0
- fix(dotnet,go,search): make the probes and lists say what they mean
  
  KuSh
  and
  claude
  committed
  e622c2e
- fix(git,search,dotnet): correct the flag classes the last round got wrong
  
  KuSh
  and
  claude
  committed
  7931993
- refactor(core): one value spec per flag instead of three predicates
  
  KuSh
  and
  claude
  committed
  eec9f33
- style: name each predicate after the grammar it encodes, not the file
  
  KuSh
  and
  claude
  committed
  63da972
- fix(git,search,go,dotnet): adapt the consumers tuned to the old parse
  
  KuSh
  and
  claude
  committed
  b6755bf
- fix(git): stop the log limit and the body suppressor overreaching
  
  KuSh
  and
  claude
  committed
  1b33154
- fix(git): let the user's own command be the one that reports
  
  KuSh
  and
  claude
  committed
  4a2e58d
- fix(git): --quiet is not a third spelling of -s in git show
  
  KuSh
  and
  claude
  committed
  f126a11
- test(git): pin the diff/show/log routing against real git
  
  KuSh
  and
  claude
  committed
  7272290
Commits on Sep 12, 2026
- Merge upstream/develop into feat/arg-tokenizer
  
  KuSh
  and
  claude
  committed
  6e16ad8

Commits

Commits on Sep 7, 2026

Commits on Sep 12, 2026