Skip to content

fix(tool_parser): keep literal </arg_value> inside GLM-4 arg values - #1942

Closed
santhreal wants to merge 2 commits into
smg-project:mainfrom
santhreal:upstream/glm4-arg-value-close-clean
Closed

santhreal wants to merge 2 commits into
smg-project:mainfrom
santhreal:upstream/glm4-arg-value-close-clean

Conversation

@santhreal

@santhreal santhreal commented Jul 19, 2026 •

Copy link
Copy Markdown
Contributor

Description

Problem

GLM-4 MoE argument parsing used a non-greedy </arg_value> match. A value that mentions the close tag (for example use </arg_value> in docs) was truncated at the first occurrence, so later keys were lost or values were wrong.

Solution

Accept a candidate </arg_value> only when what follows is end-of-args or a real <arg_key>…</arg_key><arg_value> pair. Applies to both GLM-4.5 and GLM-4.7 extractors.

Changes

  • crates/tool_parser/src/parsers/glm4_moe.rs: scan-based close detection instead of non-greedy regex capture
  • Regression tests in tool_parser_glm4_moe.rs and tool_parser_glm47_moe.rs

Test Plan

Before: <arg_value>use </arg_value> in docs</arg_value> stores only use.

After: stores use </arg_value> in docs, and a following real key still parses.

cargo +nightly fmt --all -- --check
cargo clippy -p tool-parser --all-targets -- -D warnings
cargo test -p tool-parser --test tool_parser_glm4_moe --test tool_parser_glm47_moe
Checklist
  • cargo +nightly fmt passes
  • cargo clippy --all-targets on tool-parser with -D warnings passes
  • (Optional) Documentation updated
  • (Optional) Please join us on Slack #sig-smg to discuss, review, and merge PRs

Summary by CodeRabbit

  • Bug Fixes

    • Improved tool-call argument parsing when tag-like text appears inside argument values.
    • Preserved embedded closing and opening argument tags as literal content instead of ending values prematurely.
    • Continued supporting schema-based value conversion, including numeric arguments.
  • Tests

    • Added coverage for complete and incremental parsing scenarios involving embedded tag text.

Non-greedy </arg_value> matching truncated values that mention the close
tag. Accept a candidate close only when what follows is end-of-args or a
real <arg_key>/<arg_value> pair.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>
@github-actions github-actions Bot added tests Test changes tool-parser Tool/function call parser changes labels Jul 19, 2026
@coderabbitai

coderabbitai Bot commented Jul 19, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@santhreal, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 53 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: c847db16-36b5-4986-abc4-7d9344d8d233

📥 Commits

Reviewing files that changed from the base of the PR and between 273bd7c and 37eac62.

📒 Files selected for processing (1)
  • crates/tool_parser/src/parsers/glm4_moe.rs
📝 Walkthrough

Walkthrough

Glm4MoeParser now scans argument tags contextually so tag-like text inside values is preserved. Schema-aware coercion remains intact, and GLM-4/4.7 integration tests cover embedded closing and argument-key tags.

Changes

GLM argument parsing

Layer / File(s) Summary
Context-aware argument scanning
crates/tool_parser/src/parsers/glm4_moe.rs
Replaces regex extraction with sequential tag scanning, validates closing tags against real subsequent arguments, and preserves existing coercion and tool-call parsing.
Embedded tag regression coverage
crates/tool_parser/tests/tool_parser_glm4_moe.rs, crates/tool_parser/tests/tool_parser_glm47_moe.rs
Adds coverage for literal </arg_value> and <arg_key>...</arg_key> content inside values, including parsing a following numeric argument.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested reviewers: slin1237, catherinesue

Poem

I’m a rabbit parsing tags in a row,
Keeping literal snippets wherever they go.
Close tags may hide, key tags may gleam,
Numbers still hop through the schema stream.
The GLM burrow is tidy and bright!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: preserving literal </arg_value> text during GLM-4 tool argument parsing.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request replaces the regex-based argument extraction in Glm4MoeParser with a manual string scanning parser to handle edge cases where argument values contain literal <arg_key> or </arg_value> tags, and adds corresponding unit tests. The reviewer suggested a code improvement in find_arg_value_close to make the parser more robust against trailing garbage by treating the current tag as closing if no subsequent closing tags exist.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

while let Some(rel) = value_and_rest[from..].find(needle) {
let abs = from + rel;
let after = &value_and_rest[abs + needle.len()..];
if after.trim_start().is_empty() || Self::follows_with_real_arg_key(after) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To make the parser more robust against trailing garbage or extra characters at the end of the tool call block, we can also accept the current </arg_value> if there are no subsequent </arg_value> tags in the remaining text. This ensures that the last </arg_value> is always treated as the closing tag even if there is trailing non-whitespace text.

Suggested change
if after.trim_start().is_empty() || Self::follows_with_real_arg_key(after) {
if after.trim_start().is_empty() || Self::follows_with_real_arg_key(after) || !after.contains(needle) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Accepted the final </arg_value> when no later close remains (!after.contains(needle)) in 37eac62.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 273bd7c9ab

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +107 to +109
let Some(close_rel) = Self::find_arg_value_close(rest) else {
break;
};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve complete arguments before malformed trailing fields

When a model closes a valid argument but then emits an incomplete next field before </tool_call> (for example, <arg_key>a</arg_key><arg_value>1</arg_value><arg_key>b</arg_key>), find_arg_value_close rejects a's real close because the follower is neither end-of-args nor a complete key/value pair, then returns None and this break drops a as well. The previous regex retained all complete preceding pairs, so this reduces recovery from malformed model output; accept the current close when the following key is incomplete, or otherwise preserve already-complete arguments.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The same change covers this: an incomplete following <arg_key> no longer blocks the prior complete close (37eac62).

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/tool_parser/src/parsers/glm4_moe.rs`:
- Around line 122-134: Update find_arg_value_close to retain the index of the
first matched </arg_value> while scanning for a strict closing-tag match. Return
the strict match when after is empty or follows_with_real_arg_key(after)
succeeds; otherwise, after the loop, return the stored first-match index so
trailing garbage does not drop the final argument.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: f30e91b5-528f-40a7-930b-115969e02ac8

📥 Commits

Reviewing files that changed from the base of the PR and between a1ed168 and 273bd7c.

📒 Files selected for processing (3)
  • crates/tool_parser/src/parsers/glm4_moe.rs
  • crates/tool_parser/tests/tool_parser_glm47_moe.rs
  • crates/tool_parser/tests/tool_parser_glm4_moe.rs

Comment thread crates/tool_parser/src/parsers/glm4_moe.rs
… remains

Keeps literal close tags inside values, while recovering when trailing
garbage or an incomplete next key follows the real close.

Signed-off-by: santhreal <64453045+santhreal@users.noreply.github.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 37eac62f6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +131 to +133
if after.trim_start().is_empty()
|| Self::follows_with_real_arg_key(after)
|| !after.contains(needle)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid recognizing literal argument examples as boundaries

When a string value contains an unescaped complete argument example—for example, documentation text containing </arg_value><arg_key>sample</arg_key><arg_value>x</arg_value> before the value's actual closing tag—follows_with_real_arg_key accepts the first literal close as the boundary. This truncates the enclosing value and parses sample as a spurious tool argument, despite this change intending to preserve literal </arg_value> text in values.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as-is. A value that embeds a full </arg_value><arg_key>…</arg_key><arg_value>… example is indistinguishable from a real next field without a richer grammar, and fixing it would reopen the literal-close case this PR targets. Happy to follow a maintainer preference if you want a different tradeoff.

@santhreal santhreal closed this Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests Test changes tool-parser Tool/function call parser changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant