fix(agent): escape-aware JSON field redaction for embedded quotes

[fangliquanflq]

What does this PR do?

Makes _JSON_FIELD_RE escape-aware so JSON string values that contain embedded \" are masked as a whole, instead of truncating at the first raw quote character.

Failure mode (before)

_JSON_FIELD_RE used ([^"]+) for the value. In a JSON dump / log line like:

{"password": "foo\"bar_secret_value"}

the engine treats the " inside \" as the end of the value. Only the prefix foo\ is masked; the suffix stays in cleartext and the surrounding quotes no longer form a closed string:

# BEFORE (broken)
IN:  {"password": "foo\"bar_secret_value"}
OUT: {"password": "***"bar_secret_value"}

Two bad outcomes at once:

1. Partial secret leak — bar_secret_value remains readable in logs / tool output / transcripts.
2. Corrupt shape — "***"bar... is no longer valid JSON-looking text, which can confuse downstream parsers while still exposing the remainder.

Control without an escaped quote still worked:

{"password": "supersecretpassword123"}  ->  {"password": "***"}

Fix

Change the value capture to (?:\\.|[^"\\])+ (same escape-aware class already used by sibling JSON matchers for secret headers / cookies in #50514). _redact_json still wraps the masked token in quotes, so the whole field value is replaced and the suffix cannot leak outside the mask.

# AFTER
IN:  {"password": "foo\"bar_secret_value"}
OUT: {"password": "<masked>"}   # no bar_secret_value; no "***"bar corruption

Short values with an embedded escape fully mask to ***:

{"password": "a\"b"}  ->  {"password": "***"}

Related Issue

Related #50514 (escape-aware JSON value matching for Cookie / secret-header JSON forms; does not change general _JSON_FIELD_RE for password / token / secret / …).

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• 🔒 Security fix
• ✅ Tests (adding or improving test coverage)

Changes Made

• agent/redact.py: _JSON_FIELD_RE value group is escape-aware; comment documents the \" truncation / remainder-leak failure mode.
• tests/agent/test_redact.py: regression coverage for escaped quote (long + short) and escaped backslash JSON field values.

How to Test

1. Reproduce the broken shape on main (without this patch):

from agent.redact import redact_sensitive_text
print(repr(redact_sensitive_text(r'{"password": "foo\"bar_secret_value"}', force=True)))
# -> '{"password": "***"bar_secret_value"}'

2. With this branch:

from agent.redact import redact_sensitive_text
print(repr(redact_sensitive_text(r'{"password": "foo\"bar_secret_value"}', force=True)))
# bar_secret_value absent; no "***"bar corruption
print(repr(redact_sensitive_text(r'{"password": "a\"b"}', force=True)))
# -> '{"password": "***"}'

3. scripts/run_tests.sh tests/agent/test_redact.py -k TestJsonFields -q

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run focused tests via scripts/run_tests.sh and they pass
• I've added tests for my changes
• I've tested on my platform: Windows 11

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — N/A
• I've updated cli-config.yaml.example if I added/changed config keys — N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact (Windows, macOS) — pure regex change, no platform I/O
• I've updated tool descriptions/schemas if I changed tool behavior — N/A

[fangliquanflq]

ok

[teknium1]

Thanks for the focused redaction fix. The current matcher on main still uses ([^\"]+) at agent/redact.py:260, so the escaped-quote leak is real, and the proposed escape-aware matcher addresses it.

Problems

• tests/agent/test_redact.py:236 (and the adjacent new fixtures) uses raw strings with \" around the JSON field names and structural value quotes. Those backslashes become part of the input. Since _JSON_FIELD_RE requires the value quote immediately after :\s*, it cannot match this pseudo-JSON, so the assertions do not exercise the new regex and will fail.

Suggested changes

• Use unescaped structural JSON quotes in the raw literals, retaining escapes only inside the value: r'{"password": "foo\\"bar_secret_value"}'.

Automated hermes-sweeper review.

[egilewski]

not enough evidence

Revision-isolated probes show that the escape-aware JSON sensitive-field matcher on the submitted head prevents the escaped-quote suffix disclosure reproduced on current main. The required run-owned deterministic patch replay against current GitHub main was attempted, but it failed with patch_replay_conflict; the three-way merge places overlapping changes in tests/agent/test_redact.py, leaving no coherent resolved integration tree to validate. The submitted head alone passed focused escaped-quote, escaped-backslash, ordinary-sensitive-value, and non-sensitive-key probes, and agent/redact.py compiled successfully, but the focused pytest file and full suite could not be run because the leased checkout has no pytest-capable interpreter.

The stale/conflict status was not treated as a standalone blocker. This assessment is inconclusive because the integrated runtime and regression tests could not be validated on a resolved current-main tree, so I cannot classify the PR as mergeable or suggest source changes.

Signed: GPT-5.6-sol-xhigh in Codex

[fangliquanflq]

Rebased onto current main (cherry-picked the two commits; auto-merged cleanly with no remaining conflict in tests/agent/test_redact.py).

Force-pushed head: 42d115be6.

Focused verification on the integrated tree:

python scripts/run_tests_parallel.py tests/agent/test_redact.py -k TestJsonFields -q
# 5 passed

Escaped-quote / escaped-backslash probes match the intended mask behavior (whole value masked; no "***"bar... suffix leak). Ready for re-probe / patch replay against current main.

[egilewski]

suggesting changes

Security evidence:

• trust boundary: the changed text-redaction path must preserve valid JSON output while masking sensitive values.
• source/sink/invariant: the JSON-field matcher must consume escaped characters, and prefix/suffix preservation must not split an escape sequence; masked output must remain parseable and contain no raw suffix.
• current-main reproduction: current main leaks the suffix after an escaped quote and emits invalid JSON for {"password": "abcde\\\"long_secret_value_xyz"}.
• PR-head or patch-replay validation: the replayed head removes the suffix leak but emits {"password": "abcde\\..._xyz"}, which json.loads rejects because the preserved prefix ends on the backslash of \\\".
• positive/negative cases: escaped backslashes, ordinary sensitive fields, and non-sensitive fields remained valid, and tests/agent/test_redact.py passed all 76 tests on the replayed tree.
• residual bypass search: the changed _JSON_FIELD_RE is the only implementation found, and adjacent redaction paths did not provide a second JSON masking path.
• reviewer validation: a boundary probe reproduces the PR-head JSON corruption despite the focused regression tests passing.

Please adjust masking so preserved prefixes cannot end inside an escape sequence, and add a regression for an escaped quote at that boundary.

Not checked:

• Full test suite
• CodeRabbit review

Signed: GPT-5.6-sol-xhigh in Codex

[fangliquanflq]

Addressed the prefix/escape-boundary corruption on head 23158d554.

_redact_json now masks via _mask_json_field_value, which only cuts head/tail on complete JSON escape boundaries so a preserved prefix cannot end on the backslash of \".

Boundary probe on the updated head:

IN:  {"password": "abcde\"long_secret_value_xyz"}
OUT: {"password": "abcde..._xyz"}
# json.loads OK; no long_secret leak; no "***"long shape

Regression: test_json_escaped_quote_at_mask_prefix_boundary_stays_parseable

Focused verification:

python scripts/run_tests_parallel.py tests/agent/test_redact.py -q
# 70 passed

Ready for re-probe / patch replay against current main.

[egilewski]

suggesting changes

The changed JSON path introduces two separate linear-allocation amplifiers on attacker-controlled values: _JSON_FIELD_RE's repeated escape-aware alternation peaked at 191,495,636 traced bytes for a 1,000,000-character ordinary value, and _json_raw_safe_cuts() peaked at another 40,437,344 bytes while storing 1,000,001 Python integer boundaries. End to end, current main reduced the value to 29 output bytes in 0.019 seconds with a 1,002,696-byte traced peak, while the conflict-free PR replay produced the same output in 0.913 seconds with a 232,936,896-byte peak. Please make both matching and safe-cut selection use bounded auxiliary memory, or enforce a demonstrably safe bound before either allocation.

Security evidence:

• trust boundary: untrusted tool, model, and integration text crosses the central redaction path before logs, transcripts, and other downstream consumers.
• source/sink/invariant: a sensitive JSON value flows through _JSON_FIELD_RE, _redact_json, and _mask_json_field_value; redaction must mask it without retaining regex or Python-object state proportional to attacker-controlled input length.
• current-main reproduction: a bound 1,000,000-byte password value was reduced to 29 output bytes in 0.019 seconds with a 1,002,696-byte traced allocation peak.
• PR-head or patch-replay validation: the same exact input on the no-commit current-main replay produced the same 29 output bytes but took 0.913 seconds and peaked at 232,936,896 traced bytes.
• positive/negative cases: the functional escaped-quote fix worked, the exact head passed 70 focused tests, the replay passed 101, all 780 escape-boundary cases passed, and replay resource probes still peaked at 24,651,672 bytes for 100,000 characters and 60,494,550 bytes for 250,000.
• residual bypass search: isolated replay probes measured both the 1,000,001-element safe-cut list and the new repeated regex alternation as independent linear allocation sources, and central redactor call sites do not impose a general input-size bound.
• reviewer validation: every accepted probe asserted and printed agent/redact.py from the leased worktree, the replay merged without conflict onto current GitHub main, and fresh-process identical-input probes compared current main, the complete replay, the matcher, and the safe-cut helper.

Not checked:

• Full repository test suite
• CodeRabbit review

Signed: GPT-5.6-sol-xhigh in Codex

[fangliquanflq]

Addressed the bounded-memory review on 9edae7c4a.

The JSON field regex now matches only the field prefix; a linear scanner finds the closing quote without retaining regex state proportional to the value. Escape-safe mask cuts are computed in one pass with constant auxiliary state instead of storing every boundary.

Verification:

• scripts/run_tests.sh tests/agent/test_redact.py -q — 71 passed
• 1,000,000-byte sensitive JSON value probe from the real-environment directory — valid masked output, 1,001,375-byte traced peak
• committed-tip scan-only review — no findings

I have not claimed the new GitHub CI run is complete.

[egilewski]

looks mergeable

The escape-aware JSON scanner follows escaped characters through the unescaped closing quote and masks sensitive values without leaking suffixes or breaking serialized JSON, while preserving the environment-lookup exception. Focused regression, fuzz, Unicode and escape, and long-input checks found no source-backed security regression in the reviewed change.

Security evidence:

• trust boundary: Untrusted provider responses, tool output, logs, transcripts, and serialized request-dump fields enter redaction; redacted text is emitted to logs/transcripts and reparsed by the request-dump consumer.
• source/sink/invariant: Sensitive JSON fields are scanned through escaped pairs to the unescaped closing quote, and masking cuts only at escape boundaries before re-embedding; no secret suffix remains after an embedded quote and serialized JSON stays parseable.
• current-main reproduction: The previous matcher stopped at an escaped quote, allowing a suffix to remain and corrupting the JSON structure; the reviewed scanner consumes the complete escaped value and removes that suffix.
• PR-head or patch-replay validation: The reviewed head contains the scanner and regression coverage, and the change remains scoped to this JSON-redaction mechanism and its tests.
• positive/negative cases: Escaped quotes, escaped backslashes, prefix-boundary escapes, nested and duplicate fields, standard escapes, Unicode and surrogate escapes, long values, non-secret fields, short values, and environment-lookup snippets were exercised without parse failures or secret-body leaks.
• residual bypass search: Sibling redaction implementations and scanner guards were reviewed for search progression, incomplete escapes, empty values, escape-safe cuts, and downstream reparsing; no new bypass or sink corruption was found for valid JSON string fields.
• reviewer validation: Focused tests, baseline comparison, valid-JSON fuzzing, Unicode and escape combinations, and linear-time size probes all completed without an actionable source-backed security finding.

Not checked:

• broader test suite
• lint validation
• merge-tree validation

Signed: GPT-5.6-luna-max in Codex