fix(redact): stop masking from eating container delimiters

[ZHJay]

What does this PR do?

redact_sensitive_text() corrupts the document it is redacting. Five patterns bound the secret with a "not whitespace" value class ((\S+), or [^\s&]+? / [^\s&]++ for the config and YAML forms), which also admits the quote that closes an enclosing document. Because _mask_token masks anything under its 18-char floor to a bare ***, that quote is deleted rather than merely displaced.

This is the same defect class as #43083 — "_AUTH_HEADER_RE's greedy \S+ credential class ate a closing quote … turning value corruption into syntax corruption" — at the sibling patterns that fix did not touch: _ENV_ASSIGN_RE, _CFG_DOTTED_RE / _CFG_ANCHORED_RE (via _CFG_VALUE), _SECRET_HEADER_RE, and _YAML_ASSIGN_RE.

It needs no JSON to bite. On current main:

sh -c 'export MY_TOKEN=xyz'        ->  sh -c 'export MY_TOKEN=***      # unterminated quote
sh -c 'export MY_TOKEN='           ->  sh -c 'export MY_TOKEN=***      # empty value, same
docker run -e 'DB_PASSWORD=pw1'    ->  docker run -e 'DB_PASSWORD=***  # shell EOF
curl -H 'x-api-key: abc123'        ->  curl -H 'x-api-key: ***         # shell EOF

Why it matters: three consumers reparse redacted text, and each degrades differently

consumer	handler	user-visible result
agent_runtime_helpers.dump_api_request_debug	except Exception: return None	no request dump lands at all, for exactly the API failures the dump exists to explain
agent.trace_upload._tool_calls_to_blocks	raises TraceRedactionError	the whole upload is refused
tools.kanban_tools	except json.JSONDecodeError: pass	leaves the original unredacted dict bound — the secret is persisted verbatim

The third is a redaction bypass, not data loss.

Related Issue

Refs #77472 — the "redacted only by regex (force=True makes it a controlled residual)" item in cluster R-DUMP. The file-mode items from that issue are covered separately by #77520 / #77655 / #77717, and its unbounded-growth item by #78395; the "exact-value redaction on every persistence path" ask remains deliberately not implemented, per #43083 (masking a credential in a replayed path poisons the replay — guarded by tests/agent/test_tool_call_arg_no_redaction.py).

Type of Change

• 🔒 Security fix
• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

This body describes the mechanism as of 59bb144ff. Earlier pushes on this branch used different mechanisms and this body described one of them. Review found defects in each; see "Review found three defects across my earlier pushes" and "An adversarial re-audit found a P0 in this PR" below. I have corrected the claims rather than quietly replacing them.

• agent/redact.py
  • _DocumentStateScanner — tracks, for one sub() pass, which quote is open and which containers are open at a position. re.sub calls its callback on non-overlapping matches in increasing positional order, so the state is carried forward incrementally and a whole pass stays O(L). Escape-aware (the \" inside serialized JSON does not flip parity); { / [ are only counted while no quote is open, so a brace inside a string is content, not structure.
  • _justified_delimiter_len(run, stack) — how much of a trailing }]}} run is consistent with closing the containers actually open there, in order.
  • _split_trailing_delimiter(value, *, quote="", open_quote=None, stack=None) — splits a trailing delimiter off an unquoted capture and returns (secret, suffix); the suffix is re-emitted after the mask. Two requirements, both about what is open: the captured trailing quote must be the open one, and the structural run after it is truncated to its justified prefix. The rest is secret and is dropped — not re-emitted, and not handed back to the masker where the head/tail window could republish it.
  • _mask_preserving_delimiter(value, *, open_quote=None, stack=None) — for bare-value captures with no quote group of their own. Adds the balanced-pair case so x-api-key: "abc" → x-api-key: "***" instead of an unbalanced ***". That branch is reachable only from _SECRET_HEADER_RE; _YAML_ASSIGN_RE's (?!['\"]) lookahead rejects a value that starts with a quote. Verified by instrumentation — 0 firings from _redact_yaml over 2,872 inputs.
  • Wired into _redact_env (shared by _ENV_ASSIGN_RE, _CFG_DOTTED_RE, _CFG_ANCHORED_RE) and the _SECRET_HEADER_RE and _YAML_ASSIGN_RE passes. One scanner per sub() pass, since each pass runs over a freshly rebuilt subject.
  • _escapes_the_quote_at(text, quote_pos) — is the quote at that position backslash-escaped? Keyed on parity, not presence: json.dumps doubles a literal backslash, so a value genuinely ending in \ serializes to \\ and leaves an EVEN run before the real delimiter. Only an ODD run means the last backslash introduces an escape. This is the one rule behind every escape-aware branch in the module, and it is wired into three sites (see the P0 section below).
• tests/agent/test_redact.py — 73 → 356 tests.

The mechanism, and why it is bounded by document state rather than by a cap

The structural bytes a capture absorbs are document bytes — they must be re-emitted or the document loses them. But a secret's random }}}} run is distinguishable from a document's closing run, because the document's run has to be consistent with what is actually open at that position:

capture	open containers	emitted	masked
xyz"}]}}}	{ { { [ {	"}]}}} — fully justified	xyz
a'}}}}	{	'} — one closer justified	a + the rest dropped
pw',	{	', — separator, inside a container	pw
pw'}]},	(nothing open)	no split at all	the whole value

A separator is accepted only as the final character and only with a container open. The stack bounds closers, but , closes nothing, so a run of them is bounded by nothing — my own differential fuzzer caught ',,,, being republished whole before this landed.

Why not copy #43083's mechanism

The obvious move is to reuse that fix's shape: exclude quotes from the value class ((\S+) → ([^\s\"']+)). I implemented it and measured it head-to-head, then rejected it. It truncates at an interior quote and re-emits the tail verbatim:

                                    #43083 mechanism          this PR
PGPASSWORD=p'ass psql        ->  PGPASSWORD=***'ass       PGPASSWORD=*** psql
MY_SECRET=he"llo             ->  MY_SECRET=***"llo        MY_SECRET=***
DB_PASSWORD=x'y'z            ->  DB_PASSWORD=***'y'z      DB_PASSWORD=***

It also cannot cross the escaped \" inside serialized JSON, which leaks the whole value and still corrupts the document:

in   {"c": "MY_TOKEN=\"quotedshortvalue\""}
out  {"c": "MY_TOKEN=***"quotedshortvalue\""}     # leaked + unparseable

#43083's stated rationale — "real credentials never contain " or '" — holds for the opaque bearer tokens it was written for. It does not hold for passwords, which is what PGPASSWORD / _CFG_* match. Making a redactor leak more is the wrong direction, so the fix goes in the masking callbacks instead.

Review found three defects across my earlier pushes

I had this branch adversarially reviewed before asking for yours, twice. Both earlier commits were wrong, and one of my own claims in this body was wrong. Stating it plainly rather than editing history:

ca158df8b — the split was itself a disclosure channel. _TRAILING_DELIMITER_RE matched on shape alone (["'][}\],]*$) and the run was re-emitted verbatim, so a secret whose own tail looked structural got republished: MY_TOKEN=a'}}}… (33 chars) printed 32 of them, all of which the bare *** had covered on main. My first body described this as a one-character delta. That was simply wrong.

2df71234e — "present" is not "open", and the check was quadratic. That commit required the quote to appear earlier in the subject (quote in text[:pos]). Two further defects:

1. A prose apostrophe (it's, couldn't) satisfies a membership test, so the leak came straight back — 2,335 of 6,000 fuzz cases.
2. text[:pos] copies a prefix per match, i.e. O(N·L) per pass — 8.7 GB copied on a 1 MB input.

aad60d98c (this push) — the cap was not a property of anything. An intermediate iteration replaced presence with real quote parity (correct, kept) but bounded the structural run with a {0,3} cap. json.dumps' default separators on a five-level payload close with "}]}}} — five structural characters — so the cap rejected the genuine container tail and left exactly the corruption this PR exists to remove (5 consumer tests failed), while still admitting three characters of a secret's own }}}}. Replaced with the open-container-stack test above.

Measured across the three, same metric, same 2,772-input corpus — inputs where the candidate discloses more than the open-container stack justifies:

	ca158df8b	2df71234e	aad60d98c
inputs over the justified bound	321	128	0
valid-JSON inputs that reparse (of 462)	—	—	462 (base: 123)

Known limitations, stated plainly

1. Quote parity is syntactic, so an unbalanced quote earlier in the subject inverts it. A prose apostrophe, or a quote inside an earlier secret, leaves a quote nominally open for everything after it. Two consequences, both no worse than main:

• A genuine shell quote after prose is not repaired: it's fine, sh -c 'export MY_TOKEN=xyz' still loses its closing quote. main eats it too — verified in a clean worktree — so this is an un-repaired case, not a regression.
• A secret ending in that quote re-emits it. Bounded by 1 + the open container depth, drawn from "'}],; in flat prose that is a single character. Above the 18-char mask floor mask_secret's head/tail window dominates and base and fix disclose the same 4-char window; the residual only appears below the floor, where base emitted a bare ***.

This is asserted, not excused: test_odd_prefix_discloses_at_most_one_quote_beyond_the_baseline pins the output to base or base + the one quote, so the residual cannot grow silently.

2. A truly compact serializer (separators=(",", ":")) leaves no whitespace after the value, so the capture ends in } preceded by non-structural bytes — no delimiter-only suffix to split. That input is still corrupted, and the helper's docstring says so. It is unreachable from the three reparsing callers: dump_api_request_debug passes indent=2; trace_upload and kanban_tools use json.dumps' default ", ". Of the 52 separators=(",", ":") sites in the tree, none feed the redactor (wire framing, cache keys, canonical hashing). plugins/platforms/google_chat/adapter.py:1410 redacts an un-indented dump but only logs it, never reparses. Handling it generally means walking the parsed structure instead of running a text redactor across JSON boundaries — a caller-side change at three sites, so not here.

3. One shape stays unbalanced, identically to base: sh -c "MY_TOKEN="quotedshort"" — the input is genuinely ambiguous about which quote closes what. Byte-identical to main.

Intentional behavior deltas

Container-free output is byte-identical to main on 17 of 22 probe inputs; the 5 deltas are the repairs:

• A quoted header value keeps both quotes: x-api-key: "quoted" → x-api-key: "***" (was x-api-key: ***). Matches the contract Passwords get replaced by *** but model reads back its own conversation history and fails on second tool call. #43083 already pins for Authorization: (assert result.count('"') == 2).
• A long value's preserved tail now comes from the value rather than from the delimiter: MY_TOKEN=averylongsecretvalue inside JSON shows …alue where base showed …lue". Same 4-char budget, correct bytes.

An adversarial re-audit found a P0 in this PR, and two more sites of the same defect

I had this branch attacked rather than reviewed. It found a regression I introduced, plus two pre-existing siblings of the same mechanism. All three are fixed in 5e0656bdf, 971dca686, 59bb144ff.

The P0 (mine). _split_trailing_delimiter compared the captured trailing quote to the quote open at the match, but never asked whether that quote was escaped. Inside a serialized JSON string the enclosing quote appears as \", and the (\S+) value classes absorb both bytes — so the split cut between the backslash and the quote. The backslash went into the secret handed to _mask_token (deleted outright below the 18-char floor, where the mask is a bare ***) and the quote was re-emitted bare, terminating the string early:

in    {"content": "sh -c \"export MY_TOKEN=9\" && echo done"}
main  {"content": "sh -c \"export MY_TOKEN=*** && echo done"}   parses
prev  {"content": "sh -c \"export MY_TOKEN=***" && echo done"}  BROKEN
now   {"content": "sh -c \"export MY_TOKEN=***\" && echo done"} parses

Why every sweep in this body missed it. The defect needs whitespace and more content after the escaped quote, so the capture terminates exactly at \". When the escaped quote is the last thing in the string the capture runs past it and the $-anchored search lands on the document's real quote instead — which is the shape all my earlier corpora generated. json.dumps of a plain value never produces the failing shape either, so 11,664 valid-JSON probes also reported clean. My corpus was structurally blind to the family, and the fuzz numbers above should be read with that in mind.

Why it was a merge blocker, not a cosmetic regression. tools/kanban_tools.py keeps the original unredacted dict on JSONDecodeError, so every corrupted document is a redaction bypass that persists the secret verbatim. Measured on 4,800 cases (3 families × lengths 1-40 × quote " and ' × indent None/2 × last / not-last / nested):

metric	base 1be70d635	pre-guard aad60d98c	guarded 59bb144ff
reparse OK	3882/4800	4032/4800	4800/4800
kanban raw secret persisted	918	768	0
reparse regressions vs base	—	288	0

Independently re-measured on a smaller focused grid (320 P0+authz probes, 120 _JSON_FIELD_RE probes), importing agent.redact from each worktree with an asserted __file__: reparse failures 32 → 128 → 0, kanban raw-persisted 32 → 128 → 0, JSON-field failures 48 → 48 → 0.

Sibling 1 — _AUTH_HEADER_RE, pre-existing on main. Its credential class [^\s\"']+ excludes the quote (the #43083 exclusion, doing its job) but not the backslash that escapes it. The capture therefore ends …secret\, masking destroys the escape, and the document's own " is left unescaped. It only corrupted credentials under the mask floor, which made it look like a floor bug — above the floor _mask_token's 4-char tail happened to include the backslash, so the escape survived by accident of the window:

len=16  captured='…abcdefgh\'   mask='***'           reparse FAIL
len=17  captured='…abcdefghi\'  mask='k9v3zq...ghi\' reparse OK

Fixed with the same parity rule rather than by widening the value class — excluding \ would stop the capture at any backslash and re-emit the rest verbatim, the lossy outcome test_interior_quote_value_is_fully_masked already rules out for the sibling patterns. authz reparse 1920/2400 → 1920/2400 → 2400/2400.

Sibling 2 — _JSON_FIELD_RE, pre-existing on main. [^"]+ stops at the first " even when it is escaped, so this is the one site reachable without a serialized document nested inside a string — an ordinary one-level {"password": "abc\""} corrupts:

main  {"password": "***""}     broken
now   {"password": "***\""}    parses

Scope, pinned by a test: the fix makes the document parseable, it does not extend the masked span — [^"]+ still ends the value at the first quote, so bytes after an interior quote are re-emitted verbatim as on main. The improvement is still strict, because an unparseable document makes kanban_tools discard the redaction and persist the whole original value. Field cases 12/30 → 30/30, no new leaks. Deliberately not switched to (?:[^"\\]|\\.)+: that regex does not match at all when the closing quote is missing (truncated or malformed JSON, which a redactor does see), turning a masked value into an unmasked one — a disclosure regression on exactly the inputs least able to afford it.

Disclosure delta, stated rather than buried. Both escape fixes hand the masker the correct token instead of token + \, which shifts _mask_token's tail window left by one byte. Measured: 378 of 1600 cases disclose exactly one more byte than the pre-guard revision, all at credential length ≥ 18, all inside the true secret's own last-4 window, 0 outside it, and 0 cases gain disclosure while still failing to reparse. Below the floor both mask to ***, so lengths 1-17 disclose strictly less than main. This is head6/tail4 applied to the right token, not a new channel — the same effect the delimiter split has, one byte instead of four. The _split_trailing_delimiter docstring now states this bound; the earlier suffix-only bound (delimiter-only, ≤ 1 + depth) was true but did not describe it.

Knowingly left, all unchanged from main:

• An inner shell quote nested inside a JSON string (sh -c 'export MY_TOKEN=x') still loses its '. json.dumps does not escape ', so the quote open at the match is the JSON string's own and a text redactor cannot tell that ' from content. The document parses; only the inner shell quote is lost.
• _JSON_FIELD_RE truncation (above) — parseable, not fully masked. Pinned by test_json_field_interior_quote_parses_but_still_truncates so it is a recorded limit rather than a surprise.
• Two pre-existing defects outside this PR's premise, both filed as separate work: x-api-key: <val>' at depth 0 loses its quote to _YAML_ASSIGN_RE → _SECRET_HEADER_RE double-masking within one call, and api_key: <secret> inside a JSON string is redacted by no tree (_SECRET_HEADER_NAMES lists api-key/apikey but not api_key).

Cost. The authz pass now builds a scanner: ~4 ms on a synthetic 112 KB payload with 2,000 Authorization matches (10.3 → 14.3 ms). Gate-skipped text is unaffected (23.4 → 24.9 ms on a 297 KB secret-free log).

How to Test

1. scripts/run_tests.sh tests/agent/test_redact.py — 356 passed (was 73 at the merge base; an earlier revision of this body said 79, which was wrong — I re-measured at 3aeff239b, f5be9236e, 4075c8fd5 and 1be70d635 and got 73 every time).

2. Teeth check. Copy the new test file onto clean upstream/main and run it: 83 of 259 fail at the pre-guard revision; against the three trees, the escaped-quote class alone gives base 37 failed / 33 passed, pre-guard 46 failed / 24 passed, guarded 70 passed.

3. Differential fuzz, base vs fix, 2,772 inputs (11 container shapes × 7 prose prefixes × 6 key patterns × 6 secret shapes, both quote kinds, both serializers, secrets above and below the mask floor):

   • 0 inputs where the fix discloses more than the open-container stack justifies (ca158df8b: 321, 2df71234e: 128).
   • Instrumenting the helper over the same corpus: 2,054 re-emitted suffixes, 0 containing a non-delimiter byte, 0 longer than 1 + open depth.
   • Payload bytes disclosed through the mask window: 0 inputs where the fix shows a longer run than base; 128 where it shows fewer.
   • Of the 462 corpus inputs that are valid JSON before redaction, base reparses 123, the fix reparses 462.

4. Mutation testing — 10 mutations, 10 caught: neuter parity; treat nothing-open as a match; neuter the stack check; drop the structural run; re-add the old match.start() == 0 guard; neuter escape-awareness; ignore quote state when tracking containers; make the separator non-terminal; re-add the {0,3} cap; re-emit the rejected run. Three of those survived the first run — the escaped-quote fixture used an even number of \" (parity-neutral either way) and there was no test for separator position. Both gaps are now covered, which is why this push adds test_odd_escaped_quote_count_does_not_flip_parity and test_separator_is_accepted_only_as_the_final_character.

5. Consumer regression pass: scripts/run_tests.sh tests/agent/test_redact.py tests/agent/test_trace_upload.py tests/tools/test_kanban_redaction.py tests/agent/test_tool_call_arg_no_redaction.py tests/monitoring/test_export_redaction.py — 275 passed. (test_tool_call_arg_no_redaction.py is the Passwords get replaced by *** but model reads back its own conversation history and fails on second tool call. #43083 guard: this PR does not re-introduce history redaction.)

6. Blast radius over every redaction-touching test file — scripts/run_tests.sh $(rg -ln "redact" tests/ | rg -v " 2\.py") — 84 files, 3012 passed, 2 failed; both reproduce identically on clean upstream/main (test_approval.py::…nonrecursive_verification_artifact_cleanup…, test_api_server.py::…health_detailed_returns_ok).

7. Sibling audit — 24 patterns × 4 container shapes (both serializers + both shell quotings): base corrupts 22 cases, the fix 1 (limitation 3 above, byte-identical to base). That audit was run before the escape work below. It is now out of date in one direction: _AUTH_HEADER_RE and _JSON_FIELD_RE are no longer unaffected — both carried the same escape-destruction defect on main and both are now fixed (see below). The remaining patterns are still byte-for-byte unaffected, each for a stated reason: _TELEGRAM_RE and _PRIVATE_KEY_RE emit fixed replacement text; _JWT_RE, _SIGNAL_PHONE_RE and _PREFIX_RE have value classes that exclude both " and \; _DB_CONNSTR_RE and _URL_BARE_TOKEN_RE do admit both, but their captures are bounded by a mandatory following @, so an escape and its quote are always consumed together — that deletes content but never leaves a bare quote, so no syntax corruption.

8. Perf. The scanner is incremental, so a pass is O(L). Measured on the shape that actually reaches the split path (every value ends in a quote, so every match pays the cost), min of 3:

   matches	base	2df71234e	this push
   2,000	19.3 ms	16.8 ms	21.4 ms
   4,000	39.2 ms	40.8 ms	43.4 ms
   8,000	78.5 ms	93.8 ms	88.7 ms
   16,000	157.4 ms	218.4 ms	175.2 ms
   µs/match, 2k → 16k	9.64 → 9.84	8.42 → 13.65	10.69 → 10.95
   scaling per doubling	×2.00	×2.33	×1.98

   A realistic 637 KB indent=2 request dump (2,400 messages, three secret shapes per message): base 85 ms → fix 129 ms, reparses under both. The ~1.1 µs/match constant is the scanner step; the slope is flat.

9. Full CI-parity suite — scripts/run_tests.sh — 3,093 files, 28,599 passed, 52 failed. None are redaction-related (no failing test name or file matches redact). I re-ran every failing file against clean upstream/main to attribute them: 32 reproduce identically there, and the remaining 20 are all files that do not exist on upstream/main — stale untracked "… 2.py" duplicates in my local tree. Of the three tracked files that differ, two are whole-tree AST scans (test_update_zip_two_phase.py, test_managed_runtime_resolution.py) that fail on a stray untracked hermes_cli/update_cmd 2.py in my working copy, not on anything this PR touches, and the third (test_telegram_start_polling_timeout.py) fails on the parent commit 2df71234e as well. Nothing in the failure set is attributable to this change.

Platforms tested: macOS 27.0 (Darwin 27.0.0, arm64), Python 3.11.15 (venv) — pure-stdlib re and string handling, no platform-conditional code, no filesystem or path behavior, so Linux / WSL2 / Windows behavior is unchanged. No config keys added, no new env vars, no docs impact.

CI has never run on this PR. Fork PRs from contributors without a merged PR are gated (check-suites = action_required), so every number above is a local run on this commit, not a CI result.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix
• I've run the suite and all tests pass (pre-existing failures unchanged — see How to Test 6)
• I've added tests for my changes
• I've tested on my platform: macOS 27.0 (Darwin 27.0.0, arm64), Python 3.11.15

Documentation & Housekeeping

• Documentation — N/A (internal helpers; behavior documented in docstrings, including the residual and the known limitations)
• cli-config.yaml.example — N/A (no config keys added)
• CONTRIBUTING.md / AGENTS.md — N/A (no architecture or workflow change)
• Cross-platform impact considered — see Platforms tested

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[ZHJay]

Independent verification of the second commit (2df71234e), run by a second pair of eyes against a clean worktree at that SHA rather than by the author. Posting it because this PR carries sweeper:risk-security-boundary and the first commit's disclosure claim turned out to be wrong — that's exactly the kind of thing worth re-measuring rather than taking on trust.

The disclosure channel is closed. For each probe below, I searched the output for any suffix of the secret of length ≥ 2, not just the whole run:

--- 33-char structural tail
  IN : "MY_TOKEN=a'}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}"
  OUT: 'MY_TOKEN=***'
  secret bytes re-emitted: 0
--- amp + brace tail
  IN : "MY_TOKEN=Tr0ub4dor&3'}"
  OUT: 'MY_TOKEN=***'
  secret bytes re-emitted: 0
--- YAML pass in shell
  IN : "sh -c '\npassword: xyz'"
  OUT: "sh -c '\npassword: ***'"
  secret bytes re-emitted: 0   squotes=2 balanced=True
--- empty value
  IN : "sh -c 'export MY_TOKEN='"
  OUT: "sh -c 'export MY_TOKEN=***'"
  secret bytes re-emitted: 0   squotes=2 balanced=True
--- compact-ish json empty value
  IN : {"c": "MY_TOKEN=", "b": 1}
  OUT: {"c": "MY_TOKEN=***", "b": 1}
  reparses=True

So the four rows in the PR's own before/after table hold on the real redact_sensitive_text() path: 32 bytes → 0, 2 bytes → 0, and both unterminated cases now balanced.

Worth noting for reviewers, because it changes how the first commit should be read: I had independently flagged _YAML_ASSIGN_RE as a missed sibling of this bug class while reviewing ca158df8b. Its value class is ([^\s&]++) — structurally the same "not whitespace" bound as _ENV_ASSIGN_RE and _SECRET_HEADER_RE — but it went through _redact_yaml, which the first commit didn't touch. On ca158df8b that reproduced as sh -c "\napi_key: abc" → sh -c "\napi_key: *** (one quote in, zero out). The ^[ \t]* anchor under re.MULTILINE does make it unreachable from serialized JSON specifically — json.dumps escapes a literal newline to \n, so ^ never matches mid-string, which is what the note at tests/agent/test_redact.py:865 is about — but it is plainly reachable from raw text, and the PR body's own gateway.run._redact_approval_command path (raw command string, code_file=False) is a concrete caller. Now fixed, and the fifth pattern is covered.

Suite at the new head:

$ ./scripts/run_tests.sh tests/agent/test_redact.py
=== Summary: 1 files, 155 tests passed, 0 failed (100% complete) in 45.2s (24 workers) ===

Two things I did not verify, stated so nobody reads more into this than it supports: I didn't re-run the 54-failures-on-clean-main teeth check, and I didn't re-run the full 3089-file suite — so the "identical failure set to upstream/main" claim in How to Test 6 is the author's measurement, not mine.

Separately, and this is the honest framing rather than a status complaint: no workflow run has ever executed on this PR. The check-suites sit at conclusion=action_required, which is this repo's approval gate for contributors without a merged PR — not a failure, and not something the branch can fix. Every number in this PR and in this comment is local evidence. Copilot's review also never ran here (it returned a quota-limit stub), so there is currently no automated review signal on the branch at all. If a maintainer approves the workflow runs, CI can speak for itself.

The branch merges cleanly into current upstream/main — verified by local test-merge, so the mergeable: UNKNOWN / BLOCKED GitHub shows is review-approval state, not a conflict. No rebase needed unless you want one behind #78138.

[ZHJay]

Correcting my own claims: review found defects in both of my earlier commits, and the PR body described a mechanism I no longer ship. Pushed aad60d98c with the replacement. Nothing was force-pushed or amended — the earlier commits stay in history so the sequence is auditable.

What was wrong

ca158df8b split any trailing run matching ["'][}\],]*$ and re-emitted it verbatim. Shape is not evidence: MY_TOKEN=a'}}}… (33 chars) republished 32 of them, all of which the bare *** had covered on main. My original body called this a one-character delta. That was wrong, and it is the more serious of the two because a redactor that emits more plaintext than before is a regression, not a fix.

2df71234e required the quote to appear earlier in the subject (quote in text[:pos]). Two further defects:

1. Presence is not openness. A prose apostrophe (it's, couldn't) satisfies a membership test, so the leak came straight back — 2,335 of 6,000 fuzz cases.
2. The check was quadratic. text[:pos] copies a prefix per match, i.e. O(N·L) per pass — 8.7 GB copied on a 1 MB input, and the ENV pass went 124 ms → 2.50 s at 2 MB.

An intermediate iteration I did not push fixed both (real escape-aware quote parity, carried forward incrementally so a pass stays O(L)) but bounded the structural run with a {0,3} cap. The cap was not a property of anything. json.dumps' default separators on a five-level payload close with "}]}}} — five structural characters — so the cap rejected the genuine container tail and reinstated exactly the corruption this PR exists to remove (5 consumer tests failed), while still admitting three characters of a secret's own }}}}.

What aad60d98c does instead

Parity is kept. The structural run is bounded by the containers actually open at the match rather than by a number. The scanner now tracks an open-container stack alongside the open quote (pushing on { / [ only while no quote is open, so a brace inside a string is content), and only the prefix of the run that closes those containers in order is re-emitted. The rest cannot be the document's — nothing is open for it to close — so it is secret, and it is dropped rather than handed back to the masker where the head/tail window could republish it.

capture xyz"}]}}}  open { { { [ {   ->  emit "}]}}}   mask xyz        # json.dumps defaults, 5 deep
capture a'}}}}     open {           ->  emit '}       mask a + drop   # secret's run, cut at open depth
capture pw'}]},    nothing open     ->  no split      mask everything

A separator is accepted only as the final character and only inside a container. The stack bounds closers, but , closes nothing, so a run of them is bounded by nothing — my differential fuzzer caught ',,,, being republished whole before this landed.

Evidence

Differential fuzz, 2,772 inputs, same metric across all three commits — inputs disclosing more than the open-container stack justifies:

	ca158df8b	2df71234e	aad60d98c
over the justified bound	321	128	0
µs/match, 2k → 16k matches	—	8.42 → 13.65	10.69 → 10.95 (×1.98/doubling)

Instrumenting the helper over the same corpus: 2,054 re-emitted suffixes, 0 containing a non-delimiter byte, 0 longer than 1 + open depth. Of the 462 corpus inputs that are valid JSON before redaction, main reparses 123 and this reparses 462. Sibling audit (24 patterns × 4 container shapes): main corrupts 22, this corrupts 1 (an ambiguous input, byte-identical to main), and 0 immune patterns changed.

Mutation testing: 10 mutations, 10 caught. Three survived the first run, and that is worth reporting rather than burying — my escaped-quote fixture used an even number of \", which is parity-neutral whether or not the scanner honours the backslash, so it could not catch a scanner that ignores escapes; and there was no test for separator position. Both gaps are covered now.

The residual, stated honestly

Quote parity is syntactic, so an unbalanced quote earlier in the subject inverts it for everything after. Two consequences, both no worse than main, both verified against a clean worktree:

• A genuine shell quote after prose is not repaired — it's fine, sh -c 'export MY_TOKEN=xyz' still loses its closing quote. main eats it too, so this is an un-repaired case rather than a regression.
• A secret ending in that quote re-emits it, bounded by 1 + open container depth and drawn from "'}],. In flat prose that is one character, and only below the 18-char mask floor; above the floor the head/tail window dominates and base and fix disclose the same 4-char window.

That is asserted rather than excused — test_odd_prefix_discloses_at_most_one_quote_beyond_the_baseline pins the output to base or base + the one quote, so it cannot grow silently — and it is in the helper docstring and the PR body.

Tests: 155 → 259, of which 83 fail against clean upstream/main. Blast radius unchanged (84 files, 3012 passed, the same 2 pre-existing failures). CI still has not run on this PR — fork PRs from contributors without a merged PR are gated — so all of the above is local, on this commit.