fix(responses): strip ChatGPT citation control markers before the client - #3195
Conversation
The ChatGPT backend delimits inline citations with private-use characters (U+E200 open, U+E202 separate, U+E201 close). The desktop client renders them as source chips; the Codex TUI prints them literally, so a Copilot-routed answer showed citeturn1view0turn1view1 and the markers persisted into the saved transcript. OpenCodex neither emits nor understands that grammar - it arrives as ordinary assistant text - but the proxy is the last place that can remove it before a client that cannot render it. A span can straddle a delta boundary, so the streaming path uses a stateful filter that withholds an unterminated tail and releases it at close: a stream dying mid-marker must not swallow words the model produced. The accumulated text is stripped separately, because closeCurrentMessage re-sends it in output_text.done, content_part.done and output_item.done. Strip only. The turnNviewN ids are turn-scoped and opaque with no URL mapping in the response, so there is nothing to convert them into. Structured url_citation annotations are untouched, so desktop Sources chips keep working. Closes #3150
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
✅ Deterministic PR hygiene checks passed. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 00d1bce344
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // A citation span can straddle a delta boundary, so the filter withholds an | ||
| // unterminated tail and releases it at close (#3150). The accumulator above | ||
| // keeps the raw text; it is stripped once in closeCurrentMessage. | ||
| const visible = currentMsg.citationFilter.push(event.text); |
There was a problem hiding this comment.
For the reported github-copilot/gpt-5.6-sol route, src/providers/registry.ts:2872-2879 selects the openai-responses adapter, whose passthrough: true causes src/server/responses/core.ts:3611 to return through the native passthrough branch. That branch applies createGithubCopilotResponsesBlockRewrite at src/server/responses/core.ts:4615-4617 and never calls bridgeToResponsesSSE, so this new filter is bypassed and the citation markers still reach the Codex client in the exact scenario this commit claims to fix. Add equivalent stateful filtering to the Copilot passthrough block rewrite, including delta, done, and completed snapshot text fields.
Useful? React with 👍 / 👎.
| const start = combined.lastIndexOf(CITATION_MARKER_START); | ||
| if (start === -1) return stripCitationMarkers(combined); | ||
| const endAfterStart = combined.indexOf(CITATION_MARKER_END, start + 1); | ||
| if (endAfterStart !== -1) return stripCitationMarkers(combined); | ||
| // The trailing span is still open: emit everything before it, hold the rest. | ||
| held = combined.slice(start); | ||
| return stripCitationMarkers(combined.slice(0, start)); |
There was a problem hiding this comment.
When an open span receives another START before an END, such as chunks `${S}a` and `${S}b${E}`, lastIndexOf selects the second opener and stripCitationMarkers(combined.slice(0, start)) emits the first opener and a verbatim. The suffix is then removed, while closeCurrentMessage strips the entire raw span from its done snapshot, so live deltas leak a private-use marker and disagree with the finalized transcript. Preserve the already-held earliest opener, or otherwise make streamed output match stripCitationMarkers for every chunking.
Useful? React with 👍 / 👎.
…ent (lidge-jun#3195) The ChatGPT backend delimits inline citations with private-use characters (U+E200 open, U+E202 separate, U+E201 close). The desktop client renders them as source chips; the Codex TUI prints them literally, so a Copilot-routed answer showed citeturn1view0turn1view1 and the markers persisted into the saved transcript. OpenCodex neither emits nor understands that grammar - it arrives as ordinary assistant text - but the proxy is the last place that can remove it before a client that cannot render it. A span can straddle a delta boundary, so the streaming path uses a stateful filter that withholds an unterminated tail and releases it at close: a stream dying mid-marker must not swallow words the model produced. The accumulated text is stripped separately, because closeCurrentMessage re-sends it in output_text.done, content_part.done and output_item.done. Strip only. The turnNviewN ids are turn-scoped and opaque with no URL mapping in the response, so there is nothing to convert them into. Structured url_citation annotations are untouched, so desktop Sources chips keep working. Closes lidge-jun#3150 Co-authored-by: jun <jun@lidge.dev>
…ent (lidge-jun#3195) The ChatGPT backend delimits inline citations with private-use characters (U+E200 open, U+E202 separate, U+E201 close). The desktop client renders them as source chips; the Codex TUI prints them literally, so a Copilot-routed answer showed citeturn1view0turn1view1 and the markers persisted into the saved transcript. OpenCodex neither emits nor understands that grammar - it arrives as ordinary assistant text - but the proxy is the last place that can remove it before a client that cannot render it. A span can straddle a delta boundary, so the streaming path uses a stateful filter that withholds an unterminated tail and releases it at close: a stream dying mid-marker must not swallow words the model produced. The accumulated text is stripped separately, because closeCurrentMessage re-sends it in output_text.done, content_part.done and output_item.done. Strip only. The turnNviewN ids are turn-scoped and opaque with no URL mapping in the response, so there is nothing to convert them into. Structured url_citation annotations are untouched, so desktop Sources chips keep working. Closes lidge-jun#3150 Co-authored-by: jun <jun@lidge.dev>
1 participant
Summary
Codex CLI routed through OpenCodex to
github-copilot/gpt-5.6-solrendered assistant text as:Those are Unicode private-use characters —
U+E200opens a citation span,U+E202separates references,U+E201closes it. The desktop client renders them as source chips; the Codex TUI prints them literally, in both commentary and the final answer, and they persist into the saved transcript.Where they come from.
rgacrosssrc/for those codepoints,citeturn, and private-use handling returns nothing: OpenCodex neither emits nor recognizes this grammar. Its citation support is entirely structured (OcxUrlCitation,takeWebAnnotations()). So of the three origins the reporter proposed, it is the first — the markers are already literal text in the upstream response, because Copilot's backend is ChatGPT-derived.That still makes it ours to fix. The proxy is the last place that can see this text before a client that cannot render it.
The part that is easy to get wrong
Assistant text reaches the client through two paths, and a span can straddle a delta boundary —
\uE200citein one chunk, the rest in the next. A stateless per-delta strip emits the tail of a span it never recognized.So the streaming path uses a stateful filter that withholds an unterminated tail and releases it at close. Withholding rather than dropping matters: if a stream dies mid-marker, the bytes still reach the user instead of vanishing.
The accumulated text is stripped separately, because
closeCurrentMessage()re-sends it inoutput_text.done,content_part.done, andoutput_item.done— filtering only the deltas would leave the markers in the transcript even with a clean-looking stream.Scope
Strip only.
turn1view0is a turn-scoped opaque id and the response carries no mapping to a URL, so there is nothing to convert it into — the reporter's option 3 is the honest one. Structuredurl_citationannotations are untouched, so desktop Sources chips keep working.Closes #3150.
Verification
Exact head
00d1bce34:bun test ./tests/citation-markers.test.ts— 10 pass, 0 fail (unit: whole-string, multi-span, unchanged plain text, unterminated span preserved, stray delimiters left alone, one-character-at-a-time deltas, mid-span stream end, prefix emitted immediately).bun test ./tests/bridge.test.ts— 66 pass, 0 fail, including three new end-to-end cases asserting the codepoints andturn1view0are absent from every emitted event, not just the deltas.bun test ./tests/bridge-lifecycle.test.ts ./tests/bridge-live-delivery.test.ts ./tests/bridge-nonstreaming-terminal.test.ts ./tests/bridge-terminal-singleness.test.ts ./tests/responses-parser.test.ts— 84 pass, 0 fail.bun x tsc --noEmit— clean for the touched files.Red-green on both failure modes, since each protects a different half:
One honesty note: I could not obtain a live Copilot citation response, so this is verified against synthesized marker text matching the exact grammar the reporter documented, plus the real bridge end-to-end. If a live capture shows a variant shape, that is worth a follow-up.
Checklist
devurl_citationpath untouched; desktop rendering unaffected