fix(telegram): prevent surrogate pair splitting in truncate_message - #11649
Closed
truenorth-lj wants to merge 1 commit into
Closed
truenorth-lj wants to merge 1 commit into
truenorth-lj wants to merge 1 commit into
Conversation
When model APIs deliver astral-plane characters (emoji like 🎩) as JSON-escaped surrogate pairs (\uD83C\uDFA9), some decoders store them as two separate code units in the Python string. truncate_message() could slice between the high and low surrogate, producing a lone surrogate that causes UnicodeEncodeError on delivery. Two fixes: - utf16_len: use 'surrogatepass' error handler so lone surrogates are counted instead of raising - truncate_message: after finding a natural break point, check that split_at does not land between a high and low surrogate Closes #11467 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5 tasks
alt-glitch
added
type/bug
19 tasks
Collaborator
|
Closing — issue #11467 is resolved on current |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #11467 — Telegram message splitting can slice UTF-16 surrogate pairs, causing
UnicodeEncodeErrordelivery failures.When model APIs deliver astral-plane characters (emoji like 🎩
U+1F3A9) as JSON-escaped surrogate pairs (\uD83C\uDFA9), some decoders store them as two separate code units in the Python string.truncate_message()could slice between the high and low surrogate, producing a lone surrogate that causesUnicodeEncodeError: 'utf-16-le' codec can't encode character '\udfa9'on delivery.Changes
utf16_len(): usesurrogatepasserror handler so lone surrogates are counted as one UTF-16 code unit each instead of raisingtruncate_message(): after finding a natural break point (newline/space/backtick adjustment), check thatsplit_atdoes not land between a high surrogate (U+D800..U+DBFF) and its low surrogate (U+DC00..U+DFFF); if it does, step back by one to keep the pair togetherFiles changed
gateway/platforms/base.py— core fix inutf16_lenandtruncate_messagetests/gateway/test_platform_base.py— regression test that constructs a string with surrogate-pair code units and verifies no chunk contains a lone surrogate after splittingTest plan
test_platform_base.pypasstest_no_lone_surrogates_after_splitvalidates the fix against the exact scenario from Telegram message splitting can slice UTF-16 surrogate pairs, causing delivery failures #11467🤖 Generated with Claude Code