fix(telegram): use UTF-16 code units for message length splitting - #8698
Closed
teknium1 wants to merge 1 commit into
Closed
fix(telegram): use UTF-16 code units for message length splitting#8698teknium1 wants to merge 1 commit into
teknium1 wants to merge 1 commit into
Conversation
Port from nearai/ironclaw#2304: Telegram's 4096 character limit is measured in UTF-16 code units, not Unicode codepoints. Characters outside the Basic Multilingual Plane (emoji like 😀, CJK Extension B, musical symbols) are surrogate pairs: 1 Python char but 2 UTF-16 units. Previously, truncate_message() used Python's len() which counts codepoints. This could produce chunks exceeding Telegram's actual limit when messages contain many astral-plane characters. Changes: - Add utf16_len() helper and _prefix_within_utf16_limit() for UTF-16-aware string measurement and truncation - Add _custom_unit_to_cp() binary-search helper that maps a custom-unit budget to the largest safe codepoint slice position - Update truncate_message() to accept optional len_fn parameter - Telegram adapter now passes len_fn=utf16_len when splitting messages - Fix fallback truncation in Telegram error handler to use _prefix_within_utf16_limit instead of codepoint slicing - Update send_message_tool.py to use utf16_len for Telegram platform - Add comprehensive tests: utf16_len, _prefix_within_utf16_limit, truncate_message with len_fn (emoji splitting, content preservation, code block handling) - Update mock lambdas in reply_mode tests to accept **kw for len_fn
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Port from nearai/ironclaw#2304: Telegram's 4096 character limit is measured in UTF-16 code units, not Unicode codepoints. Characters outside the Basic Multilingual Plane (emoji like 😀, CJK Extension B, musical symbols) are surrogate pairs — 1 Python char but 2 UTF-16 code units.
The Bug
truncate_message()used Python'slen()which counts codepoints. A message with 2049+ emoji (2049 codepoints = 4098 UTF-16 units) would pass thelen(msg) <= 4096check but exceed Telegram's actual limit, causing API errors.The Fix
utf16_len(s)— counts UTF-16 code units vialen(s.encode('utf-16-le')) // 2_prefix_within_utf16_limit(s, limit)— binary-search truncation respecting surrogate pair boundaries_custom_unit_to_cp(s, budget, len_fn)— maps a custom-unit budget to the largest safe codepoint slice positiontruncate_message(..., len_fn=utf16_len)— optional length function parameter; Telegram passesutf16_len, all other platforms use defaultlen()(zero behavior change)_prefix_within_utf16_limitinstead of codepoint slicing for the too-long fallbackFiles Changed
gateway/platforms/base.pyutf16_len,_prefix_within_utf16_limit,_custom_unit_to_cp; updatetruncate_messagewithlen_fnparamgateway/platforms/telegram.pylen_fn=utf16_lentotruncate_message; use_prefix_within_utf16_limitin error handlertools/send_message_tool.pyutf16_lenwhen splitting for Telegram insend_messagetooltests/gateway/test_platform_base.pyTestUtf16Len,TestPrefixWithinUtf16Limit,TestTruncateMessageUtf16tests/gateway/test_*_reply_mode.py**kwfor newlen_fnparameterTest Plan
test_platform_base.pypass (22 new)Practical Severity
Low-medium for typical agent output (mostly ASCII/Latin). Would trigger for users who receive messages with many emoji or CJK Extension B characters that push total UTF-16 length past 4096 while Python
len()shows ≤ 4096.Discovered via IronClaw PR Scout cron job — scanning nearai/ironclaw for features to port.