Skip to content

fix(gateway): keep Telegram UTF-16 split points safe after natural break fallback - #11522

Closed
dwc1997 wants to merge 1 commit into
NousResearch:mainfrom
dwc1997:fix/telegram-utf16-split-boundary
Closed

fix(gateway): keep Telegram UTF-16 split points safe after natural break fallback#11522
dwc1997 wants to merge 1 commit into
NousResearch:mainfrom
dwc1997:fix/telegram-utf16-split-boundary

Conversation

@dwc1997

@dwc1997 dwc1997 commented Apr 17, 2026

Copy link
Copy Markdown

Re-clamp custom-length split positions after newline/space fallback so Telegram chunks remain UTF-16 safe. Add a regression test to ensure emoji-heavy natural-break chunking always stays encodable and within limits.

What does this PR do?

This PR fixes a Telegram message chunking edge case in truncate_message when len_fn=utf16_len is used.
After selecting a natural break (\n or space), the final split point could drift from the original UTF-16-safe boundary logic. This change re-clamps the split point after natural-break adjustment to ensure every produced chunk remains valid under UTF-16 constraints and safely encodable.
A regression test is also added to verify that emoji-heavy messages split by natural breaks remain UTF-16 safe and within configured limits.

Related Issue

Fixes #11467

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • Updated gateway/platforms/base.py:
    • In BasePlatformAdapter.truncate_message(...), added a post-natural-break UTF-16 re-clamp step:
      • Recompute a safe prefix using _prefix_within_utf16_limit(...)
      • Fallback to _custom_unit_to_cp(...) when needed to avoid invalid/empty split points
  • Added regression test in tests/gateway/test_platform_base.py:
    • test_natural_break_split_keeps_utf16_safe_boundaries
    • Verifies chunk encodability (utf-16-le) and UTF-16 length constraints for emoji-heavy content

How to Test

  1. Run:
    • python -m pytest -o addopts="-m 'not integration'" tests/gateway/test_platform_base.py -q
  2. Confirm the new test test_natural_break_split_keeps_utf16_safe_boundaries passes.
  3. (Optional) Reproduce with a long emoji-heavy message and verify chunking remains valid and no encoding-related delivery failure occurs.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Linux (RHEL-based), Python 3.11 venv

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery platform/telegram Telegram bot adapter labels Apr 24, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Competing fix with #11649 for the same issue (#11467). Both address UTF-16 surrogate pair splitting in Telegram message chunking — this PR focuses on the natural-break fallback re-clamping path.

@dwc1997 dwc1997 closed this Apr 29, 2026
@dwc1997 dwc1997 reopened this May 1, 2026
…tion

Re-clamp custom-length split positions after newline/space fallback so Telegram chunks remain UTF-16 safe. Add a regression test to ensure emoji-heavy natural-break chunking always stays encodable and within limits.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/telegram Telegram bot adapter type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Telegram message splitting can slice UTF-16 surrogate pairs, causing delivery failures

2 participants