Skip to content

fix: use Unicode-aware word boundaries for non-ASCII language detection - #15708

Merged
diegosouzapw merged 3 commits into
diegosouzapw:release/v3.8.52from
dajiaohuang:fix/language-detector-non-ascii
Oct 7, 2026
Merged

diegosouzapw merged 3 commits into
diegosouzapw:release/v3.8.52from
dajiaohuang:fix/language-detector-non-ascii

Conversation

@dajiaohuang

Copy link
Copy Markdown
Contributor

Summary

Fixes language detection for non-ASCII scripts by replacing ASCII-only \b word boundaries with Unicode-aware lookarounds using (?<![\p{L}\p{N}]) and (?![\p{L}\p{N}]) with the u (Unicode) flag.

Problem

JavaScript's \b word boundary only works with ASCII letters [a-zA-Z0-9_]. This meant:

  • Cyrillic words (Russian) with accented or non-ASCII characters were not correctly matched at word boundaries
  • Accented Latin characters (Italian: perché, ciòe; Portuguese: você, código; Spanish: código; etc.) were also not correctly matched

Fix

  • Replace all \b boundaries with Unicode-aware lookarounds: (?<![\p{L}\p{N}]) (lookbehind: not preceded by letter/number) and (?![\p{L}\p{N}]) (lookahead: not followed by letter/number)
  • Add u flag to all regexes to enable Unicode property escapes
  • Update CJK regexes to also use u flag for consistency

Affected languages

  • Italian (it)
  • Brazilian Portuguese (pt-BR)
  • Spanish (es)
  • German (de)
  • French (fr)
  • Russian (ru)
  • Indonesian (id)
  • Japanese (ja) (flag update only)

Replace ASCII-only \b word boundaries with Unicode-aware lookarounds using
(?<![\p{L}\p{N}]) and (?![\p{L}\p{N}]) with 'u' flag to correctly match
words in Cyrillic (Russian) and accented Latin scripts (Italian, Portuguese,
Spanish, French, German, Indonesian).

- Fixes false negatives when non-ASCII words are adjacent to other letters
- Adds 'u' flag to all regex patterns for proper Unicode property escapes
- Maintains same matching logic for ASCII words while extending support to non-ASCII
@diegosouzapw
diegosouzapw changed the base branch from main to release/v3.8.52 October 7, 2026 18:29
dajiaohuang and others added 2 commits October 7, 2026 15:56
Refresh onto release/v3.8.52. Hungarian hints use the same lookbehind
boundaries, and an ASCII-adjacent fragment does not match.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
@diegosouzapw
diegosouzapw merged commit 3b644d6 into diegosouzapw:release/v3.8.52 Oct 7, 2026
7 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants