-
Notifications
You must be signed in to change notification settings - Fork 29.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The behavior for unassigned codepoint of Shift_JIS is incompatible with WHATWG spec #43962
Labels
confirmed-bug
Issues with confirmed bugs.
encoding
Issues and PRs related to the TextEncoder and TextDecoder APIs.
Comments
daeyeon
added
the
encoding
Issues and PRs related to the TextEncoder and TextDecoder APIs.
label
Jul 23, 2022
cola119
changed the title
The behavior for unassigned codepoint of Shift_JIS is Incompatible with WHATWG spec
The behavior for unassigned codepoint of Shift_JIS is incompatible with WHATWG spec
Jul 23, 2022
Able to reproduce this on
@cola119 are you looking into |
@hemanth Lines 370 to 377 in 7ef069e
|
nodejs-github-bot
pushed a commit
that referenced
this issue
Jul 29, 2022
PR-URL: #43999 Fixes: #43962 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: LiviaMedeiros <[email protected]> Reviewed-By: Feng Yu <[email protected]>
danielleadams
pushed a commit
that referenced
this issue
Aug 16, 2022
PR-URL: #43999 Fixes: #43962 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: LiviaMedeiros <[email protected]> Reviewed-By: Feng Yu <[email protected]>
ruyadorno
pushed a commit
that referenced
this issue
Aug 23, 2022
PR-URL: #43999 Fixes: #43962 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: LiviaMedeiros <[email protected]> Reviewed-By: Feng Yu <[email protected]>
targos
pushed a commit
that referenced
this issue
Sep 5, 2022
PR-URL: #43999 Fixes: #43962 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: LiviaMedeiros <[email protected]> Reviewed-By: Feng Yu <[email protected]>
Fyko
pushed a commit
to Fyko/node
that referenced
this issue
Sep 15, 2022
PR-URL: nodejs#43999 Fixes: nodejs#43962 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: LiviaMedeiros <[email protected]> Reviewed-By: Feng Yu <[email protected]>
juanarbol
pushed a commit
that referenced
this issue
Oct 10, 2022
PR-URL: #43999 Fixes: #43962 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: LiviaMedeiros <[email protected]> Reviewed-By: Feng Yu <[email protected]>
juanarbol
pushed a commit
that referenced
this issue
Oct 11, 2022
PR-URL: #43999 Fixes: #43962 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: LiviaMedeiros <[email protected]> Reviewed-By: Feng Yu <[email protected]>
guangwong
pushed a commit
to noslate-project/node
that referenced
this issue
Jan 3, 2023
PR-URL: nodejs/node#43999 Fixes: nodejs/node#43962 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: LiviaMedeiros <[email protected]> Reviewed-By: Feng Yu <[email protected]>
guangwong
pushed a commit
to noslate-project/node
that referenced
this issue
Jan 3, 2023
PR-URL: nodejs/node#43999 Fixes: nodejs/node#43962 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Mohammed Keyvanzadeh <[email protected]> Reviewed-By: Darshan Sen <[email protected]> Reviewed-By: LiviaMedeiros <[email protected]> Reviewed-By: Feng Yu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
confirmed-bug
Issues with confirmed bugs.
encoding
Issues and PRs related to the TextEncoder and TextDecoder APIs.
Version
v18.5.0
Platform
No response
Subsystem
No response
What steps will reproduce the bug?
How often does it reproduce? Is there a required condition?
Always
What is the expected behavior?
According to WHATWG spec, any decoder should use
�(U+FFFD)
when an unassigned codepoint is found during decoding.What do you see instead?
From my investigation, ICU intentionally uses
\x1A
for unassigned codepoint on Shift_JIS encoding, and Node.js uses it as it is.Conversion Data - ICU Documentation
Which substitution character is used if a character cannot be converted?
Additional information
ICU provides the utility
ucnv_setSubstChars
to specify substitution characters for any encoding, and Node.js already has it in library. I'm working on this.The text was updated successfully, but these errors were encountered: