Read logic must consider unicode character lengths - #1221
Closed
jaredpar wants to merge 1 commit into
Closed
Conversation
The SourceTextStream type was operating under the assumption that Encoder.Convert was a non-throwing method so long as it was passed a destination buffer with at least one byte available for writing. The actual contract for Convert is it will not throw so long as it is able to write the result of converting at least one character to the destination buffer (or there is nothing to convert). In that case it will throw an ArgumentException indicating it attempting to do work but was unable to do so. The SourceTextStream type processes the characters in chunks according to the count passed into Read. This caused a bug when a character which was represented with more than one byte value was at the end of a logical chunk of text. The Converter would convert all the chars except the last one. But SourceTextStream continued processing because there was at least one byte left in the destination buffer and hence an exception was thrown. The fix is to not check for count > 0 when processing but instead count >= the maximum number of bytes the encoding could produce for a single character. Note: I did consider calling GetByteCount here instead but decided against it. It essentially forces the encoder to do the work of decoding the lead byte twice on every iteration of the loop. Seemed better to keep the simple worst case check here. closes dotnet#1197
Member
|
👍 This is classic! |
Member
|
LGTM |
1 similar comment
Member
|
LGTM |
jjonescz
pushed a commit
to jjonescz/roslyn
that referenced
this pull request
Apr 28, 2026
…3.1-preview1-to-release/3.1 [automated] Merge branch 'release/3.1-preview1' => 'release/3.1'
This was referenced May 7, 2026
dibarbet
pushed a commit
that referenced
this pull request
Aug 18, 2026
* Add .NET Reference assemblies Addeds .NET 10 references. In draft until I resolve the references for other platforms. * Update other packages * Update .NET 9 assemblies
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The SourceTextStream type was operating under the assumption that
Encoder.Convert was a non-throwing method so long as it was passed a
destination buffer with at least one byte available for writing. The
actual contract for Convert is it will not throw so long as it is able
to write the result of converting at least one character to the
destination buffer (or there is nothing to convert). In that case it
will throw an ArgumentException indicating it attempting to do work but
was unable to do so.
The SourceTextStream type processes the characters in chunks according
to the count passed into Read. This caused a bug when a character which
was represented with more than one byte value was at the end of a
logical chunk of text. The Converter would convert all the chars except
the last one. But SourceTextStream continued processing because there
was at least one byte left in the destination buffer and hence an
exception was thrown.
The fix is to not check for count > 0 when processing but instead count
>=the maximum number of bytes the encoding could produce for a singlecharacter.
Note: I did consider calling GetByteCount here instead but decided
against it. It essentially forces the encoder to do the work of
decoding the lead byte twice on every iteration of the loop. Seemed
better to keep the simple worst case check here.
closes #1197