Skip to content

Read logic must consider unicode character lengths - #1221

Closed
jaredpar wants to merge 1 commit into
dotnet:masterfrom
jaredpar:fix-1197
Closed

Read logic must consider unicode character lengths#1221
jaredpar wants to merge 1 commit into
dotnet:masterfrom
jaredpar:fix-1197

Conversation

@jaredpar

Copy link
Copy Markdown
Member

The SourceTextStream type was operating under the assumption that
Encoder.Convert was a non-throwing method so long as it was passed a
destination buffer with at least one byte available for writing. The
actual contract for Convert is it will not throw so long as it is able
to write the result of converting at least one character to the
destination buffer (or there is nothing to convert). In that case it
will throw an ArgumentException indicating it attempting to do work but
was unable to do so.

The SourceTextStream type processes the characters in chunks according
to the count passed into Read. This caused a bug when a character which
was represented with more than one byte value was at the end of a
logical chunk of text. The Converter would convert all the chars except
the last one. But SourceTextStream continued processing because there
was at least one byte left in the destination buffer and hence an
exception was thrown.

The fix is to not check for count > 0 when processing but instead count
>= the maximum number of bytes the encoding could produce for a single
character.

Note: I did consider calling GetByteCount here instead but decided
against it. It essentially forces the encoder to do the work of
decoding the lead byte twice on every iteration of the loop. Seemed
better to keep the simple worst case check here.

closes #1197

The SourceTextStream type was operating under the assumption that
Encoder.Convert was a non-throwing method so long as it was passed a
destination buffer with at least one byte available for writing.  The
actual contract for Convert is it will not throw so long as it is able
to write the result of converting at least one character to the
destination buffer (or there is nothing to convert).  In that case it
will throw an ArgumentException indicating it attempting to do work but
was unable to do so.

The SourceTextStream type processes the characters in chunks according
to the count passed into Read.  This caused a bug when a character which
was represented with more than one byte value was at the end of a
logical chunk of text.  The Converter would convert all the chars except
the last one.  But SourceTextStream continued processing because there
was at least one byte left in the destination buffer and hence an
exception was thrown.

The fix is to not check for count > 0 when processing but instead count
>= the maximum number of bytes the encoding could produce for a single
character.

Note: I did consider calling GetByteCount here instead but decided
against it.  It essentially forces the encoder to do the work of
decoding the lead byte twice on every iteration of the loop. Seemed
better to keep the simple worst case check here.

closes dotnet#1197
@jaredpar

Copy link
Copy Markdown
Member Author

CC @pharring, @gafter , @VSadov, @agocke

@gafter

gafter commented Mar 12, 2015

Copy link
Copy Markdown
Member

👍 This is classic!

@agocke

agocke commented Mar 12, 2015

Copy link
Copy Markdown
Member

LGTM

1 similar comment
@VSadov

VSadov commented Mar 12, 2015

Copy link
Copy Markdown
Member

LGTM

@jaredpar jaredpar closed this in 6313aa3 Mar 12, 2015
@jaredpar
jaredpar deleted the fix-1197 branch August 21, 2018 00:50
jjonescz pushed a commit to jjonescz/roslyn that referenced this pull request Apr 28, 2026
…3.1-preview1-to-release/3.1

[automated] Merge branch 'release/3.1-preview1' => 'release/3.1'
dibarbet pushed a commit that referenced this pull request Aug 18, 2026
* Add .NET Reference assemblies

Addeds .NET 10 references. In draft until I resolve the references for other platforms.

* Update other packages

* Update .NET 9 assemblies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Csc.exe crashes when compiling Roslyn with latest Roslyn

6 participants