You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The solution proposed for issue #25 does not, in fact, accurately detect file format when BOM is absent.
If the 32 first characters are pure ASCII, then 'ascii' encoding is "detected". When other characters are present, an encoding compatible with that character is chosen, but it needs not be the correct one.
For instance, with character é present in the first line, I get encoding 'cp-1252' on Windows, while other characters are present later in the file that can't be represented in that encoding (box drawing characters).
The classical coding specification -*- coding: utf-8 -*- is not considered at all.
The text was updated successfully, but these errors were encountered:
cfrings
changed the title
Unicode: encoding detection doesn't work
Unicode: encoding detection is inaccurate
Mar 6, 2018
The solution proposed for issue #25 does not, in fact, accurately detect file format when BOM is absent.
If the 32 first characters are pure ASCII, then 'ascii' encoding is "detected". When other characters are present, an encoding compatible with that character is chosen, but it needs not be the correct one.
For instance, with character
é
present in the first line, I get encoding 'cp-1252' on Windows, while other characters are present later in the file that can't be represented in that encoding (box drawing characters).The classical coding specification
-*- coding: utf-8 -*-
is not considered at all.The text was updated successfully, but these errors were encountered: