Fixed encoding issue. See #35 #81
Conversation
| @@ -97,7 +97,8 @@ def decodeString(encoding, encodestr): | |||
| return encodestr.decode(encoding) | |||
| except UnicodeDecodeError as e: | |||
| print >> sys.stderr, encodestr, ":", e | |||
| return encodestr.decode(encoding, "ignore") | |||
| ascii_only = re.sub(r'[^\x00-\x7f]',r' ',encodestr) | |||
| return encodestr.decode(ascii_only, "ignore") | |||
There was a problem hiding this comment.
I'm just curious, isn't the whole point of "ignore" here to stop errors being thrown?
There was a problem hiding this comment.
I agree with @charleso,
and the fact that the original code does not work seems to indicate that the exception thrown is not the one you are trying to catch here.
According to the docs the exception to catch is
UnicodeError
instead of
UnicodeDecodeError
bytes.decode(encoding='utf-8', errors='strict')
bytearray.decode(encoding='utf-8', errors='strict')
Return a string decoded from the given bytes. Default encoding is 'utf-8'. errors may be given to set a different error handling scheme. The default for errors is 'strict', meaning that encoding errors raise a UnicodeError. Other possible values are 'ignore', 'replace' and any other name registered via codecs.register_error(), see section Error Handlers. For a list of possible encodings, see section Standard Encodings.
Just trying to see if that works...
Fixed encoding issue due to non-utf8 characters in commit messages.