-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Minor optimization of fix_utf8() #4234
Conversation
…need to catch that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR!
this change seems to optimize away a code path that is still very possible to hit, so we probably would want to keep the code in question for now
if six.PY2: | ||
text = unicode.translate(vistir.misc.to_text(text), UNICODE_TO_ASCII_TRANSLATION_MAP) # noqa | ||
return text | ||
return decode_output(text) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What leads you to believe this is not a possible code path? The above code block always attempts to encode the provided text into the desired encoding without any regard for what that encoding is; it might encode it to ascii
for example, or mbcs
. It is still very capable of raising this error:
>>> decode_output("✗ FAILED")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 13, in decode_output
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Previously I thought UnicodeDecodeError is already caught and it will be processed via vistir.misc.to_text() or output.translate(). Am I wrong?
Also I can not reproduce your problem, with python 2.7.17:
>>> "✗ FAILED".encode("utf-8")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
>>> decode_output("✗ FAILED")
u'\u2717 FAILED'
decode_output() does not raise UnicodeDecodeError, is it related with locale setting or terminal setting? Thanks again.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hrm... I swapped computers since yesterday and I can't reproduce it here; I suspect this relates in some way to #3131
Since I think we can still encounter this I am going to close this PR for now. I do feel this code has significant room for improvement but we should be careful not to accidentally break python 2 workarounds in the interim Thanks again for taking the time to dig into this, I'm pretty sure there are pieces of the terminal output encoding puzzle we can unravel a bit to improve the codebase for anyone willing to dive in |
Since decode_output() will not raise UnicodeDecodeError, there is no need to catch that.