Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor optimization of fix_utf8() #4234

Closed
wants to merge 1 commit into from
Closed

Minor optimization of fix_utf8() #4234

wants to merge 1 commit into from

Conversation

ideal
Copy link

@ideal ideal commented May 4, 2020

Since decode_output() will not raise UnicodeDecodeError, there is no need to catch that.

@frostming frostming requested a review from techalchemy May 6, 2020 05:10
Copy link
Member

@techalchemy techalchemy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

this change seems to optimize away a code path that is still very possible to hit, so we probably would want to keep the code in question for now

if six.PY2:
text = unicode.translate(vistir.misc.to_text(text), UNICODE_TO_ASCII_TRANSLATION_MAP) # noqa
return text
return decode_output(text)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What leads you to believe this is not a possible code path? The above code block always attempts to encode the provided text into the desired encoding without any regard for what that encoding is; it might encode it to ascii for example, or mbcs. It is still very capable of raising this error:

>>> decode_output("✗ FAILED")                 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 13, in decode_output
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
                                                                                                                                 
'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

Copy link
Author

@ideal ideal May 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Previously I thought UnicodeDecodeError is already caught and it will be processed via vistir.misc.to_text() or output.translate(). Am I wrong?

Also I can not reproduce your problem, with python 2.7.17:

>>> "✗ FAILED".encode("utf-8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
>>> decode_output("✗ FAILED")
u'\u2717 FAILED'

decode_output() does not raise UnicodeDecodeError, is it related with locale setting or terminal setting? Thanks again.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hrm... I swapped computers since yesterday and I can't reproduce it here; I suspect this relates in some way to #3131

@techalchemy
Copy link
Member

Since I think we can still encounter this I am going to close this PR for now. I do feel this code has significant room for improvement but we should be careful not to accidentally break python 2 workarounds in the interim

Thanks again for taking the time to dig into this, I'm pretty sure there are pieces of the terminal output encoding puzzle we can unravel a bit to improve the codebase for anyone willing to dive in

@techalchemy techalchemy closed this May 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants