Proper Unicode handling in port process output #2212
Conversation
The problem is that the shell util handlers receive a byte stream, which we handle as a list coming from a port. However the byte stream is utf-8 encoded in most systems, which breaks encoding expectations (binaries only) for Erlang. This patch turns the list of characters to a binary for all shell calls, before re-encoding it properly as unicode lists to keep the format compatible. Fixes erlang#2205
|
@lpil if you want to double-check it |
|
Great stuff, thank you. I'll take a look as soon as I can, though may not be until tomorrow or Monday as I'm currently traveling |
|
Works perfectly! Thank you! :) |
|
Hi gang, did this ever get released? I still have the same problem. |
|
Yeah it was released and I don't believe it was changed after the fact. Might need to go bisect to find which version reverted the change. |
|
Thank you. Would you like me to make another PR with these changes? |
|
Yeah if you feel like making the fix I'd be glad to review and get it through. |
|
Sorry @ferd I was looking at your repo rather than this one by mistake and thought that the patch had got lost in a bad rebase or similar. I don't know what the issue is and I'm too busy to dig into it properly. |
The problem is that the shell util handlers receive a byte stream, which
we handle as a list coming from a port. However the byte stream is utf-8
encoded in most systems, which breaks encoding expectations (binaries
only) for Erlang.
This patch turns the list of characters to a binary for all shell calls,
before re-encoding it properly as unicode lists to keep the format compatible.
Fixes #2205