rebar3 incorrectly printing stdout from rebar_utils:sh/pre_hook command

[lpil]

I have a shell command (the Gleam compiler) that prints text to stdout. When run within rebar3 using a pre_hook or rebar_utils:sh/2 some of the output is garbled and is incorrectly printed to the console.

Here is the expected output:

   ┌── /src/tetromino.gleam:2:10 ───
   │
 2 │   [1] == [1.0]
   │          ^^^^^
   │

Here is the output when run within rebar3.

===> Verifying dependencies...
===> Compiling unicode_print_error
===> Performing EUnit tests...
   ��� /src/tetromino.gleam:2:10 ���
   �
 2 �   [1] == [1.0]
   �          ^^^^^
   �

Finished in 0.005 seconds
0 tests

I have a reproduction of the issue here: lpil/learning@c6d9bfe

An example of how I am using rebar_util:sh/2 can be found here: https://github.com/gleam-lang/rebar_gleam/blob/2e9f8d4726f4c38db08e5dbc894c67da6159489f/src/rebar_gleam.erl#L15

Thank you

Environment

Rebar3 report
 version 3.13.0
 generated at 2020-01-14T20:33:27+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: eunit
Entered as:
  eunit
-----------------
Operating System: x86_64-apple-darwin18.6.0
ERTS: Erlang/OTP 21 [erts-10.3.5] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe]
Root Directory: /Users/louis/.asdf/installs/erlang/21.3.8
Library directory: /Users/louis/.asdf/installs/erlang/21.3.8/lib
-----------------
Loaded Applications:
bbmustache: 1.8.0
certifi: 2.5.1
cf: 0.2.2
common_test: 1.17.2
compiler: 7.3.2
crypto: 4.4.2
cth_readable: 1.4.6
dialyzer: 3.3.2
edoc: 0.10
erlware_commons: 1.3.1
eunit: 2.3.7
eunit_formatters: 0.5.0
getopt: 1.0.1
hipe: 3.18.3
inets: 7.0.7
kernel: 6.3.1
providers: 1.8.1
public_key: 1.6.6
relx: 3.33.0
sasl: 3.3
snmp: 5.2.12
ssl_verify_fun: 1.1.5
stdlib: 3.8.2
syntax_tools: 2.1.7
tools: 3.1

-----------------
Escript path: undefined
Providers:
  app_discovery as autotest clean compile compile cover ct cut deps dialyzer do docs edoc escriptize eunit get-deps help install install_deps key list lock new owner path pkgs publish release relup repo report repos retire revert search shell state tar tree unlock update upgrade upgrade upgrade user version xref

[ferd]

Yeah I think I get it. The problem is that the shell util handlers receive a byte stream, which we handle as a list coming from a port. However, that same format makes for invalid unicode. The fix should be rather straightforwards.

[lpil]

Can we re-open this? The fix was never released as there was a regression before the next version.

[ferd]

Sure. I forgot it was linked to the other issue. I have not had time to spend on this either recently but it's fine to reopen and track properly.

[lpil]

Thanks. I totally forgot as well!