Erlang/OTP 26 stdin fails on file:read/file:read_line

[josevalim]

Describe the bug

$ echo "duπa" | erl -noshell -eval "io:setopts(standard_io, [{encoding, latin1}, {binary, true}]), io:format('~p~n', [file:read_line(standard_io)]), erlang:halt()"
{error, collect_line}

Expected behavior

On Erlang/OTP 25 it worked without problems:

$ echo "duπa" | erl -noshell -eval "io:setopts(standard_io, [{encoding, latin1}, {binary, true}]), io:format('~p~n', [file:read_line(standard_io)]), erlang:halt()"
{ok,<<100,117,207,128,97,10>>}

The goal is to read a bytestream, "duπa" is just an example.

Affected versions

Only on Erlang/OTP 26. This patch fixes stdout, perhaps similar is needed for stdin?

[lukaszsamson]

@frazze-jobb when can we expect a release with fix for that? This one is breaking elixirLS (Elixir Language Server)

[frazze-jobb]

Yes, am on parental leave this week, but will look into it next week

[frazze-jobb]

Would it be okey to start the shell later?
echo "duπa" | erl -noinput -eval "io:setopts(standard_io, [{encoding, latin1}, {binary, true}]), shell:start_interactive(noshell), io:format('~p~n', [file:read_line(standard_io)]), erlang:halt()"

In that case it would be possible to solve this similarly to #7211

[lukaszsamson]

@frazze-jobb I can start the shell later but I need something that will work on lower OTP versions. Your example crashes OTP 25

echo "duπa" | erl -noinput -eval "io:setopts(standard_io, [{encoding, latin1}, {binary, true}]), shell:start_interactive(noshell), io:format('~p~n', [file:read_line(standard_io)]), erlang:halt()"
{"init terminating in do_boot",{undef,[{shell,start_interactive,[noshell],[]},{erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,744}]},{erl_eval,exprs,6,[{file,"erl_eval.erl"},{line,136}]},{init,start_it,1,[]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({undef,[{shell,start_interactive,[noshell],[]},{erl_eval,do_apply,7,[{_},{_}]},{erl_eval,exprs,6,[{_},{_}]},{init,start_it,1,[]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

Crash dump is being written to: erl_crash.dump...done

[josevalim]

@frazze-jobb just so I understand better the problem, it is not possible to change the encoding after the shell is started, even with -noshell?

[frazze-jobb]

It is possible to change the encoding, problem is that the driver has already read stdin before "io:setopts" has been evaluated

[lukaszsamson]

Alternatively, is it possible to read from stdin device without starting a shell?

[josevalim]

@frazze-jobb I see! What do you think about a VM flag or app configuration? -kernel stdin_encoding latin1? This way you can read the default configuration, it can be set early on, and it should be backwards compatible.

[ferd]

For what it's worth, you can access the direct binary stream by setting stdio as a file descriptor when using the erlang:open_port/2 BIFs:

cat ~/Desktop/Audio.wav | head -n1
RIFF�sWAVEJUNK4fmt D��data��s������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������

$ cat ~/Desktop/Audio.wav | head -n1 | erl -noinput -eval 'P = erlang:open_port({fd,0,1}, [binary]), receive {P,{data,X}} -> erlang:port_command(P, ["input was: ", X]) end, halt()'
input was: RIFF�sWAVEJUNK4fmt D��data��s������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������

This will preserve the exact encoding that was passed in the input. If it was unicode, it remains unicode, if it was binary garbage, it remains binary garbage. The 'port_command' value does expect iodata() out, so any unicode output subsequences you have have to be flattened out to valid binaries before being sent out (so avoid any charlists that have unicode codepoints in the values and force it down to valid utf8 binaries with unicode:characters_to_binary if you do have unicode)

[lukaszsamson]

@ferd will it work in windows?