FAKE doesn't support Unicode output redirection

[Balinth]

Description

Exact same problem as #2401 I see that got stale because the original reporter could not reproduce it any longer.

In my case, the problem persists without fail, so I thought to upload the dump if it helps:

gdrive file to dump

Repro steps

FAKE script:
let runToolWithOutputFiltering cmd args workingDir = let arguments = args |> String.split ' ' |> Arguments.OfArgs Command.RawCommand (cmd, arguments) |> CreateProcess.fromCommand |> CreateProcess.withWorkingDirectory workingDir |> CreateProcess.ensureExitCode |> CreateProcess.redirectOutput |> CreateProcess.withOutputEventsNotNull Trace.trace Trace.traceError |> Proc.run |> ignore
with
Fake.Core.Process (5.19.1)

• Operating system Win 10 64bit
• .NET Core 3.1

[github-actions]

Welcome to the FAKE community! Thank you so much for creating your first issue and therefore improving the project!

[matthid]

I took a quick look at the dump.
So you get the same FAKE output At least one redirection task did not finish: and when setting FAKE_DEBUG_PROCESS_HANG=true it crashes just like in #2401?

If not please make sure the process actually exits, because Proc.run will wait for the process to exit. Which tool are you running?
If yes, can you upload the .fake folder as well? (In particular build_B7F6EB2B8C895616F30387EC4628378648196358CFE017FA3C43B9932D4AED36.dll, note that this is your compiled script code!)

[Balinth]

No, the define does not generate "At least one redirection task did not finish:" output.

The tool I try to run is yarn (basic SAFE stack template) and with the redirect output, the bottom left bar keeps showing Building... and never starts actually debugging the client.
build_4600597A4A0F284DB6028AADE10BF4BE03634C5A76B285C236B566E70BC90C20.zip

[matthid]

@Balinth As far as I can see it behaves as designed. You apparently are running yarn webpack-dev-server as far as I know this process will keep living as long as you exit it (for example with CTRL+C). Therefore FAKE keeps waiting for it.
If you have questions to the SAFE template itself we probably should move that issue to them?

[Balinth]

@matthid I had some time to look further into the problem, and got closer to the cause (but not the root, or a solution)

The more accurate problem is, that the redirected output screws up unicode characters, and VS Code looks for a specific console output to know when the yarn task is done building and entered watch mode: "i ｢wdm｣: Compiled"

This gets mangled to "i ?wdm?: Compiled" with the FAKE redirection

In trying to circumvent the problem, I tried setting Console.OutputEncoding <- System.Text.Encoding.Unicode; in build.fsx, writing out the problem chars with character codes: Console.Write '\uFF62' but any output I try to produce from FAKE mangles those fancy unicode characters.

[matthid]

@Balinth Maybe you can contribute a failing test case so it is easier for me to look into it?

[Balinth]

The simple case: putting Trace.trace "｢" will output ? to the console. If this is not the case on your side, I will look into trying out building FAKE and creating a test case for it, but probably weeks later.

[matthid]

@Balinth I'm still not 100% sure what the scenario is or how to reproduce myself.

[Balinth]

The most basic case, create a new FAKE build.fsx and inside it anywhere call Trace.trace "｢" and the console output will be "?".
See repo: https://github.com/Balinth/fake_trace_unicode

[matthid]

Are you sure this has something to do with FAKE?
When I do:

dotnet new console -lang f#
# add printfn "｢"
dotnet run

E:\Projects\FAKE\test>dotnet run
?
Hello World from F#!

I'm not exactly sure what's going on but it looks like it is way out of our league...

[matthid]

The following changes that behavior

[<EntryPoint>]
let main argv =
    Console.OutputEncoding <- System.Text.Encoding.UTF8
    printfn "｢"
    0 // return an integer exit code

I'm not sure how 'safe' it is to add this to FAKE globally. You can always add that like to the top of your script. But I might add it and see if someone starts complaining. Given that Unicode is a superset of the default (at least on English systems?) I assume it should work without problems.

[matthid]

Ok scratch that. Adding those lines to the script doesn't seem to work for some reason.

[matthid]

One thing to consider when fixing this: #1461

[matthid]

I'm not quite sure why the same line of code works when running at the start of main but fails when run as part of the script. I guess OutputEncoding stops working as soon as you use some other APIs (unfortunately it doesn't seem to be documented)

[matthid]

From testing around locally I'm pretty sure the OutputEncoding loses it's effect as soon as you use the Console.Out or Console.Error properties (which we do for some setup purposes).
I changed some internal logic and I enabled the changes by default (with an option to disable, just to be sure).
I'll probably release an alpha so you can test and see if it fixes the problem (make sure to update the runner, not the packages).

So in the new version:

• It hopefully works by default
• It behaves like today with FAKE_DISABLE_UTF8=true
  -> You can set up the encoding within the script

Thanks for the report and the detailed research with the repro steps :).

[matthid]

Please test with 5.20.1-alpha.1590 (should be available in a couple of minutes)

[Balinth]

Tested locally, and solves the issue. FAKE_DISABLE_UTF8=true also unsolves the issue if that was someone's poison, thanks for taking the time!