Chinese messages appear garbled in debug console

[isidorn]

As reported by @DFVSQY here microsoft/vscode#35235

1. Have the following program

using System;

namespace files
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("文件 编辑 选择 查看 转到 调试 任务 帮助");
            Console.WriteLine("哈哈");
        }
    }
}

2. Have your windows machine setup to use simplified chineese
3. Have vscode setup to use simplified chineese
4. Run the program, notice garbled characters
5. Notice garbled output in the C# extension output channel

Work around:

Add the following at the start of the program:

    Console.OutputEncoding = System.Text.Encoding.UTF8;

Alternative work around:

Edit launch.json to change "console" to use the integrated terminal:

    "console": "integratedTerminal"

Notes:

• this is not happening with other debug adapters thus I am assuming that the C# debug backend is sending wrong characters to vscode.
• I was not able to reproduce this with just setting vscode to use chineese, I think the whole OS language has to be changed.
• I get lots of bugs around this in the vscode repo from chineese people every mileston only for C# debugging.

Please ping if you need more details.

[DustinCampbell]

@isidorn: This is a VS Code issue: microsoft/vscode#3550.

[gregg-miskelly]

@DFVSQY what is your Console.OutputEncoding set to in your program? (You can evaluate the property under the debugger to find out).

If it isn't System.Text.UTF8Encoding, does the issue go away if you explicitly set it to that?

[DFVSQY]

@gregg-miskelly It seems like that the console depends on system encoding, if i change console.outputencoding to utf8, debug console is ok but the output window is still messy code, the default encoding is System.Text.OSEncoding,like below:

In addition, i'm a simplified chinese user, simplified chinese win10 and simplified chinese vscode.

[gregg-miskelly]

@DFVSQY Thanks. In this case, I think there are multiple bugs but with the same root cause -- non-UTF8 characters written to tools that expect UTF8 because on Windows there aren't great conversions for specifying output encoding. I will reopen this issue to track seeing if there is anything in the debugger that we can do to try and convince the implementation of Console.OutputEncoding to default to UTF-8.

For the output window, by chance does it work if you add /consoleloggerparameters:Encoding=UTF-8 in the build command in tasks.json?

[gregg-miskelly]

Debugger implementation notes: I believe this is where CoreFX works out the default console output page: https://github.com/dotnet/corefx/blob/7ce0acee7eaadc2f3f9776a76c166b29724c8acb/src/System.Console/src/System/ConsolePal.Windows.cs#L104
https://github.com/dotnet/corefx/blob/bb068f3be887ed34e7f8bedc4c12ec4284884441/src/Common/src/System/Text/EncodingHelper.Windows.cs#L27

So we may be able to solve this by calling SetConsoleCP(CP_UTF8).

[isidorn]

@gregg-miskelly thanks for reopening. The issue in vscode which @DustinCampbell pointed to is task specifc and is related to code pages and how windows determines printable characters in the terminal. This is however not exactly the same as this issue - unprintable characters in the output and debug console since those characters are coming directly from your extension.

[DFVSQY]

@gregg-miskelly where i should put the /consoleloggerparameters:Encoding=UTF-8 in ? please give me an example, thank you.

[DustinCampbell]

@isidorn: The non-debug output example above is simply the VS Code Output pane running the build task. C# for VS Code doesn't control that.

[isidorn]

@DustinCampbell yes the output is a dupilcate of the issue you linked. Thanks.

[gregg-miskelly]

@DFVSQY try setting tasks[0].args to [ "${workspaceRoot}/files.csproj", "/consoleloggerparameters:Encoding=UTF-8" ]. I am not sure if that will work or if we need something else to try and convince the .NET CLI to output UTF-8 text.

[DFVSQY]

@gregg-miskelly i have a try for that parameter, but there is no difference in output window, it's still messy code.

[gregg-miskelly]

@DFVSQY Thanks for trying. One last idea:

{
    "version": "0.1.0",
    "tasks": [
        {
            "taskName": "build",
            "command": "cmd",
            "suppressTaskName": true,
            "args": [
                "/c chcp 65001 >nul && dotnet build ${workspaceRoot}/files.csproj"
            ],
            "isBuildCommand": true,
            "problemMatcher": "$msCompile"
        }
    ]
}

[DFVSQY]

@gregg-miskelly i have a try for last idea, now it's ok, like below:

[gregg-miskelly]

Excellent.

@dbaeumer: there might be a pretty trivial fix for microsoft/vscode#3550 by calling SetConsoleCP(CP_UTF8) in the vscode process that spawns task processes.

[dbaeumer]

@gregg-miskelly thanks. With the support of running tasks in the terminal this is not needed anymore since the terminal itself already allows to specify encodings.

[bayesrule]

@dbaeumer @gregg-miskelly indeed everything is fine within the new terminal runner. HOWEVER, this debug console UTF-8 output issue itself has not been resolved... looking for a fix... THANK YOU

[gregg-miskelly]

@bayesrule I will see if we can fix this in our next release. In the mean time, use the work around at the top of this issue.

[Svetomech]

Have the same problem (ru locale).

[DustinCampbell]

This should be fixed when we moving test to be a task. cc @TheRealPiotrP

[dbaeumer]

@Svetomech run your tests using task version 2.0.0. Then they execute in a terminal for which either the correct code page is already set or you can set it using chcp.

[DFVSQY]

@dbaeumer yes, i have a try for the new task version, The terminal output is ok but if i use the Console default encoding, the debug console output is still messy code.

[Svetomech]

@dbaeumer how'd I do so?
This is what dotnet new xunit command leaves me with:

P.S. dotnet --version is 2.0.2

[DustinCampbell]

I think @dbaeumer assumed you had a custom task to run this. In this case, C# for VS Code is creating this particular output channel, so this is a bug on us to fix.

[Svetomech]

This should be fixed when we moving test to be a task. cc @TheRealPiotrP

@DustinCampbell sounds not really related to this issue. I mean, I gave running tests as an example, but in actuality all output in the Output window is encoded incorrectly. E.g. just pressing F5 results in this:

P.S. Same for Debug console window. Only in Terminal window everything seems to be displaying correctly.

[DustinCampbell]

@Svetomech : From an implementation perspective build and test are very different in C# for VS Code. The key difference is that the build task is defined in your project (in tasks.json) and can be easily converted to to tasks version 2.0. Then it will run in the Terminal as @dbaeumer. Testing with the "run/debug test" Code Lenses is not a task that is defined in your project. In that case, C# for VS Code runs specific code to handle this (not tasks) that output to a specific output channel created by the extension.

[Svetomech]

@DustinCampbell So, as I understand, the current workaround is to always prefer "console": "integratedTerminal" over "console": "internalConsole"? Can I set it up globally or do I have to edit launch.json on a per project basis?

[DustinCampbell]

I'm not sure whether that can be set globally. My goal in here was to point out that running tests with C# for VS Code is different and the workarounds in this issue might not help.

[gregg-miskelly]

@Svetomech : You can't set it globally.

I have updated the title of this bug to hopefully reflect the fact that this bug is only tracking fixing output in the debug console. If there are other items that need to be fixed lets get separate issues opened.

[gregg-miskelly]

@DustinCampbell do we need a similar issue for dealing with build command output for unit tests?

[DustinCampbell]

Yes, I think so. However, I think it is also related to @TheRealPiotrP's work on the task provider API. @TheRealPiotrP?

[gregg-miskelly]

@TheRealPiotrP do you want me to open another issue for unit tests? Or do you have this covered already?

[gregg-miskelly]

Fixed in 1.14.0-beta2

[donaldnevermore]

This bug is reproduced. Check out #3437