Update file encoding to UTF-8 with BOM

[333fred]

Since integrating our repos, there have been a number of changes in PRs that touch the encoding of the file, adding or removing a BOM as the editor in question decides. I've standardized on UTF-8 with BOM, as we do in roslyn, and put it in the .gitattributes so it should hopefully stay consistent.

[333fred]

For reference, the script I used to convert all files:

using System.Diagnostics;
using System.Text;

var baseDir = "/home/fred/git/razor";

var utf8Preamble = Encoding.UTF8.GetPreamble();

var lsFilesOutput = Process.Start(new ProcessStartInfo("git", "ls-files --eol") { WorkingDirectory = baseDir, RedirectStandardOutput = true })!.StandardOutput.ReadToEnd();

var eolLines = lsFilesOutput.Split('\n', StringSplitOptions.RemoveEmptyEntries).Select(line => line.Split(' ', StringSplitOptions.RemoveEmptyEntries)).ToArray();

foreach (var eolLine in eolLines)
{
    var file = Path.Combine(baseDir, eolLine[3].Trim());

    if (file.EndsWith("sh") || file.Contains("node_modules") || file.EndsWith("json") || file.EndsWith("snap") || eolLine[0].Contains("-text"))
        continue;

    var bytes = File.ReadAllBytes(file);
    if (bytes.Length < utf8Preamble.Length)
        goto update;

    for (int i = 0; i < utf8Preamble.Length; i++)
    {
        if (bytes[i] != utf8Preamble[i])
            goto update;
    }

    continue;

update:
    var updatedBytes = new byte[bytes.Length + utf8Preamble.Length];
    utf8Preamble.CopyTo(updatedBytes, 0);
    bytes.CopyTo(updatedBytes, utf8Preamble.Length);
    File.WriteAllBytes(file, updatedBytes);
}

[davidwengier]

GitHub is struggling to let me review this, but I approve!

[ryanbrandenburg]

Looks good except it seems like your script might have modified some binary files (a picture in this case) and made them invalid: https://dev.azure.com/dnceng-public/public/_build/results?buildId=131223&view=logs&j=1f132584-ab08-5fbf-e5d0-5685de454342&t=76c675cd-a096-5dd5-8431-7aa49ff610be&l=345.

[333fred]

I updated the script to be more intelligent about not updating binary files, using git's interpretation of what encoding a file has.

[333fred]

Removed json files from the conversion, as the BOM is invalid JSON.

[jjonescz]

put it in the .gitattributes so it should hopefully stay consistent.

It seems that encoding=UTF-8 disappeared from .gitattributes in the first force push.

Shouldn't .editorconfig's encoding = utf-8-bom be applied to more files than just *.{cs,csx,vb,vbx} to be consistent with this change?

Shouldn't we create .git-blame-ignore-revs file for this commit?

[DustinCampbell]

Note: This PR breaks CMD files.

[333fred]

It seems that encoding=UTF-8 disappeared from .gitattributes in the first force push.

Yeah, looks like I forgot to reapply it in one of my force pushes.

Shouldn't .editorconfig's encoding = utf-8-bom be applied to more files than just *.{cs,csx,vb,vbx} to be consistent with this change?

I don't know that this would actually do anything.

Shouldn't we create .git-blame-ignore-revs file for this commit?

Sure.