[Spec] Specify if UTF-8 encoding should have BOM

[Tuupertunut]

Suggestion

Specify in the Ultrastar format specification whether files should use UTF-8 BOM or UTF-8 without BOM.

Use case

The Ultrastar format specification speficies that encoding should be UTF-8, but does not distinguish between UTF-8 BOM and UTF-8 without BOM. There are currently tools in the Ultrastar ecosystem that accept or produce only one of these.

Examples:

• Ultrastar Play 0.9.0 song editor can read both formats, but always saves the file as UTF-8 BOM.
• Performous Composer 2.0 can only open UTF-8 without BOM.

Therefore if you save a song in Ultrastar Play song editor and try to open it in Performous Composer, it will fail to open, even though both are UTF-8.

Extra info/examples/attachments

No response

[Baklap4]

as far as previous discussions assembled i think we came onto without bom

[bohning]

BOM does only really make sense for UTF16 and UTF32. It was only introduced by Vocaluxe afaik to detect UTF8 and differentiate CP1252 (as alternative to the Vocaluxe-specific #ENCODING tag). The Vocaluxe reasoning is: if there is no #ENCODING tag, it is UTF8 if there is a BOM, otherwise it's CP1252. But there are definitely better ways to implement encoding detection and if the standard proposes UTF8, it should not even be necessary anymore to detect encodings.

I strongly suggest that some Vocaluxe developers change the logic to default to UTF8 (and use other encodings via the #ENCODING tag, as long as anything other than UTF8 is not deprecated yet).

[basisbit]

I also vote for (and strongly suggest) UTF-8 without BOM. There is no need for it.

[Baklap4]

@marwin89 can you make a pr to add this :)?

[marwin89]

Here is the pull request. Please approve and merge 👋

• Update Spec - UTF-8 without BOM #45

FYI: there is an estimated issue for implementing support for UTF-8 (without BOM) in vocaluxe repository

• Implement UTF-8 (without BOM) encoding support Vocaluxe/Vocaluxe#633

I close this issue. thanks @Tuupertunut for discussion and refining the spec.

[codello]

The current version of the formal specification has a slightly more relaxed phrasing wrt to the BOM, acknowledging that applications may ignore a BOM if one is present. Is this phrasing in line with the result of this discussion?

format/spec.md

Lines 64 to 67 in 23bf930

	Songs are plain text files
	The UTF-8 encoding MUST be used.
	Implementations MUST NOT add a byte order mark to the beginning of a file.
	In the interests of interoperability, implementations MAY ignore the presence of a byte order mark rather than treating it as an error.