Proposal: Add new canonical json v2 that is valid JSON in all circumstances

[cgwalters]

Moving this from containers/ocidir-rs#10

The canonical JSON spec hasn't changed in ~9 years; maybe there's a better place to engage but as this repo seems actively maintained I'd like to start here.

Basically, canonical JSON encoding newlines literally is problematic as there are valid real-world reasons to want to have newlines in strings. (And I guess theoretically carriage returns)

What do you think about trying to define a new version of the spec that is defined to always escape control characters instead?

[jku]

I'm not a tough developer but I expect that for any software that intends to be TUF specification compliant, the expectation would be that discussion starts there: https://github.com/theupdateframework/specification

I do have to warn that TUF spec is in a bit of a limbo WRT changes that are not backwards-compatible:

• theoretically none of the changes so far have been backwards incompatible so no clients or repositories have yet built mechanisms to deal with that
• because of the nature of TUF metadata, software will have to support the old format for a very long time -- so no-one maintaining software or actual repositories is very eager to support multiple major versions

On the actual issue of JSON canonicalization, are you aware of the work to implement TUF over DSSE? I struggle to find a good link to this work since tap 17 is so high level that it does not even mention DSSE itself but the idea is to not canonicalize at all. Switch to DSSE is of course an even more incompatible change for all the software projects but it does seem like a reasonable direction if JSON canonicalization is being changed anyway

@adityasaky do we have any more useful links on DSSE specifically for TUF use case?

[adityasaky]

Nothing springs to mind at the moment. As you say, if a change to JSON canonicalization is being considered, I think considering a switch to DSSE altogether is a better idea to remove the requirement of canonicalizing altogether. This ensures that as yet unverified (and thus untrusted) envelopes are not parsed prior to verification. For more, see DSSE's motivations.

Side note: TAP-17 is purposely kept at a high level so as to not mandate any one envelope spec over another. However, perhaps it should be updated to recommend (and not require) DSSE, as we do on the in-toto side.

cc @mnm678