feat: MessagePack serialisation for circuits

[aakoshh]

Description

Problem*

While trying to figure out how to compile the C++ code generated by protoc in AztecProtocol/aztec-packages#12640 we discussed with @ludamad that MessagePack is already used in Barretenberg, so we agreed that I'd prepare comparisons to bincode similar to #7513

Summary*

The PR adds msgpack_serialize and msgpack_deserialize, and refactors the bincode_serialize and proto_serialize to be standalone functions as well.

msgpack_serialize accepts a compact flag:

• if compact is false then the names of struct fields are serialized along the data, which provides maximum backwards compatibility: old clients ignore new fields
• if compact is true then the encoding of structs is based on their position: in this case old clients (at least using this particular library), cannot read data written by newer clients if new fields have been added, which could actually be the safer option for us

Both modes handle enums identically at the moment, with their variant name written to the data. Apparently the library used to work based on positions, but this changed, despite the docs still talking about numerical indexes.

I figured the compact = true option would potentially be the best of both worlds when it comes to opcodes: a new field on an opcode means it's probably not OK to execute it with an older VM, although this could also be indicated with a semantic version number. OTOH a new field on Program could be something safe to ignore 🤷 Added a test to show the different scenarios of modifying enums.

Should we go with MessagePack and want to ignore Brillig in Barretenberg, I would consider again the two-step encoding process, where we have a DTO that represents both the ACIR and the Brillig part as opaque bytes, and the client decides which one they want to deserialise.

I'm not yet sure which format the C++ side would expect with the MSGPACK_FIELDS macro annotation.

Additional Context

Here are the comparisons:

Going from bincode to msgpack with map encoding (compact=false) is similar to CBOR:

Going from bincode to msgpack with tuple encoding (compact=true) can be even better than bincode, with some backwards compatibility (e.g. new (unused) or removed enum members, renamed fields):

Relative reduction in size going from compact=false (x-axis) to compact=true (y-axis):

Documentation*

Check one:

• No documentation needed.
• Documentation included in this PR.
• [For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist*

• I have tested the changes locally.
• I have formatted the changes with Prettier and/or cargo fmt on default settings.

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Test Suite Duration'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.

Benchmark suite	Current: 59876b0	Previous: 6f0b119	Ratio
zkemail_noir-jwt_	6 s	3 s	2

This comment was automatically generated by workflow using github-action-benchmark.

CC: @TomAFrench

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Compilation Time'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.

Benchmark suite	Current: 1d214b4	Previous: c30bcd7	Ratio
rollup-base-private	21.02 s	15.62 s	1.35

This comment was automatically generated by workflow using github-action-benchmark.

CC: @TomAFrench