feat(DRAFT): Protobuf bindings in Barretenberg

[aakoshh]

Add support for protobuf serialisation introduced in noir-lang/noir#7511

• Add scripts to generate C++ bindings for the Protobuf schemas in Noir
• Add protoc to the devbox image used by EC2 to run the build in chore: Add protoc to the devbox image #12644
• Generate bindings during the build
• Map the protobuf DTOs to the existing classes generated from serde
• Deserialise the data with a fallback to the old format

Unlike with the serde based approach, code generation is done on the aztec-packages side instead of Noir (see noir-lang/noir#7590), but we still keep the serde based classes as well as they are what we actually work with in Rust, whereas the protobuf are just DTOs for backwards compatibility.

WIP

Currently ./barretenberg/cpp/bootstrap.sh will generate the bindings, but it can be done explicitly with ./barretenberg/cpp/scripts/codegen_dsl.sh as well.

After that think the native build succeeds:

% ./bootstrap.sh build_native                                                                        
Not using cache for barretenberg-release-disabled-cache.tar.gz due to uncommitted changes/files.
./src/barretenberg/dsl/acir_format/acir_to_constraint_buf.hpp:2:1: error: code should be clang-formatted [-Wclang-format-violations]
#include "acir_format.hpp"
^
./src/barretenberg/dsl/acir_format/convert/proto_to_serde.hpp:1:1: error: code should be clang-formatted [-Wclang-format-violations]
#include "../serde/index.hpp"
^

I think these are just warnings, as the first one was already like it is.

The WASM build fails:

% ./bootstrap.sh build_wasm                                                                     
...
[2/44] Building CXX object src/barretenberg/dsl/CMakeFiles/dsl_objects.dir/acir_format/acir_to_constraint_buf.cpp.obj
FAILED: src/barretenberg/dsl/CMakeFiles/dsl_objects.dir/acir_format/acir_to_constraint_buf.cpp.obj 
/opt/wasi-sdk/bin/clang++ --sysroot=/opt/wasi-sdk/share/wasi-sysroot -DDISABLE_ASM=1 -DDISABLE_AZTEC_VM=1 -DNO_MULTITHREADING -DNO_PAR_ALGOS -D_WASI_EMULATED_PROCESS_CLOCKS=1 -I/mnt/user-data/akosh/aztec-packages/barretenberg/cpp/src -I/mnt/user-data/akosh/aztec-packages/barretenberg/cpp/build-wasm/_deps/msgpack-c/src/msgpack-c/include -I/mnt/user-data/akosh/aztec-packages/barretenberg/cpp/build-wasm/_deps/tracy-src/public -I/mnt/user-data/akosh/aztec-packages/barretenberg/cpp/build-wasm/_deps/lmdb/src/lmdb_repo/libraries/liblmdb -DBB_VERBOSE -Oz -DNDEBUG -std=gnu++20 -fbracket-depth=1024 -fno-exceptions -fno-slp-vectorize -Werror -Wall -Wextra -Wconversion -Wsign-conversion -Wfatal-errors -fcolor-diagnostics -fconstexpr-steps=100000000 -Wno-vla-cxx-extension -Wno-missing-field-initializers -MD -MT src/barretenberg/dsl/CMakeFiles/dsl_objects.dir/acir_format/acir_to_constraint_buf.cpp.obj -MF src/barretenberg/dsl/CMakeFiles/dsl_objects.dir/acir_format/acir_to_constraint_buf.cpp.obj.d -o src/barretenberg/dsl/CMakeFiles/dsl_objects.dir/acir_format/acir_to_constraint_buf.cpp.obj -c /mnt/user-data/akosh/aztec-packages/barretenberg/cpp/src/barretenberg/dsl/acir_format/acir_to_constraint_buf.cpp
In file included from /mnt/user-data/akosh/aztec-packages/barretenberg/cpp/src/barretenberg/dsl/acir_format/acir_to_constraint_buf.cpp:1:
In file included from /mnt/user-data/akosh/aztec-packages/barretenberg/cpp/src/barretenberg/dsl/acir_format/acir_to_constraint_buf.hpp:4:
In file included from /mnt/user-data/akosh/aztec-packages/barretenberg/cpp/src/barretenberg/dsl/acir_format/convert/proto_to_serde.hpp:2:
/mnt/user-data/akosh/aztec-packages/barretenberg/cpp/src/barretenberg/dsl/acir_format/convert/../proto/acir/native.pb.h:10:10: fatal error: 'google/protobuf/port_def.inc' file not found
   10 | #include <google/protobuf/port_def.inc>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
[32/44] Building CXX object src/barretenberg/stdlib/honk_verifier/CMakeFiles/stdlib_honk_verifier_objects.dir/ultra_recursive_verifier.cpp.objgit st^C
ninja: build stopped: interrupted by user.

The files it's looking for should be there after apt install protobuf-compiler, but maybe for wasm they are not included:

% ls /usr/include/google/protobuf                                                                                                                                                                    ~/aztec-packages af/bb-dsl-acir-proto akosh-box
any.h          arenaz_sampler.h       empty.pb.h                field_mask.proto                  has_bits.h               map_field_inl.h     port.h                 service.h             timestamp.proto      wrappers.proto
any.pb.h       descriptor.h           empty.proto               generated_enum_reflection.h       implicit_weak_message.h  map_field_lite.h    port_def.inc           source_context.pb.h   type.pb.h
any.proto      descriptor.pb.h        endian.h                  generated_enum_util.h             inlined_string_field.h   map_type_handler.h  port_undef.inc         source_context.proto  type.proto
api.pb.h       descriptor.proto       explicitly_constructed.h  generated_message_bases.h         io                       message.h           reflection.h           struct.pb.h           unknown_field_set.h
api.proto      descriptor_database.h  extension_set.h           generated_message_reflection.h    map.h                    message_lite.h      reflection_internal.h  struct.proto          util
arena.h        duration.pb.h          extension_set_inl.h       generated_message_tctable_decl.h  map_entry.h              metadata.h          reflection_ops.h       stubs                 wire_format.h
arena_impl.h   duration.proto         field_access_listener.h   generated_message_tctable_impl.h  map_entry_lite.h         metadata_lite.h     repeated_field.h       text_format.h         wire_format_lite.h
arenastring.h  dynamic_message.h      field_mask.pb.h           generated_message_util.h          map_field.h              parse_context.h     repeated_ptr_field.h   timestamp.pb.h        wrappers.pb.h

I tried to instruct cmake to use the FindProtobuf module to set the variables, needed for inclusion, but it doesn't work. Some suggest that's because I did not install Protobuf from source, so I tried to do that like so:

# instead of `apt install libprotobuf-dev protobuf-compiler`
# https://github.com/protocolbuffers/protobuf/blob/v29.3/cmake/README.md
# using the same version as the Rust build
git clone --branch v29.3 https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive
cmake . 
cmake --build . --parallel 10
sudo cmake --install .

And using this protobuf.cmake:

#list(APPEND CMAKE_PREFIX_PATH "/usr/local/lib/cmake/protobuf")
set(Protobuf_DIR "/usr/local/lib/cmake/protobuf")
find_package(Protobuf 29.3.0 EXACT CONFIG REQUIRED)
include_directories(${Protobuf_INCLUDE_DIRS})

Unfortunately it looks like the version I installed can't be used by Wasm because of 32bit vs 64bit arch:

% ./barretenberg/cpp/bootstrap.sh build_wasm 
...
-- Could NOT find Protobuf (missing: Protobuf_LIBRARIES Protobuf_INCLUDE_DIR) 
CMake Error at cmake/protobuf.cmake:13 (find_package):
  Could not find a configuration file for package "Protobuf" that exactly
  matches requested version "29.3.0".

  The following configuration files were considered but not accepted:

    /usr/local/lib/cmake/protobuf/protobuf-config.cmake, version: 29.3.0 (64bit)

Call Stack (most recent call first):
  CMakeLists.txt:164 (include)

[aakoshh]

These are the issues I'm facing:

• I need the protoc compiler to turn .proto schemas into C++ files.
• Initially I used apt install protobuf-compiler which installed some older version, which I invoke from my custom Makefile , where I also massage the files a bit so the import paths work.
• apt also installed files into /usr/include which I think build_native found, but build_wasm did not.
• I tried adding a protobuf.cmake where I use include(FindProtobuf) and find_package(Protobuf REQUIRED) to try to set the include paths to include_directories(${Protobuf_INCLUDE_DIRS}), but for wasm it didn't find anything.
• Some forums mentioned I had to install from source, so I did that: clone the repo, specifically the v29.3 version I used in Rust (two weeks ago v30.0 hasn't been released yet). This overwrote my includes and also added custom cmake files such as protobuf-config.cmake, which I can look for with find_package(Protobuf CONFIG REQUIRED)
• This is where it turned out that the generated code was with an older version of the compiler; now refreshed.
• With this the build_wasm I think either doesn't find anything, or at some point indicated that the problem is that it found it, but the built source is for 64 bits, and it's looking for 32 bit version.
• Just to make sure, I added include_directories("/usr/include") to the top level CMakeLists.txt to see if it finds the includes, but it wants to include pthread.h, so this isn't going to work in Wasm.
• Then I tried removing the installed protobuf and protoc, and instead use FetchContent_Declare to get it from GitHub as part of the cmake build, which is how other dependencies are fetched into e.g. cpp/build/_deps and cpp/build-wasm/_deps.
• Curiously for build_native I can't actually see any _deps/protobuf, yet it seems to install it into /usr/local/include and find_package finds it. I'm not sure what protoc I would use to compile the schemas before the build, but that's another problem.
• However for build_wasm, even though _deps/protobuf-src and _deps/protobuf-build are downloaded, it doesn't build a 32 bit version like I hoped, nor does find_package report any paths, so I still can't build for Wasm

There seems to be a dearth of information about building protobuf for wasm in C++. There is https://github.com/dsyer/protobuf-wasm as an alternative set of libraries to use, but it's old, not sure it's still maintained, or how to get it into the build.

[aakoshh]

We discussed a bit with Adam and he suggested that since Barretenberg already uses MessagePack for backwards compatible message exchange in its "napi API", and it's also used between the AVM and TypeScript, that we could try that instead of adding Protobuf as a dependency. CBOR started as a fork of MessagePack, so the performance should be similar to this