Prototype Mithril/Cardano bundle in Docker image

[jpraynaud]

Why

We want to evaluate the feasibility of bundling Mithril and Cardano node together in the same Docker image (and maybe also the DMQ node when it is available). This will allow smooth (seamless?) integration of Mithril with Cardano for existing Cardano SPOs.

What

Prototype the implementation of a Docker image bundling the Cardano node (and tools) and the Mithril signer.

How

Phase 1

• Build static binaries for signer and client
• Assess the modifications that would be needed in the crates configuration (in particular usage of default features)
• Use the existing Cardano Docker image and add the Mithril static binaries in it
• Start the signer when the Cardano node is started
• Create a mapping stored in the image of the Mithril network configurations by default for preview, preprod and mainnet (for seamless integration)
• Use the default configurations mapping for the configuration of the signer based on the Cardano network which is used
• Run the same command that is used to start a Cardano node on the new Docker image and make sure that Cardano and Mithril signer nodes are running properly

Next Phases

• TBD:
  • How to integrate Phase 1 in the Hydra CI?
  • How to support Mithril relay and IP obfuscation of the block producer?

Ideas / Questions

• Use the existing Cardano image and add the Mithril part in it?
• Use the existing Mithril signer image and add the Cardano part in it?
• How to support opt out Mithril signer?
• How to start both nodes in the same Docker?
• What if some Mithril parameters are missing: do we start only Cardano or does it fail?
• Assess the friction introduced by the Mithril signer configuration parameters not needed for the Cardano node.
• Try to mirror the way configuration switch by network is done with the Cardano node (if any)
• Do we support default values with release-mainnet configuration (e.g. aggregator endpoint)?
• What about the DMQ node (own socket path, connects to socket of the Cardano node, downloads a bootstrap peers list specific to the network)?
• Use the Mithril client to fast bootstrap the Cardano node
• ⚠️ How to deal with IP obfuscation of the block producer (vs current use of a proxy on the relay)?

Resources

• https://github.com/IntersectMBO/cardano-node/blob/master/nix/docker/README.md
• https://github.com/IntersectMBO/cardano-node/blob/master/.github/workflows/release-ghcr.yaml
• https://github.com/containers/skopeo
• https://github.com/IntersectMBO/cardano-node/blob/4bb0fd1cd376e404d64ce4de95c212ec6e713027/nix/pkgs.nix#L93
• https://github.com/IntersectMBO/cardano-node/blob/4bb0fd1cd376e404d64ce4de95c212ec6e713027/nix/docker/default.nix
• https://nixos.org/manual/nixpkgs/stable/#sec-pkgs-dockerTools
• https://stackoverflow.com/a/68006519

[Alenar]

Investigations

Ideally we would prefer to extend the actual Cardano docker image as it would allow drop in replacement for users, but this pose several challenges:

• We would need to support all functionality of the image, even those who can't run alongside a Mithril node
• The image may not have the required dependency for Mithril nodes (glibc 2.35+)
• This requires overriding the entry-point of the image, something that is difficult to do without referencing files from the base image which could be modified easily from upstream

About Cardano Node docker image

The Cardano node docker image is built using nix dockerTools.buildImage, as shown here: https://github.com/IntersectMBO/cardano-node/blob/master/nix/docker/default.nix

We can also see that it's using the scratch base image since it does not specify a base image (as specified on the buildImage fromImage parameter doc).
Using scratch base image means that every dependencies must be provided, I've confirmed that Mithril nodes can't be run on it as glibc is missing (by attaching a volume to my locally compiled binaries):

bash-5.2# ldd --version
bash: ldd: command not found
bash-5.2# ./mithril-client --version
bash: ./mithril-client: cannot execute: required file not found
bash-5.2# ./mithril-signer --version
bash: ./mithril-signer: cannot execute: required file not found
bash-5.2# ./mithril-aggregator --version
bash: ./mithril-aggregator: cannot execute: required file not found

The entrypoint is set to a entrypoint script which is copied to /usr/local/bin.

The image have several modes of operations, script / custom / merge / cli

• it make no sense to run a Mithril node the cli mode is used, since it's for "one shot" commands against the cardano-cli
• script, custom and merge provide different ways to start a node, the difference is on how much configuration is needed (only a network env var for script, custom to provide the full config, merge to customized part of the config and reuse base config from a network)

By looking at the run-node script (used by entrypoint), we can see that the block producer mode if a CARDANO_BLOCK_PRODUCER=true is provided, else the node start up as a relay.
Mithril signer node requires to be in a block producer mode, else a operational certificate won't be available to identify the node.
Mithril aggregator node should be able to operate in a relay mode.

Key findings

• Extending the image would requires to add at least glibc as it's a missing required dependency for the Mithril nodes
• As currently a Mithril aggregator or signer mode requires a active Cardano node, for the Mithril node startup we could replace the entrypoint with a custom script that would start simultaneously the entrypoint script from the base image and a process that would do a wait loop for the node to delay the start of the Mithril node
• Not all modes of the base image are compatible with Mithril nodes:
  • cli is compatible with none
  • the other are compatible with the Mithril aggregator but requires to be run as a block producer to run the Mithril signer (which can be identified with the CARDANO_BLOCK_PRODUCER env var)
• If we want to be fully integrated we would need to use nix dockerTools.buildImage as well, this pose additional challenges (how to get the mithril binaries ? from which source, from hydra or from our fully controlled github releases ?)

[jpraynaud]

We have run a successful experiment to build fully static Mithril signer and client CLI binaries and run them in the Cardano node Docker image:

• with the x86_64-unknown-linux-musl target
• with the num-integer-backend feature activated in common
• with the bundle_tls feature activated in signer and client CLI

The static builds are successfully created for both nodes:

$ cargo build -p mithril-signer --release --target x86_64-unknown-linux-musl
warning: dropping unsupported crate type `cdylib` for target `x86_64-unknown-linux-musl`

warning: `mithril-stm` (lib) generated 1 warning
warning: `mithril-common` (lib) generated 1 warning (1 duplicate)
warning: `mithril-cardano-node-chain` (lib) generated 1 warning (1 duplicate)
warning: `mithril-resource-pool` (lib) generated 1 warning (1 duplicate)
warning: `mithril-signed-entity-lock` (lib) generated 1 warning (1 duplicate)
warning: `mithril-cardano-node-internal-database` (lib) generated 1 warning (1 duplicate)
warning: `mithril-persistence` (lib) generated 1 warning (1 duplicate)
warning: `mithril-ticker` (lib) generated 1 warning (1 duplicate)
warning: `mithril-metric` (lib) generated 1 warning (1 duplicate)
warning: `mithril-signed-entity-preloader` (lib) generated 1 warning (1 duplicate)
warning: `mithril-era` (lib) generated 1 warning (1 duplicate)
    Finished `release` profile [optimized] target(s) in 0.32s

$ ldd target/x86_64-unknown-linux-musl/release/mithril-signer
statically linked

and

$ cargo build -p mithril-client-cli --release --target x86_64-unknown-linux-musl
   Compiling mithril-stm v0.4.11 (/home/jp/Works/Cardano/Mithril/mithril/mithril-stm)
warning: dropping unsupported crate type `cdylib` for target `x86_64-unknown-linux-musl`

warning: `mithril-stm` (lib) generated 1 warning
   Compiling mithril-common v0.6.15 (/home/jp/Works/Cardano/Mithril/mithril/mithril-common)
warning: `mithril-common` (lib) generated 1 warning (1 duplicate)
   Compiling mithril-cardano-node-internal-database v0.1.5 (/home/jp/Works/Cardano/Mithril/mithril/internal/cardano-node/mithril-cardano-node-internal-database)
warning: `mithril-cardano-node-internal-database` (lib) generated 1 warning (1 duplicate)
   Compiling mithril-client v0.12.27 (/home/jp/Works/Cardano/Mithril/mithril/mithril-client)
warning: `mithril-client` (lib) generated 1 warning (1 duplicate)
   Compiling mithril-client-cli v0.12.26 (/home/jp/Works/Cardano/Mithril/mithril/mithril-client-cli)
    Finished `release` profile [optimized] target(s) in 24.82s

$ ldd target/x86_64-unknown-linux-musl/release/mithril-client
statically linked

Then we have build a Docker image from the Cardano node 10.5.1 image and we have copied the signer and client nodes in it:

FROM ghcr.io/intersectmbo/cardano-node:10.5.1

# Copy static build of the Mithril client
COPY ./mithril-client /usr/local/bin/mithril-client
RUN chmod +x /usr/local/bin/mithril-client

# Copy static build of the Mithril signer
COPY ./mithril-signer /usr/local/bin/mithril-signer
RUN chmod +x /usr/local/bin/mithril-signer

Then we have built the docker image for the Cardano/Mithril bundle:

$ docker build . -t mithril-cardano-bundle

And we started the Cardano/Mithril bundle node:

$ docker run -v cardano-node-ipc:/ipc -v cardano-node-data:/data --mount type=bind,source="$(pwd)/db",target=/data/db/ -e NETWORK=preview mithril-cardano-bundle

The bundle image has started successfully:

$ docker ps
CONTAINER ID   IMAGE               COMMAND        CREATED          STATUS          PORTS     NAMES
06eac51d79f6   mithril-cardano-1   "entrypoint"   18 minutes ago   Up 18 minutes             youthful_raman

We have verified that the Cardano node was running properly and that the tip of the chain was moving forward:

$ docker exec -it youthful_raman cardano-cli query tip --testnet-magi
c 2 --socket-path /ipc/node.socket
{
    "block": 285245,
    "epoch": 68,
    "era": "Babbage",
    "hash": "7bb24789dcf6b7ad7684dd591a6c83a2bb3267fd19f4ca77cf96710c50d19706",
    "slot": 5950558,
    "slotInEpoch": 75358,
    "slotsToEpochEnd": 11042,
    "syncProgress": "6.63"
}

Finally, we have connected to the docker image and attempted to run the Mithril nodes from the shell:

$ docker exec -it youthful_raman sh
sh-5.2# /usr/local/bin/mithril-client --version               
mithril-client 0.12.26
sh-5.2# /usr/local/bin/mithril-signer --version               
mithril-signer 0.2.267

We have also made sure that we were able to list the latest snapshots from an aggregator with the client.

[jpraynaud]

We have created a prototype implementation of a Docker image made from the Cardano node image where:

• the statically built Mithril signer is embedded
• with its configuration files
• the Cardano node endpoint has been wrapped so that the Mithril signer can be started in the background (if running Cardano node as a block producer)

The developments are available in this branch:
main...ctl/2616-prototype-mithril-cardano-bundle-in-docker

Next steps:

• Answer questions:
  • How to gracefully support IP obfuscation of the block producer (by not making any HTTP calls without going through the Mithril relay (proxy))? Maybe using the topology file of the block producer can help automatically discover and setup the configuration for the Mithril relay in the container?
  • How to deal with logs (Mithril logs are just written in a mounted volume)
  • How use the Cardano configuration which are currently computed inside the Cardano node endpoint (env vars)?
• Build static Mithril signer in Hydra CI
• Embed the Mithril signer in the Cardano node Docker image build proces