Spike: Hydra Head in Head

[ch1bo]

Why

When a Hydra head is opened a part of the underlying ledger state gets locked up and made available off-chain to a small group of participants. While this is already quite a "small world" to process transactions, we encountered use cases where it would make sense to even open further heads with different/even smaller groups of participants on parts of the L2 state into L3 heads.

While the Interhead Hydra protocol eventually would allow to create L3 heads across multiple L2 heads, this spike is deliberately only exploring opening a head in-side an existing head.

What

• Demonstrate a 2-party head operating on top of a 3-party head
• Create a PR with results of the spike, but do not merge it

How

• Create a Hydra.Chain.Hydra chain backend that can be used as an alternative to the cardano-node and Blockfrost chain layer  #1305
• Use a websocket client: interpret SnapshotConfirmed as blocks and submit transactions using NewTx commands
• Switch to it using the command line --inception <hydra-node-api-host>
• Crash if --start-chain-from given

[ch1bo]

Notes from exploration today:

• Started by writing an end-to-end test that starts a hydra-node pointing to another hydra-node
• Hydra-based/Inception chain config is very similar to DirectChainConfig. This will be also the case with blockfrost, so we should re-use all the hydraScriptsTxId, cardanoSigningKey … etc parts better.
• Tiny wallet is also going to be duplicated across multiple Cardano-based chain components
• On the L2 we need to publish the hydra scripts too. Need to generalize the publishHydraScripts to work on any "Chain"?
  • Factored publishing of scripts into a way we can query utxo and relevant parameters separately
• Many things from the direct chain layer are actually not specific to the direct connection to cardano, but cardano-specific. One example: switching on PostChainTx and building the right cardano Tx.

[ch1bo]

Worked on the End-to-end test today and achieved it in #1603.

Notes from exploration:

• Started putting together TinyWallet for the inception chain component to keep track of UTxOs to use as seedInput
• The hydra client needed in Hydra.Chain.Inception is very much like the HydraNode utilities we have in hydra-cluster. We should create a hydra-client package in course of implementing any head-in-head chain component.
• When trying to use ServerOutput in my ad-hoc hydra client, it was requiring an IsChainState instance. But why is that a superclass of FromJSON ServerOutput? Found at least InvalidStateToPost in PostTxError tx, but that is not even used!
• Not using SnapshotConfirmed as I want to avoid book-keeping (lazy) and snapshot only contains transaction ids. We should consider flipping it the other way around and have a TxValid with a TxId and SnapshotConfirmed holding [Tx] and snapshot only contains transaction ids.
• Is the ChainContext useful still / outside of Hydra.Chain.Direct.State?
• Seing PostTxError in return type of draftCommitTx is surprising.
• When creating the collect transaction the query snapshot UTxO is seemingly missing the commit outputs. Maybe it's because I'm invoking callback already on TxValid and the query returns only confirmed UTxO?
• Ticks / time advancing in a Hydra head is an issue! There is no API output when the time of the underlying chain advances. So the L3 head could not possible know what time it is. Consequently, we can also not validate closing of the L3 head on the L2 head -> there are no deadlines.
• When writing the Hydra client for the inception chain layer I was again lazy and just acted on TxValid messages and invoked the callback whenever a Hydra transaction was observed. This is too early as any subsequent GET /snapshot/utxo would not include the outputs of the just "observed" transaction. Hence, a client would need to keep books about transactions seen in TxValid, but likely only act on them once the txId is included in a SnapshotConfirmed -> what an awkward UX! In the spike, I quickly hacked the Snapshot in the SnapshotConfirmed to include full transactions. We had it this way in the past, but moved away from it when we also made ReqSn not include full transactions

[ch1bo]

Next steps:

• Make L2 a three party head and L3 a two party head
• Make the setup easy to do manually
  • Could visualize the ledger state of each layer
    • L1: watch cardano-cli query utxo
    • L2/L3: hydra-tui
  • Could visualize each layer by observing transactions and classifying them
    • L1: using hydra-chain-observer with improved output?
    • L2: websocat + hydra-tx observe utility?

[ch1bo]

Bigger chunks that need solving / would help a proper implementation of this significantly:

• Make transaction creation less Chain.Direct specific
  • The hydra-tx library as we create it will help here
  • ScriptRegistry goes into the right direction as it quite modular
• Make script publishing independent of Chain.Direct (see early commits of the PR)
  • Maybe make reference scripts optional when building txs?
• Improve the API to be easier to consume (e.g. SnapshotConfirmed includes full txs, synchronous NewTx via HTTP)
  • And/or create a hydra-client package.
• Easy re-usable balancing / covering fees instead of a full blown TinyWallet
  • The cardano-api has improved here
  • Maybe relevant [FR] - Balance existing transaction IntersectMBO/cardano-cli#867
• Add ticks / time to the Hydra client API

[ch1bo]

Wrote a how-to open a head in a head interactively using this branch. While it's in the docs/how-to, copying here too for later reference:

How-to: Open a head in a head

Start the L1 devnet and an L2 head:

nix run .#hydra-cluster -- --devnet --publish-hydra-scripts --state-directory inception-demo

Explain briefly the UTxO on the L1 (and keep open):

watch -n1 cardano-cli query utxo --whole-utxo --socket-path inception-demo/node.socket --testnet-magic 42

Show L2 utxo state and demonstrate 1-2 transactions:

nix run .#hydra-tui -- -c 0.0.0.0:4001 -k inception-demo/wallet.sk

Publish hydra scripts on L2:

mkdir -p inception-demo/l3
curl localhost:4001/protocol-parameters > inception-demo/l3/protocol-parameters.json

function publishScripts() {
  nix run .#hydra-node -- publish-scripts \
    --inception 0.0.0.0:4001 \
    --cardano-signing-key inception-demo/wallet.sk
}

export HYDRA_SCRIPTS_TX_ID_ON_L2="$(publishScripts)"

Prepare another key to separate fuel from funds:

mkdir -p inception-demo/l3
cardano-cli address key-gen --normal-key \
  --verification-key-file inception-demo/l3/wallet.vk \
  --signing-key-file inception-demo/l3/wallet.sk
  
echo "Send some funds to commit to L3 to:"
cardano-cli address build --testnet-magic 42 --payment-verification-key-file inception-demo/l3/wallet.vk

Use the TUI on the L2 to separate fuel from funds using the interactive [n]ew transaction.

Start a hydra-node on L2:

nix run .#hydra-node -- \
  --node-id "l3" \
  --api-port 4003 \
  --port 5003 \
  --inception 0.0.0.0:4001 \
  --hydra-scripts-tx-id ${HYDRA_SCRIPTS_TX_ID_ON_L2} \
  --persistence-dir inception-demo/l3 \
  --cardano-signing-key inception-demo/wallet.sk \
  --hydra-signing-key demo/bob.sk \
  --ledger-protocol-parameters inception-demo/l3/protocol-parameters.json

Open the head on L2 with the tui:

nix run .#hydra-tui -- -c 0.0.0.0:4003 -k inception-demo/wallet.sk

Then, inside this TUI you must initialize the head by pressing i.

Commit via terminal (tui only supports direct node)

export WALLET_L3_ADDR=$(cardano-cli address build --testnet-magic 42 --payment-verification-key-file inception-demo/l3/wallet.vk)

cd inception-demo

curl 0.0.0.0:4001/snapshot/utxo \
  | jq "with_entries(select(.value.address == \"${WALLET_L3_ADDR}\"))" \
  > l2-commit-utxo.json

curl -X POST 0.0.0.0:4003/commit \
  --data @l2-commit-utxo.json \
  > commit-tx.json

cardano-cli transaction sign \
  --tx-file commit-tx.json \
  --signing-key-file l3/wallet.sk \
  --out-file commit-tx-signed.json

cat commit-tx-signed.json | jq -c '{tag: "NewTx", transaction: .}' | websocat "ws://0.0.0.0:4001?history=no"

[ch1bo]

Recorded a demo for this: https://www.youtube.com/watch?v=Y_Pw3MVooxg

[ch1bo]

Was successfully reviewed by @noonio and this concludes this spike.