Skip to content

Stabilization of cardano db v2 (aggregator) #2610

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jul 2, 2025
Merged

Stabilization of cardano db v2 (aggregator) #2610

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
913db1a
feature(aggregator, signer, common): keep only MithrilStakeDistributi…
Jun 25, 2025
0fda445
refactor(client-cli, e2e): rename cardano-db download directory
Jun 26, 2025
21632a5
feat(docs): add cardano-db download example with a range of immutables
Jun 27, 2025
7d4236d
fix(docs): default backend for cardano-db verify is v2
Jun 27, 2025
29321a7
feat(client-cli): remove unstable in CardanoDbCommandsBackend V2 clap…
Jun 27, 2025
404e14b
refactor(aggregator): switch from immutableFilesFull to cardanoDataba…
Jul 1, 2025
1a05752
refactor(aggregator): remove MithrilStakeDistribution from example
Jul 2, 2025
77385e4
chore: upgrade crate versions
Jul 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/backward-compatibility.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ on:
description: "Signed entity types parameters (discriminants names in an ordered comma separated list)"
required: true
type: string
default: "CardanoTransactions,CardanoStakeDistribution,CardanoDatabase"
default: "CardanoTransactions,CardanoStakeDistribution,CardanoDatabase,CardanoImmutableFilesFull"
workflow_call:
inputs:
total-releases:
Expand All @@ -28,7 +28,7 @@ on:
default: "10.4.1"
signed-entity-types:
type: string
default: "CardanoTransactions,CardanoStakeDistribution,CardanoDatabase"
default: "CardanoTransactions,CardanoStakeDistribution,CardanoDatabase,CardanoImmutableFilesFull"

jobs:
prepare-env-variables:
Expand Down
22 changes: 11 additions & 11 deletions .github/workflows/test-client.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ jobs:
- name: Cardano Database Snapshot / download & restore latest
shell: bash
working-directory: ./bin
run: ./mithril-client ${{ steps.prepare.outputs.debug_level }} --origin-tag CI cardano-db download $CDB_SNAPSHOT_DIGEST --backend v1 ${{ matrix.extra_args }} 2>&1 | tee cdb-download-output.txt
run: ./mithril-client ${{ steps.prepare.outputs.debug_level }} --origin-tag CI cardano-db download $CDB_SNAPSHOT_DIGEST --backend v1 --download-dir v1 ${{ matrix.extra_args }} 2>&1 | tee cdb-download-output.txt

- name: Cardano Database Snapshot / verify Cardano node starts successfully
if: runner.os == 'Linux'
Expand All @@ -173,19 +173,19 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
shell: bash
working-directory: ./bin
run: ./mithril-client ${{ steps.prepare.outputs.debug_level }} --unstable tools utxo-hd snapshot-converter --db-directory db --cardano-node-version latest --utxo-hd-flavor LMDB --commit
run: ./mithril-client ${{ steps.prepare.outputs.debug_level }} --unstable tools utxo-hd snapshot-converter --db-directory v1/db --cardano-node-version latest --utxo-hd-flavor LMDB --commit

- name: Remove downloaded artifacts to free up disk space (Linux, Windows)
if: runner.os != 'macOS'
shell: bash
working-directory: ./bin
run: rm --force db/immutable/*.{chunk,primary,secondary}
run: rm --force v1/db/immutable/*.{chunk,primary,secondary}

- name: Remove downloaded artifacts to free up disk space (macOs)
if: runner.os == 'macOS'
shell: bash
working-directory: ./bin
run: sudo rm -rf db/
run: sudo rm -rf v1/db/

- name: Mithril Stake Distribution / list and get last hash
shell: bash
Expand Down Expand Up @@ -259,13 +259,13 @@ jobs:
if: steps.aggregator_capability_unix.outputs.cardano_database_v2_enabled == 'true' || steps.aggregator_capability_windows.outputs.cardano_database_v2_enabled == 'true'
shell: bash
working-directory: ./bin
run: ./mithril-client ${{ steps.prepare.outputs.debug_level }} --origin-tag CI cardano-db download --backend v2 $CARDANO_DATABASE_V2_SNAPSHOT_HASH ${{ matrix.extra_args }} 2>&1 | tee cdb-v2-download-output.txt
run: ./mithril-client ${{ steps.prepare.outputs.debug_level }} --origin-tag CI cardano-db download --backend v2 --download-dir v2 $CARDANO_DATABASE_V2_SNAPSHOT_HASH ${{ matrix.extra_args }} 2>&1 | tee cdb-v2-download-output.txt

- name: Cardano Database V2 Snapshot / verify immutables
if: steps.aggregator_capability_unix.outputs.cardano_database_v2_enabled == 'true' || steps.aggregator_capability_windows.outputs.cardano_database_v2_enabled == 'true'
shell: bash
working-directory: ./bin
run: ./mithril-client ${{ steps.prepare.outputs.debug_level }} --origin-tag CI cardano-db verify --backend v2 --db-dir db_v2 $CARDANO_DATABASE_V2_SNAPSHOT_HASH | tee cdb-v2-verify-output.txt
run: ./mithril-client ${{ steps.prepare.outputs.debug_level }} --origin-tag CI cardano-db verify --backend v2 --db-dir v2/db $CARDANO_DATABASE_V2_SNAPSHOT_HASH | tee cdb-v2-verify-output.txt

- name: Cardano Database V2 Snapshot / verify Cardano node starts successfully
if: runner.os == 'Linux' && steps.aggregator_capability_unix.outputs.cardano_database_v2_enabled == 'true'
Expand Down Expand Up @@ -329,16 +329,16 @@ jobs:

- name: Cardano Database Snapshot / download & restore latest
shell: bash
run: ${{ steps.command.outputs.mithril_client }} ${{ steps.prepare.outputs.debug_level }} --origin-tag CI cardano-db download $CDB_SNAPSHOT_DIGEST --backend v1 --download-dir /app/data ${{ matrix.extra_args }}
run: ${{ steps.command.outputs.mithril_client }} ${{ steps.prepare.outputs.debug_level }} --origin-tag CI cardano-db download $CDB_SNAPSHOT_DIGEST --backend v1 --download-dir /app/data/v1 ${{ matrix.extra_args }}

- name: Ledger state snapshot conversion from InMemory to LMDB
if: matrix.extra_args == '--include-ancillary'
shell: bash
run: ${{ steps.command.outputs.mithril_client }} ${{ steps.prepare.outputs.debug_level }} --unstable tools utxo-hd snapshot-converter --db-directory /app/data/db --cardano-node-version latest --utxo-hd-flavor LMDB --commit
run: ${{ steps.command.outputs.mithril_client }} ${{ steps.prepare.outputs.debug_level }} --unstable tools utxo-hd snapshot-converter --db-directory /app/data/v1/db --cardano-node-version latest --utxo-hd-flavor LMDB --commit

- name: Remove downloaded artifacts to free up disk space
shell: bash
run: sudo rm -rf $PWD/data/db/
run: sudo rm -rf $PWD/data/v1/db/

- name: Mithril Stake Distribution / list and get last hash
shell: bash
Expand Down Expand Up @@ -401,12 +401,12 @@ jobs:
- name: Cardano Database V2 Snapshot / download & restore latest (Full restoration)
if: steps.aggregator_capability.outputs.cardano_database_v2_enabled == 'true'
shell: bash
run: ${{ steps.command.outputs.mithril_client }} ${{ steps.prepare.outputs.debug_level }} --origin-tag CI cardano-db download --backend v2 $CARDANO_DATABASE_V2_SNAPSHOT_HASH --download-dir /app/data ${{ matrix.extra_args }}
run: ${{ steps.command.outputs.mithril_client }} ${{ steps.prepare.outputs.debug_level }} --origin-tag CI cardano-db download --backend v2 $CARDANO_DATABASE_V2_SNAPSHOT_HASH --download-dir /app/data/v2 ${{ matrix.extra_args }}

- name: Cardano Database V2 Snapshot / verify immutables
if: steps.aggregator_capability.outputs.cardano_database_v2_enabled == 'true'
shell: bash
run: ${{ steps.command.outputs.mithril_client }} ${{ steps.prepare.outputs.debug_level }} --origin-tag CI cardano-db verify --backend v2 $CARDANO_DATABASE_V2_SNAPSHOT_HASH --db-dir /app/data/db_v2
run: ${{ steps.command.outputs.mithril_client }} ${{ steps.prepare.outputs.debug_level }} --origin-tag CI cardano-db verify --backend v2 $CARDANO_DATABASE_V2_SNAPSHOT_HASH --db-dir /app/data/v2/db

test-mithril-client-wasm:
strategy:
Expand Down
10 changes: 5 additions & 5 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docs/website/root/manual/develop/nodes/mithril-aggregator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -527,7 +527,7 @@ Here is a list of the available parameters for the serve command:
| `era_reader_adapter_type` | `--era-reader-adapter-type` | - | `ERA_READER_ADAPTER_TYPE` | Era reader adapter type that can be `cardano-chain`, `file` or `bootstrap`. | `bootstrap` | - | - |
| `era_reader_adapter_params` | `--era-reader-adapter-params` | - | `ERA_READER_ADAPTER_PARAMS` | Era reader adapter params that is an optional JSON encoded parameters structure that is expected depending on the `era_reader_adapter_type` parameter | - | - | - |
| `ancillary_files_signer_config` | - | - | `ANCILLARY_FILES_SIGNER_CONFIG` | Configuration of the ancillary files signer<br/><br/>Can either be a secret key or a key stored in a Google Cloud Platform KMS account.<br/><br/>**IMPORTANT**: The cryptographic scheme used is ED25519 | - | - secret-key:<br/>`{ "type": "secret-key", "secret_key": "136372c3138312c3138382c3130352c3233312c3135" }`<br/>- Gcp kms:<br/>`{ "type": "gcp-kms", "resource_name": "projects/project_name/locations/_location_name/keyRings/key_ring_name/cryptoKeys/key_name/cryptoKeyVersions/key_version" }` | - |
| `signed_entity_types` | `--signed-entity-types` | - | `SIGNED_ENTITY_TYPES` | Signed entity types parameters (discriminants names in an ordered comma separated list) | - | `MithrilStakeDistribution,CardanoImmutableFilesFull,CardanoStakeDistribution` | - |
| `signed_entity_types` | `--signed-entity-types` | - | `SIGNED_ENTITY_TYPES` | Signed entity types parameters (discriminants names in an ordered comma separated list) | - | `MithrilStakeDistribution,CardanoImmutableFilesFull,CardanoStakeDistribution,CardanoDatabase,CardanoTransactions` | - |
| `snapshot_compression_algorithm` | `--snapshot-compression-algorithm` | - | `SNAPSHOT_COMPRESSION_ALGORITHM` | Compression algorithm of the snapshot archive | `zstandard` | `gzip` or `zstandard` | - |
| `zstandard_parameters` | - | - | `ZSTANDARD_PARAMETERS__LEVEL` and `ZSTANDARD_PARAMETERS__NUMBER_OF_WORKERS` | Zstandard specific parameters | - | `{ level: 9, number_of_workers: 4 }` | - |
| `allow_unparsable_block` | `--allow-unparsable-block` | - | `ALLOW_UNPARSABLE_BLOCK` | If set no error is returned in case of unparsable block and an error log is written instead. Will be ignored on (pre)production networks. | `false` | - | - |
Expand Down
2 changes: 1 addition & 1 deletion docs/website/root/manual/develop/nodes/mithril-client.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -578,7 +578,7 @@ Here is a list of the available parameters:

| Parameter | Command line (long) | Command line (short) | Environment variable | Description | Default value | Example | Mandatory |
| -------------------------- | ---------------------------- | :------------------: | -------------------------- | ------------------------------------------------------------------------------- | ------------- | ------- | :----------------: |
| `backend` | `--backend` | `-b` | - | - | `v1` | - | - |
| `backend` | `--backend` | `-b` | - | - | `v2` | - | - |
| `json` | `--json` | - | - | Enable JSON output for command results | `false` | - | - |
| `digest` | - | - | - | Digest of the Cardano db snapshot to verify or `latest` for the latest artifact | - | - | :heavy_check_mark: |
| `db_dir` | `--db-dir` | - | - | Directory from where the immutable will be verified | - | - | - |
Expand Down
19 changes: 18 additions & 1 deletion ...sioned_docs/version-maintained/manual/getting-started/bootstrap-cardano-node.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -446,7 +446,9 @@ You will see more information about the snapshot:

### Step 4: Download the selected Cardano DB snapshot

To download the selected snapshot from the remote location to your remote location, run:
#### Step 4.1 : Download the full Cardano node database

To download the full selected snapshot from the remote location to your local location, run:

```bash
./mithril-client cardano-db download --include-ancillary $SNAPSHOT_DIGEST
Expand Down Expand Up @@ -493,6 +495,21 @@ Upgrade and replace the restored ledger state snapshot to 'LMDB' flavor by runni
mithril-client --unstable tools utxo-hd snapshot-converter --db-directory db --cardano-node-version 10.4.1 --utxo-hd-flavor Legacy --cardano-network preview --commit
```

#### Step 4.2 : Download a partial range of immutable files

To download a range of immutable files (e.g. from 17,000 to 18,000) from the remote location to your local location, prepare the following environment variables:

```bash
export IMMUTABLE_START=17000
export IMMUTABLE_END=18000
```

And run the command:

```bash
./mithril-client cardano-db download $SNAPSHOT_DIGEST --include-ancillary --backend v2 --start $IMMUTABLE_START --end $IMMUTABLE_END
```

### Step 5 (optional): Convert the ledger state snapshot to another flavor

After restoring a snapshot with the `--include-ancillary` option, the ledger state is in the `InMemory` format. You can convert it to another UTxO-HD flavor (e.g., `LMDB` or `Legacy`) using the Mithril client `tools utxo-hd snapshot-converter` command.
Expand Down
2 changes: 1 addition & 1 deletion mithril-aggregator/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "mithril-aggregator"
version = "0.7.70"
version = "0.7.71"
description = "A Mithril Aggregator server"
authors = { workspace = true }
edition = { workspace = true }
Expand Down
5 changes: 2 additions & 3 deletions mithril-aggregator/src/configuration.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -495,9 +495,8 @@ pub struct ServeCommandConfiguration {
/// Signed entity types parameters (discriminants names in an ordered, case-sensitive, comma
/// separated list).
///
/// The values `MithrilStakeDistribution` and `CardanoImmutableFilesFull` are prepended
/// automatically to the list.
#[example = "`MithrilStakeDistribution,CardanoImmutableFilesFull,CardanoStakeDistribution`"]
/// The value `MithrilStakeDistribution` is prepended is automatically to the list.
#[example = "`CardanoImmutableFilesFull,CardanoStakeDistribution,CardanoDatabase,CardanoTransactions`"]
pub signed_entity_types: Option<String>,

/// Compression algorithm used for the snapshot archive artifacts.
Expand Down
Loading
Loading