test(masternode-upgrade): standalone masternode-identity upgrade harness + spec

docs/testing/masternode-identity-upgrade/SPEC.md

@@ -0,0 +1,103 @@
+# Masternode-identity upgrade test — spec
+
+## Goal
+
+Prove that a masternode (evonode) **identity added as a standalone identity in
+DET 0.9.3** survives an in-place upgrade to a newer DET — the identity, its keys,
+and its on-chain state must all still be usable after the app and the platform
+network are upgraded, the way a real user's install would be.
+
+## Versions under test
+
+| Role | DET | pin |
+|------|-----|-----|
+| baseline (old) | `v0.9.3` — commit `3268b736` | dash-sdk platform rev `29f7492e` (platform 2.1.0, grovedb 3.1) |
+| upgrade (new)  | PR #887 `fix/qa-followups-885` — commit `8a47745a`, base `feat/legacy-identity-migration` | dash-sdk platform rev `93b967f9` (platform 4.0) |
+
+Network: local `dashmate` group, **3 masternodes**. Started on dashmate **v3**
+(protocol 11), then upgraded in place to **v4** (protocol 12) — mirroring a real
+mainnet protocol upgrade between the two DET versions.
+
+## What we did
+
+1. **Baseline on 0.9.3 (v3 network / protocol 11).** Loaded an evonode identity
+   by proTxHash + owner/voting/payout WIFs (for a masternode the platform owner
+   identity id *is* the proTxHash). Created a DET wallet, funded it from the
+   local_seed Core wallet, topped the identity up via asset lock, and **withdrew
+   credits to L1 Dash** — confirmed the asset-unlock landed on the Core chain.
+   Snapshotted the DET profile (`backup-1` post-load, `backup-2` post-withdrawal).
+
+2. **Upgraded the network v3 -> v4 in place.** `dashmate@4 group start` migrated
+   the config and images; validators signalled protocol 12, which activated at
+   the next epoch boundary. From then on Drive emits `GroveDBProof::V1`, which
+   0.9.3's SDK cannot decode (expected — this is why the app must be upgraded).
+
+3. **Ran the new DET (PR #887) on a copy of the 0.9.3 profile.** The DB migrated
+   cleanly (schema v11 -> v38, no data loss) and the evonode identity appeared in
+   the new **Masternodes** section: Active, all three key roles (Voting / Owner /
+   Payout) and the derived voter identity intact.
+
+## Results
+
+- **Migration: pass.** Identity + keys + alias survive the 0.9.3 -> PR-887 DB
+  migration.
+- **On-chain liveness: pass.** Direct DAPI `getIdentity`/`getIdentityBalance`
+  (bypassing DET) shows the identity live on protocol 12 with a real, growing
+  balance — proof the migrated reference points at the genuine on-chain identity.
+- **Key functionality: pass.** DET's *Key Info -> Sign Message* signs a known
+  message with the migrated **owner** and **payout (TRANSFER)** keys; each
+  signature (a) recovers to the key's expected address, (b) is byte-identical to
+  what `dashd signmessage` produces with the genuine private key (deterministic
+  RFC6979), and (c) is accepted by `dashd verifymessage`. This proves the private
+  key material survived migration and still signs correctly.
+
+## Additional scenario: password-encrypted wallet survives the upgrade
+
+Separately verifies that a **password-protected wallet** created in DET 0.9.3
+survives the upgrade and can still be unlocked with the same password.
+
+**What we did.** In DET 0.9.3, created an HD wallet with a strong password
+(0.9.3 encrypts the seed with AES-256-GCM, key derived from the password via
+Argon2). Confirmed at the storage layer — `data.db` `wallet.uses_password = 1`
+for that wallet (vs `0` for a no-password wallet). Snapshotted the profile, then
+ran the new DET on a copy.
+
+**Result: pass.** After the DB migration the wallet appears in the new DET as
+`HD: <name>`, is correctly recognized as encrypted (it starts **locked** with an
+**Unlock** button), and **unlocks with the original password** (the button flips
+to **Lock** and address-derivation controls appear). Because the seed is stored
+with *authenticated* encryption (AES-256-GCM), a successful unlock is conclusive:
+a wrong password or a seed corrupted by migration would fail the GCM auth tag and
+refuse to unlock. So the encrypted seed and the password both survive the
+migration intact. (Address rows don't populate in the UI while SPV is unsynced —
+same dip0024 cause as below — but that's independent of the unlock proof.)
+
+## Known limitation (candidate finding for the PR)
+
+A **withdrawal through the new DET cannot be exercised on a local network.** The
+new DET verifies platform proofs *trustlessly* via its own SPV-synced masternode
+list (`SpvProvider` in `src/context_provider.rs`; no core-RPC fallback — the old
+`core_backend_mode` was dropped, migration v38 notes "SPV-only now"). That SPV
+masternode sync uses the DIP24 `qrinfo` path, which needs an `llmq_test_dip0024`
+rotation quorum. That quorum is **type 103, size 4, minSize 4** (per dash-spv
+`test_utils/masternode_network.rs`), so it cannot form on a 3-masternode local
+network, and regtest exposes no param to shrink it (only `llmq_test`,
+`llmq_test_instantsend`, `llmq_test_platform` are resizable). Result: the new
+DET's `masternodes_ready()` never becomes true, every DAPI address gets banned
+("servers unreachable"), and no proof-verified op (balance refresh, withdrawal)
+can run locally.
+
+DET 0.9.3 was unaffected because it fetched quorum keys straight from its local
+Core node over RPC (`core.get_quorum_public_key`), which has no dip0024 dependency.
+
+**Open question for the PR authors:** is a standard `dashmate` local network a
+supported target for the new SPV-only verifier, or should the SPV masternode sync
+fall back to the legacy `mnlistdiff` path (or accept trusted core quorum data)
+when no rotation quorum exists? Until then, an end-to-end withdrawal has to be
+tested on devnet/testnet (which have rotation quorums), and the **key-signing
+check above is the local substitute** that proves the migrated keys are usable.
+
+## How to reproduce
+
+See `harness/README.md`. The scripts regenerate the network, keys, and profiles
+locally; secrets (private keys, wallet DBs) are gitignored and never committed.

docs/testing/masternode-identity-upgrade/harness/.gitignore

@@ -0,0 +1,10 @@
+# Never commit regenerable local secrets / fixtures — they contain private keys
+# and recoverable wallet seeds (the test wallet has no password).
+keys.json
+*.db
+*.sqlite
+*.sqlite-*
+backups/
+home-*/
+*.tgz
+network-backup*/

docs/testing/masternode-identity-upgrade/harness/01_setup_v3_network.sh

@@ -0,0 +1,38 @@
+#!/bin/sh
+# Phase 0 — stand up a local platform network on dashmate v3 (protocol 11).
+#
+# Why v3: v3 Drive (grovedb 4) only emits GroveDBProof::V0, which DET 0.9.3's
+# SDK (platform 2.1, grovedb 3.1) can decode. That keeps the 0.9.3 baseline
+# working with no time pressure. The network is later upgraded to v4 in
+# 04_upgrade_to_v4.sh, which is the actual v3->v4 protocol-11->12 migration.
+#
+# Requires: docker running, npx available. Wipes any existing ~/.dashmate.
+set -e
+
+DASHMATE_V3="${DASHMATE_V3:-dashmate@3.0.2}"   # last stable v3 line (3.1 never released)
+NODES="${NODES:-3}"
+
+echo ">> Removing any existing local network + ~/.dashmate"
+docker ps -a --format '{{.Names}}' | grep '^dashmate_' | xargs -r docker rm -f >/dev/null 2>&1 || true
+docker volume ls --format '{{.Name}}' | grep '^dashmate_' | xargs -r docker volume rm >/dev/null 2>&1 || true
+rm -rf ~/.dashmate
+
+echo ">> dashmate setup local (${DASHMATE_V3}, ${NODES} masternodes)"
+npx -y "$DASHMATE_V3" setup local --node-count="$NODES" --no-debug-logs --miner-interval=1m --verbose
+
+# Fast, steady platform blocks so DKG/sync and manual UI steps don't stall.
+for cfg in local_1 local_2 local_3 local_seed; do
+  npx -y "$DASHMATE_V3" config set --config="$cfg" \
+    platform.drive.tenderdash.consensus.createEmptyBlocksInterval 10s
+done
+
+echo ">> Starting group"
+npx -y "$DASHMATE_V3" group start --verbose
+
+echo ">> Waiting for platform blocks to flow"
+until curl -s --max-time 3 http://127.0.0.1:46657/status \
+      | python3 -c "import json,sys; sys.exit(0 if json.load(sys.stdin)['sync_info']['latest_block_height']!='1' else 1)" 2>/dev/null; do
+  sleep 5
+done
+curl -s http://127.0.0.1:46657/status | python3 -c "import json,sys; d=json.load(sys.stdin); print('platform height', d['sync_info']['latest_block_height'], 'protocol', d['node_info']['protocol_version']['app'])"
+echo ">> v3 network ready (expect protocol 11)."

docs/testing/masternode-identity-upgrade/harness/02_extract_masternode_keys.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""Extract each local masternode's platform-identity keys from the running
+network, so they can be loaded into DET as a standalone evonode identity.
+
+For a masternode/evonode, the platform OWNER identity id == the proTxHash, and
+its keys are the owner / voting / payout address keys. dashmate only persists
+the operator BLS key in config.json; the owner/voting/payout private keys live
+in the local_seed Core wallet ("main"), retrievable via dumpprivkey.
+
+Writes keys.json next to this script. keys.json is gitignored (contains WIF
+private keys) — regenerate it locally, never commit it.
+
+Requires: the network from 01_setup_v3_network.sh running.
+"""
+import json
+import os
+import subprocess
+
+DASHMATE_CONFIG = os.path.expanduser("~/.dashmate/config.json")
+SEED_RPC = "http://127.0.0.1:20302/"
+OUT = os.path.join(os.path.dirname(__file__), "keys.json")
+
+
+def rpc_password():
+    cfg = json.load(open(DASHMATE_CONFIG))
+    return cfg["configs"]["local_seed"]["core"]["rpc"]["users"]["dashmate"]["password"]
+
+
+def call(pw, method, params=None, wallet=None):
+    url = SEED_RPC + (f"wallet/{wallet}" if wallet else "")
+    body = json.dumps({"jsonrpc": "1.0", "id": "h", "method": method, "params": params or []})
+    r = subprocess.run(
+        ["curl", "-s", "--user", f"dashmate:{pw}", "--data-binary", body,
+         "-H", "content-type: text/plain;", url],
+        capture_output=True, text=True,
+    )
+    d = json.loads(r.stdout)
+    if d.get("error"):
+        raise RuntimeError(d["error"])
+    return d["result"]
+
+
+def main():
+    pw = rpc_password()
+    out = {}
+    for p in call(pw, "protx", ["list", "registered", True]):
+        st = p["state"]
+        entry = {
+            "proTxHash": p["proTxHash"],          # == platform owner identity id
+            "type": p.get("type"),
+            "ownerAddress": st["ownerAddress"],
+            "votingAddress": st["votingAddress"],
+            "payoutAddress": st["payoutAddress"],
+            "operatorPubKey": st["pubKeyOperator"],
+        }
+        for role, addr_key in (("ownerPrivkey", "ownerAddress"),
+                               ("votingPrivkey", "votingAddress"),
+                               ("payoutPrivkey", "payoutAddress")):
+            entry[role] = call(pw, "dumpprivkey", [st[addr_key]], wallet="main")
+        out[p["proTxHash"][:8]] = entry
+    json.dump(out, open(OUT, "w"), indent=2)
+    print(f"wrote {OUT} ({len(out)} masternode identities)")
+    for k, v in out.items():
+        print(f"  {k} {v['type']} owner={v['ownerAddress']} payout={v['payoutAddress']}")
+
+
+if __name__ == "__main__":
+    main()

docs/testing/masternode-identity-upgrade/harness/03_fund_address.py

@@ -0,0 +1,54 @@
+#!/usr/bin/env python3
+"""Fund a DET wallet receive address from the local_seed Core wallet ("main"),
+then wait for one confirmation. Used in phase 1 to give the DET wallet UTXOs so
+it can top up the identity via asset lock.
+
+Usage: 03_fund_address.py <dash_address> [amount_dash]
+Requires: the network running.
+"""
+import json
+import os
+import subprocess
+import sys
+import time
+
+DASHMATE_CONFIG = os.path.expanduser("~/.dashmate/config.json")
+SEED_RPC = "http://127.0.0.1:20302/"
+
+
+def call(pw, method, params=None, wallet=None):
+    url = SEED_RPC + (f"wallet/{wallet}" if wallet else "")
+    body = json.dumps({"jsonrpc": "1.0", "id": "h", "method": method, "params": params or []})
+    r = subprocess.run(
+        ["curl", "-s", "--user", f"dashmate:{pw}", "--data-binary", body,
+         "-H", "content-type: text/plain;", url],
+        capture_output=True, text=True,
+    )
+    d = json.loads(r.stdout)
+    if d.get("error"):
+        raise RuntimeError(d["error"])
+    return d["result"]
+
+
+def main():
+    if len(sys.argv) < 2:
+        print("usage: 03_fund_address.py <dash_address> [amount_dash]")
+        raise SystemExit(2)
+    address = sys.argv[1]
+    amount = float(sys.argv[2]) if len(sys.argv) > 2 else 10.0
+    pw = json.load(open(DASHMATE_CONFIG))["configs"]["local_seed"]["core"]["rpc"]["users"]["dashmate"]["password"]
+
+    txid = call(pw, "sendtoaddress", [address, amount], wallet="main")
+    print(f"sent {amount} DASH to {address}: {txid}")
+    for _ in range(30):
+        conf = call(pw, "gettransaction", [txid], wallet="main").get("confirmations", 0)
+        if conf >= 1:
+            print(f"confirmed ({conf} conf)")
+            return
+        time.sleep(10)
+    print("WARNING: not confirmed within timeout")
+    raise SystemExit(1)
+
+
+if __name__ == "__main__":
+    main()

docs/testing/masternode-identity-upgrade/harness/04_upgrade_to_v4.sh

@@ -0,0 +1,31 @@
+#!/bin/sh
+# Phase 2 — upgrade the SAME running network from v3 to v4 in place.
+#
+# dashmate v4 migrates the config file (configFormatVersion 3 -> 4) and bumps
+# every image tag to v4 (drive:4, tenderdash 1.6.0, rs-dapi:4). v4 Drive reads
+# the v3 on-disk state (the mainnet-supported upgrade path). v4 validators
+# signal protocol 12, which activates at the next epoch boundary; from then on
+# Drive emits GroveDBProof::V1 and DET 0.9.3 can no longer decode proofs
+# (expected — mirrors a real mainnet upgrade).
+#
+# BACK UP FIRST if you want to be able to return to the v3 state:
+#   tar czf dashmate-home.tgz -C ~ .dashmate
+#   for v in $(docker volume ls --format '{{.Name}}' | grep '^dashmate_'); do
+#     docker run --rm -v "$v":/vol -v "$PWD":/b alpine tar czf "/b/$v.tgz" -C /vol .
+#   done
+set -e
+DASHMATE_V4="${DASHMATE_V4:-dashmate@4.0.0}"
+
+echo ">> Stopping v3 group"
+npx -y dashmate@3.0.2 group stop --force || true
+
+echo ">> Starting group with ${DASHMATE_V4} (migrates config + images)"
+npx -y "$DASHMATE_V4" group start --verbose
+
+echo ">> Waiting for protocol 12 to activate (next epoch boundary)"
+until curl -s --max-time 3 http://127.0.0.1:46657/block \
+      | python3 -c "import json,sys; sys.exit(0 if json.load(sys.stdin)['block']['header']['version']['app']=='12' else 1)" 2>/dev/null; do
+  sleep 30
+  curl -s http://127.0.0.1:46657/block | python3 -c "import json,sys; h=json.load(sys.stdin)['block']['header']; print('height', h['height'], 'active', h['version']['app'], 'proposed', h.get('proposed_protocol_version'))" 2>/dev/null || true
+done
+echo ">> Network is now on protocol 12 (v4). Proofs are V1."

docs/testing/masternode-identity-upgrade/harness/README.md

@@ -0,0 +1,88 @@
+# Harness — masternode-identity upgrade test
+
+Reproducible scripts for the test described in [`../SPEC.md`](../SPEC.md). They
+regenerate everything locally; **no secrets or binaries are committed** (see
+`.gitignore`). The DET UI steps are still manual today — the value here is the
+deterministic network/keys/verification tooling an automated test can build on.
+
+## Prerequisites
+
+- Docker running; `npx` (Node) available.
+- A checkout of `dashpay/platform` (for DAPI `.proto` files) — default path
+  `~/Projects/dashpay/platform`.
+- `grpcurl` (for `query_identity_dapi.sh`).
+- Two DET builds: `v0.9.3` and the PR branch, each run against an **isolated
+  profile** via `HOME` override, e.g.
+  `HOME=/path/to/profile /path/to/dash-evo-tool` — DET derives its data dir from
+  `$HOME` (`~/Library/Application Support/Dash-Evo-Tool` on macOS), so this keeps
+  the test off your real profile. Snapshot the profile dir to back it up.
+
+## Order of operations
+
+| Step | Script / action | Phase |
+|------|-----------------|-------|
+| 1 | `01_setup_v3_network.sh` — dashmate v3 local net, protocol 11 | 0 |
+| 2 | `02_extract_masternode_keys.py` — writes `keys.json` (gitignored) | 0 |
+| 3 | *DET 0.9.3*: Load Identity (Evonode) with a proTxHash + its owner/voting/payout WIFs from `keys.json`; create wallet; fund from Core; top up; withdraw to L1. Snapshot the profile. | 1 |
+| 4 | `04_upgrade_to_v4.sh` — in-place v3->v4, waits for protocol 12 | 2 |
+| 5 | `p2p_proxy.py` — **required for the new DET** (see gotcha) | 3 |
+| 6 | *New DET (PR)*: run on a copy of the 0.9.3 profile; confirm the identity migrated (Masternodes screen). | 3 |
+| 7 | `query_identity_dapi.sh <proTxHash>` — on-chain liveness/balance | 3 |
+| 8 | *New DET*: Masternodes -> identity -> Owner key / Payout key -> Sign Message; copy the signature; verify with `verify_signed_message.py` | 3 |
+
+## Key-functionality check (the local substitute for a withdrawal)
+
+In the new DET, open a migrated key (Owner or Payout), type a known message,
+click **Sign Message**, copy the Base64 signature, then:
+
+```
+python3 verify_signed_message.py <key_address> <base64_signature> "<message>"
+```
+
+Exit 0 / `MATCH` means the migrated private key still produces a valid signature
+for its address. For a byte-for-byte cross-check, `dashd signmessage <address>
+"<message>"` (the address is in the local_seed `main` wallet) yields the
+identical signature — deterministic RFC6979 — proving DET holds the same key.
+
+## Additional check — encrypted wallet survives the upgrade
+
+Verifies a password-protected wallet created in 0.9.3 still unlocks in the new
+DET (see SPEC "Additional scenario"). No network is needed for the wallet itself;
+run it alongside or independently of the identity flow.
+
+1. **In DET 0.9.3** — Wallets -> **Create Wallet**. Hover the entropy grid, click
+   **Generate**, tick **"I wrote it down"**, give it a **name** and a **strong
+   password** (the strength meter must read at least "Strong"), then **Save
+   Wallet**. Record the name + password.
+2. **Confirm it's encrypted at the storage layer** (profile still on 0.9.3):
+   ```
+   sqlite3 "<profile>/Library/Application Support/Dash-Evo-Tool/data.db" \
+     "SELECT alias, uses_password FROM wallet;"
+   ```
+   The new wallet must show `uses_password = 1` (a no-password wallet shows `0`).
+3. **Snapshot** the profile dir, then **run the new DET on a copy** of it. Choose
+   *Just Explore*, dismiss the sync modal (*Continue in the background*), open
+   **Wallets**, and select the wallet.
+4. **Unlock it** — click **Unlock**, enter the original password, confirm. Pass =
+   the wallet is present and the **Unlock button flips to Lock** (authenticated
+   AES-256-GCM decryption succeeded, so the password + seed survived migration).
+   A wrong password shows an error and the button stays **Unlock**.
+
+Note: address rows may stay empty while SPV is unsynced (the dip0024 limitation
+below) — that does not affect the unlock result.
+
+## Gotchas
+
+- **P2P port proxy (step 5).** The PR's SPV client hardcodes the regtest Core P2P
+  port `19899`, but dashmate's `local_1` Core listens on `20001`. Run
+  `python3 p2p_proxy.py` (bridges `127.0.0.1:19899 -> 20001`) before starting the
+  new DET, or its SPV client can't connect at all.
+- **dip0024 / withdrawal.** See SPEC "Known limitation" — the new DET's SPV
+  verifier needs an `llmq_test_dip0024` rotation quorum that a 3-MN local net
+  can't form, so proof-verified ops (including withdrawal) fail locally with
+  "servers unreachable". The Sign Message check above is the workaround.
+- **Signing a *fresh* message.** Always sign a message you control and verify
+  against *that* message. Recovering a stale signature against the wrong message
+  yields a valid-but-wrong address and looks like a failure.
+- **Text entry on non-US keyboard layouts.** Automated typing tools may fail to
+  map characters; paste via the clipboard instead.