feat: Validators sentinel by spalladino · Pull Request #12818 · AztecProtocol/aztec-packages

spalladino · 2025-03-17T21:54:17Z

Adds a new Sentinel component that runs in the Aztec Node. On every new slot, with a 2-slot delay, the sentinel gathers all attestations seen (both on p2p and L1) and flags each validator for the current committee as having attested or not. It also flags the proposer based on whether the block was mined, on the block being proposed but not mined, or not mined at all.

The sentinel then computes stats on each validator based on the data last N epochs, so a client can query those and decide whether to push for slashing.

Sample output from stats endpoint:

{
  "stats": {
    "initialSlot": 48,
    "lastProcessedSlot": 60,
    "slotWindow": 1152,
    "stats": {
      "0x037bbe29c2766243a070a4b3cbf3891767097308": {
        "address": "0x037bbe29c2766243a070a4b3cbf3891767097308",
        "history": [
          {
            "slot": 49,
            "status": "block-missed"
          },
          {
            "slot": 50,
            "status": "attestation-missed"
          },
          {
            "slot": 51,
            "status": "attestation-sent"
          },
          {
            "slot": 52,
            "status": "block-mined"
          },
          {
            "slot": 53,
            "status": "attestation-sent"
          },
          {
            "slot": 54,
            "status": "block-mined"
          },
          {
            "slot": 55,
            "status": "attestation-sent"
          },
          {
            "slot": 56,
            "status": "attestation-missed"
          },
          {
            "slot": 57,
            "status": "attestation-sent"
          },
          {
            "slot": 58,
            "status": "attestation-sent"
          },
          {
            "slot": 59,
            "status": "attestation-sent"
          },
          {
            "slot": 60,
            "status": "attestation-sent"
          }
        ],
        "lastAttestation": {
          "date": "2025-03-18T22:30:25.000Z",
          "slot": 60,
          "timestamp": 1742337025
        },
        "lastProposal": {
          "date": "2025-03-18T22:29:13.000Z",
          "slot": 54,
          "timestamp": 1742336953
        },
        "missedAttestations": {
          "count": 2,
          "currentStreak": 0,
          "rate": 0.2222222222222222
        },
        "missedProposals": {
          "count": 1,
          "currentStreak": 0,
          "rate": 0.3333333333333333
        },
        "totalSlots": 12
      },
      "0x4af126598417209bfa57bab856963c42c2fee5db": {
        "address": "0x4af126598417209bfa57bab856963c42c2fee5db",
        "history": [
          {
            "slot": 49,
            "status": "attestation-missed"
          },
          {
            "slot": 50,
            "status": "attestation-missed"
          },
          {
            "slot": 51,
            "status": "block-mined"
          },
          {
            "slot": 52,
            "status": "attestation-sent"
          },
          {
            "slot": 53,
            "status": "block-mined"
          },
          {
            "slot": 54,
            "status": "attestation-sent"
          },
          {
            "slot": 55,
            "status": "block-mined"
          },
          {
            "slot": 56,
            "status": "attestation-missed"
          },
          {
            "slot": 57,
            "status": "attestation-sent"
          },
          {
            "slot": 58,
            "status": "block-mined"
          },
          {
            "slot": 59,
            "status": "block-mined"
          },
          {
            "slot": 60,
            "status": "attestation-sent"
          }
        ],
        "lastAttestation": {
          "date": "2025-03-18T22:30:25.000Z",
          "slot": 60,
          "timestamp": 1742337025
        },
        "lastProposal": {
          "date": "2025-03-18T22:30:13.000Z",
          "slot": 59,
          "timestamp": 1742337013
        },
        "missedAttestations": {
          "count": 3,
          "currentStreak": 0,
          "rate": 0.42857142857142855
        },
        "missedProposals": {
          "count": 0,
          "currentStreak": 0,
          "rate": 0
        },
        "totalSlots": 12
      },
      "0x61b13c781397bdb68284f02961e16602b9ce8923": {
        "address": "0x61b13c781397bdb68284f02961e16602b9ce8923",
        "history": [
          {
            "slot": 49,
            "status": "attestation-missed"
          },
          {
            "slot": 50,
            "status": "block-missed"
          },
          {
            "slot": 51,
            "status": "attestation-missed"
          },
          {
            "slot": 52,
            "status": "attestation-missed"
          },
          {
            "slot": 53,
            "status": "attestation-missed"
          },
          {
            "slot": 54,
            "status": "attestation-missed"
          },
          {
            "slot": 55,
            "status": "attestation-missed"
          },
          {
            "slot": 56,
            "status": "block-missed"
          },
          {
            "slot": 57,
            "status": "attestation-missed"
          },
          {
            "slot": 58,
            "status": "attestation-missed"
          },
          {
            "slot": 59,
            "status": "attestation-missed"
          },
          {
            "slot": 60,
            "status": "attestation-missed"
          }
        ],
        "missedAttestations": {
          "count": 10,
          "currentStreak": 10,
          "rate": 1
        },
        "missedProposals": {
          "count": 2,
          "currentStreak": 2,
          "rate": 1
        },
        "totalSlots": 12
      },
      "0xa1153393636750230f4db253300e59ca8bd3a9c0": {
        "address": "0xa1153393636750230f4db253300e59ca8bd3a9c0",
        "history": [
          {
            "slot": 49,
            "status": "attestation-missed"
          },
          {
            "slot": 50,
            "status": "attestation-missed"
          },
          {
            "slot": 51,
            "status": "attestation-sent"
          },
          {
            "slot": 52,
            "status": "attestation-sent"
          },
          {
            "slot": 53,
            "status": "attestation-sent"
          },
          {
            "slot": 54,
            "status": "attestation-sent"
          },
          {
            "slot": 55,
            "status": "attestation-sent"
          },
          {
            "slot": 56,
            "status": "attestation-missed"
          },
          {
            "slot": 57,
            "status": "attestation-sent"
          },
          {
            "slot": 58,
            "status": "attestation-sent"
          },
          {
            "slot": 59,
            "status": "attestation-sent"
          },
          {
            "slot": 60,
            "status": "block-mined"
          }
        ],
        "lastAttestation": {
          "date": "2025-03-18T22:30:13.000Z",
          "slot": 59,
          "timestamp": 1742337013
        },
        "lastProposal": {
          "date": "2025-03-18T22:30:25.000Z",
          "slot": 60,
          "timestamp": 1742337025
        },
        "missedAttestations": {
          "count": 3,
          "currentStreak": 0,
          "rate": 0.2727272727272727
        },
        "missedProposals": {
          "count": 0,
          "currentStreak": 0,
          "rate": 0
        },
        "totalSlots": 12
      },
      "0xc223b10ef31d74b74ac4931da23c76dfd154e43f": {
        "address": "0xc223b10ef31d74b74ac4931da23c76dfd154e43f",
        "history": [
          {
            "slot": 49,
            "status": "attestation-missed"
          },
          {
            "slot": 50,
            "status": "attestation-missed"
          },
          {
            "slot": 51,
            "status": "attestation-sent"
          },
          {
            "slot": 52,
            "status": "attestation-sent"
          },
          {
            "slot": 53,
            "status": "attestation-sent"
          },
          {
            "slot": 54,
            "status": "attestation-sent"
          },
          {
            "slot": 55,
            "status": "attestation-sent"
          },
          {
            "slot": 56,
            "status": "attestation-missed"
          },
          {
            "slot": 57,
            "status": "block-mined"
          },
          {
            "slot": 58,
            "status": "attestation-sent"
          },
          {
            "slot": 59,
            "status": "attestation-sent"
          },
          {
            "slot": 60,
            "status": "attestation-sent"
          }
        ],
        "lastAttestation": {
          "date": "2025-03-18T22:30:25.000Z",
          "slot": 60,
          "timestamp": 1742337025
        },
        "lastProposal": {
          "date": "2025-03-18T22:29:49.000Z",
          "slot": 57,
          "timestamp": 1742336989
        },
        "missedAttestations": {
          "count": 3,
          "currentStreak": 0,
          "rate": 0.2727272727272727
        },
        "missedProposals": {
          "count": 0,
          "currentStreak": 0,
          "rate": 0
        },
        "totalSlots": 12
      }
    }
  },
  "validators": [
    {
      "amount": 100000000000000000000,
      "attester": "0xa1153393636750230f4dB253300E59Ca8bD3a9c0",
      "proposer": "0xad6855aDD35F78bc594102E0aC781527cDEb9c52",
      "withdrawer": "0xa1153393636750230f4dB253300E59Ca8bD3a9c0"
    },
    {
      "amount": 100000000000000000000,
      "attester": "0xC223B10ef31d74B74ac4931da23c76dfD154e43F",
      "proposer": "0x6C7810A6A724f0A3c5882e7149b90D23E8bF6e63",
      "withdrawer": "0xC223B10ef31d74B74ac4931da23c76dfD154e43F"
    },
    {
      "amount": 100000000000000000000,
      "attester": "0x4aF126598417209bFa57bab856963c42C2fee5Db",
      "proposer": "0xC6edFc31693E921DF3C826B102a999DB5cC1cA96",
      "withdrawer": "0x4aF126598417209bFa57bab856963c42C2fee5Db"
    },
    {
      "amount": 100000000000000000000,
      "attester": "0x037BBe29C2766243A070a4b3cbf3891767097308",
      "proposer": "0x7D2Cd31b604B34e289E02538D468948eC515b10C",
      "withdrawer": "0x037BBe29C2766243A070a4b3cbf3891767097308"
    },
    {
      "amount": 100000000000000000000,
      "attester": "0x61B13C781397Bdb68284F02961e16602B9ce8923",
      "proposer": "0xB50043ea007490e0a0250afF54831D8443D1393f",
      "withdrawer": "0x61B13C781397Bdb68284F02961e16602B9ce8923"
    }
  ]
}

Fixes #12682

To selecting validators to be slashed based on the output of this API, we can use the following:

$ for VALIDATOR in $(curl -s $AZTEC_NODE_URL -XPOST -d '{"method": "node_getValidatorsStats"}' | jq -r '.result.stats | to_entries[] | select((.value.missedAttestations.rate > 0.8 or .value.missedProposals.rate > 0.8) and  .value.totalSlots > 30) | .value.address'); do echo $VALIDATOR; done

Or to slash them right away:

$ for VALIDATOR in $(curl -s $AZTEC_NODE_URL -XPOST -d '{"method": "node_getValidatorsStats"}' | jq -r '.result.stats | to_entries[] | select((.value.missedAttestations.rate > 0.8 or .value.missedProposals.rate > 0.8) and  .value.totalSlots > 30) | .value.address'); do echo "Removing $VALIDATOR"; cast send $ROLLUP_ADDRESS "initiateWithdraw(address)" $VALIDATOR --rpc-url $HOST --private-key $WITHDRAWER_PK; done

yarn-project/p2p/src/client/p2p_client.ts

Maddiaa0 · 2025-03-25T11:02:03Z

yarn-project/aztec-node/src/sentinel/sentinel.ts

+    await this.l2TipsStore.handleBlockStreamEvent(event);
+    if (event.type === 'blocks-added') {
+      for (const block of event.blocks) {
+        this.slotNumberToArchive.set(block.block.header.getSlot(), block.block.archive.root.toString());


slotNumberToArchive should be pruned with behind slotHistoryLength too

yarn-project/aztec-node/src/sentinel/sentinel.ts

Maddiaa0

Overall approach is great, awesome stuff, will likely need to add the cli option too

spalladino · 2025-03-25T12:45:56Z

will likely need to add the cli option too

I was thinking we could just use the curl + jq version for the time being

Maddiaa0 · 2025-03-25T14:42:43Z

Will need to add the test to bootstrap test-cmds

spalladino · 2025-03-25T14:46:27Z

Will need to add the test to bootstrap test-cmds

I keep forgetting those, thanks!

🤖 I have created a new Aztec Packages release --- ## [0.82.3](v0.82.2...v0.82.3) (2025-03-27) ### Features * `msgpack` encoding for `Program` and `WitnessStack` ([#12841](#12841)) ([1e58eb1](1e58eb1)) * 64 bit log type id, 64 bit log metadata ([#12956](#12956)) ([20d734a](20d734a)) * AVM parsing tag validation ([#12936](#12936)) ([56b1f0d](56b1f0d)) * **avm:** add calldata & returndata to context ([#13008](#13008)) ([f03b2e5](f03b2e5)) * **avm:** merkle db hints (part 1) ([#12922](#12922)) ([34ec9e8](34ec9e8)) * **avm:** merkle hints (part 2) ([#13077](#13077)) ([fbbc6c7](fbbc6c7)) * **avm:** vm2 initial context ([#12972](#12972)) ([e2b1361](e2b1361)) * benchmark avm simulator ([#12985](#12985)) ([00fae1b](00fae1b)) * client flows benchmarks ([#13007](#13007)) ([9bf7568](9bf7568)) * gas benchmark for "normal usage" ([#13073](#13073)) ([4eb1156](4eb1156)) * Implement merkle writes in the merkle check gadget ([#13050](#13050)) ([c94fe50](c94fe50)) * LogEncryption trait ([#12942](#12942)) ([0b7e564](0b7e564)) * Node snapshot sync ([#12927](#12927)) ([afde851](afde851)), closes [#12926](#12926) * **p2p:** capture all gossipsub metrics ([#12930](#12930)) ([cc940cb](cc940cb)) * Prover node snapshot sync ([#13097](#13097)) ([1e77efb](1e77efb)) * staking asset handler ([#12968](#12968)) ([af48184](af48184)), closes [#12932](#12932) * stream crs data to disk ([#12996](#12996)) ([d016e4d](d016e4d)), closes [#12948](#12948) * track failed tests. add flake. ([f4936d7](f4936d7)) * Track test history. ([#13037](#13037)) ([036bb32](036bb32)) * track total tx fee ([#12601](#12601)) ([9612a4e](9612a4e)) * Validators sentinel ([#12818](#12818)) ([770695c](770695c)) ### Bug Fixes * added #[derive(Eq)] to EcdsaPublicKeyNote ([#12966](#12966)) ([0c21c74](0c21c74)) * Allow use of local blob sink client ([#13025](#13025)) ([ba8d654](ba8d654)) * **avm:** semicolons are hard ([#12999](#12999)) ([8871c83](8871c83)) * bootstrap network and sponsored fpc devnet ([#13044](#13044)) ([8a47d8b](8a47d8b)) * Bump tsc target ([#13052](#13052)) ([985e83b](985e83b)) * cycle_group fuzzer ([#12921](#12921)) ([69f426e](69f426e)) * **docs:** Fix import errors in aztec.js tutorial ([#12969](#12969)) ([856208a](856208a)) * **docs:** Load token artifact from the compiled source in the sample dapp tutorial ([#12802](#12802)) ([0838084](0838084)), closes [#12810](#12810) * **docs:** Update sponsored fpc docs to use 82.2 syntax ([#13054](#13054)) ([e5d425b](e5d425b)) * **e2e:** p2p ([#13002](#13002)) ([1ece539](1ece539)) * extend e2e 2 pxes timeout. strip color codes for error_regex. ([73820e4](73820e4)) * flake ([6cc9e81](6cc9e81)) * fuzzer on staking asset handler constructor test ([#13101](#13101)) ([d936285](d936285)) * invalid getCommittee function ([#13072](#13072)) ([327341f](327341f)) * mac publish should use clang 18 like x-compiler, and use it ([#12983](#12983)) ([7b83c45](7b83c45)) * make circuit parsing deterministic ([#11772](#11772)) ([76ef873](76ef873)) * parse away trailing slash from consensus host ([#12577](#12577)) ([6701806](6701806)) * prerelease versions should be pushed to install.aztec.network ([#13086](#13086)) ([c4e6039](c4e6039)) * smoke ([#13060](#13060)) ([7756b15](7756b15)) * some flake additions ([58638f1](58638f1)) * sponsored fpc arg parsed correctly ([#12976](#12976)) ([#12977](#12977)) ([a85f530](a85f530)) * starting the sandbox with no pxe should still deploy initial test accounts ([#13047](#13047)) ([d92d895](d92d895)) * Syntax error when running tests via jest after tsc build ([#13051](#13051)) ([f972db9](f972db9)) * Use the correct image in aztec start ([#13058](#13058)) ([06285cd](06285cd)) * yolo fix ([91e2f4b](91e2f4b)) * yolo fix nightly ([b3b3259](b3b3259)) * yolo fix obvious thing to track fails. ([2fee630](2fee630)) * yolo flakes ([e3b030a](e3b030a)) * yolo set -x ([bfd3205](bfd3205)) * yolo we suspect the halt is making tests fail that would have passed ([04e3fa2](04e3fa2)) ### Miscellaneous * `getIndexedTaggingSecretAsSender` oracle cleanup ([#13015](#13015)) ([8e71e55](8e71e55)) * Add a script to generate cpp files for AVM2 ([#13091](#13091)) ([7bb43a9](7bb43a9)) * add default native proving for cli wallet ([#12855](#12855)) ([c0f773c](c0f773c)) * add default native proving for cli wallet retry ([#13028](#13028)) ([b2f4785](b2f4785)) * Alpha testnet into master ([#13033](#13033)) ([d98fdbd](d98fdbd)) * AVM TS - move tag validation outside of instruction constructors ([#13038](#13038)) ([45548ab](45548ab)), closes [#12934](#12934) * **avm:** final codegen nuking ([#13089](#13089)) ([9c82f3f](9c82f3f)) * **avm:** remove codegen (all but flavor) ([#13079](#13079)) ([e1f2bdd](e1f2bdd)) * **bb:** minor acir buf C++ improvements ([#13042](#13042)) ([1ebd044](1ebd044)) * boxes dep cleanup ([#12979](#12979)) ([6540b7c](6540b7c)) * **ci:** less catch all e2e_p2p flakes ([#12737](#12737)) ([2134634](2134634)) * comprehensive cleanup of translator flavor and use inheritance properly in flavors ([#13041](#13041)) ([dc5f78f](dc5f78f)) * compress storage footprint ([#12871](#12871)) ([58c110f](58c110f)) * display warning when installing bb versions < 0.82.0 ([#13027](#13027)) ([7247fe7](7247fe7)) * **docs:** Update docs on fees and various other updates ([#12929](#12929)) ([1dec907](1dec907)) * dump dmesg/net/cpu/mem usage at end of ci run ([#12967](#12967)) ([8877792](8877792)) * fix governance util issue ([#13043](#13043)) ([d768d26](d768d26)) * redundant if in affine from projective constructor ([#13045](#13045)) ([3a7ba2d](3a7ba2d)) * remove addition of dummy ops in mock circuit producer ([#13003](#13003)) ([a64d1dc](a64d1dc)) * remove dummy ops in decider pk ([#13049](#13049)) ([da6d021](da6d021)) * replace relative paths to noir-protocol-circuits ([e1b88f6](e1b88f6)) * replace relative paths to noir-protocol-circuits ([849b4b0](849b4b0)) * replace relative paths to noir-protocol-circuits ([18a02d6](18a02d6)) * Revert "chore: add default native proving for cli wallet ([#12855](#12855))" ([#13013](#13013)) ([98e2576](98e2576)) * Speed up and deflake sentinel test ([#13078](#13078)) ([27f1eca](27f1eca)) * **testnet:** making consensus host mandatory input ([#12716](#12716)) ([d47c74a](d47c74a)) * towards no more mock op_queues ([#12984](#12984)) ([fefffa7](fefffa7)) * update bb version for noir 1.0.0-beta.0+ ([#13026](#13026)) ([dd68074](dd68074)) * update CODEOWNERS to reflect new sync method ([#12998](#12998)) ([a3d1915](a3d1915)) ### Documentation * Add fees to cli reference ([#12884](#12884)) ([4a0fd58](4a0fd58)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

spalladino force-pushed the palla/identify-slashable branch from 1d5d5ce to 98f53de Compare March 18, 2025 22:01

spalladino marked this pull request as ready for review March 18, 2025 22:02

spalladino force-pushed the palla/identify-slashable branch 7 times, most recently from 18a62bd to 6c5e30c Compare March 21, 2025 14:18

LHerskind mentioned this pull request Mar 21, 2025

test scenario: Slashing - Large number of dead validators #11828

Closed

Maddiaa0 reviewed Mar 25, 2025

View reviewed changes

yarn-project/p2p/src/client/p2p_client.ts Outdated Show resolved Hide resolved

Maddiaa0 reviewed Mar 25, 2025

View reviewed changes

yarn-project/aztec-node/src/sentinel/sentinel.ts Outdated Show resolved Hide resolved

Maddiaa0 reviewed Mar 25, 2025

View reviewed changes

spalladino force-pushed the palla/identify-slashable branch from 6c5e30c to 16afc62 Compare March 25, 2025 12:57

spalladino added 5 commits March 25, 2025 11:04

feat: Validators sentinel

f07e163

Add some unit tests

eea1e24

A few more unit tests

e5b355f

Apply suggestions

90ab35f

Cleanup log

72d023b

spalladino force-pushed the palla/identify-slashable branch from 48024b1 to 72d023b Compare March 25, 2025 14:05

Handle edge case with empty committee

9c76886

spalladino added 2 commits March 25, 2025 11:49

Address odd off-by-one in test

462a30c

Add test

b2e1c02

spalladino requested a review from charlielye as a code owner March 25, 2025 14:49

Maddiaa0 approved these changes Mar 25, 2025

View reviewed changes

Maddiaa0 enabled auto-merge (squash) March 25, 2025 14:53

Fix test

97bb0f2

spalladino force-pushed the palla/identify-slashable branch from 978166d to 97bb0f2 Compare March 25, 2025 15:06

Fix test

a875bb3

Maddiaa0 merged commit 770695c into master Mar 25, 2025
7 checks passed

Maddiaa0 deleted the palla/identify-slashable branch March 25, 2025 15:51

AztecBot mentioned this pull request Mar 25, 2025

chore(master): release 0.82.3 #12980

Merged

DanielKotov pushed a commit that referenced this pull request Mar 27, 2025

feat: Validators sentinel (#12818)

6b30cd5

DanielKotov pushed a commit that referenced this pull request Mar 27, 2025

feat: Validators sentinel (#12818)

d8ae7f3

FilipHarald mentioned this pull request Apr 22, 2025

polished validator dashboard aztec-scan/chicmoz#406

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Validators sentinel#12818

feat: Validators sentinel#12818
Maddiaa0 merged 10 commits intomasterfrom
palla/identify-slashable

spalladino commented Mar 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

Maddiaa0 Mar 25, 2025

Uh oh!

Uh oh!

Maddiaa0 left a comment

Uh oh!

spalladino commented Mar 25, 2025

Uh oh!

Maddiaa0 commented Mar 25, 2025

Uh oh!

spalladino commented Mar 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

spalladino commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Maddiaa0 Mar 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Maddiaa0 left a comment

Choose a reason for hiding this comment

Uh oh!

spalladino commented Mar 25, 2025

Uh oh!

Maddiaa0 commented Mar 25, 2025

Uh oh!

spalladino commented Mar 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

spalladino commented Mar 17, 2025 •

edited

Loading