feat: merge-train/spartan

.claude/agents/analyze-logs.md

@@ -39,14 +39,14 @@ Return a condensed summary:
 
 ## Key Events Timeline
 
-**IMPORTANT**: Include BOTH absolute timestamps (for referencing original logs) AND relative offsets from test start (for easier understanding and cross-run comparison).
+**IMPORTANT**: Include BOTH absolute timestamps (for referencing original logs) AND relative offsets from test start (for easier understanding and cross-run comparison). Include the **actor** column when logs come from multi-node tests.
 
-| Time | Offset | Level | Module | Event |
-|------|--------|-------|--------|-------|
-| 11:18:42 | +0.0s | INFO | e2e | Running test my_test |
-| 11:18:44 | +2.1s | INFO | sequencer | Building block |
-| 11:18:47 | +5.3s | ERROR | sequencer | Failed to build block |
-| 11:18:50 | +8.0s | WARN | p2p | Connection timeout |
+| Time | Offset | Actor | Level | Module | Event |
+|------|--------|-------|-------|--------|-------|
+| 11:18:42 | +0.0s | | INFO | e2e | Running test my_test |
+| 11:18:44 | +2.1s | validator-0 | INFO | sequencer | Building block |
+| 11:18:47 | +5.3s | validator-0 | ERROR | sequencer | Failed to build block |
+| 11:18:50 | +8.0s | node-0 | WARN | p2p | Connection timeout |
 
 (Offset = seconds since "Running test" marker. Makes it easy to compare timing between failed and successful runs.)
 
@@ -75,15 +75,27 @@ Return a condensed summary:
 
 Aztec logs follow this format:
 ```
-HH:MM:SS [HH:MM:SS.mmm] LEVEL: module:submodule Message {optional json}
+HH:MM:SS [HH:MM:SS.mmm] LEVEL: module [actor] [instanceId] Message {optional json}
 ```
 
-Example:
+- **module**: Always present. The component emitting the log (colon-separated hierarchy).
+- **actor**: Optional. Identifies *which process/node* is emitting the log in multi-node tests (e.g., `validator-0`, `node-1`, `prover-0`). Only set in e2e tests that spin up multiple nodes.
+- **instanceId**: Optional. Identifies *which instance* of a component within a single process (e.g., `checkpoint-5`, `epoch-3`). Used when multiple instances of the same component exist concurrently (e.g., one epoch-proving-job per epoch).
+
+Both `actor` and `instanceId` appear between the module name and the message in pretty-printed output.
+
+Examples:
 ```
 11:18:42 [11:18:42.518] WARN: node:blob-client:client No L1 consensus host urls configured
 11:18:42 [11:18:42.562] INFO: world_state Created world state synchroniser with block history of 2
+11:18:44 [11:18:44.100] INFO: sequencer validator-0 Building block 1
+11:18:45 [11:18:45.200] INFO: checkpoint-builder validator-1 checkpoint-5 Building block 1
 ```
 
+In the last two examples:
+- `validator-0` / `validator-1` is the **actor** (which node)
+- `checkpoint-5` is the **instanceId** (which checkpoint instance)
+
 **Levels** (in priority order for investigation):
 1. `ERROR` - Always investigate
 2. `WARN` - Often important
@@ -165,16 +177,25 @@ FAIL src/e2e_something.test.ts
 ## Multi-Actor Awareness
 
 Tests may have multiple actors running concurrently:
-- Multiple nodes
-- Sequencers
-- Validators
-- Provers
+- Multiple nodes (`node-0`, `node-1`, ...)
+- Validators (`validator-0`, `validator-1`, ...)
+- Provers (`prover-0`, `prover-1`, ...)
+
+The **actor** field in logs identifies which node/process emitted a log line. It appears between the module name and the message:
+```
+INFO: sequencer validator-0 Building block 1     # Sequencer on validator-0
+INFO: sequencer validator-1 Building block 1     # Sequencer on validator-1
+INFO: p2p node-0 Received block proposal         # P2P on node-0
+```
+
+**Filtering by actor**: When investigating a specific node's behavior, filter logs by the actor name (e.g., grep for `validator-0`) to isolate that node's log stream.
 
-Log entries belong to different actors, sometimes (but not always) identified by a keyword or number in the module. Example:
+**Instance IDs**: Within a single actor, the **instanceId** field distinguishes multiple instances of the same component. For example, epoch-proving-jobs spawn one per epoch:
 ```
-INFO: node:MAIN-aztec-node Starting...      # Main node
-INFO: node:SECONDARY-aztec-node Starting... # Secondary node
+INFO: epoch-proving-job prover-0 epoch-3 Starting proof generation
+INFO: epoch-proving-job prover-0 epoch-4 Starting proof generation
 ```
+Here `prover-0` is the actor and `epoch-3`/`epoch-4` are instance IDs.
 
 ## High-Value Log Sources
 

.github/workflows/publish-misc-pages.yml

@@ -0,0 +1,36 @@
+name: Publish misc GitHub Pages
+on:
+  push:
+    branches:
+      - next
+    paths:
+      - "yarn-project/scripts/latency-explorer/**"
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout source repo
+        uses: actions/checkout@v4
+
+      - name: Checkout benchmark-page-data repo
+        uses: actions/checkout@v4
+        with:
+          repository: AztecProtocol/benchmark-page-data
+          token: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
+          path: benchmark-page-data
+
+      - name: Copy latency explorer
+        run: |
+          mkdir -p benchmark-page-data/misc/tx-latency-explorer
+          cp yarn-project/scripts/latency-explorer/index.html benchmark-page-data/misc/tx-latency-explorer/index.html
+
+      - name: Push changes
+        working-directory: benchmark-page-data
+        run: |
+          git config user.name "AztecBot"
+          git config user.email "tech@aztecprotocol.com"
+          git add .
+          git diff --cached --quiet && echo "No changes to publish" && exit 0
+          git commit -m "Update tx-latency-explorer from ${{ github.sha }}"
+          git push

ci3/run_test_cmd

@@ -47,8 +47,9 @@ if [ "$CI" -eq 1 ]; then
   allow_flakes=1
   track_test_fail=1
 
-  # Track the test in the history tracker if this is in merge queue or targeting a version branch.
-  if [[ "$is_merge_queue" -eq 1 || ("${TARGET_BRANCH:-}" =~ ^v[0-9]) ]]; then
+  # Track the test in the history tracker if this is in merge queue, targeting a version branch, or
+  # targeting a merge-train.
+  if [[ "$is_merge_queue" -eq 1 || ("${TARGET_BRANCH:-}" =~ ^v[0-9]) || ("${TARGET_BRANCH:-}" == merge-train/*) ]]; then
     track_test_history=1
   fi
 

spartan/aztec-node/templates/_pod-template.yaml

@@ -248,6 +248,10 @@ spec:
         - name: SLASH_DUPLICATE_PROPOSAL_PENALTY
           value: {{ .Values.node.slash.duplicateProposalPenalty | quote }}
         {{- end }}
+        {{- if .Values.node.slash.duplicateAttestationPenalty }}
+        - name: SLASH_DUPLICATE_ATTESTATION_PENALTY
+          value: {{ .Values.node.slash.duplicateAttestationPenalty | quote }}
+        {{- end }}
         {{- if .Values.node.slash.attestDescendantOfInvalidPenalty }}
         - name: SLASH_ATTEST_DESCENDANT_OF_INVALID_PENALTY
           value: {{ .Values.node.slash.attestDescendantOfInvalidPenalty | quote }}

spartan/bootstrap.sh

@@ -153,6 +153,7 @@ function network_bench {
 
   echo_header "spartan bench"
   gcp_auth
+  export K8S_ENRICHER=${K8S_ENRICHER:-1}
   network_bench_cmds | parallelize 1
 }
 
@@ -165,6 +166,7 @@ function proving_bench {
 
   echo_header "spartan proving bench"
   gcp_auth
+  export K8S_ENRICHER=${K8S_ENRICHER:-1}
   proving_bench_cmds | parallelize 1
 }
 
@@ -219,6 +221,7 @@ case "$cmd" in
     # Run the network deploy script
     DENOISE=1 denoise "./scripts/network_deploy.sh $env_file"
 
+    export K8S_ENRICHER=${K8S_ENRICHER:-1}
     if [[ "${RUN_TESTS:-}" == "true" ]]; then
       if [[ -n "$test_set" ]]; then
         network_tests_$test_set "$env_file"
@@ -281,8 +284,8 @@ case "$cmd" in
     source scripts/source_network_env.sh
     source_network_env ${KIND_ENV:-kind-provers}
     namespace="upgrade-rollup-version${NAME_POSTFIX:-}"
-    INSTALL_METRICS=false \
-      ./scripts/test_kind.sh src/spartan/upgrade_rollup_version.test.ts "$namespace"
+    export K8S_ENRICHER=${K8S_ENRICHER:-1}
+    ./scripts/test_kind.sh src/spartan/upgrade_rollup_version.test.ts "$namespace"
     ;;
   "network_teardown")
     env_file="$1"

spartan/environments/mbps-net.env

@@ -0,0 +1,60 @@
+CREATE_ETH_DEVNET=false
+GCP_REGION=us-west1-a
+CLUSTER=aztec-gke-private
+NETWORK=next-net
+NAMESPACE=mbps-net
+DESTROY_NAMESPACE=true
+ETHEREUM_CHAIN_ID=11155111
+ETHEREUM_RPC_URLS=REPLACE_WITH_GCP_SECRET
+ETHEREUM_CONSENSUS_HOST_URLS=REPLACE_WITH_GCP_SECRET
+ETHEREUM_CONSENSUS_HOST_API_KEYS=REPLACE_WITH_GCP_SECRET
+ETHEREUM_CONSENSUS_HOST_API_KEY_HEADERS=REPLACE_WITH_GCP_SECRET
+FUNDING_PRIVATE_KEY=REPLACE_WITH_GCP_SECRET
+LABS_INFRA_MNEMONIC=REPLACE_WITH_GCP_SECRET
+ROLLUP_DEPLOYMENT_PRIVATE_KEY=REPLACE_WITH_GCP_SECRET
+OTEL_COLLECTOR_ENDPOINT=REPLACE_WITH_GCP_SECRET
+VERIFY_CONTRACTS=false
+ETHERSCAN_API_KEY=REPLACE_WITH_GCP_SECRET
+DEPLOY_INTERNAL_BOOTNODE=true
+STORE_SNAPSHOT_URL=
+BLOB_BUCKET_DIRECTORY=${BLOB_BUCKET_DIRECTORY:-next-net/blobs}
+R2_ACCESS_KEY_ID=REPLACE_WITH_GCP_SECRET
+R2_SECRET_ACCESS_KEY=REPLACE_WITH_GCP_SECRET
+PROVER_FAILED_PROOF_STORE=gs://aztec-develop/next-net/failed-proofs
+TEST_ACCOUNTS=true
+SPONSORED_FPC=true
+SEQ_MIN_TX_PER_BLOCK=0
+SEQ_MAX_TX_PER_BLOCK=8
+AZTEC_EPOCH_DURATION=32
+REAL_VERIFIER=false
+PROVER_REAL_PROOFS=false
+
+SEQ_BUILD_CHECKPOINT_IF_EMPTY=true
+SEQ_BLOCK_DURATION_MS=6000
+LOG_LEVEL=verbose
+
+AZTEC_LAG_IN_EPOCHS_FOR_VALIDATOR_SET=2
+AZTEC_LAG_IN_EPOCHS_FOR_RANDAO=2
+
+VALIDATOR_REPLICAS=4
+VALIDATORS_PER_NODE=12
+PUBLISHERS_PER_VALIDATOR_KEY=2
+VALIDATOR_PUBLISHER_MNEMONIC_START_INDEX=5000
+
+PUBLISHERS_PER_PROVER=2
+PROVER_PUBLISHER_MNEMONIC_START_INDEX=8000
+
+BOT_TRANSFERS_REPLICAS=1
+BOT_TRANSFERS_TX_INTERVAL_SECONDS=8
+BOT_TRANSFERS_FOLLOW_CHAIN=PENDING
+
+BOT_SWAPS_REPLICAS=1
+BOT_SWAPS_FOLLOW_CHAIN=PENDING
+BOT_SWAPS_TX_INTERVAL_SECONDS=8
+
+REDEPLOY_ROLLUP_CONTRACTS=true
+
+DEBUG_P2P_INSTRUMENT_MESSAGES=true
+
+VALIDATOR_HA_REPLICAS=1
+VALIDATOR_RESOURCE_PROFILE="prod-spot"

spartan/environments/network-defaults.yml

@@ -139,6 +139,8 @@ slasher: &slasher
   SLASH_ATTEST_DESCENDANT_OF_INVALID_PENALTY: 10e18
   # Penalty for proposing two different block or checkpoint proposal for the same position.
   SLASH_DUPLICATE_PROPOSAL_PENALTY: 10e18
+  # Penalty for signing attestations for different proposals at the same slot.
+  SLASH_DUPLICATE_ATTESTATION_PENALTY: 10e18
   # Penalty for unknown offenses.
   SLASH_UNKNOWN_PENALTY: 10e18
   # Penalty for broadcasting an invalid block.
@@ -240,6 +242,7 @@ networks:
     SLASH_INACTIVITY_PENALTY: 10e18
     SLASH_PROPOSE_INVALID_ATTESTATIONS_PENALTY: 10e18
     SLASH_DUPLICATE_PROPOSAL_PENALTY: 10e18
+    SLASH_DUPLICATE_ATTESTATION_PENALTY: 10e18
     SLASH_ATTEST_DESCENDANT_OF_INVALID_PENALTY: 10e18
     SLASH_UNKNOWN_PENALTY: 10e18
     SLASH_INVALID_BLOCK_PENALTY: 10e18
@@ -278,6 +281,7 @@ networks:
     SLASH_INACTIVITY_PENALTY: 10e18
     SLASH_PROPOSE_INVALID_ATTESTATIONS_PENALTY: 10e18
     SLASH_DUPLICATE_PROPOSAL_PENALTY: 10e18
+    SLASH_DUPLICATE_ATTESTATION_PENALTY: 10e18
     SLASH_ATTEST_DESCENDANT_OF_INVALID_PENALTY: 10e18
     SLASH_UNKNOWN_PENALTY: 10e18
     SLASH_INVALID_BLOCK_PENALTY: 10e18
@@ -334,6 +338,7 @@ networks:
     SLASH_INACTIVITY_PENALTY: 2000e18
     SLASH_PROPOSE_INVALID_ATTESTATIONS_PENALTY: 2000e18
     SLASH_DUPLICATE_PROPOSAL_PENALTY: 2000e18
+    SLASH_DUPLICATE_ATTESTATION_PENALTY: 2000e18
     SLASH_ATTEST_DESCENDANT_OF_INVALID_PENALTY: 2000e18
     SLASH_UNKNOWN_PENALTY: 2000e18
     SLASH_INVALID_BLOCK_PENALTY: 2000e18

spartan/scripts/deploy_network.sh

@@ -484,6 +484,7 @@ SLASH_PRUNE_PENALTY = ${SLASH_PRUNE_PENALTY:-null}
 SLASH_DATA_WITHHOLDING_PENALTY = ${SLASH_DATA_WITHHOLDING_PENALTY:-null}
 SLASH_PROPOSE_INVALID_ATTESTATIONS_PENALTY = ${SLASH_PROPOSE_INVALID_ATTESTATIONS_PENALTY:-null}
 SLASH_DUPLICATE_PROPOSAL_PENALTY = ${SLASH_DUPLICATE_PROPOSAL_PENALTY:-null}
+SLASH_DUPLICATE_ATTESTATION_PENALTY = ${SLASH_DUPLICATE_ATTESTATION_PENALTY:-null}
 SLASH_ATTEST_DESCENDANT_OF_INVALID_PENALTY = ${SLASH_ATTEST_DESCENDANT_OF_INVALID_PENALTY:-null}
 SLASH_UNKNOWN_PENALTY = ${SLASH_UNKNOWN_PENALTY:-null}
 SLASH_INVALID_BLOCK_PENALTY = ${SLASH_INVALID_BLOCK_PENALTY:-null}

spartan/scripts/k8s_enricher.ts

@@ -35,6 +35,12 @@ if (cacheLogMode) {
 
 const namespace = args[0];
 
+// Previously this also ran, then Alex got bombarded running hundreds of prover agents.
+// This is now set on the relevant tests.
+if (process.env.K8S_ENRICHER !== '1') {
+  process.exit(0);
+}
+
 if (!namespace) {
   console.error('Usage: k8s_enricher.ts <namespace> [--cache-log]');
   process.exit(1);

spartan/terraform/cloudflare/main.tf

@@ -37,24 +37,42 @@ resource "cloudflare_r2_custom_domain" "aztec_labs_snapshots_com" {
   enabled     = true
 }
 
+locals {
+  top_level_folders = toset([
+    "devnet",
+    "ignition-sepolia",
+    "next-net",
+    "staging-ignition",
+    "staging-public",
+  ])
+}
+
 # Lifecycle rules to automatically delete old objects
 resource "cloudflare_r2_bucket_lifecycle" "cleanup" {
   account_id  = var.R2_ACCOUNT_ID
   bucket_name = cloudflare_r2_bucket.bucket.name
 
-  rules = [
-    {
-      id                        = "delete-snapshot-files"
-      enabled                   = true
-      conditions                = { suffix = ".db" }
-      delete_objects_transition = { days = var.SNAPSHOT_RETENTION_DAYS }
-    },
-    {
-      id                        = "delete-blob-files"
-      enabled                   = true
-      conditions                = { suffix = ".data" }
-      delete_objects_transition = { days = var.BLOB_RETENTION_DAYS }
-    },
-  ]
+  rules = flatten([
+    for folder in local.top_level_folders : [
+      {
+        id                        = "delete-snapshots-${folder}"
+        enabled                   = true
+        conditions                = { prefix = "${folder}/aztec" }
+        delete_objects_transition = { days = var.SNAPSHOT_RETENTION_DAYS }
+      },
+      {
+        id                        = "delete-blobs-${folder}"
+        enabled                   = true
+        conditions                = { prefix = "${folder}/blobs" }
+        delete_objects_transition = { days = var.BLOB_RETENTION_DAYS }
+      },
+      {
+        id                        = "delete-txs-${folder}"
+        enabled                   = true
+        conditions                = { prefix = "${folder}/txs" }
+        delete_objects_transition = { days = var.TX_RETENTION_DAYS }
+      },
+    ]
+  ])
 }