test(l1): add per-phase timing breakdown to multisync Slack notifications

[pablodeymo]

Motivation

The multisync monitoring script (docker_monitor.py) sends Slack notifications at the end of each sync run, but they only report the total sync time per network. When investigating performance regressions or comparing runs, we had to manually SSH into the server and parse raw container logs to figure out which phase was slow. This is time-consuming and error-prone.

The sync logs already contain per-phase completion markers like:

✓ BLOCK HEADERS complete: 25,693,009 headers in 0:29:00
✓ STORAGE HEALING complete: 87,414 storage accounts healed in 1:42:00

This PR surfaces that data directly in the Slack notification, so performance bottlenecks are visible at a glance.

Description

Adds three things to tooling/sync/docker_monitor.py:

1. PHASE_COMPLETION_PATTERNS dict — Regex patterns for all 8 snap sync phases:

   • Block Headers, Account Ranges, Account Insertion, Storage Ranges, Storage Insertion, State Healing, Storage Healing, Bytecodes

2. parse_phase_timings(run_id, container) function — Reads saved container log files from multisync_logs/run_{run_id}/{container}.log and extracts (phase_name, item_count, duration) for each completed phase. Returns an empty list if logs are missing or if a phase didn't complete (e.g., on a failed run), so the behavior is graceful.

3. Phase breakdown in Slack and run logs — After the per-instance status line, a code block is appended showing the full phase timing table. The same breakdown is also written to run_history.log and the per-run summary.txt.

Expected Slack output (successful run)

The Slack message will now include a section like this for each network instance:

📊 Phase Breakdown — mainnet
Block Headers       0:29:00  (25,693,009)
Account Ranges      0:45:12  (12,345,678)
Account Insertion   0:12:34  (12,345,678)
Storage Ranges      0:38:45  (1,234,567)
Storage Insertion   0:08:23  (1,234,567)
State Healing       0:15:00  (87,414)
Storage Healing     1:42:00  (87,414)
Bytecodes           0:05:30  (45,678)

Phase names are left-aligned with padding for readability. The count in parentheses corresponds to the number of items processed (headers, accounts, storage slots, etc.).

Expected Slack output (failed run with partial phases)

If a run fails mid-sync (e.g., timeout during storage healing), only the phases that completed are shown:

📊 Phase Breakdown — mainnet
Block Headers       0:29:00  (25,693,009)
Account Ranges      0:45:12  (12,345,678)
Account Insertion   0:12:34  (12,345,678)
Storage Ranges      0:38:45  (1,234,567)
Storage Insertion   0:08:23  (1,234,567)

Phases that never completed (State Healing, Storage Healing, Bytecodes in this case) are simply omitted — no placeholder or "N/A" rows.

Expected text log output (summary.txt / run_history.log)

  ✅ mainnet: success (sync: 4h 32m 15s)
    Phase Breakdown:
      Block Headers       0:29:00  (25,693,009)
      Account Ranges      0:45:12  (12,345,678)
      Account Insertion   0:12:34  (12,345,678)
      Storage Ranges      0:38:45  (1,234,567)
      Storage Insertion   0:08:23  (1,234,567)
      State Healing       0:15:00  (87,414)
      Storage Healing     1:42:00  (87,414)
      Bytecodes           0:05:30  (45,678)

How it works

The flow is:

1. save_all_logs() saves container logs to disk (already existed, no changes)
2. log_run_result() now calls parse_phase_timings() and appends breakdown to text log
3. slack_notify() now calls parse_phase_timings() and appends code blocks to Slack payload

Since save_all_logs() is called before both log_run_result() and slack_notify() (lines 721→725 in main loop), the saved log files are always available for parsing.

Edge cases

Scenario	Behavior
Run fails before any phase completes	No breakdown section shown
Log file missing or unreadable	Empty list returned, no breakdown
Only some phases completed	Only completed phases listed
Multiple networks (hoodi, sepolia, mainnet)	Separate breakdown per instance

Checklist

• Updated STORE_SCHEMA_VERSION (crates/storage/lib.rs) if the PR includes breaking changes to the Store requiring a re-sync.

N/A — This PR only modifies the Python monitoring script, no Rust code or storage changes.

[greptile-apps]

Greptile Overview

Greptile Summary

Adds per-phase timing breakdown to multisync monitoring Slack notifications and run logs by parsing saved container logs for phase completion markers.

Key changes:

• Added PHASE_COMPLETION_PATTERNS dict with regex patterns matching all 8 snap sync phases from Rust logs (network.rs:389)
• Implemented parse_phase_timings() function that reads saved log files and extracts phase completion data (name, count, duration)
• Enhanced slack_notify() and log_run_result() to display phase breakdowns after per-instance status
• Gracefully handles missing logs or incomplete phases by returning empty list

Implementation quality:

• Clean separation of concerns with dedicated parsing function
• Proper error handling with try/except and file existence checks
• Regex patterns correctly match the actual log format from network.rs (verified against source)
• Consistent formatting across Slack and text logs
• No breaking changes to existing functionality

The change provides immediate visibility into performance bottlenecks without requiring manual log inspection.

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• The implementation is well-designed with proper error handling, no breaking changes, and correctly matches the actual log format from the Rust source code. The changes are isolated to the monitoring script with graceful degradation on failures.
• No files require special attention

Important Files Changed

Filename	Overview
tooling/sync/docker_monitor.py	Adds phase timing breakdown parsing and display for multisync monitoring - clean implementation with proper error handling

Sequence Diagram

sequenceDiagram
    participant Main as main()
    participant SaveLogs as save_all_logs()
    participant LogResult as log_run_result()
    participant SlackNotify as slack_notify()
    participant ParsePhase as parse_phase_timings()
    participant LogFile as Log Files

    Note over Main: Run completes (success or failed)
    Main->>SaveLogs: save_all_logs(instances, run_id, compose_file)
    SaveLogs->>LogFile: Write container logs to multisync_logs/run_{run_id}/{container}.log
    LogFile-->>SaveLogs: Logs saved
    SaveLogs-->>Main: Complete

    Main->>LogResult: log_run_result(run_id, run_count, instances, ...)
    loop For each instance
        LogResult->>ParsePhase: parse_phase_timings(run_id, container)
        ParsePhase->>LogFile: Read multisync_logs/run_{run_id}/{container}.log
        LogFile-->>ParsePhase: Log content
        ParsePhase->>ParsePhase: Apply regex patterns for 8 phases
        ParsePhase-->>LogResult: [(phase_name, count, duration), ...]
        LogResult->>LogResult: Format and append to text log
    end
    LogResult->>LogFile: Append to run_history.log and summary.txt
    LogResult-->>Main: Complete

    Main->>SlackNotify: slack_notify(run_id, run_count, instances, ...)
    loop For each instance
        SlackNotify->>ParsePhase: parse_phase_timings(run_id, container)
        ParsePhase->>LogFile: Read multisync_logs/run_{run_id}/{container}.log
        LogFile-->>ParsePhase: Log content
        ParsePhase->>ParsePhase: Apply regex patterns for 8 phases
        ParsePhase-->>SlackNotify: [(phase_name, count, duration), ...]
        SlackNotify->>SlackNotify: Format phase breakdown code block
    end
    SlackNotify->>SlackNotify: POST to Slack webhook with blocks
    SlackNotify-->>Main: Complete

Loading

[Copilot]

Pull request overview

This PR adds per-phase timing breakdown to multisync Slack notifications and log files, making it easier to identify performance bottlenecks in sync operations without manually parsing container logs.

Changes:

• Added PHASE_COMPLETION_PATTERNS dictionary with regex patterns for all 8 snap sync phases
• Added parse_phase_timings() function to extract phase timing data from saved container logs
• Enhanced Slack notifications and text logs to display phase breakdowns for each network instance

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The phase breakdown sections are added in a separate loop (lines 360-371) from the instance status sections (lines 342-357). This means in the Slack notification, all instance statuses will be shown first, followed by all phase breakdowns. This differs from the text log format (lines 456-479) where each phase breakdown immediately follows its instance status. For better readability and consistency, consider moving the phase breakdown logic inside the first loop (after line 357) so each instance's breakdown appears immediately after its status, matching the text log format and the example in the PR description.