feat: replay mechanism to sync node with execution layer

[randygrok]

Overview

Closes: #2750

[github-actions]

The latest Buf updates on your PR. Results from workflow CI and Release / buf-check (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	⏩ skipped	✅ passed	✅ passed	Oct 21, 2025, 4:45 PM

[claude]

Claude finished @randygrok's task —— View job

---

Code Review: Replay Mechanism Implementation

I've conducted a comprehensive review of the replay mechanism for synchronizing the execution layer with ev-node state. Overall, this is a well-structured implementation that addresses an important crash recovery scenario. The code is clean, well-tested, and the maintainer feedback has been appropriately addressed.

Summary Assessment

Strengths:

• Clean separation of concerns with the Replayer abstraction
• Comprehensive test coverage (82.54% on patch)
• Good error handling and validation
• Optional interface pattern (HeightProvider) allows backward compatibility
• Excellent documentation of validation assumptions

Areas for Improvement:

• Some edge cases in state management could be hardened
• Minor concurrency considerations
• A few opportunities for code clarity improvements

---

Detailed Findings

1. Code Quality & Architecture ✅

Positive:

• The Replayer struct (block/internal/common/replay.go:17-24) follows idiomatic Go patterns
• Clean separation: replay logic is isolated from executor/syncer concerns
• Optional HeightProvider interface (core/execution/execution.go:89-103) is well-designed for backward compatibility
• Integration points in executor.go:224 and syncer.go:208 are clean and non-invasive

Suggestions:

1. Naming consistency - The component logger uses "execution_replayer" (replay.go:37) but the struct is called Replayer. Consider using just "replayer" for consistency with the renamed struct.

// Current:
logger:  logger.With().Str("component", "execution_replayer").Logger(),

// Suggested:
logger:  logger.With().Str("component", "replayer").Logger(),

2. Package organization - The Replayer is in block/internal/common/ which is appropriate for shared code between executor and syncer. However, consider if this warrants its own package (block/internal/replay/) as it grows, especially if more replay strategies are added.

---

2. Error Handling & Resilience ⚠️

Positive:

• Good use of fmt.Errorf with %w for error wrapping (replay.go:62, 90, 116, etc.)
• Critical errors are properly logged with context
• Graceful handling when executor doesn't implement HeightProvider (replay.go:48-52)

Issues Found:

1. State inconsistency after partial replay failure (replay.go:88-92)

The replay loop doesn't handle partial failures well. If block 5 succeeds but block 6 fails when syncing to height 10, the execution layer is left at height 5 while the function returns an error. The caller doesn't know how far sync progressed.

// Current implementation:
for height := execHeight + 1; height <= targetHeight; height++ {
    if err := s.replayBlock(ctx, height); err != nil {
        return fmt.Errorf("failed to replay block %d to execution layer: %w", height, err)
    }
}

Recommendation: Consider one of these approaches:

• Return a structured error with the last successful height
• Use a checkpoint/resume mechanism
• Document the current behavior and recovery expectations

2. Error handling in replayBlock for genesis block (replay.go:122-130)

When replaying the genesis block, the code sets prevState.AppHash = header.AppHash (line 129) with a comment "This will be updated by InitChain". However, InitChain isn't called here - it was already called during initialization. This could be confusing.

Recommendation: Add a comment explaining why we use header.AppHash for genesis state, or verify this is correct.

3. Context cancellation check missing in loop (replay.go:88-92)

The replay loop doesn't check for context cancellation between blocks, which could delay shutdown during long replays.

// Suggested improvement:
for height := execHeight + 1; height <= targetHeight; height++ {
    select {
    case <-ctx.Done():
        return fmt.Errorf("replay cancelled at height %d: %w", height-1, ctx.Err())
    default:
    }
    if err := s.replayBlock(ctx, height); err != nil {
        return fmt.Errorf("failed to replay block %d to execution layer: %w", height, err)
    }
}

---

3. Security & Validation ✅

Positive:

• AppHash verification prevents state divergence (replay.go:163-176)
• Good documentation of validation assumptions (replay.go:105-110)
• Fail-hard on execution layer ahead of target (replay.go:70-77)

Observations:

1. Validation assumptions are well-documented (replay.go:106-110) - The comment clearly states that blocks in the store are pre-validated, and only AppHash is checked. This is appropriate for replay scenarios.

2. AppHash mismatch handling is correct (replay.go:164-176) - Returns an error with detailed logging. However, consider if this should trigger any cleanup or state reset, since it indicates serious corruption.

---

4. Performance & Resource Efficiency ✅

Positive:

• Replay is only triggered when needed (height checks)
• No unnecessary allocations in the replay loop
• Efficient use of store batch operations in caller code

Suggestions:

1. Batch progress logging - Currently logs at Info level for every block (replay.go:112, 178-180). For large replays (e.g., 10,000 blocks), consider logging every N blocks:

if height%100 == 0 || height == targetHeight {
    s.logger.Info().Uint64("height", height).Uint64("target", targetHeight).Msg("replay progress")
}

2. Memory allocation in transaction conversion (replay.go:147-150)

// Current:
rawTxs := make([][]byte, len(data.Txs))
for i, tx := range data.Txs {
    rawTxs[i] = []byte(tx)
}

This is fine, but note that []byte(tx) creates a copy. If types.Tx is already []byte, this is redundant. Verify if this copy is necessary or if you can pass data.Txs directly (after checking the type definition).

---

5. Testing & Reliability ✅

Excellent test coverage! The test file (replay_test.go) covers:

• ✅ Executor behind scenario (lines 17-94)
• ✅ In-sync scenario (lines 96-122)
• ✅ Executor ahead (error case) (lines 124-148)
• ✅ No HeightProvider implementation (lines 150-171)
• ✅ Genesis height edge case (lines 173-197)
• ✅ Multiple blocks replay (lines 199-283)
• ✅ First block replay (lines 285-331)
• ✅ AppHash mismatch (lines 333-407)

Minor suggestions:

1. Add test for context cancellation - Test that replay respects context cancellation during multi-block sync.

2. Add test for ExecuteTxs failure - Currently tests focus on happy paths and AppHash mismatch. Add a test where ExecuteTxs returns an error to verify error propagation.

3. Test naming convention - Tests use TestReplayer_ prefix, which is good. Consider adding _Success suffix to happy path tests for clarity (e.g., TestReplayer_SyncToHeight_ExecutorBehind_Success).

4. Mock verification (replay_test.go:234-273) - The string conversion string(rune('0'+height)) is clever but fragile. Consider using fmt.Sprintf for clarity:

// Instead of:
AppHash: []byte("app-hash-" + string(rune('0'+height))),

// Use:
AppHash: []byte(fmt.Sprintf("app-hash-%d", height)),

---

6. Documentation & Developer Experience ✅

Positive:

• Excellent godoc comments on Replayer struct and methods
• Validation assumptions clearly documented (replay.go:105-110)
• HeightProvider interface is well-documented (execution.go:89-103)
• Block-manager.md updated to reference the execution replayer

Suggestions:

1. Add package-level documentation - Consider adding a package comment in replay.go explaining the replay mechanism's purpose and when it's used.

2. Expand HeightProvider documentation (execution.go:89-103) - Add examples of when executors should implement this interface:

// HeightProvider is an optional interface that execution clients can implement
// to support height synchronization checks between ev-node and the execution layer.
//
// Implementations should implement this interface if:
// - The execution layer maintains persistent state across restarts
// - The execution layer could potentially be ahead/behind ev-node after crashes
// - The execution layer supports height queries (e.g., via RPC)
//
// Example implementations: EVM execution layer (Geth, Reth), CosmosSDK execution layer
type HeightProvider interface {
    ...
}

3. Document the integration points - Add comments in executor.go:224 and syncer.go:208 explaining why replay is needed at startup:

// Sync execution layer with store on startup to recover from crashes
// where ev-node may be ahead of the execution layer
execReplayer := common.NewReplayer(s.store, s.exec, s.genesis, s.logger)
if err := execReplayer.SyncToHeight(s.ctx, state.LastBlockHeight); err != nil {
    return fmt.Errorf("failed to sync execution layer on startup: %w", err)
}

---

7. Concurrency Safety ✅

Positive:

• Replayer is stateless (only has immutable dependencies)
• No shared mutable state
• Safe for concurrent use from multiple goroutines

Observation:

• The Replayer doesn't protect against concurrent calls to SyncToHeight for the same executor. This is likely fine since it's only called during initialization, but consider documenting this:

// SyncToHeight checks if the execution layer is behind ev-node and syncs it to the target height.
// This is useful for crash recovery scenarios where ev-node is ahead of the execution layer.
//
// Note: This method is not safe for concurrent calls with the same executor instance.
// It should only be called during node initialization.
//
// Returns:
// - error if sync fails or if execution layer is ahead of ev-node (unexpected state)
func (s *Replayer) SyncToHeight(ctx context.Context, targetHeight uint64) error {

---

Maintainer Feedback - Status ✅

All maintainer comments have been addressed:

1. ✅ Renamed from "execution_syncer" to "replayer" (tac0turtle's comment) - File renamed to replay.go
2. ✅ Validation assumptions documented (tac0turtle's comment at replay.go:163) - Added comprehensive comment at lines 105-110
3. ✅ Separate HeightProvider interface (tac0turtle's comment) - Kept as optional to maintain backward compatibility with ABCI
4. ✅ Documentation updated - block-manager.md includes reference to execution replayer

---

Critical Issues: None ✅

Recommendations Priority

High Priority:

1. Add context cancellation check in replay loop (replay.go:88-92)
2. Improve error reporting for partial replay failures

Medium Priority:
3. Add test for context cancellation
4. Add test for ExecuteTxs failure
5. Improve batch logging for large replays
6. Add package-level documentation

Low Priority (Polish):
7. Rename logger component to "replayer" for consistency
8. Expand HeightProvider documentation with examples
9. Document concurrent use safety expectations
10. Consider if block/internal/replay/ package makes sense as code grows

---

Conclusion

This is a solid implementation that solves an important problem (execution layer sync after crashes). The code is clean, well-tested, and follows Go best practices. The optional HeightProvider interface is a good design decision that maintains backward compatibility.

The main areas for improvement are around error handling resilience (context cancellation, partial failure recovery) and some additional test coverage for edge cases. These are not blockers but would increase robustness.

Recommendation: Approve with minor suggestions ✅

Great work on addressing all the maintainer feedback and implementing comprehensive tests!

---

[codecov]

Codecov Report

❌ Patch coverage is 82.53968% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.57%. Comparing base (d64fa9b) to head (efa1bd8).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
block/internal/common/replay.go	86.95%	10 Missing and 5 partials ⚠️
block/internal/executing/executor.go	33.33%	3 Missing and 1 partial ⚠️
block/internal/syncing/syncer.go	40.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2771      +/-   ##
==========================================
+ Coverage   61.26%   61.57%   +0.30%     
==========================================
  Files          81       82       +1     
  Lines        8589     8715     +126     
==========================================
+ Hits         5262     5366     +104     
- Misses       2830     2845      +15     
- Partials      497      504       +7     

Flag	Coverage Δ
combined	61.57% <82.53%> (+0.30%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tac0turtle]

left a few comments, otherwise looks good

[tac0turtle]

claude code has a couple good suggestions as well

[tac0turtle]

looks good, would be nice to call it replayer and then expand slightly on the docs

[github-actions]

PR Preview Action v1.6.2
Preview removed because the pull request was closed.
2025-10-21 16:59 UTC