feat: DA hints in p2p

[alpe]

Overview

Resolves #2609

The basic idea is to store an additional DAHightHint field within the p2p store.

As SignedHeader and Data are used in other places, too. I added an an DAHeightHintContainer type to wrap the tuple for the store only.
The DAHeight Hint is added by the da_submitter and read in the syncer to fetch the missing DA header/data for the most recent block as required.

Please note: this is a breaking change to the p2p network and store

[claude]

Claude finished @alpe's task in 4m 43s —— View job

---

Comprehensive Code Review: DA Hints in P2P

I've completed a thorough review of the DA hints implementation. Overall, this is a well-designed feature with strong architectural decisions, particularly the binary compatibility approach. Below are my detailed findings organized by focus area.

---

1. Code Quality & Architecture ⭐

Strengths

• Excellent binary compatibility design: The P2PSignedHeader and P2PData types with optional protobuf fields elegantly solve the migration problem
• Clean separation of concerns: DA hints are isolated to p2p layer while preserving existing types.SignedHeader and types.Data API
• Generic interface design: EntityWithDAHint[V] interface in sync_service.go:39-43 is well-abstracted
• Async DA retrieval: The worker pool pattern in AsyncDARetriever is a solid performance optimization

Issues & Recommendations

🔴 CRITICAL: Unused DAHeightHintContainer type

Location: pkg/sync/da_hint_container.go

This entire file (65 lines) appears to be dead code:

• Not referenced anywhere in the codebase except comments
• The PR initially introduced this as a wrapper type but then switched to P2PSignedHeader/P2PData
• The MarshalBinary implementation silently drops the DA hint (lines 58-60), which would be a serious bug if used

Recommendation: Remove this file entirely in a follow-up commit to avoid confusion.

🟡 Interface over-delegation in da_hint_container.go

If this type were to be used, it delegates every header.Header[H] method to the wrapped entry. This is boilerplate-heavy. Consider embedding instead:

type DAHeightHintContainer[H header.Header[H]] struct {
    H
    DAHeightHint uint64
}

🟡 Inconsistent naming: DaHeight vs DAHeight

• Protobuf uses da_height_hint (snake_case)
• Go uses DAHeightHint (correct per Go conventions)
• Code has mixed usage: DaHeightHints (syncer.go:513) vs DAHeightHint elsewhere

Recommendation: Ensure consistent casing in Go code (use DAHeightHint everywhere).

---

2. Error Handling & Resilience ⭐⭐

Strengths

• AsyncDARetriever gracefully handles failures and logs them (async_da_retriever.go:99-101)
• P2PHandler validates proposer address and data hash consistency (p2p_handler.go:90-108)
• Context cancellation properly handled throughout

Issues & Recommendations

🔴 Silent error swallowing in DA hint append

Location: da_submitter.go:201-204, da_submitter.go:250-253

if err := s.headerDAHintAppender.AppendDAHint(ctx, res.Height, hashes...); err != nil {
    s.logger.Error().Err(err).Msg("failed to append da height hint in header p2p store")
    // ignoring error here, since we don't want to block the block submission'
}

Issues:

1. Typo in comment: "block the block submission**'**" (trailing apostrophe)
2. This silently creates inconsistent state where DA submission succeeds but p2p store isn't updated
3. No metrics recorded for this failure

Recommendation:

if err := s.headerDAHintAppender.AppendDAHint(ctx, res.Height, hashes...); err != nil {
    s.logger.Error().Err(err).Msg("failed to append DA height hint to header p2p store")
    s.recordFailure(common.DAHintAppendFailure) // Add new metric
    // Continue - don't block submission, but track this degraded state
}

🟡 Missing context deadline in AsyncDARetriever

Location: async_da_retriever.go:98

The RetrieveFromDA call uses the global context without timeout. If DA retrieval hangs, workers are blocked indefinitely.

Recommendation:

func (r *AsyncDARetriever) processRetrieval(height uint64) {
    defer func() {
        r.mu.Lock()
        delete(r.inFlight, height)
        r.mu.Unlock()
    }()

    // Add timeout for DA retrieval
    ctx, cancel := context.WithTimeout(r.ctx, 30*time.Second)
    defer cancel()

    events, err := r.retriever.RetrieveFromDA(ctx, height)
    if err != nil {
        r.logger.Debug().Err(err).Uint64("height", height).Msg("async DA retrieval failed")
        return
    }
    // ...
}

🟡 No error handling for dropped requests

Location: async_da_retriever.go:73-75

When the work channel is full, requests are silently dropped with only a debug log. This could lead to data loss.

Recommendation: Add metrics and consider a buffered pending queue or backpressure mechanism.

---

3. Security 🔒

Strengths

• Proposer address validation prevents unauthorized blocks (p2p_handler.go:132-137)
• Data hash validation prevents data/header inconsistency (p2p_handler.go:104-108)
• Signature validation deferred to proper execution context (p2p_handler.go:110-111)

Issues & Recommendations

🟢 No critical security issues found

Minor observations:

• DA hints are untrusted data from p2p peers. The code correctly treats them as hints, not authoritative (good!)
• No hint validation (e.g., DA height should be ≥ block height), but this is acceptable since hints are advisory

---

4. Performance & Resource Efficiency ⚡

Strengths

• Worker pool design: 5 concurrent workers (async_da_retriever.go:43) is reasonable
• Deduplication: In-flight map prevents redundant DA fetches (async_da_retriever.go:65-67)
• Non-blocking hint requests: RequestRetrieval is non-blocking (async_da_retriever.go:59-76)
• Buffered channels: Work channel has 100 buffer (async_da_retriever.go:33)

Issues & Recommendations

🟡 Worker count hardcoded

Location: async_da_retriever.go:43

for i := 0; i < 5; i++ {

Recommendation: Make this configurable:

const defaultWorkerCount = 5

func NewAsyncDARetriever(
    retriever DARetriever,
    resultCh chan<- common.DAHeightEvent,
    logger zerolog.Logger,
    workerCount int, // Add parameter
) *AsyncDARetriever {
    if workerCount <= 0 {
        workerCount = defaultWorkerCount
    }
    // ...
}

🟡 Unbounded in-flight map

Location: async_da_retriever.go:16

The inFlight map could grow unbounded if DA retrieval consistently fails. Consider adding a maximum size or TTL.

🟡 Potential lock contention

Location: async_da_retriever.go:62-63

Every RequestRetrieval call acquires a mutex. Under heavy load, this could become a bottleneck.

Recommendation: Consider using atomic operations or a lock-free design for the common path.

---

5. Testing & Reliability ✅

Test Coverage Analysis

From codecov report: 60.53% patch coverage with 103 lines missing coverage.

Well-Tested Components

• ✅ async_da_retriever_test.go: Excellent tests for deduplication and worker pool limits
• ✅ binary_compatibility_test.go: Critical tests ensuring migration safety
• ✅ Syncer tests updated to handle DA hints

Missing Test Coverage

🔴 da_hint_container.go: 0% coverage

All 27 new lines are untested. However, since this file should be deleted (see above), this is moot.

🟡 sync_service.go:184-198 - AppendDAHint method: Low coverage

This critical method has 11 missing lines. Need tests for:

• Hash not found scenario
• Partial updates (some hashes exist, others don't)
• Error propagation from store.Append

🟡 p2p_signed_header.go & p2p_data.go: 47% coverage

Missing test cases:

• Nil pointer handling in FromProto (lines 69, 64)
• Error cases in unmarshaling
• Round-trip with DA hints set

Recommendation: Add test:

func TestP2PSignedHeaderRoundTrip(t *testing.T) {
    header, err := types.GetRandomSignedHeader("chain-id")
    require.NoError(t, err)
    
    p2pHeader := &types.P2PSignedHeader{
        SignedHeader: *header,
        DAHeightHint: 12345,
    }
    
    bytes, err := p2pHeader.MarshalBinary()
    require.NoError(t, err)
    
    var decoded types.P2PSignedHeader
    err = decoded.UnmarshalBinary(bytes)
    require.NoError(t, err)
    
    assert.Equal(t, p2pHeader.DAHeightHint, decoded.DAHeightHint)
    assert.Equal(t, p2pHeader.SignedHeader, decoded.SignedHeader)
}

---

6. Concurrency & Goroutine Safety 🔄

Strengths

• ✅ Proper WaitGroup usage in AsyncDARetriever (async_da_retriever.go:19, 55)
• ✅ Context cancellation propagation (async_da_retriever.go:51-54)
• ✅ Mutex protection for shared state (async_da_retriever.go:62-63, 92-95)
• ✅ Channel-based coordination (async_da_retriever.go:104-110)

Issues & Recommendations

🟡 Potential goroutine leak on context cancel

Location: async_da_retriever.go:104-110

for _, event := range events {
    select {
    case r.resultCh <- event:
    case <-r.ctx.Done():
        return
    }
}

If resultCh consumer is slow and context is canceled mid-loop, some events are dropped silently.

Assessment: This is acceptable for graceful shutdown, but should be documented.

🟢 P2PHandler processedHeight: Excellent CAS loop

Location: p2p_handler.go:63-72

The CompareAndSwap loop with bounded retries (1000) is well-designed. Minor suggestion: log if loop exhausts iterations (indicates severe contention).

---

7. Documentation & Developer Experience 📚

Strengths

• ✅ Clear comments explaining binary compatibility (types/p2p_signed_header.go)
• ✅ Good inline documentation in p2p_handler.go
• ✅ Comprehensive review comments from author

Issues & Recommendations

🟡 Missing godoc for exported types

Location: types/p2p_signed_header.go:14, types/p2p_data.go:14

// P2PSignedHeader wraps SignedHeader with an optional DA height hint for p2p gossip.
// The hint indicates which DA layer height contains this block's header,
// enabling targeted retrieval instead of sequential scanning.
// This type is binary-compatible with SignedHeader via optional protobuf fields.
type P2PSignedHeader struct {
    SignedHeader
    DAHeightHint uint64
}

🟡 TODO comment should be removed

Location: syncer.go:584

if event.Source == common.SourceDA { // TODO(@julienrbrt): To be reverted once DA Hints are merged (https://github.com/evstack/ev-node/pull/2891)

This PR IS the DA hints merge. This TODO is now stale. The logic should either:

1. Be removed if DA hints fully replace this pattern
2. Have the TODO updated to clarify what needs reverting

Recommendation: Clarify the migration plan in the comment or remove if obsolete.

---

Migration Strategy & Breaking Changes 🚧

Binary Compatibility: EXCELLENT ✅

The use of optional protobuf fields is the right approach:

message P2PSignedHeader {
  Header header = 1;
  bytes signature = 2;
  Signer signer = 3;
  optional uint64 da_height_hint = 4;  // ← Optional field
}

Why this works:

• Old nodes: Unmarshal P2PSignedHeader → ignore unknown field 4 → still get SignedHeader data ✅
• New nodes: Can read old messages (field 4 absent → defaults to 0) ✅
• binary_compatibility_test.go validates this bidirectional compatibility ✅

Store Migration

🟡 No migration path for existing p2p stores

Location: See PR description comment from @alpe (Dec 5)

"This PR is missing a migration strategy for the p2p store (and ideally network)"

Current status: Binary compatibility solves network compatibility, but existing stored data:

• Old stores can be read by new code (DA hint = 0)
• New nodes writing to old stores works fine

Recommendation: Document that no migration is needed (stores are forward/backward compatible), but DA hints will only be populated for new blocks.

---

Specific Code Issues

syncer.go:510-548 - DA Hint Processing Logic

Good: The logic correctly handles 3 cases:

1. Both hints present
2. Only header hint present
3. Only data hint present

Issue: Complex nested conditionals make this hard to follow.

Recommendation: Extract to helper method:

func (s *Syncer) extractDAHintsToRetrieve(event *common.DAHeightEvent) []uint64 {
    if event.DaHeightHints == [2]uint64{0, 0} {
        return nil
    }
    
    var hints []uint64
    
    if event.DaHeightHints[0] != 0 {
        if _, exists := s.cache.GetHeaderDAIncluded(event.Header.Hash().String()); !exists {
            hints = append(hints, event.DaHeightHints[0])
        }
    }
    
    if event.DaHeightHints[1] != 0 {
        if _, exists := s.cache.GetDataDAIncluded(event.Data.Hash().String()); !exists {
            // Deduplicate if both hints are same
            if len(hints) == 0 || hints[0] != event.DaHeightHints[1] {
                hints = append(hints, event.DaHeightHints[1])
            }
        }
    }
    
    return hints
}

---

Summary & Recommendations

Overall Assessment: ⭐⭐⭐⭐ (4/5)

This is a high-quality implementation of a non-trivial feature. The binary compatibility strategy is exemplary.

Must Address Before Merge

1. 🔴 Remove pkg/sync/da_hint_container.go - dead code
2. 🔴 Fix error handling in DA hint append (da_submitter.go:201-204)
3. 🔴 Update/remove stale TODO (syncer.go:584)

Strongly Recommended

4. 🟡 Add timeout to AsyncDARetriever DA calls
5. 🟡 Improve test coverage for AppendDAHint and P2P types
6. 🟡 Add metrics for dropped DA retrieval requests
7. 🟡 Document migration strategy in PR description or CHANGELOG

Nice to Have

8. Make worker count configurable
9. Add godoc to exported P2P types
10. Refactor DA hint extraction logic in syncer.go

---

Test Plan Verification

The PR adds significant test coverage:

• ✅ Binary compatibility tests
• ✅ AsyncDARetriever worker pool tests
• ✅ Integration tests updated

Missing: E2E test demonstrating full flow:

1. Block submitted to DA
2. DA hint propagated via p2p
3. Syncing node uses hint for targeted retrieval

---

Excellent work on a complex feature! The binary compatibility approach shows strong architectural thinking. Address the critical issues and this will be ready to merge.

[github-actions]

The latest Buf updates on your PR. Results from workflow CI / buf-check (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	⏩ skipped	✅ passed	✅ passed	Dec 16, 2025, 9:28 AM

[codecov]

Codecov Report

❌ Patch coverage is 60.53640% with 103 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.25%. Comparing base (95140e4) to head (f0a505f).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/sync/da_hint_container.go	0.00%	27 Missing ⚠️
types/p2p_signed_header.go	47.61%	16 Missing and 6 partials ⚠️
types/p2p_data.go	47.36%	15 Missing and 5 partials ⚠️
block/internal/syncing/syncer.go	58.62%	12 Missing ⚠️
pkg/sync/sync_service.go	69.44%	7 Missing and 4 partials ⚠️
block/internal/syncing/async_da_retriever.go	86.53%	6 Missing and 1 partial ⚠️
block/internal/submitting/da_submitter.go	80.95%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2891      +/-   ##
==========================================
+ Coverage   60.14%   60.25%   +0.11%     
==========================================
  Files          88       92       +4     
  Lines        8427     8647     +220     
==========================================
+ Hits         5068     5210     +142     
- Misses       2787     2851      +64     
- Partials      572      586      +14     

Flag	Coverage Δ
combined	60.25% <60.53%> (+0.11%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[alpe]

This is where the DA height is passed to the sync service to update the p2p store

[alpe]

This is where the "fetch from DA" is triggered for the current block event height

[alpe]

This is a data container to persist the DA hint together with the block header or data.
types.SignedHeader and types.Data are used all over the place so I did not modify them but added introduced this type for the p2p store and transfer only.

It may make sense to do make this a Proto type. WDYT?

[alpe]

Stores the DA height hints

[tac0turtle]

if da hint is not in the proto how do other nodes get knowledge of the hint?

also how would an existing network handle using this feature? its breaking so is it safe to upgrade?

[julienrbrt]

nit: gci linter

[julienrbrt]

Nice! It really makes sense.

I share the same concern as @tac0turtle however about the upgrade strategy given it is p2p breaking.

[alpe]

if da hint is not in the proto how do other nodes get knowledge of the hint?

The sync_service wraps the header/data payload in a DAHeightHintContainer object that is passed upstream to the p2p layer. When the DA height is known, the store is updated.

also how would an existing network handle using this feature? its breaking so is it safe to upgrade?

It is a breaking change. Instead of signed header or data types, the p2p network exchanges DAHeightHintContainer. This would be incompatible. Also the existing p2p stores would need migration to work.

[julienrbrt]

Could we broadcast both until every networks are updated? Then for final we can basically discard the previous one.

[alpe]

fyi: This PR is missing a migration strategy for the p2p store ( and ideally network)

[alpe]

I have added 2 new types for the p2p store that are binary compatible to the types.Data and SignedHeader. With this, we should be able to roll this out without breaking the in-flight p2p data and store.

[julienrbrt]

lgtm! I can see how useful the async retriever will be for force inclusion verification as well. We should have @auricom verify if p2p still works with Eden.

[julienrbrt]

This is going to be really useful for force inclusion checks as well.