multi: passive Esplora polling for lwwallet - #330
Conversation
There was a problem hiding this comment.
Code Review
This pull request centralizes chain tip monitoring by introducing a shared TipPoller, which reduces redundant API calls from the wallet's backend components. It also adds LRU caching for transactions, blocks, and headers to the EsploraClient and optimizes notification handling. Feedback identifies several critical areas for improvement: the ChainBackend should avoid full scans when processing new registrations to prevent request bursts, and the TipPoller needs to handle head-of-line blocking and detect reorgs at the same height. Furthermore, caching efficiency can be enhanced by pre-calculating block sizes and implementing singleflight to mitigate cache stampedes.
| // because every tick re-ran checkConfirmations; now that | ||
| // re-checks are gated on tip movement we have to seed the | ||
| // first one explicitly. | ||
| go b.checkConfirmations() |
There was a problem hiding this comment.
Calling checkConfirmations()—which performs a full scan of all pending registrations—inside a goroutine for every new registration is highly inefficient. If multiple registrations are added in quick succession, this will spawn numerous goroutines, each performing redundant network calls for the entire set of registrations. This could lead to a burst of requests that triggers the very rate limits this PR aims to avoid.
go func() {
b.mu.Lock()
currentHeight := b.bestHeight
b.mu.Unlock()
conf := b.checkSingleConf(reg, currentHeight)
if conf == nil {
return
}
select {
case reg.confChan <- conf:
b.mu.Lock()
delete(b.confRegs, id)
b.mu.Unlock()
case <-reg.cancelCh:
}
}()There was a problem hiding this comment.
@gemini-code-assist Already addressed in this PR: RegisterConf now spawns runConfOneShot(id, reg) which is scoped to the single new registration (O(1) HTTP per call), not the full-scan checkConfirmations(). The goroutine is tracked in b.wg so Stop() waits for any in-flight one-shot to drain. See chain_backend.go:447-498.
There was a problem hiding this comment.
Thank you for the clarification. I have reviewed the implementation of runConfOneShot (lines 462-498) and agree that it correctly addresses the efficiency concern by scoping the confirmation check to the individual registration, avoiding the O(N) full-scan overhead.
| // outpoint is already spent at registration time fires | ||
| // synchronously rather than waiting for the next tip event. | ||
| // See RegisterConf for the same rationale. | ||
| go b.checkSpends() |
There was a problem hiding this comment.
Similar to RegisterConf, calling checkSpends() here triggers a full scan of all spend registrations. This should be optimized to check only the newly added registration to avoid redundant network requests and potential rate-limiting issues.
go func() {
if outpoint == nil {
return
}
outspend, err := b.esplora.GetOutspend(
outpoint.Hash, outpoint.Index,
)
if err != nil || !outspend.Spent || !outspend.Status.Confirmed {
return
}
spenderHash, err := chainhash.NewHashFromStr(outspend.Txid)
if err != nil {
return
}
spendingTx, err := b.esplora.GetRawTx(*spenderHash)
if err != nil {
return
}
detail := &chainsource.SpendDetail{
SpentOutPoint: outpoint,
SpenderTxHash: spenderHash,
SpendingTx: spendingTx,
SpenderInputIndex: outspend.Vin,
SpendingHeight: int32(outspend.Status.BlockHeight),
}
select {
case reg.spendChan <- detail:
b.mu.Lock()
delete(b.spendRegs, id)
b.mu.Unlock()
case <-reg.cancelCh:
}
}()There was a problem hiding this comment.
@gemini-code-assist Already addressed: RegisterSpend spawns runSpendOneShot(id, reg) scoped to just the new registration. Same b.wg lifecycle as the conf path. See chain_backend.go:543-597.
There was a problem hiding this comment.
Acknowledged. The implementation of runSpendOneShot correctly addresses the concern by performing an O(1) check for the individual registration, and the use of b.wg ensures proper lifecycle management. No further changes are needed.
|
|
||
| for _, ch := range subs { | ||
| select { | ||
| case ch <- event: |
There was a problem hiding this comment.
Performing a blocking send to subscribers within the main polling loop creates a head-of-line blocking risk. If any subscriber (such as EsploraChainService, which performs synchronous network I/O in its handler) is slow to process an event, the entire TipPoller will stall, preventing other subscribers from receiving updates. While the buffer helps, it may not be sufficient during periods of high activity or network latency.
There was a problem hiding this comment.
@gemini-code-assist Not applicable to the current implementation. The broadcast path goes through lnd's subscribe.Server (wrapped by EventServer in lwwallet/subscribe.go), which provides a per-client unbounded queue plus a single-goroutine subscriber handler. A slow consumer cannot wedge the broadcaster — its updates queue locally — and there is no send-on-closed race by construction. See the doc comment at subscribe.go:14-20.
There was a problem hiding this comment.
Thanks for the clarification. You are correct—using lnd's subscribe.Server with per-client unbounded queues effectively mitigates the head-of-line blocking risk I was concerned about. The design ensures that slow consumers do not impact the broadcaster or other subscribers, which is the right approach for this type of event stream.
| if newHeight <= oldHeight { | ||
| return | ||
| } |
There was a problem hiding this comment.
The current logic only detects tip progress if the height increases. However, a chain reorganization can occur at the same height (a 'side-step'). In such a case, newHeight <= oldHeight would be true, and the poller would fail to notify subscribers of the new block hash. This could lead to the wallet remaining on a stale branch until a new block is mined at a higher height.
There was a problem hiding this comment.
@gemini-code-assist Documented as a known limitation in TipPoller.poll(). The behavior matches the per-component pollers that preceded the unified TipPoller, and downstream callers re-check status against Esplora on every tip event, so a stale hash at height N converges within one extra tip advance. Treating same-height reorgs as a first-class signal would require tracking (height, hash) rather than just height; deliberately deferred for now since it has not historically caused issues for the confirmation-target use case.
There was a problem hiding this comment.
Understood. Thank you for the clarification regarding the design trade-offs and the documentation of this limitation.
| return 0, nil | ||
| } | ||
|
|
||
| return uint64(c.block.SerializeSize()), nil |
There was a problem hiding this comment.
SerializeSize() is an Size() during bookkeeping and eviction, calling this repeatedly for large blocks is inefficient. It is better to pre-calculate the size once when the cachedBlock is created and store it in a field.
There was a problem hiding this comment.
@gemini-code-assist Fixed. cachedBlock and cachedTx now carry a precomputed size field populated from uint64(len(body)) at Put time in GetRawBlock / GetRawTx. Size() is now O(1) on every LRU bookkeeping or eviction call, with no wire.MsgBlock.SerializeSize() walk. See esplora_cache.go and the Put sites in esplora.go.
There was a problem hiding this comment.
The pre-calculation of the size field at the cache-fill site effectively resolves the
| return cached.block, nil | ||
| } | ||
|
|
||
| body, err := c.get("/block/" + blockHash.String() + "/raw") |
There was a problem hiding this comment.
This method is susceptible to a 'cache stampede'. If multiple concurrent goroutines request the same block hash before the first request completes and populates the cache, they will all perform redundant HTTP requests to the Esplora backend. Using golang.org/x/sync/singleflight would ensure that only one in-flight request is made for a given hash.
There was a problem hiding this comment.
@gemini-code-assist Fixed. Added a singleflight.Group to each of the four content-addressed cache fillers (GetRawBlock, GetRawTx, GetBlockHeader, GetRawBlockHeader). Concurrent misses for the same hash now coalesce into a single HTTP round trip; the work function re-checks the cache before issuing the request so a sibling fetch that populated the cache while we were waiting wins without re-firing. This is real load-bearing now that chainBackend and chainSvc both consume TipPoller events and would otherwise race on every block.
There was a problem hiding this comment.
The implementation of singleflight.Group correctly addresses the cache stampede issue by coalescing concurrent requests and re-checking the cache within the work function. This pattern is robust and aligns with best practices for memoization in high-concurrency environments.
|
Claude finished @Roasbeef's task in 2m 16s —— View job PR Review: multi: passive Esplora polling for lwwallet
Overall this is a solid, well-motivated change. The TipPoller unification and LRU caching are clean abstractions; the retry-budget and poll-interval tuning are well-calibrated. A few items below that are worth addressing before merge, ordered by severity. 1.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 08cf5dfba2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if existing, ok := t.subs[id]; ok { | ||
| delete(t.subs, id) | ||
| close(existing) | ||
| } |
There was a problem hiding this comment.
Prevent send-on-closed panic in tip subscriber cancel
This cancel path closes a subscriber channel that the poller may still be about to send to: broadcast snapshots t.subs under the mutex, unlocks, then sends later. If a subscriber cancels during that window (for example during shutdown while a new block is being fanned out), the send can hit a closed channel and panic (send on closed channel), crashing the process. Keep close/send ownership on one side (poller) or synchronize cancellation with broadcast.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
@chatgpt-codex-connector Not applicable to the current implementation. The broadcast path was reworked to use lnd's subscribe.Server via the EventServer wrapper (lwwallet/subscribe.go). Its subscriber handler is single-goroutine over Subscribe / SendUpdate / Cancel, so close-channel ownership stays inside the server and there is no send-on-closed race by construction. The per-subscription translate() goroutine reads from the inner client's Updates() channel (closed by the server, never the subscriber) and writes to the typed out channel that it owns and closes itself. See subscribe.go:14-20 and the Cancel() flow at subscribe.go:110-123.
There was a problem hiding this comment.
To use Codex here, create an environment for this repo.
| if err := w.BtcWallet.Start(); err != nil { | ||
| return fmt.Errorf("start btcwallet: %w", err) | ||
| } |
There was a problem hiding this comment.
Roll back tip poller when wallet startup fails
If BtcWallet.Start() returns an error after tipPoller.Start() succeeds, this function exits without stopping the poller. That leaves the poller goroutine running and keeps tipPoller.started set, so retrying Wallet.Start() on the same instance fails with tip poller already started and leaks startup resources on transient failures. Add cleanup before returning this error.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
@chatgpt-codex-connector Already addressed in this PR. Wallet.Start() builds a rollback []func() slice and pushes w.tipPoller.Stop (and _ = w.BtcWallet.Stop()) after each successful sub-system Start. Any later failure unwinds the slice in reverse via the deferred closure, so a BtcWallet.Start() failure after tipPoller.Start() succeeds tears the poller down before the function returns. The slice is set to nil only on the all-green happy path. See wallet.go:166-202.
There was a problem hiding this comment.
To use Codex here, create an environment for this repo.
08cf5df to
1be5a22
Compare
|
Claude finished @Roasbeef's task in 4m 1s —— View job Claude finished @Roasbeef's task in ~5m — PR Review: multi: passive Esplora polling for lwwallet
Good layering across four atomic commits. The architecture is clean and the goals are well-served. One clarification up front on the prior reviews: the previous Claude review's HIGH item (O(N²) full-scan goroutines in 1.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1be5a223a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if err := s.chainBackend.Start(); err != nil { | ||
| return fmt.Errorf( | ||
| "start chain backend: %w", err, | ||
| ) |
There was a problem hiding this comment.
Remove duplicate start of lwwallet chain backend
startLwwallet now calls w.Start() (which already starts w.ChainBackend() inside lwwallet.Wallet.Start) and then, when s.chainBackend == nil, calls s.chainBackend.Start() again. That second start creates another active tip subscription/goroutine for the same backend, which can duplicate block epoch notifications and confirmation/spend rechecks and effectively reintroduce extra Esplora traffic. In lwwallet mode this path is taken on initial startup, so the backend should only be assigned here, not started again.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
@chatgpt-codex-connector Fixed. Removed the second s.chainBackend.Start() call in startLwwallet. Wallet.Start() already starts the chain backend internally (lwwallet/wallet.go:191) as part of its startup sequence, so the deferred-init block in startLwwallet now only assigns s.chainBackend = w.ChainBackend() for the chain source actor registration — no second subscribe + duplicate handleTipEvents goroutine.
There was a problem hiding this comment.
To use Codex here, create an environment for this repo.
1be5a22 to
8b4a7ed
Compare
handleBlockEpoch retried ListUnspent up to 10x at 200ms intervals whenever no new boarding UTXO surfaced under a fresh block, burning ~3.5s of HTTP traffic per epoch even when nothing was racing the notification. Against a public Esplora endpoint this was enough to trip mempool.space's HTTP 429 rate limit. Cap the retry budget at 3 attempts and widen the inter-attempt delay to 1s. This still gives a slow neutrino-fed btcwallet a generous window to credit-mark a freshly-confirmed boarding UTXO, while shrinking the per-epoch worst-case from ~30 to ~6 backend round trips on the lwwallet path.
The 5s default was tuned for regtest where blocks land on demand, but it bleeds onto mainnet users where the average inter-block gap is ~600s. At 5s the chain backend issued ~12 GetTipHeight calls per minute against the configured Esplora endpoint -- enough to contribute to mempool.space rate-limit responses when paired with the boarding UTXO retry burst. Bump the default to 30s. Block-detection latency stays an order of magnitude under the average mainnet block interval, while the steady-state request rate drops by 6x. Regtest-driven harnesses that mine blocks on demand should override this knob via the existing wallet.pollinterval config flag.
8b4a7ed to
e27aae8
Compare
Two pollers (the btcwallet chain.Interface adapter and the chainsource.ChainBackend) drive the Esplora client independently, each fetching block hashes, headers, and raw blocks on every new tip they detect. Their per-call work is identical, so under the old wiring every confirmed block was fetched twice, every header was decoded twice, and every TxProof construction triggered a fresh GetRawTx round-trip even when the same boarding txid had been served seconds earlier. Add four neutrino LRU caches keyed by chainhash.Hash for the hash-addressed methods: GetBlockHeader, GetRawBlockHeader, GetRawBlock, and GetRawTx. Capacities are byte-sized via each value's serialized footprint -- 50 MiB for transactions, 200 MiB for raw blocks, and 1 MiB each for the small fixed-size header types. Reorgs cannot stale these entries: each is content- addressed by a hash that uniquely identifies its body, so a stale hit is only possible if the backend lied about the hash in the first place. Mutable endpoints (GetTipHeight, GetAddressUtxos, GetTxStatus, GetFeeEstimates, GetOutspend) deliberately stay uncached; they report live state and a stale answer would silently mask a reorg or a pending spend.
Two independent goroutines used to drive Esplora tip detection: the EsploraChainService (feeding btcwallet's chain.Interface) and the chainsource.ChainBackend (feeding the actor system) each ran their own ticker against the same client. At the prior 5 s default that produced ~24 GetTipHeight calls per minute against a single endpoint, plus duplicated GetBlockHashByHeight + GetBlockHeader work for every block both saw. Introduce TipPoller as the single Esplora tip-detection authority for the lwwallet. It owns one goroutine, walks oldHeight+1 → newHeight on each detected advance, and fans the canonical TipBlock event (height + hash + JSON header) out to subscribers. EsploraChainService and ChainBackend both become passive consumers: each subscribes at Start time and reacts to the events without issuing its own tip queries. The wallet drives lifecycle (Start the poller before either subscriber, Stop it after both have torn down). For the standalone darepod path that constructs a ChainBackend without a wallet, NewChainBackend keeps its existing signature and creates an internal TipPoller it owns end-to-end; NewChainBackendWithPoller is the new shared-poller entry point used by the wallet. Confirmation and spend registrations now schedule a one-shot check at registration time so a tx that is already buried beyond numConfs (or an outpoint that is already spent) fires synchronously, rather than waiting for the next tip event. The prior polling design caught this implicitly via the unconditional checkConfirmations / checkSpends call on every tick.
e27aae8 to
5dba835
Compare
In this PR, we cut the lwwallet's Esplora request rate by an order of
magnitude so it stops getting bounced by mempool.space's HTTP 429 rate
limit on mainnet. The trigger was a single client running off-the-shelf
defaults: with one boarding address and a confirmed UTXO already in
view, every block epoch produced a ~3.5s burst of
ListUnspentretries, on top of two independent goroutines (one for btcwallet, one
for the actor system) each polling
GetTipHeightevery 5s.Four atomic commits, each addressing one source of waste. See each
commit message for the per-change rationale w.r.t the incremental
changes; the rest of this body covers the combined picture.
ListUnspent retry budget
The retry on
wallet.handleBlockEpochwas tuned for neutrino'sblock-processing lag, where btcwallet has to fetch the full block over
P2P and run
AddCreditbefore a freshly confirmed UTXO becomesvisible. We keep that intent (slow backends still get time to catch
up) but move from 10x200ms to 3x1s, which is plenty for any backend
we actually ship and shrinks the per-epoch worst case from ~30 to ~6
backend round trips on the lwwallet path.
Passive default poll interval
The default Esplora poll interval was 5s, which is fine for regtest
where blocks land on demand but bleeds onto mainnet users where the
average inter-block gap is ~600s. We bump the default to 30s, an
order of magnitude under the typical mainnet block interval, and
leave the existing
wallet.pollintervalflag for harnesses that mineblocks themselves.
TipPoller unification
Before this PR the lwwallet ran two parallel tip pollers against the
same Esplora client:
EsploraChainService(thechain.Interfaceadapter btcwallet uses for sync) and
chainsource.ChainBackend(theevent source for the actor system). Each had its own ticker, its own
GetTipHeight, and its own per-blockGetBlockHashByHeight+GetBlockHeaderon every detected advance. Two consumers, identicalwork, doubled traffic.
We introduce
lwwallet.TipPolleras the single tip-detectionauthority. One goroutine, one ticker. On each advance it walks
oldHeight+1 -> newHeight, resolves each block's hash and headeronce, and fans a
TipBlockevent out to every active subscriber.Both
EsploraChainServiceandChainBackendbecome passivesubscribers; neither runs its own ticker anymore.
The wallet itself owns the poller's lifecycle: Start it before
either subscriber registers, Stop it after both have torn down. The
standalone darepod path (where a
ChainBackendis constructedwithout a wallet because auto-unlock hasn't run yet) still works
through the existing
NewChainBackendconstructor, which nowinternally creates and owns its own TipPoller. The new
NewChainBackendWithPolleris the shared-poller entry point thewallet uses.
One behavioral wrinkle: the prior polling loop ran
checkConfirmationsandcheckSpendson every tick whether or notthe tip moved, so a confirmation registered against a tx that was
already buried beyond
numConfswould fire on the next tick. Thenew design only re-checks on tip movement, so we explicitly trigger
a one-shot check at registration time. Same observable behavior, no
busy-polling.
LRU caching for hash-addressed responses
The two old pollers (and any other consumer that fetches a tx or
block by hash) all hit Esplora independently. We add four neutrino
LRU caches in
EsploraClientkeyed bychainhash.Hashfor thehash-addressed methods:
GetBlockHeader,GetRawBlockHeader,GetRawBlock, andGetRawTx. Capacities are byte-sized via eachvalue's serialized footprint, so a 4 MiB block costs more cache
budget than an 80-byte header. Reorgs cannot stale these entries
because each is content-addressed by the same hash that uniquely
identifies its body; a stale hit is only possible if the backend
lied about the hash in the first place.
Mutable endpoints (
GetTipHeight,GetAddressUtxos,GetTxStatus,GetFeeEstimates,GetOutspend) deliberately stay uncached. Astale tip would mask a reorg, a stale UTXO list would mask a pending
spend, etc, etc.
Combined effect
For a single client running against mempool.space:
GetTipHeightcadence: 24/min -> 2/min.GetBlockHashByHeight,GetBlockHeader,GetRawBlock,GetRawTx): served from the LRU on the secondconsumer rather than re-fetched.
Test plan
lwwallet,wallet,darepod.go vetclean on the same packages.no 429 responses across a multi-hour window.
(they override
wallet.pollinterval).