-
Notifications
You must be signed in to change notification settings - Fork 955
Only mark block lookups as pending if block is importing from gossip #8112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Only mark block lookups as pending if block is importing from gossip #8112
Conversation
eserilev
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Allowing double imports seems fine to me, especially if these are rare. Wondering if there's a metric we could add somewhere to keep track of instances where a block has been imported twice? Can't find any issues with this PR, but another set of eyes on this would be nice.
| | BlockImportSource::HttpApi => { | ||
| // Lookup, RangeSync or HttpApi block import don't emit the GossipBlockProcessResult | ||
| // event. If a lookup happens to be created during block import from one of | ||
| // those sources just import the block twice. Otherwise the lookup will get |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a comment on the doc here:
- I think re-imports wouldn't happen to lookup responses because of
duplicate_cachehere
let Some(handle) = duplicate_cache.check_and_insert(block_root) else { - for range sync its unlikely to race with lookups
- http api block imports are much less frequent, so the impact of re-imports is much smaller
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also the impact of duplicate imports is not as bad since michael made the fix in #8050
As described in the PR - longer term we'll have @ethDreamer's block status tracker to prevent block re-imports properly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
longer term we'll have @ethDreamer's block status tracker to prevent block re-imports properly.
I noticed in this PR that we'll need to re-engineer sync a bit to for that to work. But working with Mark about it
jimmygchen
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Niice, thanks for fixing this!
The fix looks good to me, be great to add some regression test for this as well.
|
Merging this now since the fix is simple and makes sense. Great work finding the issue! 🕵️ |
…igp#8112) - PR sigp#8045 introduced a regression of how lookup sync interacts with the da_checker. Now in unstable block import from the HTTP API also insert the block in the da_checker while the block is being execution verified. If lookup sync finds the block in the da_checker in `NotValidated` state it expects a `GossipBlockProcessResult` message sometime later. That message is only sent after block import in gossip. I confirmed in our node's logs for 4/4 cases of stuck lookups are caused by this sequence of events: - Receive block through API, insert into da_checker in fn process_block in put_pre_execution_block - Create lookup and leave in AwaitingDownload(block in processing cache) state - Block from HTTP API finishes importing - Lookup is left stuck Closes sigp#8104 - sigp#8110 was my initial solution attempt but we can't send the `GossipBlockProcessResult` event from the `http_api` crate without adding new channels, which seems messy. For a given node it's rare that a lookup is created at the same time that a block is being published. This PR solves sigp#8104 by allowing lookup sync to import the block twice in that case. Co-Authored-By: dapplion <[email protected]>
commit 9b2de09 Author: Pawan Dhananjay <[email protected]> Date: Mon Oct 6 17:22:44 2025 -0700 Rethink peer scoring commit 5c562c6 Author: Pawan Dhananjay <[email protected]> Date: Mon Oct 6 11:17:57 2025 -0700 Fix some more issues commit 15df3d2 Merge: c491856 26575c5 Author: Pawan Dhananjay <[email protected]> Date: Thu Oct 2 13:15:59 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit c491856 Merge: 826a06e ffa7b2b Author: Pawan Dhananjay <[email protected]> Date: Thu Sep 25 16:23:27 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit 826a06e Author: Pawan Dhananjay <[email protected]> Date: Thu Sep 25 16:21:55 2025 -0700 Fix variant name commit 421e954 Author: Pawan Dhananjay <[email protected]> Date: Thu Sep 25 16:05:45 2025 -0700 Revert "Revert type change in UnexpectedRequestId" This reverts commit 6ea14016f3d164456bc4c3cae0355ab532fe1a86. commit 3f8998f Author: Lion - dapplion <[email protected]> Date: Thu Sep 25 05:52:27 2025 +0200 Only mark block lookups as pending if block is importing from gossip (sigp#8112) - PR sigp#8045 introduced a regression of how lookup sync interacts with the da_checker. Now in unstable block import from the HTTP API also insert the block in the da_checker while the block is being execution verified. If lookup sync finds the block in the da_checker in `NotValidated` state it expects a `GossipBlockProcessResult` message sometime later. That message is only sent after block import in gossip. I confirmed in our node's logs for 4/4 cases of stuck lookups are caused by this sequence of events: - Receive block through API, insert into da_checker in fn process_block in put_pre_execution_block - Create lookup and leave in AwaitingDownload(block in processing cache) state - Block from HTTP API finishes importing - Lookup is left stuck Closes sigp#8104 - sigp#8110 was my initial solution attempt but we can't send the `GossipBlockProcessResult` event from the `http_api` crate without adding new channels, which seems messy. For a given node it's rare that a lookup is created at the same time that a block is being published. This PR solves sigp#8104 by allowing lookup sync to import the block twice in that case. Co-Authored-By: dapplion <[email protected]> commit d99df0a Author: Jimmy Chen <[email protected]> Date: Thu Sep 25 12:52:07 2025 +1000 Only send data coumn subnet discovery requests after peerdas is scheduled (sigp#8109) sigp#8105 (to be confirmed) I noticed a large number of failed discovery requests after deploying latest `unstable` to some of our testnet and mainnet nodes. This is because of a recent PeerDAS change to attempt to maintain sufficient peers across data column subnets - this shouldn't be enabled on network without peerdas scheduled, otherwise it will keep retrying discovery on these subnets and never succeed. Also removed some unused files. Co-Authored-By: Jimmy Chen <[email protected]> Co-Authored-By: Jimmy Chen <[email protected]> commit cf46d10 Author: Pawan Dhananjay <[email protected]> Date: Thu Sep 25 15:54:32 2025 -0700 Fix issues from review commit c2aa4ae Author: dapplion <[email protected]> Date: Thu Sep 25 23:58:31 2025 +0200 Revert type change in UnexpectedRequestId commit 7488755 Author: dapplion <[email protected]> Date: Thu Sep 25 22:09:06 2025 +0200 De-duplicate data columns by root request type commit 7650032 Author: dapplion <[email protected]> Date: Thu Sep 25 23:52:47 2025 +0200 Rename DataColumnsFromRange commit 4b0b655 Author: Pawan Dhananjay <[email protected]> Date: Wed Sep 24 12:31:06 2025 -0700 Fix nits commit aa6a1bc Author: Pawan Dhananjay <[email protected]> Date: Wed Sep 24 12:16:05 2025 -0700 Create a custom penalize_sync_peer method for clarity commit 27195ca Merge: 4a59d35 af27402 Author: Pawan Dhananjay <[email protected]> Date: Wed Sep 24 11:16:27 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit 4a59d35 Merge: 2f35c36 5928407 Author: Pawan Dhananjay <[email protected]> Date: Wed Sep 17 15:15:44 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit 2f35c36 Author: Pawan Dhananjay <[email protected]> Date: Wed Sep 17 15:13:40 2025 -0700 Add some metrics commit b3b3756 Author: Pawan Dhananjay <[email protected]> Date: Wed Sep 17 15:01:46 2025 -0700 Fix tests commit e3aed89 Author: Pawan Dhananjay <[email protected]> Date: Tue Sep 16 17:46:21 2025 -0700 Remove retry test that we do not use anymore commit baee27a Merge: 9db4c30 191570e Author: Pawan Dhananjay <[email protected]> Date: Tue Sep 16 16:39:30 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit 9db4c30 Author: Pawan Dhananjay <[email protected]> Date: Tue Sep 16 16:38:19 2025 -0700 Fix small bug commit 08bba3f Author: Pawan Dhananjay <[email protected]> Date: Thu Sep 11 18:24:25 2025 -0700 fmt commit cffbd34 Author: Pawan Dhananjay <[email protected]> Date: Thu Sep 11 18:19:16 2025 -0700 Reduce code duplication commit bf09d57 Author: Pawan Dhananjay <[email protected]> Date: Thu Sep 11 15:06:07 2025 -0700 Fix some issues from lion's review commit 04398ad Author: Pawan Dhananjay <[email protected]> Date: Mon Sep 8 15:37:26 2025 -0700 Fix some more issues from review commit 4f62a9c Merge: e259ecd 8ec2640 Author: Pawan Dhananjay <[email protected]> Date: Fri Sep 5 13:00:47 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit e259ecd Author: Pawan Dhananjay <[email protected]> Date: Fri Sep 5 12:58:49 2025 -0700 More renamings commit 6a2a33d Author: Pawan Dhananjay <[email protected]> Date: Fri Sep 5 12:48:05 2025 -0700 Fix some issues from review commit e0d8f04 Author: Pawan Dhananjay <[email protected]> Date: Thu Sep 4 18:09:07 2025 -0700 Tests compile commit 7e91eeb Merge: 29c2f83 9d2f55a Author: Pawan Dhananjay <[email protected]> Date: Thu Sep 4 18:08:52 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit 29c2f83 Author: Pawan Dhananjay <[email protected]> Date: Tue Sep 2 13:29:19 2025 -0700 Cleanup SyncingChain commit 8458df6 Author: Pawan Dhananjay <[email protected]> Date: Mon Sep 1 12:53:01 2025 -0700 Attempt sending awaitingDownload batches when restarting sync commit 7a6d0d9 Author: Pawan Dhananjay <[email protected]> Date: Sun Aug 31 19:07:39 2025 -0700 Force processing_target request commit 4f60e86 Author: Pawan Dhananjay <[email protected]> Date: Sun Aug 31 14:54:44 2025 -0700 Add logs to debug stuck range sync commit b07bc6d Author: Pawan Dhananjay <[email protected]> Date: Fri Aug 29 16:18:53 2025 -0700 Force requests if batch buffer is full under certain conditions commit 19b0a5c Merge: 8e1337d 438fb65 Author: Pawan Dhananjay <[email protected]> Date: Thu Aug 28 23:25:14 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit 8e1337d Merge: da1aaba a134d43 Author: Pawan Dhananjay <[email protected]> Date: Thu Aug 28 18:03:44 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit da1aaba Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 27 17:34:27 2025 -0700 Revise error tolerance commit 7331323 Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 27 17:30:46 2025 -0700 AwaitingValidation state only needs block peer commit 8386bd9 Merge: b4bc7fe d235f2c Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 27 17:08:42 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit b4bc7fe Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 27 14:30:46 2025 -0700 Remove more debug logs commit 05adb71 Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 27 14:26:28 2025 -0700 Fix bug with partial column responses before all column requests sent commit a97cf88 Author: Pawan Dhananjay <[email protected]> Date: Tue Aug 26 18:53:51 2025 -0700 Add docs commit 27d0b36 Author: Pawan Dhananjay <[email protected]> Date: Tue Aug 26 16:06:27 2025 -0700 Remove debug statements commit 90d319f Merge: 7c214f5 3e78034 Author: Pawan Dhananjay <[email protected]> Date: Tue Aug 26 15:38:47 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit 7c214f5 Merge: 52762b9 daf1c7c Author: Pawan Dhananjay <[email protected]> Date: Mon Aug 25 11:01:40 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit 52762b9 Author: Pawan Dhananjay <[email protected]> Date: Fri Aug 22 11:52:56 2025 -0700 Handle 0 blobs per epoch case commit da27441 Merge: 521778b cee30d8 Author: Pawan Dhananjay <[email protected]> Date: Thu Aug 21 11:02:22 2025 -0700 Merge branch 'unstable' into blocks-then-columns commit 521778b Author: Pawan Dhananjay <[email protected]> Date: Thu Aug 21 07:03:58 2025 -0700 Random logs commit 4540195 Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 13 17:33:47 2025 -0700 Request columns from global peer pool commit fdce537 Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 20 18:37:44 2025 -0700 Implement responsible peer tracking commit 17c4e34 Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 20 14:18:15 2025 -0700 Avoid root requests for backfill sync commit 1a0df30 Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 20 14:14:48 2025 -0700 Also penalize all batch peers for availability check errors commit 6da924b Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 20 14:14:26 2025 -0700 Fix bug in initialization code commit 68cce37 Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 20 14:14:11 2025 -0700 Try to avoid chains failing for rpc errors commit ca9cfd5 Author: Pawan Dhananjay <[email protected]> Date: Tue Aug 19 12:17:59 2025 -0700 Small fixes commit 5337e46 Author: Pawan Dhananjay <[email protected]> Date: Mon Aug 18 22:59:38 2025 -0700 Add a function to retry column requests that could not be made commit 9455153 Author: Pawan Dhananjay <[email protected]> Date: Mon Aug 18 14:48:44 2025 -0700 Without retries commit 6bd8944 Author: Pawan Dhananjay <[email protected]> Date: Fri Aug 15 14:23:48 2025 -0700 Reduce backfill buffer size commit 156449c Author: Pawan Dhananjay <[email protected]> Date: Thu Aug 14 09:10:28 2025 -0700 Increase columns_by_root quota commit 836f9c6 Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 13 16:37:12 2025 -0700 Priorotize status v2 commit 490b627 Author: Pawan Dhananjay <[email protected]> Date: Wed Aug 13 16:36:56 2025 -0700 Penalize if invalid EL block
…igp#8112) - PR sigp#8045 introduced a regression of how lookup sync interacts with the da_checker. Now in unstable block import from the HTTP API also insert the block in the da_checker while the block is being execution verified. If lookup sync finds the block in the da_checker in `NotValidated` state it expects a `GossipBlockProcessResult` message sometime later. That message is only sent after block import in gossip. I confirmed in our node's logs for 4/4 cases of stuck lookups are caused by this sequence of events: - Receive block through API, insert into da_checker in fn process_block in put_pre_execution_block - Create lookup and leave in AwaitingDownload(block in processing cache) state - Block from HTTP API finishes importing - Lookup is left stuck Closes sigp#8104 - sigp#8110 was my initial solution attempt but we can't send the `GossipBlockProcessResult` event from the `http_api` crate without adding new channels, which seems messy. For a given node it's rare that a lookup is created at the same time that a block is being published. This PR solves sigp#8104 by allowing lookup sync to import the block twice in that case. Co-Authored-By: dapplion <[email protected]>
Issue Addressed
reqresp_pre_import_cacheintodata_availability_checker#8045 introduced a regression of how lookup sync interacts with the da_checker.Now in unstable block import from the HTTP API also insert the block in the da_checker while the block is being execution verified. If lookup sync finds the block in the da_checker in
NotValidatedstate it expects aGossipBlockProcessResultmessage sometime later. That message is only sent after block import in gossip.I confirmed in our node's logs for 4/4 cases of stuck lookups are caused by this sequence of events:
Closes #8104
Proposed Changes
GossipBlockProcessResultevent from thehttp_apicrate without adding new channels, which seems messy.For a given node it's rare that a lookup is created at the same time that a block is being published. This PR solves #8104 by allowing lookup sync to import the block twice in that case.