Skip to content

triedb/pathdb: fix an issue of history indexer deadloop#32335

Closed
jsvisa wants to merge 11 commits intoethereum:masterfrom
jsvisa:history-indexer-deadloop
Closed

triedb/pathdb: fix an issue of history indexer deadloop#32335
jsvisa wants to merge 11 commits intoethereum:masterfrom
jsvisa:history-indexer-deadloop

Conversation

@jsvisa
Copy link
Copy Markdown
Contributor

@jsvisa jsvisa commented Aug 3, 2025

In our environment, in some circumstances, we saw the below logs:

DEBUG[08-03|03:18:31.894] State history is fully indexed           last=1,075,884
DEBUG[08-03|03:18:31.894] Resume state history indexing            id=1,075,967        tail=929,308
DEBUG[08-03|03:18:31.894] State history is fully indexed           last=1,075,884
DEBUG[08-03|03:18:31.894] Resume state history indexing            id=1,075,967        tail=929,308
DEBUG[08-03|03:18:31.894] State history is fully indexed           last=1,075,884
DEBUG[08-03|03:18:31.894] Resume state history indexing            id=1,075,967        tail=929,308
DEBUG[08-03|03:18:31.894] State history is fully indexed           last=1,075,884
DEBUG[08-03|03:18:31.894] Resume state history indexing            id=1,075,967        tail=929,308
DEBUG[08-03|03:18:31.894] State history is fully indexed           last=1,075,884
DEBUG[08-03|03:18:31.894] Resume state history indexing            id=1,075,967        tail=929,308
DEBUG[08-03|03:18:31.894] State history is fully indexed           last=1,075,884
DEBUG[08-03|03:18:31.895] Resume state history indexing            id=1,075,967        tail=929,308
DEBUG[08-03|03:18:31.895] State history is fully indexed           last=1,075,884
DEBUG[08-03|03:18:31.895] Resume state history indexing            id=1,075,967        tail=929,308
DEBUG[08-03|03:18:31.895] State history is fully indexed           last=1,075,884
DEBUG[08-03|03:18:31.895] Resume state history indexing            id=1,075,967        tail=929,308
DEBUG[08-03|03:18:31.895] State history is fully indexed           last=1,075,884
DEBUG[08-03|03:18:31.895] Resume state history indexing            id=1,075,967        tail=929,308
DEBUG[08-03|03:18:31.895] State history is fully indexed           last=1,075,884
DEBUG[08-03|03:18:31.895] Resume state history indexing            id=1,075,967        tail=929,308

After some digging of the code, I found the below steps:

  1. when the beginID > lastID, the indexer will be returned immediately
  2. The initer.run checks whether the initialization of the indexer is complete. If it is not, a new indexer is created and run again.
  3. However, since lastID is not updated, the condition of beginID > lastID remains true, causing the process to loop back to step 1.

Here we restore the metadata with the lastId(which maybe it's the bottom layer's id). So the initer will be exited later, skip the deadloop.

jsvisa added 3 commits August 3, 2025 22:58
Signed-off-by: jsvisa <delweng@gmail.com>
Signed-off-by: jsvisa <delweng@gmail.com>
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa jsvisa requested a review from rjl493456442 as a code owner August 3, 2025 15:55
@rjl493456442
Copy link
Copy Markdown
Member

DEBUG[08-03|03:18:31.895] State history is fully indexed           last=1,075,884
DEBUG[08-03|03:18:31.895] Resume state history indexing            id=1,075,967        tail=929,308

I think the root cause is: the last was not reverted correctly in the chain rollback previously? It's not expected to be higher than state history.

Comment thread triedb/pathdb/history_indexer.go Outdated
jsvisa added 6 commits August 4, 2025 15:27
Signed-off-by: jsvisa <delweng@gmail.com>
This reverts commit 7184b37.

Signed-off-by: jsvisa <delweng@gmail.com>
Signed-off-by: jsvisa <delweng@gmail.com>
Signed-off-by: jsvisa <delweng@gmail.com>
Signed-off-by: jsvisa <delweng@gmail.com>
Signed-off-by: jsvisa <delweng@gmail.com>
Comment thread triedb/pathdb/database.go Outdated
jsvisa added 2 commits August 5, 2025 10:46
Signed-off-by: jsvisa <delweng@gmail.com>
Signed-off-by: jsvisa <delweng@gmail.com>
@rjl493456442 rjl493456442 self-assigned this Aug 5, 2025
@jsvisa jsvisa closed this Aug 15, 2025
rjl493456442 added a commit that referenced this pull request Sep 8, 2025
Alternative of #32335, enhancing the history indexer recovery after
unclean shutdown.
Sahil-4555 pushed a commit to Sahil-4555/go-ethereum that referenced this pull request Oct 12, 2025
Alternative of ethereum#32335, enhancing the history indexer recovery after
unclean shutdown.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants