Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fatal: Failed to register the Ethereum service: content truncated #29303

Closed
jimmyisthis opened this issue Mar 20, 2024 · 7 comments · Fixed by #29451
Closed

Fatal: Failed to register the Ethereum service: content truncated #29303

jimmyisthis opened this issue Mar 20, 2024 · 7 comments · Fixed by #29451
Labels
type:bug

Comments

@jimmyisthis
Copy link

jimmyisthis commented Mar 20, 2024 •
edited
Loading

System information

Geth version: 1.13.14-stable-2bd6bd01
CL client & version: Lighthouse/v5.1.0-10a38a8
OS & Version: Linux (Ubuntu 23.10) on Raspberry Pi 5

Expected behaviour

Node starts normally and geths starts catching up the current chain head.

Actual behaviour

Client starts, but almost immediately fails with Fatal: Failed to register the Ethereum service: content truncated, size:802805, slot:266240 message. Logs:

INFO [03-20|16:31:00.032] Starting Geth on Ethereum mainnet...
INFO [03-20|16:31:00.032] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [03-20|16:31:00.036] Maximum peer count                       ETH=50 total=50
INFO [03-20|16:31:00.037] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [03-20|16:31:00.039] Sanitizing cache to Go's GC limits       provided=4096 updated=2647
INFO [03-20|16:31:00.040] Set global gas cap                       cap=50,000,000
INFO [03-20|16:31:00.041] Initializing the KZG library             backend=gokzg
INFO [03-20|16:31:00.146] Allocated trie memory caches             clean=397.00MiB dirty=661.00MiB
INFO [03-20|16:31:00.148] Using pebble as the backing database
INFO [03-20|16:31:00.148] Allocated cache and file handles         database=/home/ethereum/.ethereum/geth/chaindata cache=1.29GiB handles=524,288
INFO [03-20|16:31:00.213] Opened ancient database                  database=/home/ethereum/.ethereum/geth/chaindata/ancient/chain readonly=false
INFO [03-20|16:31:00.215] State scheme set by user                 scheme=path
INFO [03-20|16:31:00.219] Initialising Ethereum protocol           network=1 dbversion=8
WARN [03-20|16:31:00.219] Sanitizing invalid node buffer size      provided=661.00MiB updated=256.00MiB
INFO [03-20|16:31:00.219] Failed to load journal, discard it       err="journal not found"
INFO [03-20|16:31:00.221] Opened ancient database                  database=/home/ethereum/.ethereum/geth/chaindata/ancient/state readonly=false
WARN [03-20|16:31:00.221] Path-based state scheme is an experimental feature
INFO [03-20|16:31:00.221]
INFO [03-20|16:31:00.221] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [03-20|16:31:00.222] Chain ID:  1 (mainnet)
INFO [03-20|16:31:00.222] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
INFO [03-20|16:31:00.222]
INFO [03-20|16:31:00.222] Pre-Merge hard forks (block based):
INFO [03-20|16:31:00.222]  - Homestead:                   #1150000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [03-20|16:31:00.222]  - DAO Fork:                    #1920000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
INFO [03-20|16:31:00.222]  - Tangerine Whistle (EIP 150): #2463000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [03-20|16:31:00.222]  - Spurious Dragon/1 (EIP 155): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [03-20|16:31:00.222]  - Spurious Dragon/2 (EIP 158): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [03-20|16:31:00.222]  - Byzantium:                   #4370000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [03-20|16:31:00.222]  - Constantinople:              #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [03-20|16:31:00.222]  - Petersburg:                  #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [03-20|16:31:00.222]  - Istanbul:                    #9069000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [03-20|16:31:00.222]  - Muir Glacier:                #9200000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [03-20|16:31:00.222]  - Berlin:                      #12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [03-20|16:31:00.222]  - London:                      #12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [03-20|16:31:00.222]  - Arrow Glacier:               #13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [03-20|16:31:00.222]  - Gray Glacier:                #15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [03-20|16:31:00.222]
INFO [03-20|16:31:00.222] Merge configured:
INFO [03-20|16:31:00.222]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [03-20|16:31:00.222]  - Network known to be merged: true
INFO [03-20|16:31:00.222]  - Total terminal difficulty:  58750000000000000000000
INFO [03-20|16:31:00.222]
INFO [03-20|16:31:00.222] Post-Merge hard forks (timestamp based):
INFO [03-20|16:31:00.222]  - Shanghai:                    @1681338455 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [03-20|16:31:00.222]  - Cancun:                      @1710338135 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md)
INFO [03-20|16:31:00.222]
INFO [03-20|16:31:00.222] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [03-20|16:31:00.222]
INFO [03-20|16:31:00.224] Loaded most recent local header          number=19,386,952 hash=4868c9..228db3 td=58,750,003,716,598,352,816,469 age=1w5d15h
INFO [03-20|16:31:00.225] Loaded most recent local block           number=0          hash=d4e567..cb8fa3 td=17,179,869,184                 age=55y2d16h
INFO [03-20|16:31:00.225] Loaded most recent local snap block      number=19,386,952 hash=4868c9..228db3 td=58,750,003,716,598,352,816,469 age=1w5d15h
WARN [03-20|16:31:00.226] Loaded snapshot journal                  diffs=missing
INFO [03-20|16:31:00.226] Initialized transaction indexer          range="last 2350000 blocks"
INFO [03-20|16:31:00.226] Loaded local transaction journal         transactions=0 dropped=0
INFO [03-20|16:31:00.227] Upgrading chain index                    type=bloombits percentage=0
INFO [03-20|16:31:00.228] Resuming state snapshot generation       root=d7f897..0f0544 accounts=0 slots=0 storage=0.00B dangling=0 elapsed="382.113µs"
INFO [03-20|16:31:00.237] Transaction pool stopped
Fatal: Failed to register the Ethereum service: content truncated, size:802805, slot:266240

Steps to reproduce the behaviour

  1. Used device: single Raspberry Pi 5 device (2TB SSD connected via USB3, with a 16 GB swap file stored on the SSD)
  2. Run geth on the device (in a screen session):
    geth --authrpc.addr=0.0.0.0 --authrpc.port 8551 --authrpc.vhosts=* --authrpc.jwtsecret SECRET_LOCATION --http --http.addr 0.0.0.0 --http.vhosts=* --http.api eth,net,web3 --state.scheme=path
  3. Run lighthouse on the same device (in a screen session):
    lighthouse bn --network mainnet --execution-endpoint http://localhost:8551 --execution-jwt SECRET_LOCATION --checkpoint-sync-url https://mainnet.checkpoint.sigp.io --disable-deposit-contract-sync
  4. Wait until it is fully synced, then make it lose sync (e.g., by running some heavy processes on the device or stopping the lighthouse for a few minutes).
  5. Restart the Ligthhouse and wait till it is up and running, and make sure that geth is around 3-4 minutes behind (at least this is what logs showed)
  6. Hard-kill the device by unplugging it from the power source
  7. Restart the device
  8. Run geth removedb and remove the state database, but keep the ancient state
  9. Restart both clients (although geth should be enough as lighthouse does not fully start before geth is stopped)

Remarks:

  • I tried to reproduce this behavior with a fully synced node, and this problem did not happen when the device was hard killed in a fully synced state
  • Unfortunately, it is not a vanilla state due to the geth removedb step, which I ran to check if that would fix the issue on my end
  • Even though the device may be partially responsible for this behavior, it seems more due to the "catching up" state, as I could not reproduce it with fully synced devices (despite multiple attempts). Hard-killing the device almost guaranteed database corruption before the new database model (path)
@jimmyisthis
Copy link
Author

I managed to get the logs (from a separate device but configured the same way) generated by geth with no actions taken to change the state (geth removedb was not executed):

INFO [03-20|17:35:24.376] Starting Geth on Ethereum mainnet...
INFO [03-20|17:35:24.376] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [03-20|17:35:24.380] Maximum peer count                       ETH=50 total=50
INFO [03-20|17:35:24.381] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [03-20|17:35:24.384] Sanitizing cache to Go's GC limits       provided=4096 updated=2647
INFO [03-20|17:35:24.385] Set global gas cap                       cap=50,000,000
INFO [03-20|17:35:24.385] Initializing the KZG library             backend=gokzg
INFO [03-20|17:35:24.487] Allocated trie memory caches             clean=397.00MiB dirty=661.00MiB
INFO [03-20|17:35:24.627] Using pebble as the backing database
INFO [03-20|17:35:24.627] Allocated cache and file handles         database=/home/ethereum/.ethereum/geth/chaindata cache=1.29GiB handles=524,288
INFO [03-20|17:35:26.780] Opened ancient database                  database=/home/ethereum/.ethereum/geth/chaindata/ancient/chain readonly=false
INFO [03-20|17:35:26.781] State scheme set by user                 scheme=path
INFO [03-20|17:35:26.784] Initialising Ethereum protocol           network=1 dbversion=8
WARN [03-20|17:35:26.784] Sanitizing invalid node buffer size      provided=661.00MiB updated=256.00MiB
INFO [03-20|17:35:27.282] Failed to load journal, discard it       err="unmatched journal want 6bea0a6956336007e3913ee08aa3f576a1af6937cf0362a376edb9eb44230153 got 9cb935eaa2b9e657faef606ad67e0d8cca30971225bd274ac53ac3089ec9b2dd"
INFO [03-20|17:35:27.298] Opened ancient database                  database=/home/ethereum/.ethereum/geth/chaindata/ancient/state readonly=false
WARN [03-20|17:35:27.309] Path-based state scheme is an experimental feature
INFO [03-20|17:35:27.310]
INFO [03-20|17:35:27.310] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [03-20|17:35:27.310] Chain ID:  1 (mainnet)
INFO [03-20|17:35:27.310] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
INFO [03-20|17:35:27.310]
INFO [03-20|17:35:27.310] Pre-Merge hard forks (block based):
INFO [03-20|17:35:27.310]  - Homestead:                   #1150000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [03-20|17:35:27.310]  - DAO Fork:                    #1920000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
INFO [03-20|17:35:27.310]  - Tangerine Whistle (EIP 150): #2463000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [03-20|17:35:27.310]  - Spurious Dragon/1 (EIP 155): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [03-20|17:35:27.310]  - Spurious Dragon/2 (EIP 158): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [03-20|17:35:27.310]  - Byzantium:                   #4370000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [03-20|17:35:27.310]  - Constantinople:              #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [03-20|17:35:27.310]  - Petersburg:                  #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [03-20|17:35:27.310]  - Istanbul:                    #9069000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [03-20|17:35:27.310]  - Muir Glacier:                #9200000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [03-20|17:35:27.310]  - Berlin:                      #12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [03-20|17:35:27.310]  - London:                      #12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [03-20|17:35:27.310]  - Arrow Glacier:               #13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [03-20|17:35:27.310]  - Gray Glacier:                #15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [03-20|17:35:27.310]
INFO [03-20|17:35:27.310] Merge configured:
INFO [03-20|17:35:27.310]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [03-20|17:35:27.310]  - Network known to be merged: true
INFO [03-20|17:35:27.310]  - Total terminal difficulty:  58750000000000000000000
INFO [03-20|17:35:27.310]
INFO [03-20|17:35:27.310] Post-Merge hard forks (timestamp based):
INFO [03-20|17:35:27.310]  - Shanghai:                    @1681338455 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [03-20|17:35:27.310]  - Cancun:                      @1710338135 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md)
INFO [03-20|17:35:27.310]
INFO [03-20|17:35:27.310] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [03-20|17:35:27.310]
INFO [03-20|17:35:27.317] Loaded most recent local header          number=19,476,951 hash=a24f08..9b9ab1 td=58,750,003,716,598,352,816,469 age=1h37m40s
INFO [03-20|17:35:27.317] Loaded most recent local block           number=19,476,715 hash=dbd22e..6ee300 td=58,750,003,716,598,352,816,469 age=2h24m52s
INFO [03-20|17:35:27.317] Loaded most recent local snap block      number=19,476,951 hash=a24f08..9b9ab1 td=58,750,003,716,598,352,816,469 age=1h37m40s
INFO [03-20|17:35:27.317] Loaded most recent local finalized block number=19,476,884 hash=4fc24b..e609bc td=58,750,003,716,598,352,816,469 age=1h51m4s
INFO [03-20|17:35:27.318] Loaded last snap-sync pivot marker       number=19,015,938
WARN [03-20|17:35:27.318] Enabling snapshot recovery               chainhead=19,476,715 diskbase=19,476,773
WARN [03-20|17:35:27.327] Loaded snapshot journal                  diskroot=463749..9387ea diffs=unmatched
WARN [03-20|17:35:27.327] Snapshot is not continuous with chain    snaproot=463749..9387ea chainroot=9cb935..c9b2dd
INFO [03-20|17:35:27.327] Initialized transaction indexer          range="last 2350000 blocks"
INFO [03-20|17:35:27.327] Loaded local transaction journal         transactions=0 dropped=0
INFO [03-20|17:35:27.385] Transaction pool stopped
Fatal: Failed to register the Ethereum service: content truncated, size:802805, slot:266240

@rjl493456442
Copy link
Member

@holiman It's an error raised from billy. Probably something wrong?

@holiman
Copy link
Contributor

holiman commented Mar 21, 2024 •
edited
Loading

Yes, this comes from billy. As billy is opening the shelves, it notices that a shelf-file is not of the correct size.
The data-portion is 802805 bytes, the slotsize is 266240. The data must be a multiple of the slots, but 802805 % 266240 = 4085, so the file appears to be truncated.

billy internally is able to handle this, by simply truncating the extra bytes. However, we open billy from two locations

blobpool/limbo.go

func (p *BlobPool) Init{
       ...
	store, err := billy.Open(billy.Options{Path: queuedir, Repair: true}, newSlotter(), index)
...
func newLimbo(datadir string) (*limbo, error) {
     ...
	store, err := billy.Open(billy.Options{Path: datadir}, newSlotter(), index)

If we wanted billy to just repair it and move on, we have to supply the Repair as an option:

billy.Options{Path: datadir, Repair: true}

I am not sure why the current code does not do that on both callsites, if it's just an oversight or a conscious decision. @karalabe ?

Tangential: It would be nice if the the error message could tell us what file it concerns, in case we'd want to investigate the file. Or manually truncate it to a correct size.

@holiman holiman mentioned this issue Mar 21, 2024
Merged
@holiman
Copy link
Contributor

holiman commented Mar 21, 2024

So, this must be in the second location. One of the files in /home/ethereum/.ethereum/geth/chaindata/limbo, one of the files is 802816 bytes large, and that's the culprit. It would be interesting to investigate that particular file, if you can upload it somewhere.

Other than that, I think the easiest way to resolve this is to delete the file.

@jimmyisthis
Copy link
Author

jimmyisthis commented Mar 21, 2024 •
edited
Loading

@holiman Thank you for your help.
I'm uploading a zipped file /home/ethereum/.ethereum/geth/blobpool/limbo/bkt_00266240.bag
suspect.zip

@yashutanna
Copy link

yashutanna commented Apr 4, 2024 •
edited
Loading

+1 saw the same thing just now.

1 thing to mention, I was not able to get into the machine remotely and suspect the node crashed so restarted it. once the geth container started, it resulted in the error as @jimmyisthis sees above

followed the same course as above to find the faulty file in the limbo folder and deleted it to get the node to start up again (looks like doing so and syncing now)

here is the file I deleted if that helps debug further
bkt_00135168.bag.zip

execution-client-geth-1  | INFO [04-04|04:40:31.287] Chain ID:  1 (mainnet)
execution-client-geth-1  | INFO [04-04|04:40:31.287] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
execution-client-geth-1  | INFO [04-04|04:40:31.287]
execution-client-geth-1  | INFO [04-04|04:40:31.287] Pre-Merge hard forks (block based):
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Homestead:                   #1150000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - DAO Fork:                    #1920000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Tangerine Whistle (EIP 150): #2463000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Spurious Dragon/1 (EIP 155): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Spurious Dragon/2 (EIP 158): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Byzantium:                   #4370000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Constantinople:              #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Petersburg:                  #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Istanbul:                    #9069000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Muir Glacier:                #9200000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Berlin:                      #12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - London:                      #12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Arrow Glacier:               #13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Gray Glacier:                #15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]
execution-client-geth-1  | INFO [04-04|04:40:31.287] Merge configured:
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Network known to be merged: true
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Total terminal difficulty:  58750000000000000000000
execution-client-geth-1  | INFO [04-04|04:40:31.287]
execution-client-geth-1  | INFO [04-04|04:40:31.287] Post-Merge hard forks (timestamp based):
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Shanghai:                    @1681338455 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]  - Cancun:                      @1710338135 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md)
execution-client-geth-1  | INFO [04-04|04:40:31.287]
execution-client-geth-1  | INFO [04-04|04:40:31.287] ---------------------------------------------------------------------------------------------------------------------------------------------------------
execution-client-geth-1  | INFO [04-04|04:40:31.287]
execution-client-geth-1  | INFO [04-04|04:40:31.289] Loaded most recent local header          number=19,566,711 hash=47a088..2d0d13 td=58,750,003,716,598,352,816,469 age=1d20h56m
execution-client-geth-1  | INFO [04-04|04:40:31.289] Loaded most recent local block           number=19,566,514 hash=247956..b580a6 td=58,750,003,716,598,352,816,469 age=1d21h35m
execution-client-geth-1  | INFO [04-04|04:40:31.289] Loaded most recent local snap block      number=19,566,711 hash=47a088..2d0d13 td=58,750,003,716,598,352,816,469 age=1d20h56m
execution-client-geth-1  | INFO [04-04|04:40:31.289] Loaded most recent local finalized block number=19,566,647 hash=adf628..456fda td=58,750,003,716,598,352,816,469 age=1d21h8m
execution-client-geth-1  | INFO [04-04|04:40:31.289] Loaded last snap-sync pivot marker       number=18,566,789
execution-client-geth-1  | WARN [04-04|04:40:31.290] Enabling snapshot recovery               chainhead=19,566,514 diskbase=19,566,537
execution-client-geth-1  | WARN [04-04|04:40:31.292] Loaded snapshot journal                  diskroot=2fd38b..8ac450 diffs=unmatched
execution-client-geth-1  | WARN [04-04|04:40:31.292] Snapshot is not continuous with chain    snaproot=2fd38b..8ac450 chainroot=41feb4..4dd1f5
execution-client-geth-1  | INFO [04-04|04:40:31.292] Initialized transaction indexer          range="last 2350000 blocks"
execution-client-geth-1  | INFO [04-04|04:40:31.292] Loaded local transaction journal         transactions=0 dropped=0
execution-client-geth-1  | INFO [04-04|04:40:33.059] Transaction pool stopped
execution-client-geth-1  | Fatal: Failed to register the Ethereum service: content truncated, size:22712309, slot:135168
execution-client-geth-1  | Fatal: Failed to register the Ethereum service: content truncated, size:22712309, slot:135168
execution-client-geth-1 exited with code 0```

Im running ethereum/client-go:v1.13.14

@karalabe
Copy link
Member

karalabe commented Apr 4, 2024

I am not sure why the current code does not do that on both callsites, if it's just an oversight or a conscious decision. @karalabe ?

I'd say oversight. The original issue was fixed in https://github.com/ethereum/go-ethereum/pull/29001/files and I probably forgot that we have a second Billy in there. There's no reason not to do a repair on the limbo. Worst case is the same as for the mail billy, that we lose some transactions and cannot resurrect them in our local node on a reorg. But that's fine, network has redundancy so losing a blob locally is fine. These dbs are temporary storage only, not long term.

TL;DR: We should add repair to the limbo too.

@karalabe karalabe mentioned this issue Apr 4, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

core/txpool: repair the limbo Billy too on unclean shutdowns karalabe/go-ethereum
5 participants
@karalabe @holiman @rjl493456442 @yashutanna @jimmyisthis