New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Core lightning fails on Umbrel restart if bitcoin is not synced #6924

Closed

ShahanaFarooqui opened this issue Dec 8, 2023 · 1 comment · Fixed by #7342

Assignees

Labels

Milestone

Collaborator

ShahanaFarooqui commented Dec 8, 2023

Core lightning fails on Umbrel restart if bitcoin is not synced with below error:

Core-Lightning starting
2023-12-07T05:47:12.951Z INFO    connectd: Static Tor service onion address: "wz" bound from extern port 9735
2023-12-07T05:48:38.026Z UNUSUAL plugin-bcli: Waiting for bitcoind to warm up...
2023-12-07T05:48:38.035Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
bitcoind has gone backwards from 820093 to 820080 blocks!

Reference Issues:
https://community.umbrel.com/t/problem-with-core-lightning-not-working-after-reboot/13908
ElementsProject/cln-application#33

It will be better if CLN waits for bitcoind to sync rather than failing immediately.

The text was updated successfully, but these errors were encountered:

ShahanaFarooqui added the enhancement label

ShahanaFarooqui added this to the v24.02 milestone

ShahanaFarooqui assigned rustyrussell

Member

cdecker commented Dec 15, 2023

Notice that this is not a sync issue, but rather CLN identifying that the bitcoind daemon did something that is unlikely to be correct: lost some blocks. If this were just a reorg then bitcoind would still show the number of blocks in the branch it switched to, it would never go backwards.

This is likely a bitcoind that restarted verifying from scratch, and if CLN were to run against it and wait, it could end up doing so, and being unresponsive for hours if not days at a time.

This is an abort() that is on purpose: it's very noisy, in an attempt to grab some human's attention, so they can fix the underlying issue.

I would not consider this a bug.

cdecker removed this from the v24.02 milestone

ShahanaFarooqui mentioned this issue

If there sudden power off, core lightning is in starting mode after it is on. Won't start successfully ElementsProject/cln-application#33

Closed

ShahanaFarooqui added this to the v24.08 milestone

rustyrussell added a commit to rustyrussell/lightning that referenced this issue


          pytest: test that we wait for bitcoind if it's fallen behind us.

437f224

This is better than refusing to start, especially for users not on
the cmdline.

See: ElementsProject#6924

Signed-off-by: Rusty Russell <[email protected]>

rustyrussell added a commit to rustyrussell/lightning that referenced this issue


          lightningd: wait for bitcoind at startup if it's fallen behind.

b68d17f

Fixes: ElementsProject#6924
Changelog-Changed: lightningd: we wait for bitcoind if it has somehow gone backwards (as long as header height is still ok).
Signed-off-by: Rusty Russell <[email protected]>

rustyrussell mentioned this issue

Wait for bitcoind if it's gone backwards, don't abort. #7342

Merged

rustyrussell added a commit to rustyrussell/lightning that referenced this issue


          pytest: test that we wait for bitcoind if it's fallen behind us.

4d63483

This is better than refusing to start, especially for users not on
the cmdline.

See: ElementsProject#6924

Signed-off-by: Rusty Russell <[email protected]>

rustyrussell added a commit to rustyrussell/lightning that referenced this issue


          lightningd: wait for bitcoind at startup if it's fallen behind.

fa5c477

Fixes: ElementsProject#6924
Changelog-Changed: lightningd: we wait for bitcoind if it has somehow gone backwards (as long as header height is still ok).
Signed-off-by: Rusty Russell <[email protected]>

rustyrussell added a commit to rustyrussell/lightning that referenced this issue


          pytest: test that we wait for bitcoind if it's fallen behind us.

a677edd

This is better than refusing to start, especially for users not on
the cmdline.

See: ElementsProject#6924

Signed-off-by: Rusty Russell <[email protected]>

rustyrussell added a commit to rustyrussell/lightning that referenced this issue


          lightningd: wait for bitcoind at startup if it's fallen behind.

8380bf0

Fixes: ElementsProject#6924
Changelog-Changed: lightningd: we wait for bitcoind if it has somehow gone backwards (as long as header height is still ok).
Signed-off-by: Rusty Russell <[email protected]>

ShahanaFarooqui pushed a commit to rustyrussell/lightning that referenced this issue


          pytest: test that we wait for bitcoind if it's fallen behind us.

9680cf1

This is better than refusing to start, especially for users not on
the cmdline.

See: ElementsProject#6924

Signed-off-by: Rusty Russell <[email protected]>

ShahanaFarooqui pushed a commit to rustyrussell/lightning that referenced this issue


          lightningd: wait for bitcoind at startup if it's fallen behind.

b8fbba6

Fixes: ElementsProject#6924
Changelog-Changed: lightningd: we wait for bitcoind if it has somehow gone backwards (as long as header height is still ok).
Signed-off-by: Rusty Russell <[email protected]>

rustyrussell added a commit to rustyrussell/lightning that referenced this issue


          lightningd: wait for bitcoind at startup if it's fallen behind.

1d3a92d

Fixes: ElementsProject#6924
Changelog-Changed: lightningd: we wait for bitcoind if it has somehow gone backwards (as long as header height is still ok).
Signed-off-by: Rusty Russell <[email protected]>

rustyrussell added a commit to rustyrussell/lightning that referenced this issue


          pytest: test that we wait for bitcoind if it's fallen behind us.

944287e

This is better than refusing to start, especially for users not on
the cmdline.

See: ElementsProject#6924

Signed-off-by: Rusty Russell <[email protected]>

rustyrussell added a commit to rustyrussell/lightning that referenced this issue


          lightningd: wait for bitcoind at startup if it's fallen behind.

90fd7de

Fixes: ElementsProject#6924
Changelog-Changed: lightningd: we wait for bitcoind if it has somehow gone backwards (as long as header height is still ok).
Signed-off-by: Rusty Russell <[email protected]>

rustyrussell added a commit to rustyrussell/lightning that referenced this issue


          pytest: test that we wait for bitcoind if it's fallen behind us.

26b9ea5

This is better than refusing to start, especially for users not on
the cmdline.

See: ElementsProject#6924

Also disable some tests that fail with elementsd once we rework our startup.

Signed-off-by: Rusty Russell <[email protected]>

rustyrussell added a commit to rustyrussell/lightning that referenced this issue


          lightningd: wait for bitcoind at startup if it's fallen behind.

9f23a01

Fixes: ElementsProject#6924
Changelog-Changed: lightningd: we wait for bitcoind if it has somehow gone backwards (as long as header height is still ok).
Signed-off-by: Rusty Russell <[email protected]>

rustyrussell added a commit to rustyrussell/lightning that referenced this issue


          lightningd: wait for bitcoind at startup if it's fallen behind.

56a87ce

Fixes: ElementsProject#6924
Changelog-Changed: lightningd: we wait for bitcoind if it has somehow gone backwards (as long as header height is still ok).
Signed-off-by: Rusty Russell <[email protected]>

rustyrussell closed this as completed in #7342

rustyrussell closed this as completed in

f643fb8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment