fix(drand): add null HistoricalBeaconClient for old beacons #12830

rvagg · 2025-01-20T10:18:52Z

I'm pretty sure this will work since we only need to verify, not fetch, old beacon entries. So when we remove Servers from the config (like we've done for Mainnet and Incentinet), this will satisfy drand but also be a noop.

chain/beacon/drand/drand.go

rjan90 · 2025-01-20T12:47:26Z

Checked out this branch, and tried to see if I still encountered the #11802 issue.

It got a bit longer then previous:

lotus daemon --import-chain=/home/boost/forest_snapshot_mainnet_2025-01-20_height_4635403.forest.car.zst --halt-after-import
2025-01-20T12:42:18.434+0100	INFO	lotus	lotus/daemon.go:230	lotus repo: /mnt/lotuschain/.lotus
-------
2025-01-20T13:40:45.580+0100	INFO	chainstore	store/store.go:726	clearing block validation cache...
2025-01-20T13:40:53.733+0100	INFO	chainstore	store/store.go:765	696466 block validation entries cleared.
2025-01-20T13:40:53.733+0100	INFO	lotus	lotus/daemon.go:603	setting genesis
2025-01-20T13:40:53.746+0100	INFO	chainstore	store/store.go:685	New heaviest tipset! [bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2] (height=0)
2025-01-20T13:40:53.750+0100	WARN	chainstore	store/store.go:713	no previous heaviest tipset found, using [bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2]
2025-01-20T13:40:53.754+0100	INFO	drand	drand/drand.go:118	drand beacon without pubsub
2025-01-20T13:40:53.756+0100	INFO	drand	drand/drand.go:118	drand beacon without pubsub
2025-01-20T13:40:53.757+0100	INFO	drand	drand/drand.go:118	drand beacon without pubsub
2025-01-20T13:40:53.757+0100	INFO	lotus	lotus/daemon.go:621	validating imported chain...
2025-01-20T13:40:53.757+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.HistoricalBeaconClient.Info\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-20T13:40:53.757+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.HistoricalBeaconClient.Info\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-20T13:45:53.758+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.HistoricalBeaconClient.Info\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-20T13:45:53.758+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.HistoricalBeaconClient.Info\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-20T13:50:53.759+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.HistoricalBeaconClient.Info\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-20T13:50:53.759+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.HistoricalBeaconClient.Info\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-20T13:55:53.759+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.HistoricalBeaconClient.Info\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-20T13:55:53.759+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.HistoricalBeaconClient.Info\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}

It then looped on the endpoint down when speed tested / no historical randomness available for quite some time before:

2025-01-20T13:58:20.878+0100	INFO	statemgr	stmgr/stmgr.go:487	computing state (height: 0, ts=[bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2])
2025-01-20T13:58:20.886+0100	INFO	statemgr	stmgr/stmgr.go:487	computing state (height: 1, ts=[bafy2bzacechdx6xd62lcyy7rnyc4uxcxhuwqslcxfvj77fxlwafij3nhzchpy])
2025-01-20T13:58:20.887+0100	WARN	chainstore	store/store.go:670	reorgWorker quit
2025-01-20T13:58:22.259+0100	INFO	badgerbs	[email protected]/db.go:1027	Storing value log head: {Fid:129 Len:33 Offset:576605263}
2025-01-20T13:58:23.216+0100	INFO	badgerbs	[email protected]/levels.go:1000	[Compactor: 173] Running compaction: {level:0 score:1.73 dropPrefixes:[]} for level: 0
2025-01-20T13:58:30.541+0100	INFO	badgerbs	[email protected]/levels.go:962	LOG Compact 0->1, del 8 tables, add 7 tables, took 7.324947987s
2025-01-20T13:58:30.541+0100	INFO	badgerbs	[email protected]/levels.go:1010	[Compactor: 173] Compaction for level: 0 DONE
2025-01-20T13:58:30.541+0100	INFO	badgerbs	[email protected]/db.go:550	Force compaction on level 0 done
ERROR: chain validation failed: getting block messages for tipset: failed to get messages for block: failed to load msgmeta (bafy2bzacecmwp4imjqhdg2zvc7j2s4xxahnn5jnudtrt335re24i4zim7ccfi): ipld: could not find bafy2bzacecmwp4imjqhdg2zvc7j2s4xxahnn5jnudtrt335re24i4zim7ccfi

rvagg · 2025-01-20T22:23:34Z

OK, well I don't think the drand errors were your fatal problem here it's just looping in the background and logging those entries. But unfortunately we're stuck with the optimizing client: https://github.com/drand/drand/blob/v1.5.11/client/optimizing.go#L194, so perhaps we can't take this approach at all and need it fixed inside drand otherwise it's going to continue to speed test the null-client and spam those info log messages.

I think your problem here is that you can't use a snapshot file when trying to import the full "chain". At least that's my understanding from reading the code - the main difference between importing a snapshot and importing the chain is that a chain import does a full tipset validation back to genesis, which is why it wants the beacons to be available. It calls ValidateChain, which --import-snapshot won't do, and this walks back validating each tipset, executing all the messages; so it needs those messages too! I guess --import-chain is the trustless version of starting from scratch, you don't need state because it'll build it for you from genesis. But --import-snapshot will give you the historical tipsets but not messages and it'll start you from a trusted snapshot of the state, which you're just going to have to accept without being able to validate it.

I suppose a chain export is a lotus chain export without --recent-stateroots or --skip-old-msgs, while a snapshot is with both of those. Do we happen to make those available anywhere?

rvagg · 2025-01-20T22:47:32Z

Updated this branch with a modification that attempts to use the "watcher" mechanism in drand because watchers don't get speed tested, so if I can convince it that my historical null-client is a watcher then maybe it'll not speed test it. I haven't tested this but @rjan90 if you still have that snapshot handy, would you mind running it again to see if you get any of those INFO logs or any other errors from drand?

rjan90 · 2025-01-21T10:16:55Z

I haven't tested this but @rjan90 if you still have that snapshot handy, would you mind running it again to see if you get any of those INFO logs or any other errors from drand?

Reran with the latest commit, and this is what I get:

lotus daemon --import-chain=/home/boost/forest_snapshot_mainnet_2025-01-20_height_4635403.forest.car.zst --halt-after-import
2025-01-21T09:58:59.917+0100	INFO	lotus	lotus/daemon.go:230	lotus repo: /mnt/lotuschain/.lotus
-------
2025-01-21T10:56:07.051+0100	INFO	chainstore	store/store.go:726	clearing block validation cache...
2025-01-21T10:56:07.093+0100	INFO	chainstore	store/store.go:765	0 block validation entries cleared.
2025-01-21T10:56:07.093+0100	INFO	lotus	lotus/daemon.go:603	setting genesis
2025-01-21T10:56:07.097+0100	INFO	chainstore	store/store.go:685	New heaviest tipset! [bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2] (height=0)
2025-01-21T10:56:07.100+0100	WARN	chainstore	store/store.go:713	no previous heaviest tipset found, using [bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2]
2025-01-21T10:56:07.105+0100	INFO	drand	drand/drand.go:114	drand beacon without pubsub
2025-01-21T10:56:07.107+0100	INFO	drand	drand/drand.go:114	drand beacon without pubsub
2025-01-21T10:56:07.108+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "&{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.historicalBeaconClient.Get\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-21T10:56:07.108+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "&{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.historicalBeaconClient.Get\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-21T10:56:07.110+0100	INFO	drand	drand/drand.go:114	drand beacon without pubsub
2025-01-21T10:56:07.110+0100	INFO	lotus	lotus/daemon.go:621	validating imported chain...
2025-01-21T11:01:07.109+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "&{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.historicalBeaconClient.Get\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-21T11:01:07.109+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "&{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.historicalBeaconClient.Get\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-21T11:06:07.110+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "&{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.historicalBeaconClient.Get\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-21T11:06:07.110+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "&{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.historicalBeaconClient.Get\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-21T11:11:07.111+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "&{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.historicalBeaconClient.Get\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
2025-01-21T11:11:07.111+0100	INFO	drand	client/optimizing.go:194		{"optimizing_client": "endpoint down when speed tested", "client": "&{}.(+verifier)", "err": "no historical randomness available", "errVerbose": "no historical randomness available:\n    github.com/filecoin-project/lotus/chain/beacon/drand.historicalBeaconClient.Get\n        /home/boost/lotus/chain/beacon/drand/drand.go:274"}
------
2025-01-21T11:14:55.303+0100	INFO	statemgr	stmgr/stmgr.go:487	computing state (height: 0, ts=[bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2])
2025-01-21T11:14:55.312+0100	INFO	statemgr	stmgr/stmgr.go:487	computing state (height: 1, ts=[bafy2bzacechdx6xd62lcyy7rnyc4uxcxhuwqslcxfvj77fxlwafij3nhzchpy])
2025-01-21T11:14:55.315+0100	WARN	chainstore	store/store.go:670	reorgWorker quit
2025-01-21T11:14:56.225+0100	INFO	badgerbs	[email protected]/db.go:1027	Storing value log head: {Fid:129 Len:33 Offset:578835815}

2025-01-21T11:14:57.434+0100	INFO	badgerbs	[email protected]/levels.go:1000	[Compactor: 173] Running compaction: {level:0 score:1.73 dropPrefixes:[]} for level: 0

2025-01-21T11:15:01.880+0100	INFO	badgerbs	[email protected]/levels.go:962	LOG Compact 0->1, del 9 tables, add 8 tables, took 4.446465215s

2025-01-21T11:15:01.881+0100	INFO	badgerbs	[email protected]/levels.go:1010	[Compactor: 173] Compaction for level: 0 DONE
2025-01-21T11:15:01.881+0100	INFO	badgerbs	[email protected]/db.go:550	Force compaction on level 0 done
ERROR: chain validation failed: getting block messages for tipset: failed to get messages for block: failed to load msgmeta (bafy2bzacecmwp4imjqhdg2zvc7j2s4xxahnn5jnudtrt335re24i4zim7ccfi): ipld: could not find bafy2bzacecmwp4imjqhdg2zvc7j2s4xxahnn5jnudtrt335re24i4zim7ccfi

rvagg · 2025-01-21T10:53:04Z

OK, one tiny change and this seems to work; I grabbed my own snapshot and have been running it and the only logs are the badger ones so I believe this is solved, even if it is a hack.

I'd still like @AnomalRoil to take a look and suggest alternative approaches.

Marking @Kubuxu as reviewer since he's touched a lot of this stuff previously.

BigLep · 2025-02-03T17:56:02Z

Some feedback is in https://filecoinproject.slack.com/archives/C047Z15JNEA/p1738587544472029

AnomalRoil

A few comments.

chain/beacon/drand/drand.go

rvagg · 2025-02-04T05:40:26Z

Attempted to upgrade to go-clients and drand/v2 while I was at this but we fail our new dependency check because go-clients happens to pull in these for some reason:

github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect

rvagg · 2025-02-04T05:42:15Z

argh, and a slight upgrade to urfave/cli has made docsgen-cli change, I might have to back that one out

rvagg · 2025-02-04T05:56:19Z

urfave/cli has an OK change, but a bug in that particular release, a couple more patch levels and it's good, so I've done that here and checked in the changes, I'm just not sure about the go-spew and go-difflib untagged inclusions, it might be nice to track down where those are used

rvagg · 2025-02-04T06:02:05Z

go-clients -> github.com/hashicorp/consul/[email protected] pulls in both of these, but both of those are indirect too https://github.com/hashicorp/consul/blob/main/sdk/go.mod

rvagg · 2025-02-05T22:14:08Z

Will revisit this next week: If drand/go-clients#10 isn't addressed I might wind back to my previous version here of implementing my own fake client but keep the drand/v2 upgrade here and mark the untagged transient dependencies as OK.

rvagg · 2025-02-13T05:05:36Z

I think drand/go-clients#11 resolves this. The empty client already propagates an error via Get, we just need to deal with the logging.

Fixes: #11802 Ref: #11802 (comment)

rvagg · 2025-02-17T11:14:04Z

This should be ready now. @Kubuxu would you mind taking a look?

rvagg · 2025-02-17T11:21:24Z

🤦 this pulls us from a minimum of Go 1.22.7 to 1.22.10. But hopefully we're going to bump that to 1.23 soon anyway.

rvagg mentioned this pull request Jan 20, 2025

Lotus daemon --import-chain errors out #11802

Open

11 tasks

rvagg commented Jan 20, 2025

View reviewed changes

chain/beacon/drand/drand.go Outdated Show resolved Hide resolved

rvagg commented Jan 20, 2025

View reviewed changes

chain/beacon/drand/drand.go Outdated Show resolved Hide resolved

rvagg force-pushed the rvagg/historical-beacon-client branch 2 times, most recently from cf963a5 to 30ca496 Compare January 21, 2025 10:52

rvagg marked this pull request as ready for review January 21, 2025 10:53

rvagg requested a review from Kubuxu January 21, 2025 10:53

rvagg added the skip/changelog This change does not require CHANGELOG.md update label Jan 21, 2025

BigLep assigned rvagg Jan 28, 2025

AnomalRoil reviewed Feb 3, 2025

View reviewed changes

chain/beacon/drand/drand.go Outdated Show resolved Hide resolved

chain/beacon/drand/drand.go Outdated Show resolved Hide resolved

chain/beacon/drand/drand.go Outdated Show resolved Hide resolved

chain/beacon/drand/drand.go Outdated Show resolved Hide resolved

rvagg force-pushed the rvagg/historical-beacon-client branch from 30ca496 to c0fad9d Compare February 4, 2025 05:36

rvagg added 3 commits February 17, 2025 22:08

fix(drand): add null HistoricalBeaconClient for old beacons

69d1a8b

Fixes: #11802 Ref: #11802 (comment)

chore(drand): upgrade drand to drand/v2 and drand/go-clients

c2ae6c2

chore(deps): upgrade urfave/cli

f417457

rvagg force-pushed the rvagg/historical-beacon-client branch from 153b2cc to f417457 Compare February 17, 2025 11:13

chore: update Go version minimum to 1.22.10

86b66da

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(drand): add null HistoricalBeaconClient for old beacons #12830

fix(drand): add null HistoricalBeaconClient for old beacons #12830

rvagg commented Jan 20, 2025

rjan90 commented Jan 20, 2025 •

edited

Loading

rvagg commented Jan 20, 2025

rvagg commented Jan 20, 2025

rjan90 commented Jan 21, 2025

rvagg commented Jan 21, 2025

BigLep commented Feb 3, 2025

AnomalRoil left a comment

rvagg commented Feb 4, 2025 •

edited

Loading

rvagg commented Feb 4, 2025

rvagg commented Feb 4, 2025

rvagg commented Feb 4, 2025

rvagg commented Feb 5, 2025

rvagg commented Feb 13, 2025

rvagg commented Feb 17, 2025

rvagg commented Feb 17, 2025

fix(drand): add null HistoricalBeaconClient for old beacons #12830

Are you sure you want to change the base?

fix(drand): add null HistoricalBeaconClient for old beacons #12830

Conversation

rvagg commented Jan 20, 2025

rjan90 commented Jan 20, 2025 • edited Loading

rvagg commented Jan 20, 2025

rvagg commented Jan 20, 2025

rjan90 commented Jan 21, 2025

rvagg commented Jan 21, 2025

BigLep commented Feb 3, 2025

AnomalRoil left a comment

Choose a reason for hiding this comment

rvagg commented Feb 4, 2025 • edited Loading

rvagg commented Feb 4, 2025

rvagg commented Feb 4, 2025

rvagg commented Feb 4, 2025

rvagg commented Feb 5, 2025

rvagg commented Feb 13, 2025

rvagg commented Feb 17, 2025

rvagg commented Feb 17, 2025

rjan90 commented Jan 20, 2025 •

edited

Loading

rvagg commented Feb 4, 2025 •

edited

Loading