das: Parallelise DASer

[walldiss]

Resolves

• das: Parallelise catchUp method #483
• das: Ensure there are no gaps in DASed past headers in case of stopped node #504
• tests: flaky TestDASer_catchUp_oneHeader #793
• refactor(daser): Simplify store/background store from parallel daser #1049

Distribute data availability sampling across pool of workers.

New DASer implementation bring pool of workers. Core features:

• Sampling from header with height 1 to latest known one.
• Subscription to new headers is running in parallel and pushes new headers to DASer callback on demand.
• New found headers are processed with higher priority, but from only 1 worker at time. This way catch-up workers are not block by low speed of recent headers sampling.
• Add very basic error handling: If header is impossible to sample its height is stored for retry upon restart. Amount of retries persist over checkpoint until header is sampled successfully
• Store sampling state periodically in case of force quit.

TODO:

• add tracing
• add metrics

[distractedm1nd]

just looking at docs and names for now

[renaynay]

Leaving an intermediate review for now --

I like the general direction. I think there are things that could be simplified (e.g. the backgroundStore). Also, can we rename discovery of headers to something other than discovery? Just something that indicates it's recent/received through subscription rather than some active discovery process.

Another thing is the frequency/duplicity of storing the checkpoint to disk:

2022-08-18T10:34:56.270+0200	INFO	das	das/daser.go:200	discovered new header	{"height": 181780}
2022-08-18T10:34:56.270+0200	INFO	das	das/state.go:63	added recent headers maxKnown DASer priority queue 	{"from_height": 181779, "to_height": 181780}
2022-08-18T10:35:15.924+0200	INFO	das	das/daser.go:151	stored checkpoint to disk	{"checkpoint": {"min_sampled_height":173870,"max_known_height":181780,"skipped":{}}}
2022-08-18T10:36:15.920+0200	INFO	das	das/daser.go:151	stored checkpoint to disk	{"checkpoint": {"min_sampled_height":173870,"max_known_height":181780,"skipped":{}}}
2022-08-18T10:37:15.918+0200	INFO	das	das/daser.go:151	stored checkpoint to disk	{"checkpoint": {"min_sampled_height":173870,"max_known_height":181780,"skipped":{}}}
2022-08-18T10:38:15.921+0200	INFO	das	das/daser.go:151	stored checkpoint to disk	{"checkpoint": {"min_sampled_height":173870,"max_known_height":181780,"skipped":{}}}
2022-08-18T10:39:15.916+0200	INFO	das	das/daser.go:151	stored checkpoint to disk	{"checkpoint": {"min_sampled_height":173870,"max_known_height":181780,"skipped":{}}}
2022-08-18T10:40:15.915+0200	INFO	das	das/daser.go:151	stored checkpoint to disk	{"checkpoint": {"min_sampled_height":173870,"max_known_height":181780,"skipped":{}}}

I am running your branch against mamaki and this is taken from my logs ^.

I think this PR would benefit from some further simplification as well as cleaning up of weird behaviours like the one I just mentioned. Also more documentation :) There is a doc.go file and it would be nice if you could update it.

[renaynay]

Also, when stopping the node (without force quitting), I get the following:

2022-08-18T10:45:07.166+0200	ERROR	node	node/node.go:142	Stopping Light Node: cannot close topic: outstanding event handlers or subscriptions; cannot close topic: outstanding event handlers or subscriptions
Error: cannot close topic: outstanding event handlers or subscriptions; cannot close topic: outstanding event handlers or subscriptions

[Wondertan]

Oops, sorry for making each comment a separate review. I've started using different software for reviews and not yet got used to it

[Wondertan]

As per the sync discussion, we decided to rework one by one header height handling in favor of height ranges

[Wondertan]

TODO list from sync call

• backgroundStore finish
• test documentation
• DASer state API
• Docs
• Metrics

[Wondertan]

Awesome work! The logic is correct AFAICS, and I wasn't able to find any bugs. Left mostly stylistic nits/questions and thoughts on things. Looking forward to the above TODO's changes to land

[walldiss]

More changes from review comments will come in later commits

[walldiss]

I’ve squashed DAS’er commit history so it is easier to track new changes. All previous reviews comments were addressed before the squash.

[Wondertan]

Just comments and logs fixes. The logic is correct, so a solid approval.

[Wondertan]

Reapproval

[renaynay]

Awesome progress on this PR, it looks great now and much easier to follow :) thanks a lot for being so thorough with this.

Nit that can be left for a subsequent PR:
we should godoc every exported function, struct etc. (e.g. WorkerStats) even if it seems obvious as it's best practice

[renaynay]

morge it

[Wondertan]

Let's also resolve all the comments to be nice and tidy