shares|ipld: Rework data retrieval

[Wondertan]

What and Why

This PR comes from the requirement to make tests in #598 pass deterministically by reimplementing the original goal of #517. At first, I implemented the goal by simply updating the current code without reworking it. Unfortunately, that didn't work well, and there was a bug that made me stuck for a few days. After extensive multiday debugging, I cleaned up and reworked the existing implementation of RetrieveData in studying the code for the bug. The rework landed into a new ipld.Retriever component, which now not just requests the data via sampling quadrants but also retries for other quadrants.

Eventually, the bug was about rsmt2d not being suitable for recovering over the same data square multiple times, as it sets nil shares to zeroed shares to the given flattened square slice(it actually updates the given slice in place after all!). I will report and link here the issue soon with the suggested fix. The issue celestiaorg/rsmt2d#72. Unless it is fixed, this PR has a workaround with the TODO on it.

TODO

• Rework IPLD batch committing logic
• Fix rsmt2d's zeroing of nil shares (Represent missing shares as nil internally rsmt2d#76)
• Use rsmt2d v0.4.0 with the above fix
• Parallelize downloading from different roots
• Add Debug logging
• Improve comments
• Add randomization with Col roots and retries(will fix in ipld: implement retry mechanism for block repairing if it fails with ErrUnrepairableDataSquare. #508)
• Write unit test for quadrants retries(Currently tested only with unmerged tests in tests(shares): write full node reconstruction test on the ShareService level #598)
• Manually test that a Full Node is able to sync devnet

Closes #509
Closes #508
Closes #516 ,
Partially closes #604

Note: @vgonkivs was informed about the new implementation of RetrieveData

[Wondertan]

The test failure is #608

[adlerjohn]

https://github.com/celestiaorg/rsmt2d/releases/tag/v0.4.0 is now released, which should be able to handle multiple repairs on the same square by representing missing shares as nil internally.

[liamsi]

Left some smaller comments. will try to properly review soon.

[liamsi]

Looks really good. Just a few questions and minor things. Hope we can merge this soon.

[renaynay]

Can we maybe add some debug or trace level logs in retriever.go?

[codecov-commenter]

Codecov Report

Merging #623 (7845ba9) into main (16dba97) will decrease coverage by 0.15%.
The diff coverage is 80.92%.

@@            Coverage Diff             @@
##             main     #623      +/-   ##
==========================================
- Coverage   61.70%   61.54%   -0.16%     
==========================================
  Files          93       96       +3     
  Lines        4969     5024      +55     
==========================================
+ Hits         3066     3092      +26     
- Misses       1581     1606      +25     
- Partials      322      326       +4     

Impacted Files	Coverage Δ
node/config_opts.go	27.27% <0.00%> (-8.73%)	⬇️
node/node.go	60.86% <ø> (ø)
cmd/celestia/full.go	41.93% <33.33%> (-0.93%)	⬇️
cmd/celestia/light.go	41.93% <33.33%> (-0.93%)	⬇️
cmd/celestia/bridge.go	56.89% <50.00%> (-0.80%)	⬇️
cmd/flags_rpc.go	76.00% <76.00%> (ø)
service/share/full_availability.go	93.75% <91.66%> (-6.25%)	⬇️
ipld/retriever.go	91.95% <91.95%> (ø)
ipld/read.go	77.94% <100.00%> (-7.25%)	⬇️
node/components.go	94.11% <100.00%> (-0.69%)	⬇️
... and 11 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 28020da...7845ba9. Read the comment docs.

[Wondertan]

I updated the branch with randomization and retries over columns. Better be done within this PR.

Tests are still failing due to known issue.

[Wondertan]

Rebased on main

[renaynay]

looks good - waiting for subsequent test PR

you can resolve my open comments.

[liamsi]

👏🏼