fix(ipld): fix races in the Retriever's doRequest method

[Wondertan]

Mainly this PR fixes two races:

1. One race happens when in one slice slot, two routines are trying to write the same share. Two because we have two sources for each share: Row and Col. Also, this does not happen on a happy path, and only in case, Retriever tries to fetch as many shares as possible to reconstruct the block, explaining why the race was uncovered with test(share): add Full Node reconstruction tests #702.

The mentioned race is a known one, and we had an incorrect attempt previously to fix it cca832a.

This fix is correct and achieved by keeping a mutex per share. If a share comes from any of two sources first, it locks the slot and prevents a share from another source from being written.

2. The second race happened after the square was reconstructed. During the reconstruction, some new shares could arrive and wait on the lock. Once reconstructed, they were unlocked and continued to write in an already filled-up square slice causing multiple races with the same root. Basically, the reconstructed square was returned to the user, while Retriever internal was continue to writing. This PR fixes that and ensure that after unlocking writings routines check if reconstruction happened and if so aborts them.

Closes #814

P.S. The complexity of the code in Retriever is starting to grow, and it's hard to test it. We will likely need to refactor the code(I have a brief picture in my head). However, though, this should happen after all the API improvements land in rsmt2d.

[Wondertan]

Oh shoot, I just realized I am using a feature from 1.18 here and we are not updated yet, ahhh.

[codecov-commenter]

Codecov Report

Merging #815 (7af60f3) into main (a2dbbbe) will increase coverage by 0.03%.
The diff coverage is 83.72%.

@@            Coverage Diff             @@
##             main     #815      +/-   ##
==========================================
+ Coverage   53.30%   53.33%   +0.03%     
==========================================
  Files         119      119              
  Lines        6833     6858      +25     
==========================================
+ Hits         3642     3658      +16     
- Misses       2821     2826       +5     
- Partials      370      374       +4     

Impacted Files	Coverage Δ
ipld/retriever.go	85.36% <83.72%> (+1.19%)	⬆️
fraud/bad_encoding.go	31.11% <0.00%> (-3.34%)	⬇️
das/daser.go	73.68% <0.00%> (-2.64%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a2dbbbe...7af60f3. Read the comment docs.

[Wondertan]

Ok, so this race fixing PR uncovered another race on CI. Nice

[Wondertan]

Ok, so now this PR fixes two independent races. Will update the description

[Wondertan]

Rebased on main with go1.18. Removed temporal semaphore as we have the needed feature from 1.18 now

[liamsi]

P.S. The complexity of the code in Retriever is starting to grow, and it's hard to test it. We will likely need to refactor the code(I have a brief picture in my head). However, though, this should happen after all the API improvements land in rsmt2d.

Can you please write that down (also which changes in r2mt2d are necessary and how they related to simplifying the retriever code here).

This is really hard to reason about and I can't really tell if this PR fixes things while introducing a bunch of others.

[Wondertan]

@liamsi, to reason about the code is better to check out and see the whole code, rather than changes only on GH.

[Wondertan]

I also put a lot of effort into comments. If they don't make sense, pls point that out. Otherwise, I am pretty confident in this code now.

[liamsi]

The changes with the comments make sense (besides one case I commented on).

[Wondertan]

Thank you @liamsi!