[Epic] Improving retrieval reliability

[jacobheun]

This is a parent tracking issue for issues related to retrieval reliability problems

Problem

Storage Providers and clients have been experiencing increasing trouble with retrievable reliability for various reasons (tracked below). This is critical functionality that is also currently slowing down the throughput of deal renewal programs, like Evergreen

These issues are spread out across Lotus, go-fil-markets, Boost, and Filecoin Slack. As such, this issue will be leveraged as a central hub for tracking retrieval reliability.

Latest Status

Updated: Aug 25th, 2022

• Our highest priority is currently on resolving full piece retrieval needed for deal renewal.

Tracked Issues

Known Issues

• Retrieval truncates car file causing - ERROR: given data does not match expected commP - on import lotus#8663
• Failure to retrieve when root payload CID is an identity CID #715
• ...

Bug Fixes

• Empty identity CID should be indexed when options are set ipld/go-car#316
  • Included in v1.2.0-rc2
  • Needs to be merged into Lotus

Features

• Add a boost doctor command to enable SP's to surface known issues #582

Shipped

• Added an experimental http server, booster-http, to enable easy full piece/car retrieval.
  • Full piece retrieval support in v1.2.0-rc2
• Added support for inspecting piece and payload CID's to debug retrieval issues. Available in v1.2.0-rc1

[LexLuthr]

I think https://filecoinproject.slack.com/archives/C0377FJCG1L/p1657019175554179 would need to be prioritised as it is hitting Evergreen badly.

filecoin-project/lotus#8663

[jacobheun]

@LexLuthr do we have an issue for that yet, I know the team is looking into it? I'd like to make sure we're tracking all issues here and getting them aggregated to this issue.

[dirkmc]

I believe HTTP retrieval of a piece should help with filecoin-project/lotus#8663

[LexLuthr]

Yes, I have noted that in my observation in the original issue.