[Epic] Handling non-car files within Boost

[LexLuthr]

This is the parent tracking issue related to the handling of non-car files within Boost.

Problem Statement

The current design of Boost assumes that all the deals are made with car files. There is no provision to handle a deal which might be blobs of data or concatenate car files (data segment indexing).

Change Proposal

To successfully handle the non-car files, we need to make multiple changes in Boost architecture and internal deal flow. These changes would need to be shipped in a single release to avoid breaking any existing functionality and introducing new bugs. The list of required changes is below along with their tracking issues or PRs

Merge all the new changes to a feature branch

• Create a new feature branch
• Rebase all existing PRs to the new feature branch

Handle commP for non-car file

• Boost should allow generating commP for non-car files

Handle generating index for concatenate car files (data segment indexing)

• Update PieceDirectory to generate indexes for concatenate car files

Handle deal flow error for non-car files

• nocar: update deal flow to account for non-car files #1690

Update piece-doctor

• nocar: Update piece-doctor to not generate indexing alerts in UI for non-car pieces #1691
• nocar: Update LID UI to show how many non-car pieces we have #1692

Handle LID recovery for non-car file

• nocar: Update LID recovery tool to account for non-car files #1693

Update Boost docs

• Update Boost docs to explain the new capabilities and LID changes

[willscott]

current discussion here is to support a strict PoDSI deal structure first, while continuing to understand what expectations should be made around arbitrary non-car deals