Implement SynthPoRep

[rjan90]

WIP tracking issue for SynthPoReps: https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0059.md

User story

• As a storage provider (SP) I want to leverage SynthethicPoReps to reduce the temporary data stored between PreCommit and ProveCommit from ~400GiB to ≤ 25GiB.

Acceptence criteria (SP):

• Can be enabled by setting SynthPoRep=true in the config.toml file
• All sector types sealed with SynthPoRep passes WindowPoSt/WinningPoSt

Acceptence criteria (SaaS):

Techincal scope

Synthetic PoRep Development

Preview Give feedback

1. Synthetic PoRep filecoin-ffi#400
   +0
2. Synthetic PoRep rust-fil-proofs#1701
   +0
3. Synthetic PoRep rust-filecoin-proofs-api#87
   +0
4. feat: go-wrapper for ClearSyntheticProofs filecoin-ffi#404
   +0
5. feat: Synthetic-porep #10970
   +0
   
6. fvm: Integrate Synthetic PoRep ref-fvm#1791
7. actor: Add Proof type for Synthetic PoRep builtin-actors#1314
8. ffi: update fvm deps:

We also need to introduce a network upgrade to introduce synthetic porep, thus we need to implement the barebone for nv21 in lotus to unblock testing.

add barebone nv21

Preview Give feedback

1. fvm: Integrate Synthetic PoRep ref-fvm#1791
2. actor: v12
3. gst: Implement fip 0059 go-state-types#190
4. Introduce nv21 skeleton in Lotus #10993
   1 of 1
   +0
   
5. Integrate actors v12 into Lotus using the pack.sh script

Testing

Preview Give feedback

1. Butterfly testing plan for SynthPoRep: https://www.notion.so/pl-strflt/SynthPoRep-Butterfly-network-testing-59e22e4e10a24521b50135571e67391f?pvs=4
2. Calibration testing plan for SynthPoRep

Deliverable

Preview Give feedback

1. Include segment on SP feedback of early testing [Orjan/nemo]
2. Blog post/media for explaining why all SPs want to use this [luca/irene/nemo]
3. Network upgrade timeline

Current proposals for the network upgrade timeline can be seen here: filecoin-project/core-devs#140 (comment)

[jennijuju]

on acceptance criteria, I would break it down by SaaS provider and individual SP.

[jennijuju]

added fvm and actor development handoff issue & added a tasklist for add barebone nv21 - without it, we wont be able to test the synthetic porep work.

[jennijuju]

May I get an update on the status?

[jennijuju]

Latest update from @cryptonemo

filecoin-project/rust-fil-proofs#1711 (comment) - open by Jul 7th.

[jennijuju]

Should we at least include ONE SaaS provider in the testing plan (butterfly might be a good idea), unless we believe the regular SP pipeline would fully cover that too.

[jennijuju]

@snadrus once the actor and fvm code is ready, Andy will continue #10970 in this sprint

[jennijuju]

We are hoping to have a deployable butterfly that accepts SythPoRep by the end of. the sprint (July 17th)

[jennijuju]

@rjan90 can I get a status update on this, how is testing going? are we close to landing PRs to nv integration branch?

[rjan90]

Status update:

• I would say that testing on the butterfly-network for SynthPoReps can be called completed 🟢: feat: introduce SyntheticPoRep #11057 (comment)
  • Testing verified deals on Butterfly has been hard, due to rkh-keys being hard to surface. That said SynthPoRep FIL+ sectors was tested to be successful on a 2k-network.
    • Want to confirm new FIL+ flow with DDO, so resurfacing these test-cases in the mid-september Butterfly-reset makes more sense.
  • Bug for extending SynthPoRep sectors has landed. fix: Add SynthPoReps to SealProofPoliciesV11 go-state-types#200. Confirming this fix on a local devnet will be easier once SynthPoRep is being merged into feat/nv21

Next steps:

• Get reviews and land dependencies in their nv21 specific branches.
• Rebase the Lotus introduce SynthPoRep PR to be based off v1.23.3, and get it reviewed.

[rjan90]

Closing this ticket now that the SynthPoRep-code has landed in the feat/nv21 branch..

Outstanding item from this epic, that can be seperated into their own tickets as they are more timeline dependent.

• Second round of Butterfly-testing / regression testing after all dependencies landed. Testing plan: https://www.notion.so/pl-strflt/SynthPoRep-Butterfly-network-testing-59e22e4e10a24521b50135571e67391f?pvs=4
• Blog post/media for explaining why all SPs want to use this [luca/irene/nemo]. In draft, but should be published closer to the Calibration-network upgrade IMO

[cryptonemo]

* Blog post/media for explaining why all SPs want to use this [luca/irene/nemo]. In draft, but should be published closer to the Calibration-network upgrade IMO

FYI, draft is being finalized this week and we are scheduled for blog post release on Oct 3

[rjan90]

Awesome @cryptonemo! Perfect timing👌