Skip to content

Latest commit

 

History

History
268 lines (175 loc) · 20.7 KB

fip-0090.md

File metadata and controls

268 lines (175 loc) · 20.7 KB
Raw
fip title author discussions-to status type category created
0090
Non-Interactive PoRep
luca (@lucaniz), kuba (@Kubuxu), nicola (@nicola), nemo (@cryptonemo), volker (@vmx), irene (@irenegia)
Superseded
Technical
Core
2023-12-18

FIP-0090: Non-Interactive PoRep

Simple Summary

This proposal presents a new PoRep protocol (Non-Interactive PoRep) that removes PreCommit when onboarding Committed Capacity (CC) sectors. As a result, we have

  • Simplified storage onboarding pipeline;
  • Trustless separation between storage and computing: the proving tasks associated with sector onboarding can be outsourced;
  • Unblocking SupraSeal's full potential.

Abstract

Non-Interactive PoRep (NI-PoRep) allows the removal of on-chain interaction when onboarding Committed Capacity (CC) sectors by changing the way PoRep challenges are generated.

The protocol allows Storage Providers (SPs) to locally generate PoRep challenges instead of using on-chain randomness.

On the one hand, this feature of the protocol allows for a drastically simpler onboarding pipeline. On the other hand, it requires a higher number of PoRep challenges (2268 per SDR layer instead of the current 180 per SDR layer) in order to preserve network security.

NI-PoRep is proposed as an optional feature; the previously available proof types will still be supported. NI-PoRep is restricted to CC sectors (i.e., sectors with no data).

Motivation

NI-PoRep will be beneficial for Filecoin in multiple ways, outlined below.

Cost reduction thanks to a simplified onboarding pipeline

PoRep is currently interactive (in order to complete sealing, an SP has to wait to receive a challenge seed from the chain) and requires ad-hoc collateral. These features represent a limitation when considering optimisation for the onboarding pipeline, such as Sealing-as-a-Service (SaaS) and the new SupraSeal sealing code. With NI-PoRep, no interaction is needed and this yields:

  • [Gas cost reduction] Current PoRep is composed of two steps: PreCommit and ProveCommit. With NI-PoRep there is no more PreCommit method and message, and only ProveCommit remains, i.e. only one step with one chain message is needed to onboard sectors to the network. This translates into a possible gas cost reduction when considering aggregated sectors; according to our estimation, current PoRep is 2.1x more expensive than NI-PoRep when aggregating 6 sectors.
  • [Lower hardware requirements] With NI-PoRep, there is no more waiting time between PreCommit and ProveCommit. This helps when using sealing software (like SupraSeal) that seals more sectors at the same time. Currently, memory requirements are increased by the fact that some data need to be stored during the waiting time. Having no waiting time implies lower memory requirements.

Trustless Sealing-as-a-Service (SaaS) becomes possible

NI-PoRep enables the full separation between computation and storage tasks. In particular, no PCD (PreCommit Deposit) is needed, which brings the following benefits:

  • Currently a SaaS Provider (the entity running the PoRep steps) needs to put down the PCD for the sector, along with the PreCommit message. The PCD will be re-paid or prepaid by the SP sending the final SNARK proof on the chain (i.e., the SP that will store the sector for the following PoST). With NI-PoRep, this level of payment interaction is not needed. In particular, this kind of simplification helps SPs and SaaS providers to delegate computation tasks (ie, PoRep can be split into specialized subtasks that get outsourced to specialized entities).
  • Enabling HDD wholesale: for an SP it would be possible to receive brand new drives with sectorKeys pre-generated using its miner_id.

PoRep secured cryptographically and not rationally

The cryptographic security of sectors proven with NI-PoRep is increased: NI-PoRep makes misbehaving cryptographically infeasible rather than irrational.

PoRep security now independent from consensus

Current PoRep is interactive and needs to get randomness from the chain. Moreover, in order to be secure, a 150-epochs-wait is needed between PreCommit and ProveCommit(WaitSeed). This is due to the fact that some consensus attacks need to be made infeasible (as putting those attacks in place would allow for faking storage). In NI-PoRep, since randomness is derived locally, there is no link anymore between PoRep and consensus attacks. This means that

  • Consensus attacks are not a concern anymore for NI-PoRep security;
  • PoRep can now work “agnostically” with any consensus protocol.

Backward compatibility

  • NI-PoRep would be a separate proof type with a different on-chain flow as current PoRep. Anyone can decide whether to use NI-PoRep or not.
  • No need for a new trusted setup.

Specification

The NI-PoRep protocol can be summarized as follows (see here for notation and here for basic algorithms):

Graph labelling and commitments (similar to the current PC1 and PC2 computation)

  1. Using ReplicaID, SP computes the labels for all layers and the replica R;
  2. SP computes the column commitments CommC , CommRLast and finally computes CommR = Poseidon_2(CommC, CommRLast);

SP locally generates NIChallengesNumber challenges and vanilla proofs;

  1. Each challenge is of the form NIChallenge_i = H(ReplicaID, CommR, i);
  2. SP computes responses for all the NIChallengesNumber challenges, which result in NIChallengesNumber vanilla proofs;

SP publishes the new NIProveCommitSector proof

  1. SP takes the NIChallengesNumber vanilla proofs and computes the corresponding SNARK proofs for these challenges and aggregates them into a single proof.
  2. SP publishes the aggregated SNARK proof and commitment CommR.

Chain verifies proof

  1. Using CommR as a seed, the chain generates NIChallengesNumber challenges and these are fed into proof verification.

Note that, same as with interactive PoRep, each sector has a SealRandEpoch that identifies the sealing epoch; chain verification needs to happen within a fixed number of epochs to be valid. For NI-PoRep we call this parameter sealChallengeEarliestand set it to be equal to the number of epochs in 30 days (more details in the Security Considerations section).

Actor changes

  • Add two new proof types to the list of proof types that can be used when submitting a new sector

    • StackedDrg32GiBV1_1_Feat_NiPoRep
    • StackedDrg64GiBV1_1_Feat_NiPoRep

  • Introduce a new method ProveCommitSectorsNI (method 36), which performs a non-interactive proof for CC sectors, without a preceding PreCommitSector message (ie, this method rejects sectors that were already pre-committed. Such sectors must be activated with one of the existing ProveCommit methods). The method can be used with seal proofs or aggregate proof (but only one option can be chosen per call); the return type is the same as ProveCommitAggregate.

    // Note no UnsealedCID because it must be "zero" data.
struct SectorNIActivationInfo {
    SealingNumber: SectorNumber, // Sector number used to generate replica id
    SealerID: ActorID, // Must be set to ID of receiving actor for now
    SealedCID: Cid, // CommR
    SectorNumber: SectorNumber, // unique id of sector in actor state
    SealRandEpoch: ChainEpoch,
    Expiration: ChainEpoch,
}

struct ProveCommitSectorsNIParams {
   // Information about sealing of each sector.
   Sectors: []SectorNIActivationInfo
   // Proof type for each seal (must be an NI-PoRep variant)
   SealProofType: RegisteredSealProof,
   // Proofs for each sector, parallel to activation manifests.
   // Exactly one of sector_proofs or aggregate_proof must be non-empty.
   SectorProofs: [][]byte,
   // Aggregate proof for all sectors.
   // Exactly one of sector_proofs or aggregate_proof must be non-empty.
   AggregateProof: []byte,
   // Proof type for aggregation, if aggregated
   AggregateProofType: Option<RegisteredAggregateProof>
   // Whether to abort if any sector activation fails.
   RequireActivationSuccess: bool,
 }

  • Related constants

    • sealChallengeEarliest set to be equal to the number of epochs in 30 days.

Proof changes

  • Add two new proof types to the list of proof types that can be used when pre-committing a new sector

    • RegisteredSealProof::StackedDrg32GiBV1_1_Feat_NiPoRep
    • RegisteredSealProof::StackedDrg64GiBV1_1_Feat_NiPoRep

  • Related constants

    • NI_porep_min_challenges set to 2253, the theoretical minimum number for 128 bits of security;
    • NUM_CHALLENGES computed as the smallest multiple of 18 that is also larger or equal to NI_porep_min_challenges (indeed in practice, due to existing contraints in the trusted SNARK setup and circuit design, the number of challenges used has to be a multiple of 18).

  • New challenge generation function:

    // That's the sector size in bytes divided by the node size (32 bytes).
const SECTOR_NODES: u32
const NUM_CHALLENGES = 2268
// The `tag` is a domain separation tag, which is just a list of bytes.
fn gen_porep_challenges(tag: [u8], ReplicaID: [u8; 32], CommR: [u8; 32]):
for i in 0..NUM_CHALLENGES {
    digest: [u8; 32] = sha256(tag || le_bytes(ReplicaID) || le_bytes(CommR) || le_bytes(i))
    bigint = u256::from_le_u32s(digest)
    challenge: u32 = (bigint % (SECTOR_NODES - 1)) + 1
}

NI-PoRep is an optional feature that can be opt-in for those interested. The previously available proofs types can be used with existing onboarding methods to continue interactive PoRep behavior.

Gas calculations

According to FIP-0013, for the verification of N aggregated Groth16 proofs (ie, circuits) the total gas charged is

  • 32GiB sector: 449900*N/10 + Constant32(N)
  • 64GiB sector: 359280*N/10 + Constant64(N),

where Constant32(N) and Constant64(N) are as follows:

Number of circuits aggregated, N Number of interactive PoRep proofs aggregated Number of NI-PoRep proofs aggregated, k Constant32(N) Constant64(N)
≤ 64 = 2^6 ≤ 6 - 103’994’170 102’581’240
≤ 128 = 2^7 ≤12 1 112’356’810 110’803’030
≤ 256 = 2^8 ≤ 25 2 122’912’610 120’803’700
≤ 512 = 2^9 ≤ 51 ≤ 4 137’559’930 134’642’130
≤1024 = 2^10 ≤ 102 ≤ 8 162’039’100 157’357’890
≤ 2048 = 2^11 ≤ 204 ≤ 16 210’960’780 203’017’690
≤ 4096 = 2^12 ≤ 409 ≤ 32 318’351’180 304’253’590
≤ 8192 = 2^13 ≤ 819 ≤ 65 528’274’980 509’880’640

Since one NI-PoRep proof (128 bits of security) corresponds to aggregating 126 circuits, the gas units for the verification of one NI-PoRep proof is given by:

32GiB sector: 44990*126 + 112356810 = 118025550

64GiB sector: 35928*126 + 110803030 = 115329958

Aggregation:

The minimum number of sector NI-PoRep proofs that may be aggregated is 2, and the maximum is 65. Using the same formula as above, the gas charged for verification of k NI-PoRep proofs (128 bits) is equal to

32GiB sector: 44990 * 126 * k + Constant32(126 * k)

64GiB sector: 35928 * 126 * k + Constant64(126 * k)

Usually GasUsed * BaseFee is burned for every message. However, for aggregated batches of proofs (for example ProveCommitSectorsAggregate and ProveReplicaUpdates3 messages) we charge an additional proportional fee introduced in FIP0013 and updated in FIP0024. With NI-PoRep, when aggregating at most 5 sectors (ie, k ≤ 5), the per sector gas cost is already larger than the current value (~34721049), and therefore in this case there is no need for any additional fee. On the other hand, when aggregating 6 or more sectors, the additional gas fee is needed again. In this case the fee is computed using the current function used for interactive proofs (PayBatchGasCharge function, as stated in FIP0024) but passing as input the number of proofs - 5.

Design Rationale

Current PoRep is interactive, and it is composed of two steps: PreCommit and ProveCommit. At PreCommit, SP puts down a collateral (PCD) and waits 150 epochs in order to receive a challenge seed from the chain, which enables the ProveCommit step. A first step to mitigate the downsides of the waiting time was the introduction of Synthetic PoRep (See FIP-0059), which reduces the size of the temporary data stored between PreCommit and ProveCommit. NI-PoRep is a further step forward, completely foregoing on-chain interaction (ie, the waiting time) and the need of PCD by allowing SP to locally generate challenges instead of using on-chain randomness.

NI-PoRep has little downside with respect to the status quo: it removes PreCommit at the cost of augmenting C2 (ie SNARK generation) costs, which would result in a limited cost increase looking at onboarding costs as a whole. Indeed, NI-PoRep requires 12.8x more PoRep Challenges, which translates into an 12.8x SNARK proving overhead. We analyzed how this SNARK computation overhead affects overall costs. The conclusion is that considering PC1+PC2+C1+C2 and storage costs (i.e. not considering maintenance costs), a NI-PoRep sector with 128 bits of security is 5% more expensive than an Interactive PoRep sector when sector duration is 3y. See full analysis here.

The new onboarding method ProveCommitSectorsNI is restricted to CC-sectors. We decided for this design for the following reasons:

  1. We believe that the main users of NI-PoRep will be SaaS Providers, which will use NI-PoRep for CC sectors anyway. In the SaaS scenario, the flow where the SP ships the data over to an SaaS Provider and then gets back the sealed data (replica) seems more complex (and therefore more expensive in practice) than the flow where the SaaS Provider distributes CC sectors and then the SP can snap the data later. This should be especially true if FIP0082 (SuperSnap) were deployed in the network.
  2. We think that restricting the new onboarding method to CC sectors can simplify the storage pipeline for Filecoin. Currently, there are quite a few different flows for onboarding data; having such diversity (ie, complexity) can be a source of risk and can slow down protocol development. It also complicates the SP stack software and new SP onboarding. If NI-PoRep (restricted to CC sectors) is widely adopted, we may come to consider the "CC+Snap" flow as the standard and deprecate older onboarding methods. Note that beyond a simplified pipeline, the "CC+Snap" flow allows for the possibility of new features. For a sector that is onboarded as CC, we have the SectorKey proved on-chain, meaning we can support re-snap (sector data to be replaced more than once) and proof-of-access to unsealed copy protocols.
  3. Implementation of the NI-PoRep onboarding method is simplified by only supporting CC sectors. Data sectors add significant code complexity in the miner actor to support two different ways of committing to data.

Backwards Compatibility

NI-PoRep would become a new proof type with a different on-chain flow as current PoRep (due to the removal of the PreCommit step). A new method, ProveCommitSectorNI is added and the existing ProveCommit methods (ie, ProveCommitSector, ProveCommitAggregate, ProveCommitSectors3) will not accept NI-PoRep proof.

Test Cases

Run the code for NI-PoRep and check if the resulting proof cryptographically verifies (ie, run the verification software). Test cases will be included with implementation to be presented.

Security Considerations

Fiat-Shamir Heuristic and its effects

NI-PoRep is based on the well-known Fiat-Shamir Heuristic, which allows for having a non interactive protocol starting from any 3-message interactive protocol. However, the heuristic requires the original interactive protocol to have at least 80 bits of security (instead of the current 10), and preferably 128. In order to have long term security we propose to opt for 128 bits: since the security level in PoRep is given by the number of challenges, this means that for NI-PoRep we need  NIChallengesNumber to be at least 2253.

How to take SealRandomness for ensuring sectors are anchored to the chain

The epoch in which sealing took place is identified by the corresponding chain reference, called SealRandomness which is the anchor of the sector to the chain. SealRandomness must be taken from the VRF chain, in order to ensure that no other fork can replay the Seal.

On one hand, SealRandomness needs to be taken from finalized blocks, but on the other hand it can not be taken farther back than necessary (in order to protect against long-range attacks). This means that SealRandomness needs to be verified.

Given NI-PoRep removes onchain interaction, SealRandomness verification becomes more difficult, but still necessary to protect the chain against the same class of attacks mentioned above.

We set up a validity time window which holds for all the sectors committed onchain together. We call this time window to be sealChallengeEarliest epochs and set its value to be the number of epochs in 30 days, which is coherent with the current MaxProveCommitDuration as per FIP-0013.

This means that, if PC1, PC2, C1, C2 happen locally over time resulting into different sectors sealed in different moments in time and committed on-chain at the end of the process, all sectors committed together should have a randomness which is not older than sealChallengeEarliest epochs in the past. As a result, a NI-PoRep step needs to be completed within sealChallengeEarliest epochs overall to be valid.

Addressing concerns due to interaction removal

Concerns were raised over the possibility that a malicious party wanting to take over the network could potentially keep accumulating sectors locally before onboarding them to the network all at once. Our analysis shows that no additional security risks are introduced with respect to Interactive PoRep. The security of both pathways can be improved by decoupling the power table lookback from consensus, which may be a subject for a future FIP.

Incentive Considerations

NI-PoRep will implement the batch balancer currently in place. See FIP-0013 for details.

Product Considerations

NI-PoRep represents another step forward after the introduction of Synthetic PoRep (see FIP-0059).

As already mentioned above, NI-PoRep is a prerequisite to unblock new use cases like trustless SaaS and HDD wholesale as well as allowing for SupraSeal software employment at full potential. Moreovere, note that enabling trustless SaaS can lower the entry bar for new Storage Providers to join the network.

Comparison against Interactive PoRep

In order to have a good cost estimation, we ran benchmarks on different systems. We observe that the SNARK phase of NI-PoRep is consistently 12.5x-13x slower than Interactive PoRep, in line with expectations.

Impact of SupraSeal

The SNARK phase consists of two distinct steps:

  • The synthesis, which is a memory- and CPU-heavy operation
  • The proving, which is GPU-heavy

The SupraSeal implementation only targets the proving, where we observe speedups between 6-10x. When the synthesis is also taken into account, then the overall improvements are 3-5x. This means that NI-PoRep with SupraSeal is only 2.5-4.5x slower than Interactive PoRep without SupraSeal.

The ratio between synthesis and proving time depends on the GPU, and consequently so does the speedup. In particular, synthesis time tends to be dominant in newer GPUs, reducing the overall speedup.

For benchmark data, see: SupraSeal C2 benchmarks.

Implementations

Implementation in progress (TODO).

Copyright Waiver

Copyright and related rights waived via CC0.