fip | title | author | discussions-to | status | type | category | created |
---|---|---|---|---|---|---|---|
0090 |
Non-Interactive PoRep |
luca (@lucaniz), kuba (@Kubuxu), nicola (@nicola), nemo (@cryptonemo), volker (@vmx), irene (@irenegia) |
Superseded |
Technical |
Core |
2023-12-18 |
This proposal presents a new PoRep protocol (Non-Interactive PoRep) that removes PreCommit
when onboarding Committed Capacity (CC) sectors. As a result, we have
- Simplified storage onboarding pipeline;
- Trustless separation between storage and computing: the proving tasks associated with sector onboarding can be outsourced;
- Unblocking SupraSeal's full potential.
Non-Interactive PoRep (NI-PoRep) allows the removal of on-chain interaction when onboarding Committed Capacity (CC) sectors by changing the way PoRep challenges are generated.
The protocol allows Storage Providers (SPs) to locally generate PoRep challenges instead of using on-chain randomness.
On the one hand, this feature of the protocol allows for a drastically simpler onboarding pipeline. On the other hand, it requires a higher number of PoRep challenges (2268 per SDR layer instead of the current 180 per SDR layer) in order to preserve network security.
NI-PoRep is proposed as an optional feature; the previously available proof types will still be supported. NI-PoRep is restricted to CC sectors (i.e., sectors with no data).
NI-PoRep will be beneficial for Filecoin in multiple ways, outlined below.
PoRep is currently interactive (in order to complete sealing, an SP has to wait to receive a challenge seed from the chain) and requires ad-hoc collateral. These features represent a limitation when considering optimisation for the onboarding pipeline, such as Sealing-as-a-Service (SaaS) and the new SupraSeal sealing code. With NI-PoRep, no interaction is needed and this yields:
- [Gas cost reduction] Current PoRep is composed of two steps: PreCommit and ProveCommit. With NI-PoRep there is no more
PreCommit
method and message, and onlyProveCommit
remains, i.e. only one step with one chain message is needed to onboard sectors to the network. This translates into a possible gas cost reduction when considering aggregated sectors; according to our estimation, current PoRep is 2.1x more expensive than NI-PoRep when aggregating 6 sectors. - [Lower hardware requirements] With NI-PoRep, there is no more waiting time between
PreCommit
andProveCommit
. This helps when using sealing software (like SupraSeal) that seals more sectors at the same time. Currently, memory requirements are increased by the fact that some data need to be stored during the waiting time. Having no waiting time implies lower memory requirements.
NI-PoRep enables the full separation between computation and storage tasks. In particular, no PCD
(PreCommit Deposit) is needed, which brings the following benefits:
- Currently a SaaS Provider (the entity running the PoRep steps) needs to put down the PCD for the sector, along with the PreCommit message. The PCD will be re-paid or prepaid by the SP sending the final SNARK proof on the chain (i.e., the SP that will store the sector for the following PoST). With NI-PoRep, this level of payment interaction is not needed. In particular, this kind of simplification helps SPs and SaaS providers to delegate computation tasks (ie, PoRep can be split into specialized subtasks that get outsourced to specialized entities).
- Enabling HDD wholesale: for an SP it would be possible to receive brand new drives with
sectorKeys
pre-generated using itsminer_id
.
The cryptographic security of sectors proven with NI-PoRep is increased: NI-PoRep makes misbehaving cryptographically infeasible rather than irrational.
Current PoRep is interactive and needs to get randomness from the chain. Moreover, in order to be secure, a 150-epochs-wait is needed between PreCommit
and ProveCommit
(WaitSeed). This is due to the fact that some consensus attacks need to be made infeasible (as putting those attacks in place would allow for faking storage).
In NI-PoRep, since randomness is derived locally, there is no link anymore between PoRep and consensus attacks. This means that
- Consensus attacks are not a concern anymore for NI-PoRep security;
- PoRep can now work “agnostically” with any consensus protocol.
- NI-PoRep would be a separate proof type with a different on-chain flow as current PoRep. Anyone can decide whether to use NI-PoRep or not.
- No need for a new trusted setup.
The NI-PoRep protocol can be summarized as follows (see here for notation and here for basic algorithms):
Graph labelling and commitments (similar to the current PC1 and PC2 computation)
- Using
ReplicaID
, SP computes the labels for all layers and the replica R; - SP computes the column commitments
CommC
,CommRLast
and finally computesCommR = Poseidon_2(CommC, CommRLast)
;
SP locally generates NIChallengesNumber
challenges and vanilla proofs;
- Each challenge is of the form
NIChallenge_i = H(ReplicaID, CommR, i)
; - SP computes responses for all the
NIChallengesNumber
challenges, which result inNIChallengesNumber
vanilla proofs;
SP publishes the new NIProveCommitSector
proof
- SP takes the
NIChallengesNumber
vanilla proofs and computes the corresponding SNARK proofs for these challenges and aggregates them into a single proof. - SP publishes the aggregated SNARK proof and commitment
CommR
.
Chain verifies proof
- Using
CommR
as a seed, the chain generatesNIChallengesNumber
challenges and these are fed into proof verification.
Note that, same as with interactive PoRep, each sector has a SealRandEpoch
that identifies the sealing epoch; chain verification needs to happen within a fixed number of epochs to be valid. For NI-PoRep we call this parameter sealChallengeEarliest
and set it to be equal to the number of epochs in 30 days (more details in the Security Considerations section).
-
Add two new proof types to the list of proof types that can be used when submitting a new sector
StackedDrg32GiBV1_1_Feat_NiPoRep
StackedDrg64GiBV1_1_Feat_NiPoRep
-
Introduce a new method
ProveCommitSectorsNI
(method 36), which performs a non-interactive proof for CC sectors, without a preceding PreCommitSector message (ie, this method rejects sectors that were already pre-committed. Such sectors must be activated with one of the existing ProveCommit methods). The method can be used with seal proofs or aggregate proof (but only one option can be chosen per call); the return type is the same asProveCommitAggregate
.// Note no UnsealedCID because it must be "zero" data. struct SectorNIActivationInfo { SealingNumber: SectorNumber, // Sector number used to generate replica id SealerID: ActorID, // Must be set to ID of receiving actor for now SealedCID: Cid, // CommR SectorNumber: SectorNumber, // unique id of sector in actor state SealRandEpoch: ChainEpoch, Expiration: ChainEpoch, } struct ProveCommitSectorsNIParams { // Information about sealing of each sector. Sectors: []SectorNIActivationInfo // Proof type for each seal (must be an NI-PoRep variant) SealProofType: RegisteredSealProof, // Proofs for each sector, parallel to activation manifests. // Exactly one of sector_proofs or aggregate_proof must be non-empty. SectorProofs: [][]byte, // Aggregate proof for all sectors. // Exactly one of sector_proofs or aggregate_proof must be non-empty. AggregateProof: []byte, // Proof type for aggregation, if aggregated AggregateProofType: Option<RegisteredAggregateProof> // Whether to abort if any sector activation fails. RequireActivationSuccess: bool, }
-
Related constants
sealChallengeEarliest
set to be equal to the number of epochs in 30 days.
-
Add two new proof types to the list of proof types that can be used when pre-committing a new sector
RegisteredSealProof::StackedDrg32GiBV1_1_Feat_NiPoRep
RegisteredSealProof::StackedDrg64GiBV1_1_Feat_NiPoRep
-
Related constants
NI_porep_min_challenges
set to 2253, the theoretical minimum number for 128 bits of security;NUM_CHALLENGES
computed as the smallest multiple of 18 that is also larger or equal toNI_porep_min_challenges
(indeed in practice, due to existing contraints in the trusted SNARK setup and circuit design, the number of challenges used has to be a multiple of 18).
-
New challenge generation function:
// That's the sector size in bytes divided by the node size (32 bytes). const SECTOR_NODES: u32 const NUM_CHALLENGES = 2268 // The `tag` is a domain separation tag, which is just a list of bytes. fn gen_porep_challenges(tag: [u8], ReplicaID: [u8; 32], CommR: [u8; 32]): for i in 0..NUM_CHALLENGES { digest: [u8; 32] = sha256(tag || le_bytes(ReplicaID) || le_bytes(CommR) || le_bytes(i)) bigint = u256::from_le_u32s(digest) challenge: u32 = (bigint % (SECTOR_NODES - 1)) + 1 }
NI-PoRep is an optional feature that can be opt-in for those interested. The previously available proofs types can be used with existing onboarding methods to continue interactive PoRep behavior.
According to FIP-0013, for the verification of N aggregated Groth16 proofs (ie, circuits) the total gas charged is
- 32GiB sector: 449900*N/10 + Constant32(N)
- 64GiB sector: 359280*N/10 + Constant64(N),
where Constant32(N) and Constant64(N) are as follows:
Number of circuits aggregated, N | Number of interactive PoRep proofs aggregated | Number of NI-PoRep proofs aggregated, k | Constant32(N) | Constant64(N) |
---|---|---|---|---|
≤ 64 = 2^6 | ≤ 6 | - | 103’994’170 | 102’581’240 |
≤ 128 = 2^7 | ≤12 | 1 | 112’356’810 | 110’803’030 |
≤ 256 = 2^8 | ≤ 25 | 2 | 122’912’610 | 120’803’700 |
≤ 512 = 2^9 | ≤ 51 | ≤ 4 | 137’559’930 | 134’642’130 |
≤1024 = 2^10 | ≤ 102 | ≤ 8 | 162’039’100 | 157’357’890 |
≤ 2048 = 2^11 | ≤ 204 | ≤ 16 | 210’960’780 | 203’017’690 |
≤ 4096 = 2^12 | ≤ 409 | ≤ 32 | 318’351’180 | 304’253’590 |
≤ 8192 = 2^13 | ≤ 819 | ≤ 65 | 528’274’980 | 509’880’640 |
Since one NI-PoRep proof (128 bits of security) corresponds to aggregating 126 circuits, the gas units for the verification of one NI-PoRep proof is given by:
32GiB sector: 44990*126 + 112356810 = 118025550
64GiB sector: 35928*126 + 110803030 = 115329958
Aggregation:
The minimum number of sector NI-PoRep proofs that may be aggregated is 2, and the maximum is 65. Using the same formula as above, the gas charged for verification of k NI-PoRep proofs (128 bits) is equal to
32GiB sector: 44990 * 126 * k + Constant32(126 * k)
64GiB sector: 35928 * 126 * k + Constant64(126 * k)
Usually GasUsed * BaseFee
is burned for every message. However, for aggregated batches of proofs (for example ProveCommitSectorsAggregate and ProveReplicaUpdates3 messages) we charge an additional proportional fee introduced in FIP0013 and updated in FIP0024. With NI-PoRep, when aggregating at most 5 sectors (ie, k ≤ 5), the per sector gas cost is already larger than the current value (~34721049), and therefore in this case there is no need for any additional fee. On the other hand, when aggregating 6 or more sectors, the additional gas fee is needed again. In this case the fee is computed using the current function used for interactive proofs (PayBatchGasCharge
function, as stated in FIP0024) but passing as input the number of proofs - 5.
Current PoRep is interactive, and it is composed of two steps: PreCommit and ProveCommit. At PreCommit, SP puts down a collateral (PCD) and waits 150 epochs in order to receive a challenge seed from the chain, which enables the ProveCommit step. A first step to mitigate the downsides of the waiting time was the introduction of Synthetic PoRep (See FIP-0059), which reduces the size of the temporary data stored between PreCommit and ProveCommit. NI-PoRep is a further step forward, completely foregoing on-chain interaction (ie, the waiting time) and the need of PCD by allowing SP to locally generate challenges instead of using on-chain randomness.
NI-PoRep has little downside with respect to the status quo: it removes PreCommit at the cost of augmenting C2 (ie SNARK generation) costs, which would result in a limited cost increase looking at onboarding costs as a whole. Indeed, NI-PoRep requires 12.8x more PoRep Challenges, which translates into an 12.8x SNARK proving overhead. We analyzed how this SNARK computation overhead affects overall costs. The conclusion is that considering PC1+PC2+C1+C2 and storage costs (i.e. not considering maintenance costs), a NI-PoRep sector with 128 bits of security is 5% more expensive than an Interactive PoRep sector when sector duration is 3y. See full analysis here.
The new onboarding method ProveCommitSectorsNI
is restricted to CC-sectors. We decided for this design for the following reasons:
- We believe that the main users of NI-PoRep will be SaaS Providers, which will use NI-PoRep for CC sectors anyway. In the SaaS scenario, the flow where the SP ships the data over to an SaaS Provider and then gets back the sealed data (replica) seems more complex (and therefore more expensive in practice) than the flow where the SaaS Provider distributes CC sectors and then the SP can snap the data later. This should be especially true if FIP0082 (SuperSnap) were deployed in the network.
- We think that restricting the new onboarding method to CC sectors can simplify the storage pipeline for Filecoin. Currently, there are quite a few different flows for onboarding data; having such diversity (ie, complexity) can be a source of risk and can slow down protocol development. It also complicates the SP stack software and new SP onboarding. If NI-PoRep (restricted to CC sectors) is widely adopted, we may come to consider the "CC+Snap" flow as the standard and deprecate older onboarding methods. Note that beyond a simplified pipeline, the "CC+Snap" flow allows for the possibility of new features. For a sector that is onboarded as CC, we have the
SectorKey
proved on-chain, meaning we can support re-snap (sector data to be replaced more than once) and proof-of-access to unsealed copy protocols. - Implementation of the NI-PoRep onboarding method is simplified by only supporting CC sectors. Data sectors add significant code complexity in the miner actor to support two different ways of committing to data.
NI-PoRep would become a new proof type with a different on-chain flow as current PoRep (due to the removal of the PreCommit
step). A new method, ProveCommitSectorNI
is added and the existing ProveCommit methods (ie, ProveCommitSector
, ProveCommitAggregate
, ProveCommitSectors3
) will not accept NI-PoRep proof.
Run the code for NI-PoRep and check if the resulting proof cryptographically verifies (ie, run the verification software). Test cases will be included with implementation to be presented.
NI-PoRep is based on the well-known Fiat-Shamir Heuristic, which allows for having a non interactive protocol starting from any 3-message interactive protocol. However, the heuristic requires the original interactive protocol to have at least 80 bits of security (instead of the current 10), and preferably 128. In order to have long term security we propose to opt for 128 bits: since the security level in PoRep is given by the number of challenges, this means that for NI-PoRep we need NIChallengesNumber
to be at least 2253.
The epoch in which sealing took place is identified by the corresponding chain reference, called SealRandomness
which is the anchor of the sector to the chain. SealRandomness
must be taken from the VRF chain, in order to ensure that no other fork can replay the Seal.
On one hand, SealRandomness
needs to be taken from finalized blocks, but on the other hand it can not be taken farther back than necessary (in order to protect against long-range attacks). This means that SealRandomness
needs to be verified.
Given NI-PoRep removes onchain interaction, SealRandomness
verification becomes more difficult, but still necessary to protect the chain against the same class of attacks mentioned above.
We set up a validity time window which holds for all the sectors committed onchain together. We call this time window to be sealChallengeEarliest
epochs and set its value to be the number of epochs in 30 days, which is coherent with the current MaxProveCommitDuration
as per FIP-0013.
This means that, if PC1, PC2, C1, C2 happen locally over time resulting into different sectors sealed in different moments in time and committed on-chain at the end of the process, all sectors committed together should have a randomness which is not older than sealChallengeEarliest
epochs in the past. As a result, a NI-PoRep step needs to be completed within sealChallengeEarliest
epochs overall to be valid.
Concerns were raised over the possibility that a malicious party wanting to take over the network could potentially keep accumulating sectors locally before onboarding them to the network all at once. Our analysis shows that no additional security risks are introduced with respect to Interactive PoRep. The security of both pathways can be improved by decoupling the power table lookback from consensus, which may be a subject for a future FIP.
NI-PoRep will implement the batch balancer currently in place. See FIP-0013 for details.
NI-PoRep represents another step forward after the introduction of Synthetic PoRep (see FIP-0059).
As already mentioned above, NI-PoRep is a prerequisite to unblock new use cases like trustless SaaS and HDD wholesale as well as allowing for SupraSeal software employment at full potential. Moreovere, note that enabling trustless SaaS can lower the entry bar for new Storage Providers to join the network.
In order to have a good cost estimation, we ran benchmarks on different systems. We observe that the SNARK phase of NI-PoRep is consistently 12.5x-13x slower than Interactive PoRep, in line with expectations.
The SNARK phase consists of two distinct steps:
- The synthesis, which is a memory- and CPU-heavy operation
- The proving, which is GPU-heavy
The SupraSeal implementation only targets the proving, where we observe speedups between 6-10x. When the synthesis is also taken into account, then the overall improvements are 3-5x. This means that NI-PoRep with SupraSeal is only 2.5-4.5x slower than Interactive PoRep without SupraSeal.
The ratio between synthesis and proving time depends on the GPU, and consequently so does the speedup. In particular, synthesis time tends to be dominant in newer GPUs, reducing the overall speedup.
For benchmark data, see: SupraSeal C2 benchmarks.
Implementation in progress (TODO).
Copyright and related rights waived via CC0.