Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement Synthetic PoRep #11258

Merged
merged 52 commits into from
Sep 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
c7b9633
feat: introduce local nv21 skeleton
rjan90 Jul 7, 2023
68a03c4
Set local devnet-genesis to nv20
rjan90 Jul 11, 2023
1db45f4
synthetic-porep subpackages
snadrus May 26, 2023
b43665a
synthPoRep in PreferredSeal..()
snadrus May 29, 2023
f4fe4ca
synthetic porep config
snadrus May 29, 2023
0803854
Impl synthPoRep
snadrus Jun 1, 2023
5ed8a5c
clearSynthProofs
snadrus Jun 1, 2023
7f306a5
Synthetic PoRep Impl
snadrus Jun 9, 2023
8281b26
Synthetic PoRep provider interface solution
snadrus Jun 9, 2023
1a799ae
fixes
snadrus Jun 28, 2023
a43f066
fix deps
magik6k Jul 12, 2023
c7dd001
ffiwrapper: Synth cleanup fixes
magik6k Jul 12, 2023
518c2cd
rename synth porep config to a shorter name
magik6k Jul 12, 2023
2f19a5f
gen, fix build with new state-types
magik6k Jul 12, 2023
f8c1510
itests: Add test for synth-porep pledge
magik6k Jul 12, 2023
c58ab3e
more correct actor bundles
magik6k Jul 12, 2023
50cf197
Make Synthetic PoRep work
magik6k Jul 13, 2023
85efa3e
Use state-types with correct v12 migration
magik6k Jul 13, 2023
f7d51da
docsgen
magik6k Jul 13, 2023
b649ebd
TestPledgeSynth: must-post mining
magik6k Jul 13, 2023
a396947
ffiwrapper: Test synth porep
magik6k Jul 13, 2023
34996df
update state-types to include fixed MaxProveCommitDuration policy
magik6k Jul 13, 2023
85b78f9
update ffi to use lowercased synth challenges file name
magik6k Jul 13, 2023
9eb4bb8
make gen
magik6k Jul 13, 2023
5743ed0
test: fix fevm recursive call
Stebalien Jul 14, 2023
4534416
Update butterfly-schedule
rjan90 Jul 17, 2023
9aaa00b
Add butterfly artifacts
rjan90 Jul 17, 2023
c4b469b
Update butterfly-bootstraps
rjan90 Jul 18, 2023
7adf823
add synthPoRep support to bench simple
magik6k Jul 18, 2023
1eca520
Added support for Filecoin network version 21 (v12 state types)
snissn Aug 2, 2023
4f0e2f3
Added CPU profiling option to lotus-shed
snissn Aug 2, 2023
c8f8095
update ffi with synth porep fixes
magik6k Aug 3, 2023
607e8cc
Updated Butterfly-assets
rjan90 Aug 3, 2023
7f4d8b3
add go-state-types with nv21 migration code
snissn Aug 3, 2023
dbb166b
Set upgrade epoch on Butterfly earlier
rjan90 Aug 7, 2023
9539f59
Add new butterfly-assets
rjan90 Aug 7, 2023
cd9fdcf
update go state types to use the experiment speed up branch in order …
snissn Aug 7, 2023
0b68348
initial skeleton for nv21 migrations test. migration succesfully exec…
snissn Aug 10, 2023
f57b5bd
spelling
snissn Aug 10, 2023
fa0e512
Merge pull request #11149 from filecoin-project/feat/nv21-migrations-…
arajasek Aug 14, 2023
e6b0231
Add more post-migration sanity checks
arajasek Aug 14, 2023
1eec816
Merge pull request #11168 from filecoin-project/asr/more-migration-check
arajasek Aug 17, 2023
37f6cd3
Register SynthPoRep in ResourceTable
rjan90 Aug 29, 2023
024dd5e
Merge branch 'feat/nv21' into feat/implement-SyntheticPoRep
rjan90 Sep 8, 2023
ea81c05
update ffi
magik6k Sep 19, 2023
3f076bf
fix exec traces with new ffi
magik6k Sep 19, 2023
01de96d
fix lint
magik6k Sep 19, 2023
a016f4c
mod tidy
magik6k Sep 19, 2023
3cebd99
pipeline: debugging C2 with synth porep
magik6k Sep 19, 2023
935fbc5
ci: get params in pledge tests
magik6k Sep 19, 2023
efd1e47
itests: remove obsolete nv12 pledge test
magik6k Sep 19, 2023
5ce0285
bundle-gen
magik6k Sep 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -912,6 +912,8 @@ workflows:
- build
suite: itest-sector_pledge
target: "./itests/sector_pledge_test.go"
get-params: true

- test:
name: test-itest-sector_terminate
requires:
Expand Down
2 changes: 1 addition & 1 deletion .circleci/template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -545,7 +545,7 @@ workflows:
[[- if or (eq $name "worker") (eq $name "deals_concurrent") (eq $name "wdpost_worker_config")]]
executor: golang-2xl
[[- end]]
[[- if (eq $name "wdpost")]]
[[- if or (eq $name "wdpost") (eq $name "sector_pledge")]]
get-params: true
[[end]]
[[- end ]][[- end]]
Expand Down
Binary file modified build/actors/v12.tar.zst
Binary file not shown.
4 changes: 2 additions & 2 deletions build/bootstrap/butterflynet.pi
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
/dns4/bootstrap-0.butterfly.fildev.network/tcp/1347/p2p/12D3KooWD5mtdmjHQ1Puj9Md7SEfoa7kWMpwqUhAKsyYsBP56LQC
/dns4/bootstrap-1.butterfly.fildev.network/tcp/1347/p2p/12D3KooWEoYPkm6o87ES6AppFY7d7WHJUQg7XVPRAyQZjEU31efQ
/dns4/bootstrap-0.butterfly.fildev.network/tcp/1347/p2p/12D3KooWMH8fFcfL6K7dwRo2wqa4UjdM5pxg9DAnoWPrVRx4Dioj
/dns4/bootstrap-1.butterfly.fildev.network/tcp/1347/p2p/12D3KooWArCrw1TLyJ5r37pYvHgv9kkXNC8tu2t1Sx7hyoCHpRje
280 changes: 140 additions & 140 deletions build/builtin_actors_gen.go

Large diffs are not rendered by default.

Binary file modified build/genesis/butterflynet.car
Binary file not shown.
Binary file modified build/openrpc/full.json.gz
Binary file not shown.
Binary file modified build/openrpc/gateway.json.gz
Binary file not shown.
Binary file modified build/openrpc/miner.json.gz
Binary file not shown.
Binary file modified build/openrpc/worker.json.gz
Binary file not shown.
9 changes: 5 additions & 4 deletions build/params_butterfly.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ var DrandSchedule = map[abi.ChainEpoch]DrandEnum{
0: DrandMainnet,
}

const GenesisNetworkVersion = network.Version18
const GenesisNetworkVersion = network.Version20

var NetworkBundle = "butterflynet"
var BundleOverrides map[actorstypes.Version]string
Expand Down Expand Up @@ -52,10 +52,11 @@ const UpgradeOhSnapHeight = -18
const UpgradeSkyrHeight = -19
const UpgradeSharkHeight = -20
const UpgradeHyggeHeight = -21
const UpgradeLightningHeight = -22
const UpgradeThunderHeight = -23

const UpgradeLightningHeight = 50

const UpgradeThunderHeight = UpgradeLightningHeight + 360
// 2023-08-07
const UpgradeWatermelonHeight = 900

// ??????????
const UpgradeWatermelonHeight = 999999999999999
Expand Down
29 changes: 23 additions & 6 deletions chain/actors/builtin/miner/actor.go.template
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ type Partition interface {

type SectorOnChainInfo = minertypes.SectorOnChainInfo

func PreferredSealProofTypeFromWindowPoStType(nver network.Version, proof abi.RegisteredPoStProof) (abi.RegisteredSealProof, error) {
func PreferredSealProofTypeFromWindowPoStType(nver network.Version, proof abi.RegisteredPoStProof, configWantSynthetic bool) (abi.RegisteredSealProof, error) {
// We added support for the new proofs in network version 7, and removed support for the old
// ones in network version 8.
if nver < network.Version7 {
Expand All @@ -174,17 +174,34 @@ func PreferredSealProofTypeFromWindowPoStType(nver network.Version, proof abi.Re
}
}

if nver < MinSyntheticPoRepVersion || !configWantSynthetic {
switch proof {
case abi.RegisteredPoStProof_StackedDrgWindow2KiBV1, abi.RegisteredPoStProof_StackedDrgWindow2KiBV1_1:
return abi.RegisteredSealProof_StackedDrg2KiBV1_1, nil
case abi.RegisteredPoStProof_StackedDrgWindow8MiBV1, abi.RegisteredPoStProof_StackedDrgWindow8MiBV1_1:
return abi.RegisteredSealProof_StackedDrg8MiBV1_1, nil
case abi.RegisteredPoStProof_StackedDrgWindow512MiBV1, abi.RegisteredPoStProof_StackedDrgWindow512MiBV1_1:
return abi.RegisteredSealProof_StackedDrg512MiBV1_1, nil
case abi.RegisteredPoStProof_StackedDrgWindow32GiBV1, abi.RegisteredPoStProof_StackedDrgWindow32GiBV1_1:
return abi.RegisteredSealProof_StackedDrg32GiBV1_1, nil
case abi.RegisteredPoStProof_StackedDrgWindow64GiBV1, abi.RegisteredPoStProof_StackedDrgWindow64GiBV1_1:
return abi.RegisteredSealProof_StackedDrg64GiBV1_1, nil
default:
return -1, xerrors.Errorf("unrecognized window post type: %d", proof)
}
}

switch proof {
case abi.RegisteredPoStProof_StackedDrgWindow2KiBV1, abi.RegisteredPoStProof_StackedDrgWindow2KiBV1_1:
return abi.RegisteredSealProof_StackedDrg2KiBV1_1, nil
return abi.RegisteredSealProof_StackedDrg2KiBV1_1_Feat_SyntheticPoRep, nil
case abi.RegisteredPoStProof_StackedDrgWindow8MiBV1, abi.RegisteredPoStProof_StackedDrgWindow8MiBV1_1:
return abi.RegisteredSealProof_StackedDrg8MiBV1_1, nil
return abi.RegisteredSealProof_StackedDrg8MiBV1_1_Feat_SyntheticPoRep, nil
case abi.RegisteredPoStProof_StackedDrgWindow512MiBV1, abi.RegisteredPoStProof_StackedDrgWindow512MiBV1_1:
return abi.RegisteredSealProof_StackedDrg512MiBV1_1, nil
return abi.RegisteredSealProof_StackedDrg512MiBV1_1_Feat_SyntheticPoRep, nil
case abi.RegisteredPoStProof_StackedDrgWindow32GiBV1, abi.RegisteredPoStProof_StackedDrgWindow32GiBV1_1:
return abi.RegisteredSealProof_StackedDrg32GiBV1_1, nil
return abi.RegisteredSealProof_StackedDrg32GiBV1_1_Feat_SyntheticPoRep, nil
case abi.RegisteredPoStProof_StackedDrgWindow64GiBV1, abi.RegisteredPoStProof_StackedDrgWindow64GiBV1_1:
return abi.RegisteredSealProof_StackedDrg64GiBV1_1, nil
return abi.RegisteredSealProof_StackedDrg64GiBV1_1_Feat_SyntheticPoRep, nil
default:
return -1, xerrors.Errorf("unrecognized window post type: %d", proof)
}
Expand Down
29 changes: 23 additions & 6 deletions chain/actors/builtin/miner/miner.go
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ type Partition interface {

type SectorOnChainInfo = minertypes.SectorOnChainInfo

func PreferredSealProofTypeFromWindowPoStType(nver network.Version, proof abi.RegisteredPoStProof) (abi.RegisteredSealProof, error) {
func PreferredSealProofTypeFromWindowPoStType(nver network.Version, proof abi.RegisteredPoStProof, configWantSynthetic bool) (abi.RegisteredSealProof, error) {
// We added support for the new proofs in network version 7, and removed support for the old
// ones in network version 8.
if nver < network.Version7 {
Expand All @@ -234,17 +234,34 @@ func PreferredSealProofTypeFromWindowPoStType(nver network.Version, proof abi.Re
}
}

if nver < MinSyntheticPoRepVersion || !configWantSynthetic {
switch proof {
case abi.RegisteredPoStProof_StackedDrgWindow2KiBV1, abi.RegisteredPoStProof_StackedDrgWindow2KiBV1_1:
return abi.RegisteredSealProof_StackedDrg2KiBV1_1, nil
case abi.RegisteredPoStProof_StackedDrgWindow8MiBV1, abi.RegisteredPoStProof_StackedDrgWindow8MiBV1_1:
return abi.RegisteredSealProof_StackedDrg8MiBV1_1, nil
case abi.RegisteredPoStProof_StackedDrgWindow512MiBV1, abi.RegisteredPoStProof_StackedDrgWindow512MiBV1_1:
return abi.RegisteredSealProof_StackedDrg512MiBV1_1, nil
case abi.RegisteredPoStProof_StackedDrgWindow32GiBV1, abi.RegisteredPoStProof_StackedDrgWindow32GiBV1_1:
return abi.RegisteredSealProof_StackedDrg32GiBV1_1, nil
case abi.RegisteredPoStProof_StackedDrgWindow64GiBV1, abi.RegisteredPoStProof_StackedDrgWindow64GiBV1_1:
return abi.RegisteredSealProof_StackedDrg64GiBV1_1, nil
default:
return -1, xerrors.Errorf("unrecognized window post type: %d", proof)
}
}

switch proof {
case abi.RegisteredPoStProof_StackedDrgWindow2KiBV1, abi.RegisteredPoStProof_StackedDrgWindow2KiBV1_1:
return abi.RegisteredSealProof_StackedDrg2KiBV1_1, nil
return abi.RegisteredSealProof_StackedDrg2KiBV1_1_Feat_SyntheticPoRep, nil
case abi.RegisteredPoStProof_StackedDrgWindow8MiBV1, abi.RegisteredPoStProof_StackedDrgWindow8MiBV1_1:
return abi.RegisteredSealProof_StackedDrg8MiBV1_1, nil
return abi.RegisteredSealProof_StackedDrg8MiBV1_1_Feat_SyntheticPoRep, nil
case abi.RegisteredPoStProof_StackedDrgWindow512MiBV1, abi.RegisteredPoStProof_StackedDrgWindow512MiBV1_1:
return abi.RegisteredSealProof_StackedDrg512MiBV1_1, nil
return abi.RegisteredSealProof_StackedDrg512MiBV1_1_Feat_SyntheticPoRep, nil
case abi.RegisteredPoStProof_StackedDrgWindow32GiBV1, abi.RegisteredPoStProof_StackedDrgWindow32GiBV1_1:
return abi.RegisteredSealProof_StackedDrg32GiBV1_1, nil
return abi.RegisteredSealProof_StackedDrg32GiBV1_1_Feat_SyntheticPoRep, nil
case abi.RegisteredPoStProof_StackedDrgWindow64GiBV1, abi.RegisteredPoStProof_StackedDrgWindow64GiBV1_1:
return abi.RegisteredSealProof_StackedDrg64GiBV1_1, nil
return abi.RegisteredSealProof_StackedDrg64GiBV1_1_Feat_SyntheticPoRep, nil
default:
return -1, xerrors.Errorf("unrecognized window post type: %d", proof)
}
Expand Down
38 changes: 32 additions & 6 deletions chain/actors/builtin/miner/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ import (
"github.com/filecoin-project/go-state-types/network"
)

var MinSyntheticPoRepVersion = network.Version21

func AllPartSectors(mas State, sget func(Partition) (bitfield.BitField, error)) (bitfield.BitField, error) {
var parts []bitfield.BitField

Expand All @@ -31,7 +33,7 @@ func AllPartSectors(mas State, sget func(Partition) (bitfield.BitField, error))

// SealProofTypeFromSectorSize returns preferred seal proof type for creating
// new miner actors and new sectors
func SealProofTypeFromSectorSize(ssize abi.SectorSize, nv network.Version) (abi.RegisteredSealProof, error) {
func SealProofTypeFromSectorSize(ssize abi.SectorSize, nv network.Version, synthetic bool) (abi.RegisteredSealProof, error) {
switch {
case nv < network.Version7:
switch ssize {
Expand All @@ -49,25 +51,49 @@ func SealProofTypeFromSectorSize(ssize abi.SectorSize, nv network.Version) (abi.
return 0, xerrors.Errorf("unsupported sector size for miner: %v", ssize)
}
case nv >= network.Version7:
var v abi.RegisteredSealProof
switch ssize {
case 2 << 10:
return abi.RegisteredSealProof_StackedDrg2KiBV1_1, nil
v = abi.RegisteredSealProof_StackedDrg2KiBV1_1
case 8 << 20:
return abi.RegisteredSealProof_StackedDrg8MiBV1_1, nil
v = abi.RegisteredSealProof_StackedDrg8MiBV1_1
case 512 << 20:
return abi.RegisteredSealProof_StackedDrg512MiBV1_1, nil
v = abi.RegisteredSealProof_StackedDrg512MiBV1_1
case 32 << 30:
return abi.RegisteredSealProof_StackedDrg32GiBV1_1, nil
v = abi.RegisteredSealProof_StackedDrg32GiBV1_1
case 64 << 30:
return abi.RegisteredSealProof_StackedDrg64GiBV1_1, nil
v = abi.RegisteredSealProof_StackedDrg64GiBV1_1
default:
return 0, xerrors.Errorf("unsupported sector size for miner: %v", ssize)
}

if nv >= MinSyntheticPoRepVersion && synthetic {
return toSynthetic(v)
} else {
return v, nil
}
}

return 0, xerrors.Errorf("unsupported network version")
}

func toSynthetic(in abi.RegisteredSealProof) (abi.RegisteredSealProof, error) {
switch in {
case abi.RegisteredSealProof_StackedDrg2KiBV1_1:
return abi.RegisteredSealProof_StackedDrg2KiBV1_1_Feat_SyntheticPoRep, nil
case abi.RegisteredSealProof_StackedDrg8MiBV1_1:
return abi.RegisteredSealProof_StackedDrg8MiBV1_1_Feat_SyntheticPoRep, nil
case abi.RegisteredSealProof_StackedDrg512MiBV1_1:
return abi.RegisteredSealProof_StackedDrg512MiBV1_1_Feat_SyntheticPoRep, nil
case abi.RegisteredSealProof_StackedDrg32GiBV1_1:
return abi.RegisteredSealProof_StackedDrg32GiBV1_1_Feat_SyntheticPoRep, nil
case abi.RegisteredSealProof_StackedDrg64GiBV1_1:
return abi.RegisteredSealProof_StackedDrg64GiBV1_1_Feat_SyntheticPoRep, nil
default:
return 0, xerrors.Errorf("unsupported conversion to synthetic: %v", in)
}
}

// WindowPoStProofTypeFromSectorSize returns preferred post proof type for creating
// new miner actors and new sectors
func WindowPoStProofTypeFromSectorSize(ssize abi.SectorSize, nv network.Version) (abi.RegisteredPoStProof, error) {
Expand Down
2 changes: 1 addition & 1 deletion chain/gen/genesis/genesis.go
Original file line number Diff line number Diff line change
Expand Up @@ -578,7 +578,7 @@ func MakeGenesisBlock(ctx context.Context, j journal.Journal, bs bstore.Blocksto
}

// setup Storage Miners
stateroot, err = SetupStorageMiners(ctx, cs, sys, stateroot, template.Miners, template.NetworkVersion)
stateroot, err = SetupStorageMiners(ctx, cs, sys, stateroot, template.Miners, template.NetworkVersion, false)
if err != nil {
return nil, xerrors.Errorf("setup miners failed: %w", err)
}
Expand Down
4 changes: 2 additions & 2 deletions chain/gen/genesis/miners.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ func mkFakedSigSyscalls(base vm.SyscallBuilder) vm.SyscallBuilder {
}

// Note: Much of this is brittle, if the methodNum / param / return changes, it will break things
func SetupStorageMiners(ctx context.Context, cs *store.ChainStore, sys vm.SyscallBuilder, sroot cid.Cid, miners []genesis.Miner, nv network.Version) (cid.Cid, error) {
func SetupStorageMiners(ctx context.Context, cs *store.ChainStore, sys vm.SyscallBuilder, sroot cid.Cid, miners []genesis.Miner, nv network.Version, synthetic bool) (cid.Cid, error) {

cst := cbor.NewCborStore(cs.StateBlockstore())
av, err := actorstypes.VersionForNetwork(nv)
Expand Down Expand Up @@ -132,7 +132,7 @@ func SetupStorageMiners(ctx context.Context, cs *store.ChainStore, sys vm.Syscal
i := i
m := m

spt, err := miner.SealProofTypeFromSectorSize(m.SectorSize, nv)
spt, err := miner.SealProofTypeFromSectorSize(m.SectorSize, nv, synthetic)
if err != nil {
return cid.Undef, err
}
Expand Down
64 changes: 62 additions & 2 deletions chain/types/cbor_gen.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions chain/types/execresult.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ import (
"encoding/json"
"time"

"github.com/ipfs/go-cid"

"github.com/filecoin-project/go-address"
"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/go-state-types/exitcode"
Expand All @@ -24,6 +26,9 @@ type MessageTrace struct {
Method abi.MethodNum
Params []byte
ParamsCodec uint64
GasLimit uint64
ReadOnly bool
CodeCid cid.Cid
}

type ReturnTrace struct {
Expand Down
Loading