actors v7 and venus v2 (#4658)

* feat: add modules: venus-shared & venus-devtool

* add simple test for blockHeader serializing & caching, fix test for bigint

* feat: add doc: refactor notes for venus-v2

* fix: newline in check item

* refactor: remove cachedXXXX fields inside chain.BlockHeader, re-organize
usages of Serialize()

* feat: add more types

* feat: more types in venus-shared/chain

* feat: add coverage stat for tests

* feat: impl value provider for testing

* feat: check for convertible provider, use int for all number types

* feat: test for invalid provider registration

* feat: show cbor-gened types in lotus

* feat: add value providers for chain-related types

* chore: basic test util for cbor types

* feat: move cborErBasic into testutil

* feat: add providers for +/- big.Ints

* feat: complete tests for BigInt

* feat: add more tests for FIL

* feat: add more tests for unit string

* feat: basic test for Ticket

* feat: more basic tests for cbor.Er types

* refactor: use require to replace assert

* feat: add more tests for TipSet / ElectionProof / Ticket

* feat: add tests for PaddedPieceSize / UnpaddedPieceSize

* feat: add tests for hello types

* fetch templates

* feat: complete venus-shared/actors

* feat: add FullBlock, FullTipSet in chain, add types in libp2p/exchange

* feat: remove variables for implementations in libp2p/exchange

* feat: add interfaces for libp2p components

* init venus-component module

* duplicate pkg/chainsync/exchange into venus-componet/libp2p

* restore pkg/chainsync/exchange

* fix: stuffs required by venus-component/libp2p/exchange

* feat: duplicate chainsync/exchange, make it depend on venus-shared

* feat: add type Options for exchange protocol

* feat: logger with context

* use *TaggedLogger

* add FilPeerEventType

* impl: exchange Server

* impl: exchange Client

* move client side codes into separated pkg

* feat: update exchange.Client.GetFullTipSet

* feat: impl exchange.Client

* impl & test Message.ValidForBlockInclusion

* more types in venus-shared

* impl: typechecker

* impl: typecheck.Similar & tests

* mv typecheck into venus-shared

* rename package

* impl: similar-check reason info

* duplicate v0 apis

* checkout app/client/apiface

* pick fullnode decl

* feat: checksum for api.FullNode

* go mod tidy

* add method group: chain for v0api of chain service

* add subcmd: checksum for apis

* add subcmd: diff for apis

* mv v0 to v1

* introduce api/chain/v1

* feat: api diff

* add struct field check for Similar

* add struct tag check for api-diff

* fix: api diffs caused by chain.BlockHeader

* fix: api diffs caused by json.Marshaler impl of Message & SignedMessage

* fix: api diffs caused by MessageReceipt.ReturnValue

* fix: add returned error for ChainNotify

* fix: api diffs caused by BlockTemplate

* fix: api diffs caused by PaymentInfo cbor codec impl

* show codec string in Similar reason

* impl Similar: convertible, interface all methods, recursive

* Feat/fix api permission (#4626)

* fix api permission

* fix v0 api permission

* CI integration API permission validation

* add v0 api and fix permission

* Feat/replica actos file (#4628)

* add cmd to replica actors file

* replica actors files

* Feat/replace actors (#4632)

* use actors from shared

* update shared go.mod

* complex providers (#4634)

* Feat/replace common type (#4636)

* more type and function

* replace type

* venus-devtool add tools for venus-shared api generation, (#4638)

and also ouput permisstion check to compatible-checks dir.

* add v7 actor files (#4637)

* use the shared API (#4639)

* use the shared API

* tool for auto generate api document. (#4640)

* provide esitimate-message type for doc generation (#4641)

* api mocks (#4654)

* Feat/simple api perm comment (#4655)

* simplified permission comment

* fix replica cmd

* add proxy util

* specactors: pull in lotus upgrade

* specactors: Add code not generated by the template

* upgrade vm @ network params for actor v7

* specactors: Add code not generated by the template (#4657)

* shared remove go.mod (#4659)

* VM: Circ supply should be constant per epoch (#4656)

* inline coed gen (#4663)

* inline coed gen

* fix test

* fix method map (#4666)

* Feat/venus v2 snapdeal (#4664)

* upgrade filecoin-ffi

* upgrade github.com/ipfs/go-datastore to v0.5.1

* fix go mod

* fix PrivKey Marshal

* fix api & api-doc

* fix consensus test

* fix wallet test

Co-authored-by: 一页素书 <[email protected]>

* fix func def for dsbackend

* Unify the definition of MessageReceipt (#4667)

Co-authored-by: 一页素书 <[email protected]>

* fix ci for select msg test (#4669)

Co-authored-by: 一页素书 <[email protected]>

* disable mplex stream muxer (#4668)

* fix test (#4670)

* fix test

* close dht (#4672)

* refactor: (#4673)

1: rename MsigCancel to MsigApproveTxnHash (compatible with lotus api)
2: the older MsigCancel inner invoke msigApproveOrCancelSimple.

new feat:
1: add `msig cancel` command.

others:
the compatible check files was re-generated :make compatible-all
api document is re-generated : make api-docs
api structs is re-genreated: make v2-gen-api && make gen-api

* merge type pkg (#4674)

* Feat/simplify storage interface (#4676)

* to simplify storage interface

* add method: TipSetKeyFromBytes

* chain notify return error (#4677)

* fix un-consunsus api StateMinerPreCommitDepositForPower (#4678)

* fix (#4679)

* Feat/generate v0 api doc (#4685)

* clean code

* generate v0 api doc

* circsuypply calc around null blocks (#4687)

* remove the NtwkVersionGetter (#4690)

* remove the NtwkVersionGetter

* syscalls add method: VerifyReplicaUpdate

* more inline template (#4691)

* test vector runner fixes for v6 vectors (#4692)

* add gas charge for VerifyReplicaUpdate (#4693)

* mod: update go-paramfetch (#4694)

* add temp snap params (#4697)

* support snap deals (#4696)

Co-authored-by: 一页素书 <[email protected]>

* fix sender and receiver are the same (#4703)

* Integrate proof v11.0.0 (#4705)

* support snapnet (#4702)

* support snapnet

* fix butterfly network params

* Feat/migration autobatch (#4704)

* Fast migration for v15

* fix autobatch

Co-authored-by: simlecode <[email protected]>

* opt version cmd (#4707)

* bump version to v1.2.0-rc1 (#4708)

* use expresion pattern maching version

* fix api client

* fix search msg

* update api version

* bump version to v1.2.0-rc2

* vm: fix network version not correct (#4712)

* bump version to v1.2.0-rc3 (#4713)

* No need to compare from address and to address (#4716)

* No need to compare from address and to address

* vmoption remove NetworkVersionGetter

* Feat/dtynn/compatible check for api perms (#4730)

* feat: compatible check for api perms

* add github-action: basic checks

* add ci checks for type comtatilble, api/mock api generation, (#4731)

and documents generation.

* bump version to v1.2.0-rc4 (#4733)

* Remove unused code (#4732)

* remove fixtures/proof-params/parameters.json

* remove unused api code

* remove pkg/types

* Feat/dtynn/refactor parts of the api compatible checks (#4734)

* refactor: devtool: refactor api-gen & re-generate related codes (#4737)

* feat: venus-shared: pick types & api def from venus-messager (#4738)

* feat: shared types from venus-messager

* feat: add api defs for venus-messager

* feat: generate proxy_gen.go for venus-messager

* feat: venus-shared: pick types & api def from venus-wallet (#4746)

* feat: venus-shared: pick types & api def from venus-wallet

* rename GroupId|BindId to GroupID|BindID

* remove blank line

* move DrawRandomParams to a separate file

* feat: generate client constructor for apis in venus-shared (#4747)

* upgrade ffi (#4748)

* fix wallet unlock failed (#4736)

* build: release: v1.2.0-rc5 (#4751)

* reset butterfly network

* update ffi

* udpate version to v1.2.0-rc5

* refactor: generate document for api (#4749)

* remove unrelated parmas (#4753)

* fix json marshal/unmarshal for FIL type

* fix: venus-shared: Change the type of HeadChange.Type (#4755)

* Change the type of HeadChange.Type

* feat: venus-shared: Expose RPC client options (#4756)

* Expose RPC client options

* migrate v7 version config (#4761)

* chore: dep: update ffi & go.mod (#4762)

* update ffi & go.mod

* pick venus-gateway def & api (#4769)

Co-authored-by: 一页素书 <[email protected]>

* feat: extend api & types from venus-gateway (#4770)

* feat: extend api & types from venus-gateway

* CI: fix apt install failed

Co-authored-by: 一页素书 <[email protected]>

* feat: venus-shared: introduce apis & types from venus-market (#4776)

* feat: introduce apis & types from venus-market

* opt: remove unused mod

Co-authored-by: 一页素书 <[email protected]>

* feat: remove previous api def for gateway: with proof event related methods only (#4779)

* type: venus-shared: add pad byte index (#4777)

* type: venus-shared: add pad byte index

* add v0 api for gateway (#4780)

* feat: venus-shared: api version / namespaces & helper func for rpc endpoint (#4782)

* feat: helper for formating endpoint with given addr & ver

* feat: add Version for api-gen

* feat: re-generate api clients with ver & nss

* refactor: rename v0 to gateway so that we could have a better api namespace

* feat: config: migrate maxfee (#4783)

* migrate maxfee

* Feat/remove replace for ipfscmds (#4773)

* remove replace for ipfs cmds library

* reinit peer key if not found

* fix security dependency (#4784)

* fix security dependency

* build v1.2.0-rc6 (#4785)

* tweak v15 migration params

* venus-devtool: upgrade lotus to v1.14.0

* set ohSnap mainnet upgrade epoch

* update version to v1.2.0-rc6

* upgrade venus-auth

* fix: use DealProposal inside specs-actors (#4786)

* fix comment (#4787)

* split-gateway-interfaces-by-role (#4788)

* remove unused api version info (#4795)

* update version to v1.2.0 (#4796)

* update git workflow (#4799)

* sync: fix inconsistent circ supply at the same epoch (#4802)

* Fix inconsistent circ supply at the same epoch

* bump version to v1.2.1 (#4803)

* Feat/dtynn/introduce dial args (#4804)

* feat: introduce APIInfo from venus-common-utils/apiinfo

* feat: update template for client gen, re-gen for all shared apis

Co-authored-by: dtynn <[email protected]>
Co-authored-by: zl <[email protected]>
Co-authored-by: 一页素书 <[email protected]>
Co-authored-by: Susanoo <[email protected]>
Co-authored-by: Mike <[email protected]>
Co-authored-by: hunjixin <[email protected]>

.github/workflows/baisc_checks.yml

@@ -0,0 +1,41 @@
+name: basic checks
+
+on: ["push", "pull_request"]
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - name: dependencies
+        run: sudo apt update & sudo apt upgrade & sudo apt install --reinstall mesa-opencl-icd ocl-icd-opencl-dev gcc git bzr jq pkg-config curl clang build-essential hwloc libhwloc-dev wget -y
+
+      - uses: actions/checkout@v2
+        with:
+          submodules: 'true'
+
+      - name: setup go
+        uses: actions/setup-go@v2
+        with:
+          go-version: '1.16'
+
+      - name: deps
+        run: make deps
+
+      - name: build
+        run: make all
+
+      - name: tests-shared
+        run: make test-venus-shared
+
+      - name: compatible all
+        run: |
+          make compatible-all
+
+      - name: gen all
+        run: |
+          make gen-all
+
+      - name: detect changes
+        run: |
+          git status --porcelain
+          test -z "$(git status --porcelain)"

.github/workflows/tag-workflow.yml

@@ -26,25 +26,21 @@ jobs:
         run: |
           export commit=$(git rev-parse HEAD)
           export short=$(git rev-parse --short HEAD)
-          export github_tag=${GITHUB_REF##*/}
+          export github_tag=${{github.ref_name}}
+          export tag=$github_tag
+          export branch=$github_tag
           export git_message=$(git rev-list --format=%s --max-count=1 HEAD | tail +2)
           export repo_name=${GITHUB_REPOSITORY##*/}
-          export branch=$(git symbolic-ref --short -q HEAD)
-          export tag=$(git describe --tags $(git rev-list --tags --max-count=1))
           export artifact_name=${repo_name}_$(git rev-parse --short HEAD).tar.gz
           export pub_method=pushTest
           export job_url=$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID
           export oss_exists=0
           export ftp_exists=0
-          if [[ "${{github.event_name}}" == "create" ]] && ([[ "$github_tag" =~ "v1." ]] || [[ "$github_tag" =~ "v2." ]]); then
+          export rx='^v([0-9]+\.){0,2}(\*|[0-9]+)(-rc[0-9]*){0,1}$'
+          if [[ "${{github.ref_type}}" == "tag" ]] && [[ "${{github.event_name}}" == "create" ]] && [[ "$github_tag" =~ $rx ]]; then
             export pub_method=pushRelease
           fi
-          if [[ "$branch" == "" ]]; then
-            export branch=$github_tag
-          fi
-          if [[ "$tag" == "" ]]; then
-            export tag=$github_tag
-          fi
+
           if [[ "${{secrets.OSS_KEY_ID}}" != "" && \
             "${{secrets.OSS_KEY_SECRET}}" != "" && \
             "${{secrets.OSS_ENDPOINT}}" != "" && \
@@ -102,8 +98,7 @@ jobs:
         run: sudo apt-get install make ncftp mesa-opencl-icd ocl-icd-opencl-dev gcc git bzr jq pkg-config curl clang build-essential hwloc libhwloc-dev wget -y && sudo apt upgrade -y
       - name: Build
         run: |
-          sudo make deps
-          sudo make
+          go clean --modcache && make deps && make
           mkdir ./release && mv ./venus ./release
 
       - name: Zip Release
@@ -117,6 +112,7 @@ jobs:
         id: release
         uses: ncipollo/release-action@v1
         if: ${{ steps.vars.outputs.pub_method=='pushRelease' }}
+        continue-on-error: true
         with:
           artifacts: ${{steps.vars.outputs.artifact_name}}
           tag: ${{ steps.vars.outputs.github_tag }}
@@ -151,12 +147,12 @@ jobs:
           echo '::set-output name=oss_signed_url::$(signed_url)'
 
       - name: upload artifacts
-        if: ${{ steps.vars.outputs.pub_method == 'pushTest' && steps.uploadftp.outcome != 'success' && steps.cposs.outcome != 'success' }}
+        if: ${{ steps.vars.outputs.pub_method == 'pushRelease' }}
         continue-on-error: true
         uses: actions/upload-artifact@v2
         with:
           name: ${{steps.vars.outputs.artifact_name}}
-          path: ./${{steps.vars.outputs.artifact_name}}
+          path: ./release
           if-no-files-found: error
 
       - name: push god-eye

.gitignore

@@ -8,6 +8,7 @@ localnet.json
 .task
 .idea
 *.coverprofile
+coverprofile.html
 *.out
 *.so
 
@@ -16,7 +17,7 @@ fixtures/test
 fixtures/live
 !fixtures/live/.gitkeep
 
-// Tools 
+// Tools
 tools/gengen/gengen
 tools/gengen/gensetup
 tools/fast/bin/localnet
@@ -32,6 +33,9 @@ tools/chain-util/chain-util
 .DS_Store
 **/.DS_Store
 
+// venus-dev-tool
+venus-dev-tool
+
 // IDE (vscode)
 .vscode
 debug.test

CODEWALK.md

@@ -71,8 +71,8 @@ Other patterns, we've evolving for our needs:
 - go-ipfs relies heavily on shell-based integration testing; we aim to rely heavily on unit testing and Go-based integration tests.
 - The go-ipfs package structure involves a deep hierarchy of dependent implementations; 
 we're moving towards a more Go-idiomatic approach with narrow interfaces defined in consuming packages (see [Patterns](#patterns).
-- The term "block" is heavily overloaded: a blockchain block ([`types/block.go`](https://github.com/filecoin-project/venus/tree/master/pkg/types/block.go)), 
-but also content-id-addressed blocks in the block service. 
+- The term "block" is heavily overloaded: a blockchain block ([`types/block.go`](https://github.com/filecoin-project/venus/tree/master/venus-shared/types/block_header.go)),
+but also content-id-addressed blocks in the block service.
 Blockchain blocks are stored in block service blocks, but are not the same thing.
 
 ## Architecture overview
@@ -178,7 +178,7 @@ It is expected that other implementations will match the behaviour of the Go act
 An ABI describes how inputs and outputs to the VM are encoded. 
 Future work will replace this implementation with a "real" VM.
 
-The [Actor](https://github.com/filecoin-project/venus/blob/master/pkg/types/actor.go) struct is the base implementation of actors, with fields common to all of them.
+The [Actor](https://github.com/filecoin-project/venus/blob/master/venus-shared/internal/actor.go) struct is the base implementation of actors, with fields common to all of them.
 
 - `Code` is a CID identifying the actor code, but since these actors are implemented in Go, is actually some fixed bytes acting as an identifier. 
 This identifier selects the kind of actor implementation when a message is sent to its address.
@@ -252,7 +252,7 @@ If a miner is elected, they have the right to mine a block in that round.
 
 Given the probabilistic nature of mining new blocks, more than one block may be mined in any given round. 
 Hence, a new block might have more than one parent block. 
-The parents form a set, which we call a [tipset](https://github.com/filecoin-project/venus/blob/master/pkg/types/tipset.go). 
+The parents form a set, which we call a [tipset](https://github.com/filecoin-project/venus/blob/master/venus-shared/types/tipset.go). 
 All the blocks in a tipset are at the same height and share the same parents. 
 Tipsets contain one or more blocks. 
 A null block count indicates the absence of any blocks mined in a previous round. 
@@ -374,7 +374,6 @@ The `functional-tests` directory contains some Go and Bash scripts which perform
 These are not daemon tests, but run separately.
 
 Some packages have a `testing.go` file with helpers for setting up tests involving that package’s types. 
-The [`types/testing.go`](https://github.com/filecoin-project/venus/blob/master/pkg/types/testing.go) file has some more generally useful constructors. 
 There is also a top-level [`testhelpers`](https://github.com/filecoin-project/venus/blob/master/pkg/testhelpers) package with higher level helpers, often used by daemon tests.
 
 We’re in process of creating the venus Automation and Systems Toolkit (FAST) [library](https://github.com/filecoin-project/venus/tree/master/tools/fast). 

Makefile

@@ -11,7 +11,7 @@ deps:
 lint:
 	go run ./build/*.go lint
 
-test:
+test: test-venus-shared
 	go run ./build/*.go test -timeout=30m
 
 # WARNING THIS BUILDS A GO PLUGIN AND PLUGINS *DO NOT* WORK ON WINDOWS SYSTEMS
@@ -24,10 +24,57 @@ clean:
 	rm -rf ./extern/filecoin-ffi
 	rm -rf ./extern/test-vectors
 
-gen:
-	go run ./tools/gen/api/proxygen.go
-	gofmt -s -l -w ./app/client/full.go
-	goimports -l -w ./app/client/full.go
+gen-all: cborgen gogen inline-gen api-gen
 
 gen-asset:
 	go-bindata -pkg=asset -o ./fixtures/asset/asset.go ./fixtures/_assets/car/ ./fixtures/_assets/proof-params/ ./fixtures/_assets/arch-diagram.monopic
+	gofmt -s -l -w ./fixtures/asset/asset.go
+
+### devtool ###
+cborgen:
+	cd venus-devtool && go run ./cborgen/*.go
+
+gogen:
+	cd venus-shared && go generate ./...
+
+mock-api-gen:
+	cd ./venus-shared/api/chain/v0 && go run github.com/golang/mock/mockgen -destination=./mock/full.go -package=mock . FullNode
+	cd ./venus-shared/api/chain/v1 && go run github.com/golang/mock/mockgen -destination=./mock/full.go -package=mock . FullNode
+
+inline-gen:
+	cd venus-devtool && go run ./inline-gen/main.go ../ ./inline-gen/inlinegen-data.json
+
+test-venus-shared:
+	cd venus-shared && go test -covermode=set ./...
+
+api-gen:
+	cd ./venus-devtool/ && go run ./api-gen/ proxy
+	cd ./venus-devtool/ && go run ./api-gen/ client
+	cd ./venus-devtool/ && go run ./api-gen/ doc
+
+compatible-all: compatible-api compatible-actor
+
+compatible-api: api-checksum api-diff api-perm
+
+api-checksum:
+	cd venus-devtool && go run ./compatible/apis/*.go checksum > ../venus-shared/compatible-checks/api-checksum.txt
+
+api-diff:
+	cd venus-devtool && go run ./compatible/apis/*.go diff > ../venus-shared/compatible-checks/api-diff.txt
+
+api-perm:
+	cd venus-devtool && go run ./compatible/apis/*.go perm > ../venus-shared/compatible-checks/api-perm.txt
+
+compatible-actor: actor-templates actor-sources actor-render
+
+actor-templates:
+	cd venus-devtool && go run ./compatible/actors/*.go templates --dst ../venus-shared/actors/ > ../venus-shared/compatible-checks/actor-templates.txt
+
+actor-sources:
+	cd venus-devtool && go run ./compatible/actors/*.go sources > ../venus-shared/compatible-checks/actor-sources.txt
+
+actor-render:
+	cd venus-devtool && go run ./compatible/actors/*.go render ../venus-shared/actors/
+
+actor-replica:
+	cd venus-devtool && go run ./compatible/actors/*.go replica --dst ../venus-shared/actors/