Make make gen faster

[magik6k]

Mostly make it actually work in parallel

[rvagg]

$ make gen
...
real    1m40.957s

Linux, nvme, plenty of power and resources.

If I go and time prefix every step in the makefile then I can break it down into the various stages:

make gen

go run ./gen/inline-gen . gen/inlinegen-data.json
# 0m0.176s
go run ./chain/actors/agen
# 0m0.296s
go fmt ./...
# 0m0.594s
go run ./gen/api
# 0m0.255s
goimports -w api
# 0m0.132s
goimports -w api
# 0m0.127s
go run ./gen/main.go
# 0m5.147s
go generate -x ./...
# 1m1.699s
goimports -w api/
# 0m0.122s
go run ./node/config/cfgdocgen > ./node/config/doc_gen.go
# 0m0.149s
go run ./scripts/fiximports
# 0m5.130s
go build -ldflags="-X=github.com/filecoin-project/lotus/build.CurrentCommit=+git.3daff2ed8.dirty" -o docgen-md ./api/docgen/cmd
# 0m0.307s
./docgen-md "api/api_full.go" "FullNode" "api" "./api" > documentation/en/api-v1-unstable-methods.md
# 0m0.060s
./docgen-md "api/v0api/full.go" "FullNode" "v0api" "./api/v0api" > documentation/en/api-v0-methods.md
# 0m0.048s
./docgen-md "api/api_storage.go" "StorageMiner" "api" "./api" > documentation/en/api-v0-methods-miner.md
# 0m0.052s
./docgen-md "api/api_worker.go" "Worker" "api" "./api" > documentation/en/api-v0-methods-worker.md
# 0m0.055s
go build -ldflags="-X=github.com/filecoin-project/lotus/build.CurrentCommit=+git.3daff2ed8.dirty" -o docgen-openrpc ./api/docgen-openrpc/cmd
# 0m0.304s
./docgen-openrpc "api/api_full.go" "FullNode" "api" "./api" > build/openrpc/full.json
# 0m1.958s
./docgen-openrpc "api/api_storage.go" "StorageMiner" "api" "./api" > build/openrpc/miner.json
# 0m0.792s
./docgen-openrpc "api/api_worker.go" "Worker" "api" "./api" > build/openrpc/worker.json
# 0m0.365s
./docgen-openrpc "api/api_gateway.go" "Gateway" "api" "./api" > build/openrpc/gateway.json
# 0m0.890s
go run ./scripts/fiximports
# 0m5.019s
go run ./scripts/fiximports
# 0m4.953s

• We already sped up fiximports a bit, but we also need to dedupe it in here
• go generate takes the bulk of the time and it's parallelisable, perhaps we should replace it with a new parent tool that does the same thing but forks out a go generate child for each thing it finds that needs generating (seems like something up @masih's alley)

[rvagg]

Thread on parallel go generate being rejected from core (also appearance from the ever-present Daniel): golang/go#20520

[rjan90]

Gathering some metrics/benchmarks on make gen.

• From Riba in chore: lint: enable method, function & type godoc linting #12258 (comment) on a M1 MBP:

real	2m13.809s
user	2m9.710s
sys	0m30.339s

• My machine must be cursed, or in dire need of some cleanup/: time make gen on a bit older CPU Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with plenty of RAM:

real	5m46,295s
user	14m12,453s
sys	2m9,963s

[BigLep]

Note: during #12467 the slowness of make gen was a bit painful in the pre-PR work and CI checks of attempting to rush out patches.