CI: ensure msgp generator has been run and is clean #3978

cce · 2022-05-11T17:19:11Z

Summary

This should ensure make msgp has been run for changes that impact msgp serialization on CI builds.

It looks like #3829 was merged but missed changes to agreement/msgp_gen.go and gci updates from algorand/msgp#14 were not incorporated into #3919.

Test Plan

Tests should fail in the codgen_verification.sh job for the first build on this PR, then should pass after a commit is pushed that updates the msgp-generated files with the latest on master.

scripts/travis/codegen_verification.sh

algorandskiy · 2022-05-11T19:09:57Z

scripts/buildtools/go.sum

@@ -23,6 +23,8 @@ github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuy
 github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/algorand/msgp v1.1.50 h1:Mvsjs5LCE6HsXXbwJXD8ol1Y+c+QMoFNM4j0CY+mFGo=


go mod tidy to remove the old version?

I tried and it cleared out all of go.mod! I guess all of these packages are unused. TBH I'm not really sure long-term whether it is a good idea to keep installing CLI tools this way.

I see now that install_buildtools.sh is just grepping go.mod to pull version numbers, so I changed it to a flat text file.

Does that make it harder to create though? Presumably it was created my some go tool automatically, but the versions file will need to be edited by hand. Could it grep the root go.mod instead? I don't really know what this one is, how it was made, or why it might differ.

IMO it is much easier to maintain a simple "versions" file now by hand — I would even be OK to just move the tool versions into install_buildtools.sh directly, as it's the only user. The reason it's not in root go.mod is because these tools are not dependencies of go-algorand. (They don't even really need to be implemented in go — they're just CLI tools to install)

cce · 2022-05-11T19:11:09Z

agreement/msgp_gen.go

 			if err != nil {
-				err = msgp.WrapError(err, "struct-from-array", "Sha256Commitment")
+				err = msgp.WrapError(err, "struct-from-array", "NativeSha512_256Commitment")


@Aharonee @algoidan wanted to makes sure this was as expected?

I think I know what happened.
04814e1 (#3829)

This is the commit where I've renamed the TxnRoot to TxnCommitments, and its nested fields.
While doing that I've regenerated msgp, but after noticing the fields in TxnCommitments where no longer in alphabetical order, I swapped them.

TxnCommitments struct { _struct struct{} `codec:",omitempty,omitemptyarray"` Sha256Commitment crypto.Digest `codec:"txn256"` NativeSha512_256Commitment crypto.Digest `codec:"txn"`

to

TxnCommitments struct { _struct struct{} `codec:",omitempty,omitemptyarray"` NativeSha512_256Commitment crypto.Digest `codec:"txn"` Sha256Commitment crypto.Digest `codec:"txn256"`

This doesn't break any tests of course, and wasn't caught the the codegen_verification script, but when generating msgp after this change the resulting code will be different (it seems to be generated in a sequential manner) even though its logic is identical.

algorandskiy · 2022-05-11T23:36:55Z

Any idea why it complains about protocol now?

--- a/agreement/msgp_gen_test.go
+++ b/agreement/msgp_gen_test.go
@@ -10,7 +10,6 @@ import (
 
 	"github.com/algorand/msgp/msgp"
 
-	"github.com/algorand/go-algorand/protocol"
 	"github.com/algorand/go-algorand/test/partitiontest"
 )

cce · 2022-05-12T19:53:04Z

I am trying to figure it out still, but yes.. something is weird about running make msgp in CircleCI.. even when I ssh into the CircleCI VM and run it manually it works ...

algorandskiy · 2022-05-16T15:16:08Z

Look, the msgp's master misses it:

writeImportHeader(
	testbuf,
	"github.com/algorand/msgp/msgp", "github.com/algorand/go-algorand/test/partitiontest", "testing")

algorandskiy · 2022-05-16T15:21:29Z

I think goformat that is called right before flushing the generated files was adding protocol import before and does not add it now.

cce · 2022-05-16T21:16:24Z

@algorandskiy see in the latest commit, even after downgrading msgp to the old version the protocol package is not detected. However again when I ssh in and rerun msgp manually I can get it to work (and also locally). There is something strange about the environment in CircleCI. But perhaps the easiest thing to do is to just add the protocol import to the msgp generator.

This reverts commit db418d9.

This reverts commit 4d8949d.

This reverts commit 14d3b3f.

This reverts commit 17be4e8.

This reverts commit 4745520.

cce · 2022-05-17T15:51:42Z

Updated msgp to explicitly import missing package: algorand/msgp#17

algorandskiy

LGTM but why did scripts/buildtools/go.mod and go.sum go?

cce · 2022-05-17T16:16:21Z

@algorandskiy we don't have any go code in that directory, so "go mod tidy" stopped working in there (it detected no dependencies) and we are only using go.mod to track a few version numbers used by a single shell script (install_buildtools.sh), so I switched it to use a text file containing those numbers instead.

codecov · 2022-05-17T18:06:20Z

Codecov Report

Merging #3978 (5c0a6b5) into master (91095fa) will increase coverage by 0.04%.
The diff coverage is 27.27%.

@@            Coverage Diff             @@
##           master    #3978      +/-   ##
==========================================
+ Coverage   49.77%   49.82%   +0.04%     
==========================================
  Files         409      409              
  Lines       69157    69157              
==========================================
+ Hits        34426    34460      +34     
+ Misses      31011    30983      -28     
+ Partials     3720     3714       -6

Impacted Files	Coverage Δ
crypto/merklearray/msgp_gen.go	`38.52% <ø> (ø)`
crypto/merklesignature/msgp_gen.go	`41.99% <ø> (ø)`
daemon/algod/api/spec/v2/msgp_gen.go	`20.31% <ø> (ø)`
data/account/msgp_gen.go	`40.81% <ø> (ø)`
data/basics/msgp_gen.go	`34.14% <ø> (ø)`
data/transactions/msgp_gen.go	`40.38% <ø> (ø)`
ledger/msgp_gen.go	`41.28% <ø> (ø)`
agreement/msgp_gen.go	`41.08% <27.27%> (ø)`
data/bookkeeping/msgp_gen.go	`42.60% <27.27%> (ø)`
cmd/algoh/blockWatcher.go	`77.77% <0.00%> (-3.18%)`	⬇️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 91095fa...5c0a6b5. Read the comment docs.

This should ensure make msgp has been run for changes that impact msgp serialization on CI builds. It looks like algorand#3829 was merged but missed changes to agreement/msgp_gen.go and gci updates from algorand/msgp#14 were not incorporated into algorand#3919.

cce added 2 commits May 11, 2022 13:17

upgrade msgp in scripts/buidtools/go.mod

fd2417e

run make msgp in codegen_verification.sh

468fa75

Eric-Warehime reviewed May 11, 2022

View reviewed changes

scripts/travis/codegen_verification.sh Show resolved Hide resolved

cce added the Bug-Fix label May 11, 2022

rerun msgp on latest master

ad330f1

cce requested review from Aharonee, id-ms, algorandskiy and winder May 11, 2022 18:48

algorandskiy previously approved these changes May 11, 2022

View reviewed changes

cce commented May 11, 2022

View reviewed changes

winder previously approved these changes May 11, 2022

View reviewed changes

change go.mod file in scripts/buildtools to text file

e8bff0b

cce dismissed stale reviews from winder and algorandskiy via e8bff0b May 11, 2022 20:25

move make msgp earlier

4745520

debug make msgp

17be4e8

cce added 2 commits May 16, 2022 17:05

remove unused go.sum

a5e3c49

downgrade to msgp v1.1.50 to show circleci still not working

14d3b3f

cce force-pushed the require-make-msgp branch from f35e24d to 3f67479 Compare May 16, 2022 21:43

try running msgp from inside directory

4d8949d

cce force-pushed the require-make-msgp branch from 3f67479 to 4d8949d Compare May 16, 2022 21:52

debug msgp gen

db418d9

cce mentioned this pull request May 17, 2022

Add missing protocol package import algorand/msgp#17

Merged

Revert "debug msgp gen"

3c8395a

This reverts commit db418d9.

cce added 5 commits May 17, 2022 11:46

Revert "try running msgp from inside directory"

d363809

This reverts commit 4d8949d.

Revert "downgrade to msgp v1.1.50 to show circleci still not working"

9d1f064

This reverts commit 14d3b3f.

Revert "debug make msgp"

6b1b87f

This reverts commit 17be4e8.

Revert "move make msgp earlier"

58985e2

This reverts commit 4745520.

upgrade to msgp v1.1.52

b6fdf67

algorandskiy reviewed May 17, 2022

View reviewed changes

Merge remote-tracking branch 'origin/master' into require-make-msgp

5c0a6b5

algorandskiy previously approved these changes May 17, 2022

View reviewed changes

cce dismissed algorandskiy’s stale review via 9cbc172 May 17, 2022 16:29

cce force-pushed the require-make-msgp branch from 9cbc172 to 5c0a6b5 Compare May 17, 2022 16:38

algorandskiy approved these changes May 17, 2022

View reviewed changes

Eric-Warehime approved these changes May 17, 2022

View reviewed changes

algorandskiy merged commit 8bb8a4a into master May 17, 2022

cce deleted the require-make-msgp branch May 17, 2022 19:38

This was referenced May 27, 2022

[Not Merged] go-algorand 3.7.0-beta Release PR #4049

Closed

Release: go-algorand 3.7.0-beta Release PR #4050

Merged

Algo-devops-service mentioned this pull request Jun 3, 2022

go-algorand 3.7.0-stable Release PR #4071

Merged

This was referenced Jun 8, 2022

go-algorand 3.7.1-stable Release PR #4091

Merged

go-algorand 3.7.2-stable Release PR #4118

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: ensure msgp generator has been run and is clean #3978

CI: ensure msgp generator has been run and is clean #3978

cce commented May 11, 2022 •

edited

Loading

algorandskiy May 11, 2022

cce May 11, 2022

cce May 11, 2022

jannotti May 11, 2022

cce May 11, 2022

cce May 11, 2022

Aharonee May 12, 2022

algorandskiy commented May 11, 2022

cce commented May 12, 2022 •

edited

Loading

algorandskiy commented May 16, 2022 •

edited

Loading

algorandskiy commented May 16, 2022 •

edited

Loading

cce commented May 16, 2022

cce commented May 17, 2022

algorandskiy left a comment

cce commented May 17, 2022

codecov bot commented May 17, 2022 •

edited

Loading

		@@ -23,6 +23,8 @@ github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuy
		github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
		github.com/algorand/msgp v1.1.50 h1:Mvsjs5LCE6HsXXbwJXD8ol1Y+c+QMoFNM4j0CY+mFGo=

CI: ensure msgp generator has been run and is clean #3978

CI: ensure msgp generator has been run and is clean #3978

Conversation

cce commented May 11, 2022 • edited Loading

Summary

Test Plan

algorandskiy May 11, 2022

Choose a reason for hiding this comment

cce May 11, 2022

Choose a reason for hiding this comment

cce May 11, 2022

Choose a reason for hiding this comment

jannotti May 11, 2022

Choose a reason for hiding this comment

cce May 11, 2022

Choose a reason for hiding this comment

cce May 11, 2022

Choose a reason for hiding this comment

Aharonee May 12, 2022

Choose a reason for hiding this comment

algorandskiy commented May 11, 2022

cce commented May 12, 2022 • edited Loading

algorandskiy commented May 16, 2022 • edited Loading

algorandskiy commented May 16, 2022 • edited Loading

cce commented May 16, 2022

cce commented May 17, 2022

algorandskiy left a comment

Choose a reason for hiding this comment

cce commented May 17, 2022

codecov bot commented May 17, 2022 • edited Loading

Codecov Report

cce commented May 11, 2022 •

edited

Loading

cce commented May 12, 2022 •

edited

Loading

algorandskiy commented May 16, 2022 •

edited

Loading

algorandskiy commented May 16, 2022 •

edited

Loading

codecov bot commented May 17, 2022 •

edited

Loading