Eric/use mule in jenkins by egieseke · Pull Request #1072 · algorand/go-algorand

egieseke · 2020-05-13T20:46:40Z

Summary

Create a Jenkins mule pipeline to build and test go-algorand.
https://github.com/algorand/go-algorand-internal/issues/424

Test Plan

Setup and executed Jenkins pipeline:

https://jenkins.algodev.network/job/JenkinsMulePipeline/

This includes changes that allow travis to use mule to execute our ci process

Multiple proxies are not supported in go 1.12, so since proxies fail occasionally, we'll use only direct for now. After upgrading to 1.13, we'll add proxies here

…seMuleInTravisJob

These are failing inconsistently with timeouts. Increasing it since it seems to be passing most of the time

…tests.

… eric/useMuleInJenkins

Mule.

tsachiherman · 2020-06-12T13:22:58Z

 echo Verifying a generic node will start directly
-${BINDIR}/algod -d ${DATADIR} &
+algod -d ${DATADIR} &
+sleep 1


If this does becomes an issue, this is not the correct place to fix it. you might want to try to sleep in case the pgrep returns no pids in update_running_count

I'm surprised the answer isn't to make verify_at_least_one_running have a retry loop like verify_one_running has

@tsachiherman @brianolson I just updated verify_at_least_one_running to include the retry loop and removed the sleep statements. Good thinking.

! ${NO_BUILD}.

Removed unnecessary sleep statements.

… eric/useMuleInJenkins

tsachiherman · 2020-06-14T16:05:18Z

 proc ::AlgorandGoal::TakeAccountOnline { ADDRESS FIRST_ROUND LAST_ROUND TEST_PRIMARY_NODE_DIR } {
   if { [ catch {
-       spawn goal account changeonlinestatus --address $ADDRESS --firstvalid $FIRST_ROUND --lastvalid $LAST_ROUND -d datadir
+       spawn goal account changeonlinestatus --address $ADDRESS --firstvalid $FIRST_ROUND --lastvalid $LAST_ROUND -d datadir --online


this change is supposed to be a no-op. i.e. the default for --online is true. If this is not the case, then we have a bug, which is unrelated to this PR.

@tsachiherman The --online is required by the goal account changeonlinestatus command. Without it, the command fails. I will create a ticket to fix the command and update this test case to remove the --online option.
https://github.com/algorand/go-algorand-internal/issues/541.
Created pull request: #1170

@tsachiherman Reverted the addition of the --online flag since it will no longer be necessary.

The PR(#1170) looks good to me, but I wonder how this test did not detect this issue before.
It's quite concerning that we have a test that is supposed to test exactly one thing - and the test doesn't alert us when that thing is broken.

We found this issue by running the integration tests as part of the Jenkins/Mule pipeline testing. The issue was causing the test test/e2e-go/cli/goal/expect/listExpiredParticipationKeyTest.exp to fail. What is puzzling is why this error does not surface as part of the Travis build and testing. I created this ticket to resolve: https://github.com/algorand/go-algorand-internal/issues/542

since it is no longer a required field.

onetechnical

Hmm, the mule-specific changes here look okay at a glance, but the test changes give me pause. Is it possible to add commentary around changes as to why they were made?

egieseke · 2020-06-16T21:34:32Z

Hmm, the mule-specific changes here look okay at a glance, but the test changes give me pause. Is it possible to add commentary around changes as to why they were made?

The resource-constrained platforms require more time to complete the steps of the tests. Especially for the expect tests that are running 2 node networks for goal to interact with.

btoll · 2020-06-17T05:15:30Z

+	mkdir -p $(SRCPATH)/tmp/node_pkgs && \
+	CHANNEL=$(BUILDCHANNEL) PKG_ROOT=$(SRCPATH)/tmp/node_pkgs NO_BUILD=True VARIATIONS=$(OS_TYPE)-$(ARCH) \
+	scripts/build_packages.sh $(OS_TYPE)/$(ARCH) && \
+	mkdir -p $(SRCPATH)/tmp/node_pkgs/$(BUILDCHANNEL)/$(OS_TYPE)-$(ARCH)/data && \


I'm pretty sure that this is going to break the existing mule tasks (see the package*.yaml files in the root dir) because this path is different.

If you look at the PKG_ROOT var in the deleted snippet above it's expecting the build assets to be put in a different path than the one you have now.

@btoll Please see updates.

btoll · 2020-06-17T05:17:07Z

+	export PATH=$(SRCPATH)/tmp/node_pkgs/$(BUILDCHANNEL)/$(OS_TYPE)-$(ARCH)/bin:$$PATH && \
+	export PATH=$(SRCPATH)/tmp/node_pkgs/$(BUILDCHANNEL)/$(OS_TYPE)-$(ARCH)/tools:$$PATH && \
+	export PATH=$(SRCPATH)/tmp/node_pkgs/$(BUILDCHANNEL)/$(OS_TYPE)-$(ARCH)/test-utils:$$PATH && \
 	export SRCROOT=$(SRCPATH) && \


Looks like these env vars should be precede the call to the shell script rather than exporting them before the call to the script.

Look below to see where the call to test/scripts/e2e.sh is appended to the series of exports.

The script needs the env vars, not the calling environment. It's a common shell idiom to do:

FOO=foo BAR=bar my_script.sh

Also, the ci-build target is doing it that way and it would be great to keep that consistency.

I realize this was existing code, but it would be great if you could change it.

@btoll please see updates.

Ben's package scripts depend on this structure

Eric/use mule in jenkins

btoll · 2020-06-17T17:50:56Z

@egieseke Thanks for the changes, I added one more comment.

btoll · 2020-06-17T19:00:42Z

Just approved. I'm assuming the brittleness of the tests will be addressed in future tickets.

onetechnical

I'm okay with this, but let's pay close attention to how it affects tests. This is also predicated on us doing a deeper dive into tests.

bricerisingalgorand and others added 30 commits May 1, 2020 12:43

Refactor make and mule in order to execute travis jobs using Mule

25c4982

This includes changes that allow travis to use mule to execute our ci process

Only use GOPROXY direct

718d0d9

Multiple proxies are not supported in go 1.12, so since proxies fail occasionally, we'll use only direct for now. After upgrading to 1.13, we'll add proxies here

Move modified ci-* make targets to makefile.mule

d54ef1d

Fix make target for osx jobs

5c2081b

Merge branch 'master' of github.com:algorand/go-algorand into brice/u…

59c376e

…seMuleInTravisJob

Merge branch 'master' of github.com:algorand/go-algorand into brice/u…

c611355

…seMuleInTravisJob

Added Jenkinsfile mule.yaml file for go-algorand.

7eaeb18

Updated jenkins file.

5a6f06b

Update mule.yaml with actual tasks and jobs.

d48d6da

Make timeout for upgrade tests longer

c3562ad

These are failing inconsistently with timeouts. Increasing it since it seems to be passing most of the time

Limit to mule.yaml to build step.

f622224

Include test-linux-arm64-shortest.

e274336

Include test-linux-arm64-integration.

8c4b392

Remove build dependencies from ci-integration and ci-test targets.

8534d7f

Replaced build dependencies for ci-integration and ci-test targets.

9486c08

Added jobs for architectures amd63 and arm32.

cf0c16e

extended timeouts for e2e_sub tests.

d26fc4c

Remove build task which is done as part of integration testing.

842dd76

Remove build task which is done as part of integration testing.

aa8748d

Remove build task which is done as part of integration testing.

f43c989

Increase timeout for goal network start.

7bcaf36

Increase timeout for goal network create.

d37fab4

Remove timeout for algotmpl calls.

0fb60e9

add timeout for algotmpl calls.

7342d7a

Increase wait time to 30 seconds for goal node wait commands in teal …

6bf5ff8

…tests.

Fixed error in expect script.

3565f02

Merge branch 'master' of https://github.com/egieseke/go-algorand into…

18a16ba

… eric/useMuleInJenkins

Fix merge conflict in cicd.centos.Dockerfile.

bbab6c0

Fix scripting error in createWalletTest.exp.

3e676ed

Added amd64 build job.

ee013ad

egieseke added 5 commits June 11, 2020 16:45

Revert fix to create wallet test, moved to a separate PR.

4ff2aa1

Merge branch 'master' of https://github.com/algorand/go-algorand

fecd331

Moving this fix to a different PR since it is not related to Jenkins or

19b8bb1

Mule.

Use environment variable to set S3_RELEASE_BUCKET.

1942d9d

Add S3_RELEASE_BUCKET environment var to mule.yaml docker jobs.

8477840

tsachiherman reviewed Jun 12, 2020

View reviewed changes

egieseke added 5 commits June 12, 2020 11:41

Added back removal of PKG_ROOT at the end of e2e.sh with conditional on

b35463d

! ${NO_BUILD}.

Updated verify_at_least_one_running function to loop up to 2 seconds.

5306f10

Removed unnecessary sleep statements.

Merge branch 'master' of https://github.com/algorand/go-algorand

9ab1df3

Merge branch 'master' of https://github.com/egieseke/go-algorand into…

0d4fb90

… eric/useMuleInJenkins

Adjust first timeout round from 5 to 10.

eff1999

egieseke requested a review from tsachiherman June 13, 2020 14:29

tsachiherman reviewed Jun 14, 2020

View reviewed changes

Reverted addition of --online flag for goal account changeonlinestatus

3ab549f

since it is no longer a required field.

onetechnical requested changes Jun 16, 2020

View reviewed changes

Comment thread scripts/configure_dev.sh

btoll suggested changes Jun 17, 2020

View reviewed changes

bricerisingalgorand and others added 3 commits June 17, 2020 13:03

Revert to old folder structure for build tarballs

451a947

Ben's package scripts depend on this structure

Stop exporting vars in our mule makefile

e201fd3

Merge pull request #8 from bricerisingalgorand/eric/useMuleInJenkins

6276a5f

Eric/use mule in jenkins

btoll reviewed Jun 17, 2020

View reviewed changes

Comment thread scripts/release/mule/Makefile.mule

btoll approved these changes Jun 17, 2020

View reviewed changes

bricerisingalgorand approved these changes Jun 18, 2020

View reviewed changes

egieseke requested a review from onetechnical June 18, 2020 16:24

onetechnical approved these changes Jun 18, 2020

View reviewed changes

algojohnlee merged commit a87718f into algorand:master Jun 22, 2020

cce added a commit to cce/go-algorand that referenced this pull request Jul 30, 2025

remove legacy GOCACHE NFS workaround (from algorand#1072)

f11f8d2

cce mentioned this pull request Jul 31, 2025

CI: some improvements for faster PR builds #6397

Merged

Conversation

egieseke commented May 13, 2020

Summary

Test Plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

egieseke Jun 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

egieseke Jun 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tsachiherman Jun 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

onetechnical left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

egieseke commented Jun 16, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

egieseke Jun 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

btoll commented Jun 17, 2020

Uh oh!

btoll commented Jun 17, 2020

Uh oh!

onetechnical left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

egieseke Jun 12, 2020 •

edited

Loading

egieseke Jun 14, 2020 •

edited

Loading

tsachiherman Jun 15, 2020 •

edited

Loading

egieseke Jun 17, 2020 •

edited

Loading