Add CI/testing to examples by phlax · Pull Request #12491 · envoyproxy/envoy

phlax · 2020-08-05T11:31:25Z

Commit Message: Add CI/testing to examples
Additional Description:

This PR incorporates the tests that i created for the envoy examples (originally here: https://github.com/phlax/envoy-examples)

It also adds a test to ensure that example configs have been added to the examples BUILD config (ref: #12256 (comment))

the tests are documented in #12874

original description - no longer accurate...

the tests are run inside a docker-in-docker container to ensure a clean environment, and to allow docker system pruning without affecting the hosts container setup.

The container user's umask is set to 027 to catch any potential permissions problems related to running envoy as non-root in containers

The tests can be run locally with ./ci/verify_examples.sh from the root of the envoy repo. This will mount the current directory as the envoy source

Alternatively you can do eg the following to test a different branch/source:

export ENVOY_REPO="-b some-other-branch https://github.com/phlax/envoy"
./ci/verify_examples.sh

In this case the build config test is skipped

Risk Level: low
Testing: yes
Docs Changes:
Release Notes:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]

phlax · 2020-08-05T11:52:02Z

@dio @lizan ive created this PR to add the example tests i created, and hoping it would trigger in ci etc

i added a separate job in azure pipelines but that doesnt seem to have worked.

im wondering if that is the best way to trigger this in ci - and if so - whether something needs to be added in azure to allow this job to run

lizan · 2020-08-05T20:06:47Z

/azp run

azure-pipelines · 2020-08-05T20:06:56Z

Azure Pipelines successfully started running 1 pipeline(s).

.azure-pipelines/pipelines.yml

ci/run_example_tests.sh

phlax · 2020-08-07T02:14:41Z

/azp run

azure-pipelines · 2020-08-07T02:14:45Z

Commenter does not have sufficient privileges for PR 12491 in repo envoyproxy/envoy

phlax · 2020-08-07T02:16:15Z

@lizan would you mind triggering azure. I need to see if the tests are passing

lizan · 2020-08-07T07:11:13Z

/azp run

azure-pipelines · 2020-08-07T07:11:21Z

Azure Pipelines successfully started running 1 pipeline(s).

lizan · 2020-08-07T09:36:42Z

/azp run

azure-pipelines · 2020-08-07T09:36:51Z

Azure Pipelines successfully started running 1 pipeline(s).

lizan · 2020-08-07T09:36:58Z

btw please dont force push

phlax · 2020-08-07T09:59:56Z

btw please dont force push

sorry - i figured its ok on a wip pr - im testing against this branch - but ill shift to another branch

ci/verify_examples.sh

phlax · 2020-08-08T09:03:47Z

@lizan i have updated the verify_examples script, it now runs standalone, and works on azure

i have tidied the script, and dewipped the PR - i think it should be ready for review

i have left the pipeline where it is for now so that tests are triggered/immediately.

once the script itself has approval, ill move it.

im wondering if the test for configs in BUILD needs to be separated out - i think this test probably wants to run early/always

one concern - which i think is an issue with the example itself or perhaps envoy is that while the mysql tests work (db/table created etc) they are triggering protocol errors in stats

you can see the mysql issue on travis here https://travis-ci.org/github/phlax/envoy-examples/builds/715797859#L5389 - it happens on azure/locally also

ci/verify_examples.sh

phlax · 2020-08-13T14:30:16Z

@lizan i have split the tests into verify.sh scripts in the respective examples folders

i have also added the helper scripts mentioned, and made some small improvements to the error handling - partly to compensate for moving the tests out of the scope of the error handling in verify_examples.sh

verify_examples now accepts a glob arg so you can run it with individual tests, and the tests should be runnable directly from the example folders also

this script probs needswork - but im not sure about whether the other test - the one for build configs - needs to be separated or at least runnable separately

i ran the mysql example locally and can confirm that the protocol errors occur outside of these tests when run manually

once we are certain of the format of the tests i can write a small dev doc - preferrably in a subsequent pr

phlax · 2020-08-13T14:30:39Z

...also - is it ok to rebase/force push as i think the conflict is blocking azure

examples/verify-common.sh

examples/fault-injection/verify.sh

lizan

Thanks, the script looks good.

examples/verify-common.sh

Signed-off-by: Ryan Northey <ryan@synca.io>

phlax · 2020-08-22T11:53:23Z

The example tests have now been moved to after the docker build, and i think are using the correct images.

Im wondering if publishing the docker images should be moved after these tests. If these tests fail - only on master, for some reason - you would not find out until after the images had been published

examples/verify-common.sh

Signed-off-by: Ryan Northey <ryan@synca.io>

sunjayBhatia · 2020-08-31T17:55:57Z

cc @envoyproxy/windows-dev might be interesting to get Windows coverage of these as well for more confidence etc.

phlax · 2020-09-01T07:20:40Z

@lizan are there any further changes required to land this ?

lizan

Thanks, this looks great.

lizan · 2020-09-07T02:44:32Z

.azure-pipelines/pipelines.yml

+      - bash: |
+          docker load < $(Build.StagingDirectory)/docker/envoy-docker-images.tar.xz
+          images=($(docker image list --format "{{.Repository}}"))
+          tags=($(docker image list --format "{{.Tag}}"))
+          for i in "${!images[@]}"; do
+              if [[ "${images[i]}" =~ "envoy" ]]; then
+                  docker tag ${images[$i]}:${tags[$i]} ${images[$i]}:latest
+              fi
+          done
+          docker images
+          sudo apt-get update -y
+          sudo apt-get install -y -qq --no-install-recommends redis-tools
+          export DOCKER_NO_PULL=1
+          ci/verify_examples.sh


Can you split this to a bash file?

moved to do_ci.sh - lmk if you want it somewhere else

Signed-off-by: Ryan Northey <ryan@synca.io>

phlax · 2020-09-07T19:41:17Z

@lizan should be ready for review again

lizan reviewed Aug 5, 2020

View reviewed changes

.azure-pipelines/pipelines.yml Outdated Show resolved Hide resolved

lizan reviewed Aug 5, 2020

View reviewed changes

ci/run_example_tests.sh Outdated Show resolved Hide resolved

lizan reviewed Aug 5, 2020

View reviewed changes

ci/run_example_tests.sh Outdated Show resolved Hide resolved

dio assigned lizan Aug 5, 2020

phlax force-pushed the verify-examples-reprise branch 2 times, most recently from 80c89e4 to 6fbd9db Compare August 7, 2020 08:31

phlax changed the title ~~[WIP] Add CI/testing to examples~~ Add CI/testing to examples Aug 8, 2020

phlax marked this pull request as ready for review August 8, 2020 03:52

phlax commented Aug 8, 2020

View reviewed changes

ci/verify_examples.sh Outdated Show resolved Hide resolved

phlax commented Aug 8, 2020

View reviewed changes

ci/verify_examples.sh Outdated Show resolved Hide resolved

phlax mentioned this pull request Aug 8, 2020

examples: Added a sandbox for testing HTTP caching based on the Front Proxy sandbox #12549

Merged

lizan reviewed Aug 11, 2020

View reviewed changes

ci/verify_examples.sh Outdated Show resolved Hide resolved

phlax commented Aug 13, 2020

View reviewed changes

examples/verify-common.sh Outdated Show resolved Hide resolved

phlax commented Aug 13, 2020

View reviewed changes

examples/fault-injection/verify.sh Show resolved Hide resolved

lizan reviewed Aug 13, 2020

View reviewed changes

examples/verify-common.sh Outdated Show resolved Hide resolved

phlax added 3 commits August 20, 2020 09:46

Cleanups: mostly bash formatting and syntax

79a6fa4

Signed-off-by: Ryan Northey <ryan@synca.io>

Cleanup: bash formatting

9758da4

Signed-off-by: Ryan Northey <ryan@synca.io>

Cleanup: bash vars

03c89e4

Signed-off-by: Ryan Northey <ryan@synca.io>

phlax force-pushed the verify-examples-reprise branch from 8443b8f to 03c89e4 Compare August 20, 2020 08:46

Use built images for examples ci tests

8466237

Signed-off-by: Ryan Northey <ryan@synca.io>

phlax marked this pull request as draft August 21, 2020 15:04

phlax added 3 commits August 22, 2020 11:15

Cleanup: add -s to curl in example

acc13b7

Signed-off-by: Ryan Northey <ryan@synca.io>

Tag incoming images

bec9c65

Signed-off-by: Ryan Northey <ryan@synca.io>

Move image loading to ci and ensure docker uses correct images

a57a1ef

Signed-off-by: Ryan Northey <ryan@synca.io>

phlax marked this pull request as ready for review August 22, 2020 10:16

phlax mentioned this pull request Aug 23, 2020

Docs updates and cleanups #12401

Merged

phlax commented Aug 24, 2020

View reviewed changes

examples/verify-common.sh Outdated Show resolved Hide resolved

Cleanup: docker build/pull

2d01d4f

Signed-off-by: Ryan Northey <ryan@synca.io>

This was referenced Aug 28, 2020

MySql filter example shows protocol errors #12868

Closed

Add dev docs for examples verify.sh #12874

Merged

Cleanup: bash linting

0ce19a8

Signed-off-by: Ryan Northey <ryan@synca.io>

phlax mentioned this pull request Aug 31, 2020

Examples: Change Dockerfile ENTRYPOINT/CMD commands to exec format. #12889

Merged

lizan reviewed Sep 7, 2020

View reviewed changes

phlax added 4 commits September 7, 2020 13:47

Move verify examples setup to do_ci.sh

2685a13

Signed-off-by: Ryan Northey <ryan@synca.io>

Allow do_ci to run without build setup

cae5ab8

Signed-off-by: Ryan Northey <ryan@synca.io>

check user env in azure

b7dd274

Signed-off-by: Ryan Northey <ryan@synca.io>

Set umask for azure user

28ebcb1

Signed-off-by: Ryan Northey <ryan@synca.io>

lizan approved these changes Sep 8, 2020

View reviewed changes

lizan merged commit 69a09f4 into envoyproxy:master Sep 8, 2020

phlax mentioned this pull request Sep 11, 2020

Add some docs for verify_build_configs #13058

Closed

Conversation

phlax commented Aug 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

original description - no longer accurate...

Uh oh!

phlax commented Aug 5, 2020

Uh oh!

lizan commented Aug 5, 2020

Uh oh!

azure-pipelines bot commented Aug 5, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phlax commented Aug 7, 2020

Uh oh!

azure-pipelines bot commented Aug 7, 2020

Uh oh!

phlax commented Aug 7, 2020

Uh oh!

lizan commented Aug 7, 2020

Uh oh!

azure-pipelines bot commented Aug 7, 2020

Uh oh!

lizan commented Aug 7, 2020

Uh oh!

azure-pipelines bot commented Aug 7, 2020

Uh oh!

lizan commented Aug 7, 2020

Uh oh!

phlax commented Aug 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phlax commented Aug 8, 2020

Uh oh!

Uh oh!

phlax commented Aug 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

phlax commented Aug 13, 2020

Uh oh!

Uh oh!

Uh oh!

lizan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

phlax commented Aug 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

sunjayBhatia commented Aug 31, 2020

Uh oh!

phlax commented Sep 1, 2020

Uh oh!

lizan left a comment

Choose a reason for hiding this comment

Uh oh!

lizan Sep 7, 2020

Choose a reason for hiding this comment

Uh oh!

phlax Sep 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phlax commented Sep 7, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

phlax commented Aug 5, 2020 •

edited

Loading

phlax commented Aug 7, 2020 •

edited

Loading

phlax commented Aug 13, 2020 •

edited

Loading

phlax commented Aug 22, 2020 •

edited

Loading

phlax Sep 7, 2020 •

edited

Loading