Initial support for extended integration test #1230

mfojtik · 2015-03-05T12:53:41Z

This is highly WIP atm. I have this here just to gather feedback and thoughts ;-)

mfojtik · 2015-03-05T12:56:08Z

To execute extended tests, you have to:

OS_TEST_TAGS="integration extended docker" ./hack/test-integration.sh -v=5

I don't like that I need to start the OpenShift server (all-in-one), since that breaks the parallelization of tests (I dunno if it is good idea to have multiple OpenShift server/etcd/... running simultaneously)...

The second thing I don't like is that I can't verify the application is working without having Docker registry in place... Right now, the build does not have any Output defined and I just assume that when it finishes as 'Complete' there was no error during the STI build....
I would love to have access to 'logs' (docker logs/buildLogs) at some point to verify that the 'contextDir' applied here (the files were copied from subdir).
Without this capability, I can't test properly things like '.sti/environment', because I don't have access to the output image...

mfojtik · 2015-03-05T13:01:35Z

test/integration/build_context_dir_test.go

FYI. this should be probably solved by using Watch() instead of polling, I just need to get familiar with the Watch() itself first...

csrwng · 2015-03-05T13:30:37Z

I don't like that I need to start the OpenShift server (all-in-one), since that breaks the parallelization of tests (I dunno if it is good idea to have multiple OpenShift server/etcd/... running simultaneously)...

Could we start the all-in-one server, set it up, then run all tests in parallel (each using its own namespace)?

The second thing I don't like is that I can't verify the application is working without having Docker registry in place...

I was under the impression that what we wanted was a go version of the e2e tests with everything setup, including the registry. (so e2e extended posing as an integration test)

mfojtik · 2015-03-05T13:38:40Z

@csrwng if extended tests will be a new set of tests, then I don't think I can re-use the code in integration... (dunno if I can just import the test_server.go ;-)

csrwng · 2015-03-05T13:41:22Z

That's what I thought we were going to do :-)

mfojtik · 2015-03-05T13:44:50Z

@csrwng I can still move this into its own package :-) (this is the feedback I was hoping for :-)

csrwng · 2015-03-05T13:51:33Z

@csrwng I can still move this into its own package :-) (this is the feedback I was hoping for :-)

You have my vote

mfojtik · 2015-03-05T14:47:50Z

@csrwng so the command to launch the tests is now:

OS_TEST_PACKAGE="test/extended" OS_TEST_TAGS="integration extended docker" ./hack/test-integration.sh

soltysh · 2015-03-05T15:03:59Z

test/extended/build_context_dir_test.go

Instead of having those bits and pieces here and there, let's create a test/util package and put there all the utility stuff such as RequireDocker, RequireEtcd, etc.

csrwng · 2015-03-05T15:10:32Z

@csrwng so the command to launch the tests is now:

we should add a hack/test-extended.sh to run it

soltysh · 2015-03-05T15:15:57Z

test/extended/build_context_dir_test.go

I think we are overusing those, especially in that we do copy&paste for other integration tests. I think it would make sense to clear this out doing this. Since for this test you are requiring etcd and docker this line should state:

// +build integration,extended,etcd,docker

IMHO

there should be no tags at all here...

I think you may want to leave integration,extended since you're passing those with OS_TEST_TAGS

should it just be extended? these tests aren't run when hack/test-integration.sh is run (because they're in a different package), so tagging them as integration is a bit confusing.

soltysh · 2015-03-05T15:17:57Z

🌻 for the idea itself

soltysh · 2015-03-05T15:28:32Z

we should add a hack/test-extended.sh to run it

+1

Maybe call it hack/test-integration-extended.sh to follow integration tests pattern.

bparees · 2015-03-05T18:19:14Z

I think it's reasonable to start the registry service as part of the basic integration test environment setup.

and i agree w/ @csrwng that we should be able to run in parallel by using appropriate namespaces (though that probably means we need some more generic framework that will dynamically generate unique namespaces and substitute it into templates/etc, otherwise we're relying on everyone to create their tests using unique namespaces).

also currently namespaces don't work for services...upstream issue still pending.

bparees · 2015-03-05T20:15:54Z

might be useful to have a soft-reset for openshift... ie "delete all pods/etc, clean up etcd (maybe just cleaning up etcd is sufficient), stop all docker containers"

might be faster than starting openshift again for each test.

though given that you'd need to purge/restart the registry too i guess it probably wouldn't save much time.....i just worry about how long each integration test is going to take if we have to start openshift and wait for the registry to deploy....that's like 60s just in startup time per test. I know part of the point of this is to have a place for longer tests to live, but still.

soltysh · 2015-03-05T21:50:02Z

I guess then it'll be simpler and faster at the same time, to generate new namespace for each test. Eg. we could be generating env var called TEST_NAMESPACE and all tests would be then consuming that.

soltysh · 2015-03-06T12:17:10Z

hack/test-integration.sh

Can't we have that verbosity connected with VERBOSE env var?

Why would you need the build to be verbose?

mfojtik · 2015-03-16T13:56:44Z

@bparees @csrwng

I updated this:

OpenShift server is now started by test-extended.sh
GetClient() method was added into server.go so you can obtain an osclient you use in your tests
More smaller refactorings and tweaks to have these tests running consistently

PTAL

mfojtik · 2015-03-16T15:23:11Z

FYI, now you execute the tests using:

./hack/test-extended.sh

(you can optionally set VERBOSE=1 before you run that command to get more verbose output...)

csrwng · 2015-03-16T17:01:42Z

hack/test-extended.sh

Using ${TMPDIR:-/tmp} instead of /tmp would be useful in case you want to use a different temp directory

openshift-bot · 2015-03-18T15:35:38Z

continuous-integration/openshift-jenkins/test FAILURE (https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_openshift3/1375/)

mfojtik · 2015-03-19T10:33:07Z

@deads2k can you please check the latest failure? I think it has something to do with Policies vs Namespaces but I can't figure out what broke there....

mfojtik · 2015-03-19T11:03:17Z

@deads2k this might be related:

I0319 11:01:44.033426    7246 master.go:442] No master policy found.  Creating bootstrap policy based on: /tmp/openshift-integration/policy/policy.json
E0319 11:01:44.265429    7246 master.go:445] Error creating bootstrap policy: policyBinding "openshift" not found

mfojtik · 2015-03-19T11:48:42Z

Also I wonder why TestDNS requires Docker:

W0319 11:32:07.796088    7921 node.go:96] Error running 'chcon' to set the kubelet volume root directory SELinux context: exit status 1
F0319 11:32:07.796252    7921 node.go:61] ERROR: Docker could not be reached at unix:///var/run/docker.sock.  Docker must be installed and running to start containers.

mfojtik · 2015-03-19T13:21:15Z

@csrwng @bparees @soltysh PTAL

I would really like to have this merged ASAP as this is getting broken everyday (as somebody push something into integration tests). I can fix small issues as follow up.

csrwng · 2015-03-19T13:47:43Z

LGTM but it looks like integration tests are still failing on go 1.4

mfojtik · 2015-03-19T13:54:04Z

@csrwng yeah, I saw this failure on other PR as well and @deads2k is looking into it... It does not seem that my changes made it worse/better.

mfojtik · 2015-03-19T13:54:28Z

[merge]

(will fix all issues as follow up)

openshift-bot · 2015-03-19T13:55:44Z

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/merge_pull_requests_openshift3/1223/) (Image: devenv-fedora_1081)

mfojtik · 2015-03-19T14:30:54Z

[merge] again.. seems like random failure

openshift-bot · 2015-03-19T14:35:38Z

Evaluated for origin up to 47caf77

Merged by openshift-bot

mfojtik reviewed Mar 5, 2015
View reviewed changes

soltysh reviewed Mar 5, 2015
View reviewed changes

mfojtik force-pushed the ext_int_test_poc branch 2 times, most recently from c30caa4 to 19159d0 Compare March 6, 2015 12:00

soltysh reviewed Mar 6, 2015
View reviewed changes

mfojtik mentioned this pull request Mar 11, 2015

WIP: Initial support for extended tests #1211

Closed

mfojtik force-pushed the ext_int_test_poc branch 2 times, most recently from d661722 to da2b5b3 Compare March 16, 2015 11:53

mfojtik changed the title ~~WIP [DO_NOT_MERGE]: Extended integration test~~ Initial support for extended integration test Mar 16, 2015

csrwng reviewed Mar 16, 2015
View reviewed changes

mfojtik force-pushed the ext_int_test_poc branch 3 times, most recently from 25c4464 to 443700e Compare March 19, 2015 09:48

mfojtik force-pushed the ext_int_test_poc branch 5 times, most recently from 3e1bf88 to 153b70d Compare March 19, 2015 11:01

deads2k mentioned this pull request Mar 19, 2015

bootstrapping policy bugs #1375

Merged

Add ./hack/test-extended.sh

1a25027

mfojtik force-pushed the ext_int_test_poc branch 3 times, most recently from 50e8fd0 to 864bedb Compare March 19, 2015 13:14

Reworked integration tests and added extended tests

47caf77

mfojtik force-pushed the ext_int_test_poc branch from 864bedb to 47caf77 Compare March 19, 2015 13:18

mfojtik mentioned this pull request Mar 19, 2015

make login test avoid default kubeconfig chain #1378

Merged

openshift-bot pushed a commit that referenced this pull request Mar 19, 2015

Merge pull request #1230 from mfojtik/ext_int_test_poc

814026a

Merged by openshift-bot

openshift-bot merged commit 814026a into openshift:master Mar 19, 2015

mfojtik deleted the ext_int_test_poc branch September 5, 2018 21:07

Initial support for extended integration test #1230

Initial support for extended integration test #1230

Uh oh!

Conversation

mfojtik commented Mar 5, 2015

Uh oh!

mfojtik commented Mar 5, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

csrwng commented Mar 5, 2015

Uh oh!

mfojtik commented Mar 5, 2015

Uh oh!

csrwng commented Mar 5, 2015

Uh oh!

mfojtik commented Mar 5, 2015

Uh oh!

csrwng commented Mar 5, 2015

Uh oh!

mfojtik commented Mar 5, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

csrwng commented Mar 5, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

soltysh commented Mar 5, 2015

Uh oh!

soltysh commented Mar 5, 2015

Uh oh!

bparees commented Mar 5, 2015

Uh oh!

bparees commented Mar 5, 2015

Uh oh!

soltysh commented Mar 5, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mfojtik commented Mar 16, 2015

Uh oh!

mfojtik commented Mar 16, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-bot commented Mar 18, 2015

Uh oh!

mfojtik commented Mar 19, 2015

Uh oh!

mfojtik commented Mar 19, 2015

Uh oh!

mfojtik commented Mar 19, 2015

Uh oh!

mfojtik commented Mar 19, 2015

Uh oh!

csrwng commented Mar 19, 2015

Uh oh!

mfojtik commented Mar 19, 2015

Uh oh!

mfojtik commented Mar 19, 2015

Uh oh!

openshift-bot commented Mar 19, 2015

Uh oh!

mfojtik commented Mar 19, 2015

Uh oh!

openshift-bot commented Mar 19, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects