Use custom script to invoke registry extended test suite

[miminar]

The test/extended/core.sh script does not allow to properly focus only the desired tests to run.

This script contains the necessary bits from the core.sh scripts to setup the environment and run just the registry tests.

Once openshift/origin#16846 merges, the registry tests won't be runnable using the extended test scripts because most of them are run serially.

[miminar]

/assign @Kargakis
/cc @bparees

I haven't tested it. I'm not sure what I need to do to test it in CI before merging. I'd welcome any pointers.

[bparees]

run hack/generate.sh and include those changes.

[bparees]

should not need merge_junit anymore. i'm trying to get it removed here:
https://github.com/openshift/origin/pull/16068/files

[bparees]

and for something like this that isn't going to break anyone else, we can just test it live. i'm guessing you do not have access to create jobs on the jenkins server, however?

[stevekuznetsov]

What's the developer workflow to do this from an Origin checkout?

[bparees]

@stevekuznetsov there isn't a good one. and i'm not happy about it, but go see openshift/origin#16846 for that discussion.

[stevekuznetsov]

I think we will fail to detect $OS_ROOT correctly since the script that runs is not under the Origin tree. I may have made that work in the past but I can't remember. You should test that this script, when saved in some part of the filesystem NOT in the Origin tree, works as expected.

[miminar]

Can I at least assume that $OS_ROOT is set?

[stevekuznetsov]

No, nothing sets it

[bparees]

@miminar i'm holding merging/creating this based on @stevekuznetsov's response, but if i'm wrong let's talk tomorrow and i can set it up for you and we can see what happens.

[miminar]

So the "$(pwd)" holds the truth. I've made "${OS_ROOT}" guessing a bit more robust in this script just in case...

[miminar]

Comments addressed (hopefully).

and for something like this that isn't going to break anyone else, we can just test it live

OK, let's test it live.

[miminar]

I've created new test_pull_request_origin_extended_image_registry_custom jenkins job to try out the changes. I'll update this PR once I get it working.

Update: tested & updated

[miminar]

Any other comments?

[bparees]

@miminar in light of openshift/origin#16919 is this still necessary?

[bparees]

also i only see one [registry] test? (registry.go)

Are all the registry tests in that one file?

(there's also signature.go but that is all disabled)

[miminar]

$ ack --noheading --go '\[Feature:Image|\[registry\]' -- test/extended/
test/extended/images/hardprune.go:20:var _ = g.Describe("[Feature:ImagePrune][Serial] Image hard prune", func() {
test/extended/images/prune.go:31:var _ = g.Describe("[Feature:ImagePrune][Serial] Image prune", func() {
test/extended/images/resolve.go:14:var _ = g.Describe("[Feature:ImageLookup] Image policy", func() {
test/extended/imageapis/limitrange_admission.go:24:var _ = g.Describe("[Feature:ImageQuota][Serial] Image limit range", func() {
test/extended/imageapis/quota_admission.go:29:var _ = g.Describe("[Feature:ImageQuota][Serial] Image resource quota", func() {
test/extended/registry/registry.go:28:var _ = g.Describe("[Conformance][registry][migration][Serial] manifest migration from etcd to registry storage", func() {
test/extended/registry/signature.go:15:var _ = g.Describe("[imageapis][registry][Skipped] image signature workflow", func() {

Registry tests often begin with [Feature:Image as can be seen above.

The problem with the current regexp "\[Feature:Image|\[registry\].*?\[Serial\]" is that either [registry] followed by [Serial] will be matched or just [Feature:Image will be matched. Therefor several tests will be run twice:

• [Conformance][registry][migration][Serial] manifest migration from etcd to registry storage" is matched both by FOCUS="\[Serial\].*?${t}" and FOCUS="${t}.*?\[Serial\]".
• [Feature:ImageLookup] Image policy is matched both by FOCUS="${t}.*?\[Serial\]" and the parallel test suite.
  • the same applies to the other registry tests missing [Serial] tag.

The focus of this job could be rewritten to (\[Feature:Image|\[registry\]) but I still would like this job to explicitly pick up the particular tests to run. Run them just once and run them all serial - because it's much more easy to debug. Since this job is run only against registry related PRs, it shouldn't pose a problem. Debugging a failure with a registry log containing overlapping pushes from 3 different tests run in parallel is a task I'd like to avoid.

[bparees]

@miminar how about we just consistently tag all our registry tests with [registry] or [image] and then you can set the focus to [registry] (or [image]) and be done with it?

[stevekuznetsov]

In the test env you can be certain Origin is at /data/src/github.com/openshift/origin

[miminar]

@miminar how about we just consistently tag all our registry tests with [registry] or [image] and then you can set the focus to [registry] (or [image]) and be done with it?

I'd prefer [registry]. But I'd still preserve [Feature:ImagePruning] tag for local testing.
Shall we address our desire to run registry tests serially by using [Serial] tag as well? We will eventually switch to parallel test runs when each registry test will spawn its own registry replica.

[bparees]

I'd prefer [registry]. But I'd still preserve [Feature:ImagePruning] tag for local testing.

sure, you can have as many tags as you like.

Shall we address our desire to run registry tests serially by using [Serial] tag as well?

yes.

[miminar]

Closing in favor of openshift/origin#16967