Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[It] should run a deployment to completion and then scale to zero [Conformance] #9195

Closed
bparees opened this issue Jun 7, 2016 · 9 comments
Assignees
Labels
component/apps kind/test-flake Categorizes issue or PR as related to test flakes. priority/P2

Comments

@bparees
Copy link
Contributor

bparees commented Jun 7, 2016

deploymentconfigs
/data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:213
  with test deployments
  /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:189
    should run a deployment to completion and then scale to zero [Conformance] [It]
    /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:188

    Expected
        <string>: --> pre: Running hook pod ...
        test pre hook executed
        --> pre: Success
        --> Scaling up deployment-test-2 from 0 to 1, scaling down deployment-test-1 from 0 to 0 (keep 1 pods available, don't exceed 2 pods)
            Scaling deployment-test-2 up to 1
    to contain substring
        <string>: --> Success

    /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:183
------------------------------
[k8s.io] Pods 
  should *not* be restarted with a docker exec "cat /tmp/health" liveness probe [Conformance]
  /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/test/e2e/pods.go:713

as seen in:
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/1704

@bparees bparees added component/apps kind/test-flake Categorizes issue or PR as related to test flakes. labels Jun 7, 2016
@mfojtik mfojtik assigned 0xmichalis and unassigned mfojtik Jun 7, 2016
@smarterclayton
Copy link
Contributor

The pod for the deployment comes up, but looks like it never becomes ready. It is terminated after 200s when the deployment fails.

@mfojtik
Copy link
Contributor

mfojtik commented Jun 7, 2016

I saw this flake under the rebase branch.

@stevekuznetsov stevekuznetsov changed the title deployments pre hook test flake deployments pre hook test flake on test: deploymentconfigs with test deployments [It] should run a deployment to completion and then scale to zero [Conformance] Jun 8, 2016
@stevekuznetsov
Copy link
Contributor

Changed title for searchability

@0xmichalis 0xmichalis changed the title deployments pre hook test flake on test: deploymentconfigs with test deployments [It] should run a deployment to completion and then scale to zero [Conformance] [It] should run a deployment to completion and then scale to zero [Conformance] Jun 23, 2016
@bparees
Copy link
Contributor Author

bparees commented Jul 13, 2016

@bparees
Copy link
Contributor Author

bparees commented Jul 14, 2016

@smarterclayton we're at a month on this... are we going to fix this test or disable it?

@smarterclayton
Copy link
Contributor

Fix it, because it's indicating we're broken.

@Kargakis the only way to get this is if we changed the behavior of test deployments.

• Failure [18.357 seconds]
deploymentconfigs
/data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:515
  with test deployments
  /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:254
    should run a deployment to completion and then scale to zero [Conformance] [It]
    /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:253

    Expected error:
        <*errors.errorString | 0xc821423b30>: {
            s: "deployment is complete but doesn't have expected spec replicas: 2 0",
        }
        deployment is complete but doesn't have expected spec replicas: 2 0
    not to have occurred

    /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:213

So at some point since june 7 we broke test deployments.

@smarterclayton
Copy link
Contributor

I can't see where it changed, but the easy fix is to move the check for the invariant (scale should be zero) into the end of the deployments controller loop, so we guarantee that before we ever set complete or failed scale is zero, we'd be ok.

@smarterclayton
Copy link
Contributor

Pulling up a fix.

@smarterclayton
Copy link
Contributor

It's also possible that we are triggering a failure condition (due to the last rebase slowing down certain operations) that we didn't previously. Either way, this guarantees we never mark an RC complete or failed without setting scale to zero.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/apps kind/test-flake Categorizes issue or PR as related to test flakes. priority/P2
Projects
None yet
Development

No branches or pull requests

6 participants