UPSTREAM: 23894: OOM errors when processes exit rapidly

[smarterclayton]

This is on the bubble for 1.2 but I wanted to see if it helps clear up our failures

@ncdc @derekwaynecarr @liggitt [test]

[smarterclayton]

[test]

[ncdc]

@smarterclayton upstream PR was updated and hopefully will fix the type issues in all the right places (by removing the type assertions).

[smarterclayton]

Updated against upstream

[ncdc]

Conformance tests failed, again with the update-demo scaling an RC test. But it doesn't appear to be the same failure text. Still tracking it down. But I did notice that the docker.log that's captured in the jenkins artifacts isn't complete. For example, the failing test created its first container that had an issue at 12:04, but the contents of docker.log starts at 12:08 😢

[smarterclayton]

#8441 is the other failure.

[smarterclayton]

You can extend the docker log time. Going up to 30m is probably fine.

[ncdc]

How do we do that?

I'll spin up a rhel7 vm in ec2 to try to repro manually.

[smarterclayton]

It's in the test failure trap where we shut down the server - we grab the docker logs from the journal in hack/test-end-to-end-docker.sh

[smarterclayton]

[test]

[smarterclayton]

Updated

[ncdc]

Because we run tests in parallel, each test's namespace needs to be added to the various SCCs to ensure upstream e2es can pass against OpenShift's security model. It looks like that code was resulting in each namespace stomping on the other namespaces such that only a single e2e namespace at a time was ever a member of the various SCCs.

#8465 should fix this issue.

[smarterclayton]

@jwforres @spadgett Bindata failure

[spadgett]

@smarterclayton looking at it

[spadgett]

@jwforres new font-awesome update today is breaking us

[smarterclayton]

[test]

[smarterclayton]

Flaked #8399 [test]

On Mon, Apr 11, 2016 at 7:50 PM, OpenShift Bot [email protected]
wrote:

continuous-integration/openshift-jenkins/test FAILURE (
https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/2907/)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#8412 (comment)

[openshift-bot]

Evaluated for origin test up to 9ec799d

[openshift-bot]

continuous-integration/openshift-jenkins/test FAILURE (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/2913/)

[smarterclayton]

Flaked on

Apr 11 20:39:02.217: INFO: Error running &{/data/src/github.com/openshift/origin/_output/local/bin/linux/amd64/oc [oc create --namespace=extended-test-scoped-router-27a6z-0lea8 --config=/tmp/openshift-extended-tests/extended-test-scoped-router-27a6z-0lea8-user.kubeconfig -f /data/src/github.com/openshift/origin/test/extended/fixtures/scoped-router.yaml] []   Error from server: User "extended-test-scoped-router-27a6z-0lea8-user" cannot create pods in project "extended-test-scoped-router-27a6z-0lea8"

[smarterclayton]

Have not seen OOMs reoccur - [merge]

[smarterclayton]

I think that flake is an extended flake w.r.t. the policy cache falling behind. Not sure though - @deads2k?

[openshift-bot]

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/merge_pull_requests_origin/5568/) (Image: devenv-rhel7_3953)

[openshift-bot]

Evaluated for origin merge up to 9ec799d

[deads2k]

I think that flake is an extended flake w.r.t. the policy cache falling behind. Not sure though - @deads2k?

It's likely. We have a method WaitForPolicyUpdate to avoid that problem during our integration tests.

[ncdc]

@pecameron this already merged. No need to re-test.

[deads2k]

@pecameron this already merged. No need to re-test.

Man, he meant it too. :)