vendor/k8s.io/kubernetes/test/e2e/upgrades/apps/job: List Pods in failure message

[wking]

Currently, this test can fail with the not-very-helpful:

fail [k8s.io/kubernetes/test/e2e/upgrades/apps/job.go:58]: Expected
    <bool>: false
to be true

This pull request backports kubernetes/kubernetes@96b04bfeac (kubernetes/kubernetes#77716) to get a more useful error message.

The backport didn't apply cleanly. I think I made the appropriate adjustments, but give it a careful eyeball to make sure ;).

[openshift-ci-robot]

@wking: This pull request references a valid Bugzilla bug.

Details

In response to this:

Bug 1708454: vendor/k8s.io/kubernetes/test/e2e/upgrades/apps/job: List Pods in failure message

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: wking
To complete the pull request process, please assign marun
You can assign the PR to them by writing /assign @marun in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• vendor/k8s.io/kubernetes/test/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[smarterclayton]

/hold

only to debug why this might be failing, we would merge to master first before this

(expect to get data from test runs)

[wking]

Fixed the compilation error with d44ff45 -> 795c7cb.

[openshift-ci-robot]

@wking: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-upgrade	795c7cb	link	/test e2e-aws-upgrade

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[wking]

Hey, first run :).

fail [k8s.io/kubernetes/test/e2e/upgrades/apps/job.go:49]: Expected error:
    <*errors.errorString | 0xc0022b26e0>: {
        s: "job has 0 of 2 expected running pods: ",
    }
    job has 0 of 2 expected running pods: 
not to have occurred

So that's "no Pods", not "Pods waiting to be scheduled and just not running yet". Dunno what to do about that :/

[soltysh]

This failure was caused by problems with node:

Jun 13 18:15:29.003: INFO: Waiting up to 3m0s for all (but 100) nodes to be ready
Jun 13 18:15:29.024: INFO: Condition Ready of node ip-10-0-148-26.ec2.internal is false, but Node is tainted by NodeController with [{node-role.kubernetes.io/master  NoSchedule <nil>} {node.kubernetes.io/unschedulable  NoSchedule 2019-06-13 18:14:29 +0000 UTC} {node.kubernetes.io/unreachable  NoSchedule 2019-06-13 18:15:20 +0000 UTC} {node.kubernetes.io/unreachable  NoExecute 2019-06-13 18:15:26 +0000 UTC}].

Let's keep re-testing it.
/retest

[wking]

$ curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/23161/pull-ci-openshift-origin-release-4.1-e2e-aws-upgrade/51/artifacts/e2e-aws-upgrade/nodes.json | jq -r '.items[] | .conditions = ([.status.conditions[] | {key: .type, value: .}] | from_entries) | .conditions.Ready.lastTransitionTime + " " + .conditions.Ready.status + " " + .metadata.name' | sort
2019-06-13T17:35:31Z True ip-10-0-160-149.ec2.internal
2019-06-13T17:40:33Z True ip-10-0-169-80.ec2.internal
2019-06-13T17:41:09Z True ip-10-0-130-244.ec2.internal
2019-06-13T18:13:38Z True ip-10-0-138-20.ec2.internal
2019-06-13T18:15:01Z True ip-10-0-150-197.ec2.internal
2019-06-13T18:15:20Z Unknown ip-10-0-148-26.ec2.internal

Looks like that node was just gone, with no sign of return by that cluster-teardown log collection.

[sjenning]

I can't find evidence of the pods being created. There are no pods from a foo Job in the events at all, much less that have a FailedScheduling event. Maybe the KCM isn't running or doesn't have a leader?

[wking]

Follow-up in openshift/machine-config-operator#855

[deads2k]

In this case, kubelet ip-10-0-169-80 had two scheduled job pods at 18:12 and there are more kubelet logs going back for a considerable time showing they were available. grep the worker's journals for job-upgrade to see them.

The namespace wasn't removed until 18:11 or so.

[openshift-ci-robot]

@wking: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

vendor/k8s.io/kubernetes/test/e2e/upgrades/apps/job: List Pods in failure message

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[eparis]

we have a better patch in 4.2 and more jobs. So I'm going to close this.