[ ERR] Gave up waiting for: '[[ "$(curl -s -X POST http://${FRONTEND_IP}:5432/keys/foo -d value=1337)" = "Key created" ]]'

[deads2k]

from e2e: https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/2370

[INFO] Waiting for command to finish: '[[ "$(curl -s -X POST http://${FRONTEND_IP}:5432/keys/foo -d value=1337)" = "Key created" ]]'...
[ ERR] Gave up waiting for: '[[ "$(curl -s -X POST http://${FRONTEND_IP}:5432/keys/foo -d value=1337)" = "Key created" ]]'
[ERROR] PID 31876: test/end-to-end/core.sh:306: `return 1` exited with status 1.

[deads2k]

@bparees are you familiar with the e2e app? My look at the logs suggests that it should be successfully POSTing.

[bparees]

yeah, it's not obvious to me. based on the output it appears the curl request never even got to the app (the wait_for_command timed out on the curl, it didn't return a failure code).

if we have container logs for the app pod, that might shed some light, but given that the welcome page returned successfully it seems like the app is working, unless the DB connection failed.

[deads2k]

if we have container logs for the app pod, that might shed some light, but given that the welcome page returned successfully it seems like the app is working, unless the DB connection failed.

Those are attached to the jenkins job under status and labeled with the container names. There are actually two pods started and they both seem to be getting the POSTs.

[smarterclayton]

Hrm, I've never seen that one fail before (or not in months).

[bparees]

yeah was just looking at the posts... sure looks like the app is getting the post and responding with a 200. No idea why curl isn't getting that response back.

[deads2k]

Hrm, I've never seen that one fail before (or not in months).

It was triggering the SCC flake issue people started linking recently, so something broke.

[bparees]

this happened again here:
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/2587/consoleFull

given that the app is responding with a 200, i'm inclined to question whether the service proxy is getting the response to the client...

@knobunc see the failure here:

[ ERR] Gave up waiting for: '[[ "$(curl -s -X POST http://${FRONTEND_IP}:5432/keys/foo -d value=1337)" = "Key created" ]]'

which comes from this test:
https://github.com/openshift/origin/blob/master/test/end-to-end/core.sh#L42

and you can see the app pod logs returning 200 for the POST operation here:
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/2587/artifact/origin/artifacts/test-end-to-end-docker/logs/container-frontend-1-3bk00-ruby-helloworld.log
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/2587/artifact/origin/artifacts/test-end-to-end-docker/logs/container-frontend-1-43l9r-ruby-helloworld.log

[0xmichalis]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/2657/consoleFull

[0xmichalis]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/2679/consoleFull

[gabemontero]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/2701/console

[bparees]

i've been seeing this one a lot the last couple days. I don't feel totally convinced by my service proxy theory, but I'm not sure what else it could be. Per @deads2k suggestion i'm going to update the curl command to report what it is getting back, in case that offers more insight.

[bparees]

attempting to recreate w/ full shell debug here:
#9676

[bparees]

further debugging on this issue has revealed we're getting back "key updated" instead of "key created" because apparently one of the earlier curl commands actually succeeded in creating the key, despite curl returning an RC=7(??).

will see if i can tighten up the logic that's testing the responses.

[deads2k]

Still happening: https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_integration/2854/

@stevekuznetsov @bparees suggested you.

[bparees]

dupe of #9685