Skip to content

Comments

give integration tests more time to run#2570

Merged
jschaul merged 2 commits intodevelopfrom
integrations-timeout
Jul 18, 2022
Merged

give integration tests more time to run#2570
jschaul merged 2 commits intodevelopfrom
integrations-timeout

Conversation

@jschaul
Copy link
Member

@jschaul jschaul commented Jul 18, 2022

Possibly help with some CI failures we have been seeing of the type

Error: unable to get pod logs for test-cbwmwl9q8aha-wire-server-cargohold-integration: pods "test-cbwmwl9q8aha-wire-server-cargohold-integration" not found

CI runs make kube-integration-test which is an alias for ./hack/bin/integration-tests.sh which runs this line:

helm test --logs -n "${NAMESPACE}" "${NAMESPACE}-${CHART}" --timeout 600s

helm test --logs will run all tests as defined under ./charts/<chart-e.g.brig>/templates/tests/*, then post all logs. My hunch is that tests did not have enough time, so there's actually a timeout; but the error doesn't say timeout, but "can't find this pod I'd expect to have started"

By looking at these timings, we can see the last log entry from being done setting up all the non-test pods, it's 17:57, and the first line of the log output from the test pods is 10 minutes (600s) afterwards, at 18:07:

logs

This change should give more time to integration tests, and therefore avoid the current issues.

@jschaul jschaul temporarily deployed to cachix July 18, 2022 16:34 Inactive
@jschaul
Copy link
Member Author

jschaul commented Jul 18, 2022

Indeed, galley and brig integration suites are each taking around 9 minutes (on one sample - successful - run), and with a bit of extra load on the cluster this can easily go above the 10 minute timeout currently set.

│     Command:                                                                                                                                                                               │
│       brig-integration                                                                                                                                                                     │
│       --pattern                                                                                                                                                                            │
│       !/turn/ && !/user.auth.cookies.limit/                                                                                                                                                │
│     State:          Terminated                                                                                                                                                             │
│       Reason:       Completed                                                                                                                                                              │
│       Exit Code:    0                                                                                                                                                                      │
│       Started:      Mon, 18 Jul 2022 18:22:53 +0200                                                                                                                                        │
│       Finished:     Mon, 18 Jul 2022 18:31:13 +0200 
│     Image:          quay.io/wire/galley-integration:0.0.2-pr.25                                                                                                                            │
│     Image ID:       docker-pullable://quay.io/wire/galley-integration@sha256:3557890b48a5bf132ff705b3ceba5d91f91fc7ef924fc62152f9935b40a04b30                                              │
│     Port:           <none>                                                                                                                                                                 │
│     Host Port:      <none>                                                                                                                                                                 │
│     State:          Terminated                                                                                                                                                             │
│       Reason:       Completed                                                                                                                                                              │
│       Exit Code:    0                                                                                                                                                                      │
│       Started:      Mon, 18 Jul 2022 18:31:48 +0200                                                                                                                                        │
│       Finished:     Mon, 18 Jul 2022 18:40:34 +0200  

@jschaul jschaul merged commit 95db2ba into develop Jul 18, 2022
@jschaul jschaul deleted the integrations-timeout branch July 18, 2022 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant