Failing test: X-Pack Alerting API Integration Tests.x-pack/test/alerting_api_integration/security_and_spaces/tests/alerting/alerts·ts - alerting api integration security and spaces enabled Alerting alerts superuser at space1 should unmute all instances when unmuting an alert

[kibanamachine]

A test failed on a tracked branch

Error: retry.try timeout: Error: Expected 0 tasks but received 1
    at retry.try (test/alerting_api_integration/common/lib/task_manager_utils.ts:42:15)
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at onFailure (/dev/shm/workspace/kibana/test/common/services/retry/retry_for_success.ts:28:9)
    at retryForSuccess (/dev/shm/workspace/kibana/test/common/services/retry/retry_for_success.ts:68:13)

First failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-test-triage (failed-test)

[kibanamachine]

New failure: Jenkins Build

[spalger]

Test has failed 11 times across master and PRs in the last week

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[brianseeders]

Skipped on master 51d96e5

[pmuellr]

The skipped test is here:

kibana/x-pack/test/alerting_api_integration/security_and_spaces/tests/alerting/alerts.ts

Lines 765 to 774 in 51d96e5

	it.skip(`should unmute all instances when unmuting an alert`, async () => {
	const testStart = new Date();
	const reference = alertUtils.generateReference();
	const response = await alertUtils.createAlwaysFiringAction({
	reference,
	overwrites: {
	enabled: false,
	schedule: { interval: '1s' },
	},
	});

Here's a bit of the ci log from the test. Note the test before the one that reported a failure included a message from ES about using an invalidated API key, which given the log for the actual test failure, makes me think that was the cause.

ci log from test

[00:10:13]                 └-> shouldn't schedule actions when alert instance is muted
[00:10:13]                   └-> "before each" hook: global before each
[00:10:14]                   │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1578403513476867716] Authentication using apikey failed - api key has been invalidated
[00:10:14]                   │ proc [kibana]   log   [14:38:30.411] [error][task_manager] Task actions:test.rate-limit "3c9b0a00-315b-11ea-bae7-037313b790e0" failed: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } } :: {"path":"/_security/user/_has_privileges","query":{},"body":"{\"applications\":[{\"application\":\"kibana-.kibana\",\"resources\":[\"space:space1\"],\"privileges\":[\"version:8.0.0-SNAPSHOT\",\"login:\",\"saved_object:8.0.0-SNAPSHOT:action/get\"]}]}","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/_security/user/_has_privileges]\",\"header\":{\"WWW-Authenticate\":[\"Bearer realm=\\\"security\\\"\",\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/_security/user/_has_privileges]\",\"header\":{\"WWW-Authenticate\":[\"Bearer realm=\\\"security\\\"\",\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}},\"status\":401}","wwwAuthenticateDirective":"Bearer realm=\"security\", ApiKey, Basic realm=\"security\" charset=\"UTF-8\""}
[00:10:16]                   │ debg --- retry.try error: Expected 2 but received 0.
[00:10:17]                   │ debg --- retry.try failed again with the same message...
[00:10:17]                   │ debg --- retry.try failed again with the same message...
[00:10:18]                   │ debg --- retry.try error: Expected 2 but received 1.
[00:10:18]                   │ debg --- retry.try failed again with the same message...
...
[00:10:21]                   └- ✓ pass  (8.2s) "alerting api integration security and spaces enabled Alerting alerts superuser at space1 shouldn't schedule actions when alert instance is muted"
[00:10:21]                 └-> "after each" hook

[00:10:22]                 └-> should unmute all instances when unmuting an alert

[00:10:22]                   └-> "before each" hook: global before each

[00:10:27]                   │ debg --- retry.try error: Expected 1 but received 0.

[00:10:28]                   │ debg --- retry.try failed again with the same message...

...

[00:10:29]                   │ info [o.e.x.s.a.AuthenticationService] [kibana-ci-immutable-centos-tests-xl-1578403513476867716] Authentication using apikey failed - api key has been invalidated

[00:10:29]                   │ proc [kibana]   log   [14:38:45.418] [error][task_manager] Task actions:test.index-record "6744d290-315b-11ea-bae7-037313b790e0" failed: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Bearer realm="security"" & 1="ApiKey" & 2="Basic realm="security" charset="UTF-8"" } } } :: {"path":"/_security/user/_has_privileges","query":{},"body":"{"applications":[{"application":"kibana-.kibana","resources":["space:space1"],"privileges":["version:8.0.0-SNAPSHOT","login:","saved_object:8.0.0-SNAPSHOT:action/get"]}]}","statusCode":401,"response":"{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_security/user/_has_privileges]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_security/user/_has_privileges]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}","wwwAuthenticateDirective":"Bearer realm="security", ApiKey, Basic realm="security" charset="UTF-8""}

[00:10:29]                   │ debg --- retry.try failed again with the same message...

[00:10:30]                   │ debg --- retry.try failed again with the same message...

[00:11:31]                   │ debg --- retry.try error: Expected 0 tasks but received 1

[00:11:31]                   │ debg --- retry.try failed again with the same message...

...

[00:13:31]                   └- ✖ fail: "alerting api integration security and spaces enabled Alerting alerts superuser at space1 should unmute all instances when unmuting an alert"

[pmuellr]

Note that @FrankHassanabad has seen some of the missing authentication credentials for REST request log messages as well, in the SIEM tests - guessing it's the same symptom, whatever the cause happens to be.