[ResponseOps][Alerting] Flaky test x-pack/test/alerting_api_integration/security_and_spaces/group2/tests/telemetry/index·ts

[doakalexi]

Resolves #140973
Resolves #136153

Summary

Fixes a telemetry bug and a flaky telemetry test

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2717 x 250
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2718 x 250

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

Generally LGTM. I'd like to get the equal -> gte change in the getEventLog() call - that feels like it could someday go flaky.

The others I'm not so sure about. Mainly because I'm missing some context. Could you provide some comments in the PR for:

• why the originating issues were flaky
• why they aren't flaky anymore
• it seems like we removed some rules being tested - I'm curious why

[pmuellr]

Unfortunate to have to leave that at potentially zero! Looking at the code, I think we'd have to call getEventLog() on every rule, to make sure it has at least one execute, before requesting the telemetry. Then we could use the number of rules instead of zero.

[doakalexi]

I can look into this too and see if I can use the number of rules!

[doakalexi]

Resolved in this commit a273697

[doakalexi]

Changing this to non 0 caused the test to be flaky again, I'd like to just leave leave it as > 0

[pmuellr]

heh, no problem!

[doakalexi]

• why the originating issues were flaky

The original test was flaky from this block of code for both actions and alerting telemetry

await retry.try(async () => {
        const telemetryTask = await es.get<TaskManagerDoc>({
          id: `task:Actions-actions_telemetry`,
          index: '.kibana_task_manager',
        });
        expect(telemetryTask!._source!.task?.status).to.be('idle');
        const taskState = telemetryTask!._source!.task?.state;
        expect(taskState).not.to.be(undefined);
        actionsTelemetry = JSON.parse(taskState!);
        expect(actionsTelemetry.runs).to.equal(2);
        expect(actionsTelemetry.count_total).to.equal(20);
      });

The number of actionsTelemetry.runs was not always 2, and I believe looking for an exact number of runs was causing the expect(telemetryTask!._source!.task?.status).to.be('idle') to also be flaky where sometimes it would be 'running' instead of 'idle'. This is why I removed looking for an exact count of 2 runs and only wanted to verify that there had been at least one run.

• why they aren't flaky anymore

I simplified the test, added the object remover, and updated the code block above to remove the flaky parts I mentioned.

• it seems like we removed some rules being tested - I'm curious why

I removed some rules to help clean up the test, I thought it was a hard to understand and some of the rules seemed to be verifying redundant things. I only kept the minimal amount of rules I needed to verify the telemetry results.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 3823f25

Metrics [docs]

✅ unchanged

History

• 💔 Build #145266 failed 8f37fb4
• 💔 Build #145232 failed df75869
• 💚 Build #144752 succeeded 407ff19
• 💔 Build #144731 failed c99ee36
• 💔 Build #144396 failed eaf446a
• 💚 Build #141029 succeeded 6f5643b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[pmuellr]

LGTM, thanks for changes (and attempts at changes!)