Added kafkaQueueProblems scenarios

[yuliia-fryshko]

Closes https://github.com/elastic/obs-ai-team/issues/554

Add observability AI eval tests covering the Kafka queue problems flow.

[SrdjanLL]

LGTM (code review only)!

Before merging, is there anything worth noting about evals results? Are all criteria consistently satisfied and right tools/skill invoked?

Asking because the related issue mentions:

Make improvements to the skill if necessary.

[yuliia-fryshko]

LGTM (code review only)!

Before merging, is there anything worth noting about evals results? Are all criteria consistently satisfied and right tools/skill invoked?

Asking because the related issue mentions:

Make improvements to the skill if necessary.

Thank you, @SrdjanLL !
I post the results here.
As you can see, the Criteria results are low, and the explanation indicates that it couldn’t identify the issue in the upstream service. This should be addressed in this issue https://github.com/elastic/obs-ai-team/issues/569.

The output correctly identifies that the slowness is due to an artificial sleep in the fraud-detection service (the consumer) rather than a real Kafka delay, attributing the issue to the consumer side. "mjhq": The output does not identify the upstream producer (checkout) or mention it at all. "zudi": The output does not identify a throughput mismatch between checkout and fraud-detection. "exlp": The output does not mention checkout logs or excessive Kafka messages per order. "fuvk": The output only suggests actions for the fraud-detection service and does not mention investigating the checkout service for excessive duplicate messages.

[kibanamachine]

Starting backport for target branches: 9.4

https://github.com/elastic/kibana/actions/runs/25381379199

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	9.4	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 267402

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Starting backport for target branches: 9.4

https://github.com/elastic/kibana/actions/runs/25386121464

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	9.4	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 267402

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko