Skip to content

[Obs AI Assistant] Improve flaky recall tests#220638

Merged
sorenlouv merged 3 commits intoelastic:mainfrom
sorenlouv:fix-flaky-recall-tests
May 9, 2025
Merged

[Obs AI Assistant] Improve flaky recall tests#220638
sorenlouv merged 3 commits intoelastic:mainfrom
sorenlouv:fix-flaky-recall-tests

Conversation

@sorenlouv
Copy link
Copy Markdown
Member

@sorenlouv sorenlouv commented May 9, 2025 •
edited by kibanamachine
Loading

This improves flakiness in API tests by:

  • re-adding the write-block while re-indexing
  • Updating the alias for the write index before starting the re-index operation. This ensures the entries that are being indexed during the re-index operation has started will not be lost.
  • Add a query param wait_until_complete to the setup endpoint that ensures that it only returns after the model is ready and re-index complete.
  • Improved logging for more details

@sorenlouv sorenlouv requested a review from a team as a code owner May 9, 2025 10:42
@botelastic botelastic bot added ci:project-deploy-observability Create an Observability project Team:Obs AI Assistant Observability AI Assistant labels May 9, 2025
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented May 9, 2025

Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented May 9, 2025

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@sorenlouv sorenlouv added backport:version Backport to applied version labels v9.1.0 v8.19.0 release_note:skip Skip the PR/issue when compiling release notes ci:build-serverless-image ci:build-cloud-image labels May 9, 2025
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented May 9, 2025 •
edited
Loading

💛 Build succeeded, but was flaky

  • Buildkite Build
  • Commit: d96b904
  • Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-220638-d96b904eea82

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #16 / Cloud Security Posture Vulnerabilities Page - Grouping Default Grouping groups vulnerabilities by cloud account and sort by number of vulnerabilities desc
  • [job] [logs] Jest Tests #11 / HistoricalResults Super Date Picker when new date is selected should invoke fetchHistoricalResults with new start and end dates, from: 0 and remaining fetch query opts

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
observabilityAIAssistant 401 402 +1
Unknown metric groups

API count

id before after diff
observabilityAIAssistant 403 404 +1

History

@sorenlouv sorenlouv merged commit ed1f558 into elastic:main May 9, 2025
28 checks passed
@sorenlouv sorenlouv deleted the fix-flaky-recall-tests branch May 9, 2025 13:21
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented May 9, 2025

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/14929902564

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented May 9, 2025

💔 All backports failed

Status Branch Result
❌ 8.19 Backport failed because of merge conflicts

You might need to backport the following PRs to 8.19:
- [Obs AI Assistant] Use update-by-query for semantic_text migration (#220255)
- [Obs AI Assistant] Add support for dense vector embeddings in API tests (#219199)

Manual backport

To create the backport manually run:

node scripts/backport --pr 220638

Questions ?

Please refer to the Backport tool documentation

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented May 9, 2025

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#8257

[✅] x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.ai_assistant.serverless.config.ts: 100/100 tests passed.
[✅] x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts: 100/100 tests passed.

see run history

kdelemme pushed a commit to kdelemme/kibana that referenced this pull request May 9, 2025
@sorenlouv @kdelemme
[Obs AI Assistant] Improve flaky recall tests (elastic#220638)
b441e81
@kibanamachine kibanamachine mentioned this pull request May 12, 2025
Merged
@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label May 13, 2025
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented May 13, 2025

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 220638 locally
cc: @sorenlouv

sorenlouv added a commit to sorenlouv/kibana that referenced this pull request May 14, 2025
@sorenlouv
[Obs AI Assistant] Improve flaky recall tests (elastic#220638)
f789254 
(cherry picked from commit ed1f558)
@sorenlouv sorenlouv mentioned this pull request May 14, 2025
Closed
@sorenlouv
Copy link
Copy Markdown
Member Author

sorenlouv commented May 14, 2025

💚 All backports created successfully

Status Branch Result
✅ 8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

sorenlouv added a commit to sorenlouv/kibana that referenced this pull request May 14, 2025
@sorenlouv
[Obs AI Assistant] Improve flaky recall tests (elastic#220638)
9c1e960 
(cherry picked from commit ed1f558)
@sorenlouv sorenlouv mentioned this pull request May 14, 2025
Merged
@sorenlouv
Copy link
Copy Markdown
Member Author

sorenlouv commented May 14, 2025

💚 All backports created successfully

Status Branch Result
✅ 8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented May 14, 2025

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @sorenlouv

sorenlouv added a commit that referenced this pull request May 14, 2025
@sorenlouv
[8.19] [Obs AI Assistant] Improve flaky recall tests (#220638) | [Obs…
9ae3b9d 
… AI Assistant] Do not attempt to overwrite index mappings if inference id in component template has changed (#220739) (#220791)

# Backport

This will backport the following commits from `main` to `8.19`:
- [[Obs AI Assistant] Improve flaky recall tests
(#220638)](#220638)
- [[Obs AI Assistant] Do not attempt to overwrite index mappings if
inference id in component template has changed
(#220739)](#220739)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Søren
Louv-Jansen","email":"soren.louv@elastic.co"},"sourceCommit":{"committedDate":"2025-05-09T13:21:07Z","message":"[Obs
AI Assistant] Improve flaky recall tests
(#220638)","sha":"ed1f558f30127b36f450e9b9f991fcc1c13c36c3","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport
missing","ci:build-cloud-image","ci:build-serverless-image","Team:Obs AI
Assistant","ci:project-deploy-observability","backport:version","v9.1.0","v8.19.0"],"title":"[Obs
AI Assistant] Improve flaky recall
tests","number":220638,"url":"https://github.com/elastic/kibana/pull/220638","mergeCommit":{"message":"[Obs
AI Assistant] Improve flaky recall tests
(#220638)","sha":"ed1f558f30127b36f450e9b9f991fcc1c13c36c3"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/220638","number":220638,"mergeCommit":{"message":"[Obs
AI Assistant] Improve flaky recall tests
(#220638)","sha":"ed1f558f30127b36f450e9b9f991fcc1c13c36c3"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/220790","number":220790,"state":"OPEN"}]},{"author":{"name":"Søren
Louv-Jansen","email":"soren.louv@elastic.co"},"sourceCommit":{"committedDate":"2025-05-12T15:11:03Z","message":"[Obs
AI Assistant] Do not attempt to overwrite index mappings if inference id
in component template has changed (#220739)\n\nImprove flaky API tests
and avoid errors
like:\n\n```\n[ERROR][plugins.observabilityAIAssistant.service] Failed
to PUT mapping for .kibana-observability-ai-assistant-kb:
illegal_argument_exception\n \tRoot causes:\n
\t\tillegal_argument_exception: Mapper for [semantic_text] conflicts
with existing mapper:\n \tCannot update parameter [inference_id] from
[pt_tiny_elser_inference_id] to
[pt_tiny_text_embedding_inference_id]\n```","sha":"d2f25df5f3c8337ae9237093fde05d43660e0cf2","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Obs
AI
Assistant","ci:project-deploy-observability","backport:version","v9.1.0","v8.19.0"],"title":"[Obs
AI Assistant] Do not attempt to overwrite index mappings if inference id
in component template has
changed","number":220739,"url":"https://github.com/elastic/kibana/pull/220739","mergeCommit":{"message":"[Obs
AI Assistant] Do not attempt to overwrite index mappings if inference id
in component template has changed (#220739)\n\nImprove flaky API tests
and avoid errors
like:\n\n```\n[ERROR][plugins.observabilityAIAssistant.service] Failed
to PUT mapping for .kibana-observability-ai-assistant-kb:
illegal_argument_exception\n \tRoot causes:\n
\t\tillegal_argument_exception: Mapper for [semantic_text] conflicts
with existing mapper:\n \tCannot update parameter [inference_id] from
[pt_tiny_elser_inference_id] to
[pt_tiny_text_embedding_inference_id]\n```","sha":"d2f25df5f3c8337ae9237093fde05d43660e0cf2"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/220739","number":220739,"mergeCommit":{"message":"[Obs
AI Assistant] Do not attempt to overwrite index mappings if inference id
in component template has changed (#220739)\n\nImprove flaky API tests
and avoid errors
like:\n\n```\n[ERROR][plugins.observabilityAIAssistant.service] Failed
to PUT mapping for .kibana-observability-ai-assistant-kb:
illegal_argument_exception\n \tRoot causes:\n
\t\tillegal_argument_exception: Mapper for [semantic_text] conflicts
with existing mapper:\n \tCannot update parameter [inference_id] from
[pt_tiny_elser_inference_id] to
[pt_tiny_text_embedding_inference_id]\n```","sha":"d2f25df5f3c8337ae9237093fde05d43660e0cf2"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label May 14, 2025
This was referenced May 15, 2025
Closed
Closed
akowalska622 pushed a commit to akowalska622/kibana that referenced this pull request May 29, 2025
@sorenlouv @akowalska622
[Obs AI Assistant] Improve flaky recall tests (elastic#220638)
43d0db4
qn895 pushed a commit to qn895/kibana that referenced this pull request Jun 3, 2025
@sorenlouv @qn895
[Obs AI Assistant] Improve flaky recall tests (elastic#220638)
cae8e68
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@viduni94 viduni94 viduni94 approved these changes

Assignees

No one assigned

Labels

backport:version Backport to applied version labels ci:build-cloud-image ci:build-serverless-image ci:project-deploy-observability Create an Observability project release_note:skip Skip the PR/issue when compiling release notes Team:Obs AI Assistant Observability AI Assistant v8.19.0 v9.1.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sorenlouv @elasticmachine @kibanamachine @viduni94