Skip to content

[Obs AI Assistant] Attempt to fix flaky KB migration test#206324

Merged
viduni94 merged 3 commits intoelastic:mainfrom
viduni94:attempt-to-fix-flaky-KB-migration-test
Jan 13, 2025
Merged

[Obs AI Assistant] Attempt to fix flaky KB migration test#206324
viduni94 merged 3 commits intoelastic:mainfrom
viduni94:attempt-to-fix-flaky-KB-migration-test

Conversation

@viduni94
Copy link
Copy Markdown
Contributor

@viduni94 viduni94 commented Jan 10, 2025 •
edited by kibanamachine
Loading

Summary

Closes #202926

Problem

The KB migration test appears to be flaky sometimes, because the migration takes a few seconds to run. Therefore, when the test expects to have semantic_text for all entries, it fails.

Solution

Adding a retry statement to avoid the failure in cases where the migration takes a little while to run.

Checklist

  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

@viduni94 viduni94 added release_note:skip Skip the PR/issue when compiling release notes Team:Obs AI Assistant Observability AI Assistant backport:version Backport to applied version labels v8.18.0 labels Jan 10, 2025
@viduni94 viduni94 self-assigned this Jan 10, 2025
@viduni94 viduni94 requested a review from a team as a code owner January 10, 2025 18:36
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jan 10, 2025

Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)

@botelastic botelastic bot added the ci:project-deploy-observability Create an Observability project label Jan 10, 2025
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 10, 2025

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Jan 10, 2025

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7698

[✅] x-pack/test/observability_ai_assistant_api_integration/enterprise/config.ts: 200/200 tests passed.

see run history

@sorenlouv
Copy link
Copy Markdown
Contributor

sorenlouv commented Jan 12, 2025 •
edited
Loading

What happens if you run the flaky test runner on main? I want to be sure that retrying actually improves this

@viduni94 viduni94 force-pushed the attempt-to-fix-flaky-KB-migration-test branch from b14c8bd to 36b628e Compare January 13, 2025 13:40
@viduni94
Copy link
Copy Markdown
Contributor Author

viduni94 commented Jan 13, 2025

What happens if you run the flaky test runner on main? I want to be sure that retrying actually improves this

@sorenlouv
Here are few test runs via the flaky test runner (before my changes) where this test appeared flaky:

  1. https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7682
  2. https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7676

The expectation behind adding a retry statement was to retry the test until the migration completes.

The flakiness was resolved as seen in this test run, after adding the retry statement.

@viduni94 viduni94 force-pushed the attempt-to-fix-flaky-KB-migration-test branch from 5bad10e to 6a0a28d Compare January 13, 2025 15:34
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jan 13, 2025 •
edited
Loading

💚 Build Succeeded

  • Buildkite Build
  • Commit: 6a0a28d
  • Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-206324-6a0a28d4bd62

Metrics [docs]

✅ unchanged

History

  • 💚 Build #266509 succeeded 36b628e3fb36c6c9c4331771eb8e6f685ccc357e
  • 💚 Build #266219 succeeded b14c8bd83710e88c8c1121dcde0b4e00972f3567
  • 💚 Build #266188 succeeded c2872ae85292f995dfa9b25cb115fa42538ba1e4

cc @viduni94

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Jan 13, 2025

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7708

[✅] x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts: 100/100 tests passed.

see run history

@viduni94 viduni94 merged commit 95b76dc into elastic:main Jan 13, 2025
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Jan 13, 2025

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12755275687

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Jan 13, 2025

💔 All backports failed

Status Branch Result
❌ 8.x Backport failed because of merge conflicts

You might need to backport the following PRs to 8.x:
- Observability AI Assistant Tests Deployment Agnostic (#205194)

Manual backport

To create the backport manually run:

node scripts/backport --pr 206324

Questions ?

Please refer to the Backport tool documentation

@sorenlouv sorenlouv mentioned this pull request Jan 14, 2025
Merged
@sorenlouv
Copy link
Copy Markdown
Contributor

sorenlouv commented Jan 14, 2025

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

sorenlouv added a commit that referenced this pull request Jan 14, 2025
@sorenlouv @viduni94
[8.x] [Obs AI Assistant] Attempt to fix flaky KB migration test (#206324
cb7cd55 
) (#206543)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Obs AI Assistant] Attempt to fix flaky KB migration test
(#206324)](#206324)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Viduni
Wickramarachchi","email":"viduni.wickramarachchi@elastic.co"},"sourceCommit":{"committedDate":"2025-01-13T20:12:58Z","message":"[Obs
AI Assistant] Attempt to fix flaky KB migration test (#206324)\n\n##
Summary\r\n\r\nCloses
https://github.com/elastic/kibana/issues/202926\r\n\r\n###
Problem\r\nThe KB migration test appears to be flaky sometimes, because
the\r\nmigration takes a few seconds to run. Therefore, when the test
expects\r\nto have semantic_text for all entries, it fails.\r\n\r\n###
Solution\r\nAdding a retry statement to avoid the failure in cases where
the\r\nmigration takes a little while to run.\r\n\r\n###
Checklist\r\n\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n- [x] The PR description includes
the appropriate Release Notes section,\r\nand the correct
`release_note:*` label is applied per
the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"95b76dc12b11bc5e147bb6e6628cec471fbc02e3","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Obs
AI
Assistant","ci:project-deploy-observability","backport:version","v8.18.0"],"number":206324,"url":"https://github.com/elastic/kibana/pull/206324","mergeCommit":{"message":"[Obs
AI Assistant] Attempt to fix flaky KB migration test (#206324)\n\n##
Summary\r\n\r\nCloses
https://github.com/elastic/kibana/issues/202926\r\n\r\n###
Problem\r\nThe KB migration test appears to be flaky sometimes, because
the\r\nmigration takes a few seconds to run. Therefore, when the test
expects\r\nto have semantic_text for all entries, it fails.\r\n\r\n###
Solution\r\nAdding a retry statement to avoid the failure in cases where
the\r\nmigration takes a little while to run.\r\n\r\n###
Checklist\r\n\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n- [x] The PR description includes
the appropriate Release Notes section,\r\nand the correct
`release_note:*` label is applied per
the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"95b76dc12b11bc5e147bb6e6628cec471fbc02e3"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/206324","number":206324,"mergeCommit":{"message":"[Obs
AI Assistant] Attempt to fix flaky KB migration test (#206324)\n\n##
Summary\r\n\r\nCloses
https://github.com/elastic/kibana/issues/202926\r\n\r\n###
Problem\r\nThe KB migration test appears to be flaky sometimes, because
the\r\nmigration takes a few seconds to run. Therefore, when the test
expects\r\nto have semantic_text for all entries, it fails.\r\n\r\n###
Solution\r\nAdding a retry statement to avoid the failure in cases where
the\r\nmigration takes a little while to run.\r\n\r\n###
Checklist\r\n\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n- [x] The PR description includes
the appropriate Release Notes section,\r\nand the correct
`release_note:*` label is applied per
the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"95b76dc12b11bc5e147bb6e6628cec471fbc02e3"}},{"branch":"8.x","label":"v8.18.0","labelRegex":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Viduni Wickramarachchi <viduni.wickramarachchi@elastic.co>
viduni94 added a commit to viduni94/kibana that referenced this pull request Jan 23, 2025
@viduni94
[Obs AI Assistant] Attempt to fix flaky KB migration test (elastic#20…
6d5190f 
…6324)

## Summary

Closes elastic#202926

### Problem
The KB migration test appears to be flaky sometimes, because the
migration takes a few seconds to run. Therefore, when the test expects
to have semantic_text for all entries, it fails.

### Solution
Adding a retry statement to avoid the failure in cases where the
migration takes a little while to run.

### Checklist

- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@sorenlouv sorenlouv sorenlouv approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@viduni94 viduni94

Labels

backport:version Backport to applied version labels ci:project-deploy-observability Create an Observability project release_note:skip Skip the PR/issue when compiling release notes Team:Obs AI Assistant Observability AI Assistant v8.18.0 v9.0.0

Projects

None yet

Milestone

No milestone

4 participants

@viduni94 @elasticmachine @kibanamachine @sorenlouv