Skip to content

[scout] Migrate onboarding deployment-agnostic API tests#259436

Merged
dmlemeshko merged 21 commits intoelastic:mainfrom
dmlemeshko:migrate-onboarding-da-api-tests
Mar 28, 2026
Merged

[scout] Migrate onboarding deployment-agnostic API tests#259436
dmlemeshko merged 21 commits intoelastic:mainfrom
dmlemeshko:migrate-onboarding-da-api-tests

Conversation

@dmlemeshko
Copy link
Copy Markdown
Contributor

@dmlemeshko dmlemeshko commented Mar 24, 2026

Summary

Migrates Observability onboarding deployment-agnostic API integration coverage from the FTR suite (api_integration_deployment_agnostic/apis/onboarding) to Scout API tests under observability_onboarding/test/scout/api, with @kbn/scout-oblt + synthtrace fixtures, stateful + serverless tags, and auth aligned with the old FTR setup (cookies vs API keys where behavior differs). Removes the FTR onboarding API loadTestFile bundle and wires TypeScript Pattern A (test/scout/**/* in the plugin tsconfig + Scout kbn_references) so plugin-relative imports stay valid.

FTR file (removed) Scout spec
get_progress.ts onboarding_flow_progress.spec.ts
update_progress.ts onboarding_flow_update_progress_404.spec.ts
update_progress.ts onboarding_flow_update_progress_updates.spec.ts
custom_roles/index.ts Inlined in onboarding_flow_progress.spec.ts (least-privilege / noAccessUserRole parity)

@dmlemeshko dmlemeshko force-pushed the migrate-onboarding-da-api-tests branch from d7ac3bd to 9d776b8 Compare March 25, 2026 00:21
@dmlemeshko dmlemeshko changed the title [scout] Migrate onboarding da api tests [scout] Migrate onboarding deployment-agnostic API tests Mar 25, 2026
@dmlemeshko dmlemeshko added release_note:skip Skip the PR/issue when compiling release notes backport:version Backport to applied version labels 9.4.0 v9.3.3 labels Mar 25, 2026
@dmlemeshko dmlemeshko marked this pull request as ready for review March 25, 2026 09:28
@dmlemeshko dmlemeshko requested review from a team as code owners March 25, 2026 09:28
Copy link
Copy Markdown
Contributor

@rmyz rmyz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

obs-presentation changes LGTM

Copy link
Copy Markdown
Member

@csr csr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, I left a few comments. I also ran the tests on an observability MKI and they all pass.

Comment on lines +39 to +49
await apiTest.step('updates step status', async () => {
const createFlowResponse = await apiClient.post(
'internal/observability_onboarding/flow',
{
headers: adminHeaders,
responseType: 'json',
}
);
expect(createFlowResponse).toHaveStatusCode(200);
onboardingId = (createFlowResponse.body as { onboardingFlow: { id: string } })
.onboardingFlow.id;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium tests/onboarding_flow_update_progress_updates.spec.ts:39

The test creates two onboarding flows (lines 40-49 and 78-87) but onboardingId is overwritten by the second assignment before afterEach runs, so the first saved object is never deleted. Store both IDs or nest the steps in separate tests so cleanup runs between them.

        await apiTest.step('updates step status', async () => {
-          const createFlowResponse = await apiClient.post(
+          const createFlowResponse1 = await apiClient.post(
             'internal/observability_onboarding/flow',
             {
               headers: adminHeaders,
               responseType: 'json',
             }
           );
-          expect(createFlowResponse).toHaveStatusCode(200);
-          onboardingId = (createFlowResponse.body as { onboardingFlow: { id: string } })
+          expect(createFlowResponse1).toHaveStatusCode(200);
+          onboardingId = (createFlowResponse1.body as { onboardingFlow: { id: string } })
             .onboardingFlow.id;
🤖 Copy this AI Prompt to have your agent fix this:
In file x-pack/solutions/observability/plugins/observability_onboarding/test/scout/api/tests/onboarding_flow_update_progress_updates.spec.ts around lines 39-49:

The test creates two onboarding flows (lines 40-49 and 78-87) but `onboardingId` is overwritten by the second assignment before `afterEach` runs, so the first saved object is never deleted. Store both IDs or nest the steps in separate tests so cleanup runs between them.

Evidence trail:
x-pack/solutions/observability/plugins/observability_onboarding/test/scout/api/tests/onboarding_flow_update_progress_updates.spec.ts (viewed at REVIEWED_COMMIT):
- Line 19: `let onboardingId = ''` (single variable)
- Lines 21-28: `afterEach` hook deletes saved object by `onboardingId`
- Lines 48-49: First assignment of `onboardingId` in first step
- Lines 86-87: Second assignment overwrites `onboardingId` in second step
- Both steps are inside a single test (lines 30-113), so `afterEach` only runs once after both steps complete

@dmlemeshko dmlemeshko requested a review from csr March 25, 2026 15:18
Copy link
Copy Markdown
Member

@csr csr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recent changes LGTM. Thank you!

Copy link
Copy Markdown
Contributor

@benakansara benakansara left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actionable-obs changes LGTM

@dmlemeshko dmlemeshko merged commit da2f01b into elastic:main Mar 28, 2026
18 checks passed
@kibanamachine
Copy link
Copy Markdown
Contributor

Starting backport for target branches: 9.3

https://github.com/elastic/kibana/actions/runs/23682974324

@elasticmachine
Copy link
Copy Markdown
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

✅ unchanged

History

@kibanamachine
Copy link
Copy Markdown
Contributor

💔 All backports failed

Status Branch Result
9.3 Backport failed because of merge conflicts

You might need to backport the following PRs to 9.3:
- Add kbn/test-saml-auth package (#259074)
- Add kbn/test-es-server package (#259064)
- Add shared validation runner package (#258768)
- [Skills] Add Buildkite logs skill (#258587)

Manual backport

To create the backport manually run:

node scripts/backport --pr 259436

Questions ?

Please refer to the Backport tool documentation

SoniaSanzV pushed a commit to SoniaSanzV/kibana that referenced this pull request Mar 30, 2026
)

## Summary

Migrates Observability onboarding deployment-agnostic API integration
coverage from the FTR suite
(`api_integration_deployment_agnostic/apis/onboarding`) to Scout API
tests under `observability_onboarding/test/scout/api`, with
`@kbn/scout-oblt` + `synthtrace` fixtures, `stateful` + `serverless`
tags, and auth aligned with the old FTR setup (cookies vs API keys where
behavior differs). Removes the FTR onboarding API loadTestFile bundle
and wires TypeScript Pattern A (test/scout/**/* in the plugin tsconfig +
Scout kbn_references) so plugin-relative imports stay valid.

| FTR file (removed) | Scout spec |
| --- | --- |
| `get_progress.ts` | `onboarding_flow_progress.spec.ts` |
| `update_progress.ts` | `onboarding_flow_update_progress_404.spec.ts` |
| `update_progress.ts` |
`onboarding_flow_update_progress_updates.spec.ts` |
| `custom_roles/index.ts` | *Inlined in
`onboarding_flow_progress.spec.ts` (least-privilege / `noAccessUserRole`
parity)* |

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 30, 2026
@kibanamachine
Copy link
Copy Markdown
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 259436 locally
cc: @dmlemeshko

dmlemeshko added a commit to dmlemeshko/kibana that referenced this pull request Mar 31, 2026
)

## Summary

Migrates Observability onboarding deployment-agnostic API integration
coverage from the FTR suite
(`api_integration_deployment_agnostic/apis/onboarding`) to Scout API
tests under `observability_onboarding/test/scout/api`, with
`@kbn/scout-oblt` + `synthtrace` fixtures, `stateful` + `serverless`
tags, and auth aligned with the old FTR setup (cookies vs API keys where
behavior differs). Removes the FTR onboarding API loadTestFile bundle
and wires TypeScript Pattern A (test/scout/**/* in the plugin tsconfig +
Scout kbn_references) so plugin-relative imports stay valid.

| FTR file (removed) | Scout spec |
| --- | --- |
| `get_progress.ts` | `onboarding_flow_progress.spec.ts` |
| `update_progress.ts` | `onboarding_flow_update_progress_404.spec.ts` |
| `update_progress.ts` |
`onboarding_flow_update_progress_updates.spec.ts` |
| `custom_roles/index.ts` | *Inlined in
`onboarding_flow_progress.spec.ts` (least-privilege / `noAccessUserRole`
parity)* |

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit da2f01b)

# Conflicts:
#	.github/CODEOWNERS
#	x-pack/solutions/observability/plugins/observability_onboarding/moon.yml
#	x-pack/solutions/observability/plugins/observability_onboarding/tsconfig.json
@dmlemeshko
Copy link
Copy Markdown
Contributor Author

💚 All backports created successfully

Status Branch Result
9.3

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@kibanamachine
Copy link
Copy Markdown
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @dmlemeshko

dmlemeshko added a commit that referenced this pull request Mar 31, 2026
…) (#260418)

# Backport

This will backport the following commits from `main` to `9.3`:
- [[scout] Migrate onboarding deployment-agnostic API tests
(#259436)](#259436)

<!--- Backport version: 10.2.0 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Dzmitry
Lemechko","email":"dzmitry.lemechko@elastic.co"},"sourceCommit":{"committedDate":"2026-03-28T10:09:36Z","message":"[scout]
Migrate onboarding deployment-agnostic API tests (#259436)\n\n##
Summary\n\nMigrates Observability onboarding deployment-agnostic API
integration\ncoverage from the FTR
suite\n(`api_integration_deployment_agnostic/apis/onboarding`) to Scout
API\ntests under `observability_onboarding/test/scout/api`,
with\n`@kbn/scout-oblt` + `synthtrace` fixtures, `stateful` +
`serverless`\ntags, and auth aligned with the old FTR setup (cookies vs
API keys where\nbehavior differs). Removes the FTR onboarding API
loadTestFile bundle\nand wires TypeScript Pattern A (test/scout/**/* in
the plugin tsconfig +\nScout kbn_references) so plugin-relative imports
stay valid.\n\n| FTR file (removed) | Scout spec |\n| --- | --- |\n|
`get_progress.ts` | `onboarding_flow_progress.spec.ts` |\n|
`update_progress.ts` | `onboarding_flow_update_progress_404.spec.ts`
|\n| `update_progress.ts`
|\n`onboarding_flow_update_progress_updates.spec.ts` |\n|
`custom_roles/index.ts` | *Inlined
in\n`onboarding_flow_progress.spec.ts` (least-privilege /
`noAccessUserRole`\nparity)* |\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"da2f01bb106a62a22fb0dcb57bc26fe760c939a9","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport
missing","backport:version","v9.4.0","9.4.0","v9.3.3"],"title":"[scout]
Migrate onboarding deployment-agnostic API
tests","number":259436,"url":"https://github.com/elastic/kibana/pull/259436","mergeCommit":{"message":"[scout]
Migrate onboarding deployment-agnostic API tests (#259436)\n\n##
Summary\n\nMigrates Observability onboarding deployment-agnostic API
integration\ncoverage from the FTR
suite\n(`api_integration_deployment_agnostic/apis/onboarding`) to Scout
API\ntests under `observability_onboarding/test/scout/api`,
with\n`@kbn/scout-oblt` + `synthtrace` fixtures, `stateful` +
`serverless`\ntags, and auth aligned with the old FTR setup (cookies vs
API keys where\nbehavior differs). Removes the FTR onboarding API
loadTestFile bundle\nand wires TypeScript Pattern A (test/scout/**/* in
the plugin tsconfig +\nScout kbn_references) so plugin-relative imports
stay valid.\n\n| FTR file (removed) | Scout spec |\n| --- | --- |\n|
`get_progress.ts` | `onboarding_flow_progress.spec.ts` |\n|
`update_progress.ts` | `onboarding_flow_update_progress_404.spec.ts`
|\n| `update_progress.ts`
|\n`onboarding_flow_update_progress_updates.spec.ts` |\n|
`custom_roles/index.ts` | *Inlined
in\n`onboarding_flow_progress.spec.ts` (least-privilege /
`noAccessUserRole`\nparity)* |\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"da2f01bb106a62a22fb0dcb57bc26fe760c939a9"}},"sourceBranch":"main","suggestedTargetBranches":["9.3"],"targetPullRequestStates":[{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/259436","number":259436,"mergeCommit":{"message":"[scout]
Migrate onboarding deployment-agnostic API tests (#259436)\n\n##
Summary\n\nMigrates Observability onboarding deployment-agnostic API
integration\ncoverage from the FTR
suite\n(`api_integration_deployment_agnostic/apis/onboarding`) to Scout
API\ntests under `observability_onboarding/test/scout/api`,
with\n`@kbn/scout-oblt` + `synthtrace` fixtures, `stateful` +
`serverless`\ntags, and auth aligned with the old FTR setup (cookies vs
API keys where\nbehavior differs). Removes the FTR onboarding API
loadTestFile bundle\nand wires TypeScript Pattern A (test/scout/**/* in
the plugin tsconfig +\nScout kbn_references) so plugin-relative imports
stay valid.\n\n| FTR file (removed) | Scout spec |\n| --- | --- |\n|
`get_progress.ts` | `onboarding_flow_progress.spec.ts` |\n|
`update_progress.ts` | `onboarding_flow_update_progress_404.spec.ts`
|\n| `update_progress.ts`
|\n`onboarding_flow_update_progress_updates.spec.ts` |\n|
`custom_roles/index.ts` | *Inlined
in\n`onboarding_flow_progress.spec.ts` (least-privilege /
`noAccessUserRole`\nparity)* |\n\n---------\n\nCo-authored-by:
kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"da2f01bb106a62a22fb0dcb57bc26fe760c939a9"}},{"branch":"9.3","label":"v9.3.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 31, 2026
jeramysoucy pushed a commit to jeramysoucy/kibana that referenced this pull request Apr 1, 2026
)

## Summary

Migrates Observability onboarding deployment-agnostic API integration
coverage from the FTR suite
(`api_integration_deployment_agnostic/apis/onboarding`) to Scout API
tests under `observability_onboarding/test/scout/api`, with
`@kbn/scout-oblt` + `synthtrace` fixtures, `stateful` + `serverless`
tags, and auth aligned with the old FTR setup (cookies vs API keys where
behavior differs). Removes the FTR onboarding API loadTestFile bundle
and wires TypeScript Pattern A (test/scout/**/* in the plugin tsconfig +
Scout kbn_references) so plugin-relative imports stay valid.

| FTR file (removed) | Scout spec |
| --- | --- |
| `get_progress.ts` | `onboarding_flow_progress.spec.ts` |
| `update_progress.ts` | `onboarding_flow_update_progress_404.spec.ts` |
| `update_progress.ts` |
`onboarding_flow_update_progress_updates.spec.ts` |
| `custom_roles/index.ts` | *Inlined in
`onboarding_flow_progress.spec.ts` (least-privilege / `noAccessUserRole`
parity)* |

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
paulinashakirova pushed a commit to paulinashakirova/kibana that referenced this pull request Apr 2, 2026
)

## Summary

Migrates Observability onboarding deployment-agnostic API integration
coverage from the FTR suite
(`api_integration_deployment_agnostic/apis/onboarding`) to Scout API
tests under `observability_onboarding/test/scout/api`, with
`@kbn/scout-oblt` + `synthtrace` fixtures, `stateful` + `serverless`
tags, and auth aligned with the old FTR setup (cookies vs API keys where
behavior differs). Removes the FTR onboarding API loadTestFile bundle
and wires TypeScript Pattern A (test/scout/**/* in the plugin tsconfig +
Scout kbn_references) so plugin-relative imports stay valid.

| FTR file (removed) | Scout spec |
| --- | --- |
| `get_progress.ts` | `onboarding_flow_progress.spec.ts` |
| `update_progress.ts` | `onboarding_flow_update_progress_404.spec.ts` |
| `update_progress.ts` |
`onboarding_flow_update_progress_updates.spec.ts` |
| `custom_roles/index.ts` | *Inlined in
`onboarding_flow_progress.spec.ts` (least-privilege / `noAccessUserRole`
parity)* |

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
dmlemeshko added a commit that referenced this pull request Apr 7, 2026
…259520)

## Summary:

During #259436 I faced few
challenges + broken tests in Scout, hopefully SKILLs update make
migration more robust

File | Change (high level)
-- | --
scout-api-testing/SKILL.md | Solution vs deploymentAgnostic tagging for
API tests
scout-create-scaffold/SKILL.md | Related skill link, Pattern A/B TS
layout, post-generate steps
scout-migrate-from-ftr/SKILL.md | DA tags, cookieHeader vs API key,
synthtrace API, Pattern A/B, pitfalls
kelvtanv pushed a commit to kelvtanv/kibana that referenced this pull request Apr 7, 2026
…lastic#259520)

## Summary:

During elastic#259436 I faced few
challenges + broken tests in Scout, hopefully SKILLs update make
migration more robust

File | Change (high level)
-- | --
scout-api-testing/SKILL.md | Solution vs deploymentAgnostic tagging for
API tests
scout-create-scaffold/SKILL.md | Related skill link, Pattern A/B TS
layout, post-generate steps
scout-migrate-from-ftr/SKILL.md | DA tags, cookieHeader vs API key,
synthtrace API, Pattern A/B, pitfalls
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:version Backport to applied version labels release_note:skip Skip the PR/issue when compiling release notes v9.3.3 v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants