[Oblt Onboarding] Add more errors logging for Auto Detect flow by mykolaharmash · Pull Request #222948 · elastic/kibana

mykolaharmash · 2025-06-06T09:32:04Z

This change adds more descriptive error messages inside telemetry events in the auto detect onboarding flow. The extended error message be shown to the user as well.

New telemetry event for the case when user is not running the script as root
If Agent fails to download, bot the original error from curl and the download URL will be logged
If Agent fails to extract, error message from tar command will be logged
If Agent fails to install, error message from elastic-agent install command will be logged
Error response from the integrations install endpoint will be logged

Example telemetry event:

Example terminal output:

How to test

In order to simulate errors for all scenarios, you going to need to modify the auto_detect.sh script to make it intentionally fail, like adding extra character to the EA download URL. You can modify it directly in the source if you're running Kibana locally, or you can modify it after it was downloaded and before running it.

To test just one use-case without the script manipulations, you can modify the EA download URL inside the code snippet provided by Kibana.

Whatever way you choose, still go though the flow and make sure the successful path works as expected.

You can check the telemetry events emitted from the flow on the staging telemetry cluster, but keep in mind that events delivered there with a delay of a couple of hours 😢

mykolaharmash · 2025-06-06T09:34:01Z

/oblt-deploy

mykolaharmash · 2025-06-06T09:36:37Z

x-pack/solutions/observability/plugins/observability_onboarding/public/assets/auto_detect.sh

    printf "\e[32;1m✓\e[0m %s\n" "Elastic Agent downloaded to $(pwd)/$elastic_agent_artifact_name.tar.gz"
    update_step_progress "ea-download" "complete"
  else
-    update_step_progress "ea-download" "danger" "Failed to download Elastic Agent, see script output for error."


I've removed see script output for error from everywhere as we don't show these messages to the user anymore, this was a leftover from the legacy onboarding.

…errors-during-auto-detect-flow

mykolaharmash · 2025-06-10T15:05:35Z

/oblt-deploy

…errors-during-auto-detect-flow

mykolaharmash · 2025-06-16T08:23:53Z

/oblt-deploy

…errors-during-auto-detect-flow

flash1293

Sorry it took a while to get to this.

Tested this locally and it mostly works, only issue I noticed is if the captured error is multi-line it breaks the JSON of the request body for tracking, so it never arrives. Would be great if we could fix that.

flash1293 · 2025-06-18T11:06:24Z

x-pack/solutions/observability/plugins/observability_onboarding/public/assets/auto_detect.sh

  else
-    update_step_progress "ea-extract" "danger" "Failed to extract Elastic Agent, see script output for error."
-    fail "Failed to extract Elastic Agent"
+    update_step_progress "ea-extract" "danger" "Failed to extract Elastic Agent.\nTar Error: $agent_extract_result"


I tried that and it didn't report successfully, I think because a multiline output broke the JSON payload:

0000: {"status":"danger", "message":"Failed to extract Elastic Agent.\ 0040: nTar Error: tar: Option --myabc is not supported.Usage:. List: 0080: tar -tf <archive-filename>. Extract: tar -xf <archive-filena 00c0: me>. Create: tar -cf <archive-filename> [filenames...]. Help: 0100: tar --help"}

Ah, thank you for catching this! I've added escaping for newline and backslashes to the MESSAGE.

…ing-add-more-logging-for-errors-during-auto-detect-flow

…errors-during-auto-detect-flow

flash1293 · 2025-06-24T08:19:29Z

Tested this on macos and it doesn't seem to fully work:

sed: 2: "s/\n/\\n/g
": unused label 'a;N;$!ba'
sed: 2: "s/\n/\\n/g
": unused label 'a;N;$!ba'
curl: (22) The requested URL returned error: 400

There are a bunch of errors related to sed and I still get the 400 error.

This is how I test:

Download the auto_detect.sh script
Change the extract command to

  agent_extract_result=$(tar --non-existing-flag -xzf "${elastic_agent_artifact_name}.tar.gz" 2>&1)

Run the auto_detect script

mykolaharmash · 2025-06-24T12:15:14Z

Tested this on macos and it doesn't seem to fully work:
sed: 2: "s/\n/\\n/g
": unused label 'a;N;$!ba'
sed: 2: "s/\n/\\n/g
": unused label 'a;N;$!ba'
curl: (22) The requested URL returned error: 400
There are a bunch of errors related to sed and I still get the 400 error.

This is how I test:

Download the auto_detect.sh script

Change the extract command to
  agent_extract_result=$(tar --non-existing-flag -xzf "${elastic_agent_artifact_name}.tar.gz" 2>&1)
Run the auto_detect script

😢 I only tested on linux, makes sense... will be back with the fix

…ing-add-more-logging-for-errors-during-auto-detect-flow

mykolaharmash · 2025-06-25T14:27:28Z

@flash1293 I abandoned the idea of trying to escape the error messages and replaced it with base64 encoding. This should be more robust approach. Please give it another try, I've tested on mac and linux and everything seems to work fine.

flash1293 · 2025-06-25T16:06:52Z

/oblt-deploy

flash1293 · 2025-06-25T16:07:39Z

Couldn't break it anymore locally, but I would like to do a proper e2e test from a serverless instance first and also it seems like we need t fix some unit tests anyway.

Even if FF has already passed I think we should backport this because this kind of telemetry is very important.

…errors-during-auto-detect-flow

flash1293

Tested and seems to work fine, LGTM.

Just to make sure, is POST /internal/observability_onboarding/flow/{id}/step/{name} called from somewhere else? I didn't find anything but maybe I missed it.

mykolaharmash · 2025-06-26T09:02:12Z

Tested and seems to work fine, LGTM.

Just to make sure, is POST /internal/observability_onboarding/flow/{id}/step/{name} called from somewhere else? I didn't find anything but maybe I missed it.

it was used on the FE before but now it's only called from the bash script

kibanamachine · 2025-06-26T13:29:14Z

Starting backport for target branches: 8.19, 9.0

https://github.com/elastic/kibana/actions/runs/15903200246

elasticmachine · 2025-06-26T13:29:40Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: fc3af31

Failed CI Steps

FTR Configs #21

Metrics [docs]

✅ unchanged

History

💔 Build #312925 failed e0655c2
💔 Build #312683 failed 8da76f1
💚 Build #311405 succeeded d2734c7
💚 Build #310444 succeeded 11e2d69
💛 Build #309861 was flaky 8d4cde8

…ic#222948) Closes elastic#218433 This change adds more descriptive error messages inside telemetry events in the auto detect onboarding flow. The extended error message be shown to the user as well. * New telemetry event for the case when user is not running the script as `root` * If Agent fails to download, bot the original error from `curl` and the download URL will be logged * If Agent fails to extract, error message from `tar` command will be logged * If Agent fails to install, error message from `elastic-agent install` command will be logged * Error response from the integrations install endpoint will be logged Example telemetry event: ![CleanShot 2025-06-06 at 11 29 20@2x](https://github.com/user-attachments/assets/700a3d21-8d1d-43e0-b084-198da7871dec) Example terminal output: ![CleanShot 2025-06-05 at 15 10 38@2x](https://github.com/user-attachments/assets/702e641e-ab4d-4820-8f4d-e551dde435e8) ## How to test In order to simulate errors for all scenarios, you going to need to modify the `auto_detect.sh` script to make it intentionally fail, like adding extra character to the EA download URL. You can modify it directly in the source if you're running Kibana locally, or you can modify it after it was downloaded and before running it. To test just one use-case without the script manipulations, you can modify the EA download URL inside the code snippet provided by Kibana. **Whatever way you choose, still go though the flow and make sure the successful path works as expected.** You can check the telemetry events emitted from the flow on [the staging telemetry cluster](https://telemetry-v2-staging.elastic.dev/s/observability-ux/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-24h%2Fh,to:now))&_a=(columns:!(),dataSource:(dataViewId:b4befdd3-9a15-40c2-be4d-3ac1084d6182,type:dataView),filters:!(),interval:auto,query:(language:kuery,query:'event_type%20:%20%22observability_onboarding%22'),sort:!(!('@timestamp',desc)))), but keep in mind that events delivered there with a delay of a couple of hours 😢 --------- Co-authored-by: Joe Reuter <johannes.reuter@elastic.co> (cherry picked from commit 6b452c7)

kibanamachine · 2025-06-26T13:36:43Z

💔 Some backports could not be created

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts You might need to backport the following PRs to 8.19: - [Embeddable Rebuild] Fix panel title sync with saved object when using defaultTitle (#225237) - Remaining work attributes table (#224723) - [Security Solution] Adds ability to revert prebuilt rules to their base version (#223301) - Remapping `iInCircle` and `questionInCircle` and deprecating `help` icon (#223142) - [Lens] Add internal CRUD api routes (#223296) - [Query Rules] Query rules telemetry (#225146) - [ska] relocate x-pack/test/cases_api_integration (#225056) - [Discover][Oblt] Enable Attributes tab by default (#224894) - [follow up] small ui refactoring (#225234)
✅	9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 222948

Questions ?

Please refer to the Backport tool documentation

…ic#222948) Closes elastic#218433 This change adds more descriptive error messages inside telemetry events in the auto detect onboarding flow. The extended error message be shown to the user as well. * New telemetry event for the case when user is not running the script as `root` * If Agent fails to download, bot the original error from `curl` and the download URL will be logged * If Agent fails to extract, error message from `tar` command will be logged * If Agent fails to install, error message from `elastic-agent install` command will be logged * Error response from the integrations install endpoint will be logged Example telemetry event: ![CleanShot 2025-06-06 at 11 29 20@2x](https://github.com/user-attachments/assets/700a3d21-8d1d-43e0-b084-198da7871dec) Example terminal output: ![CleanShot 2025-06-05 at 15 10 38@2x](https://github.com/user-attachments/assets/702e641e-ab4d-4820-8f4d-e551dde435e8) ## How to test In order to simulate errors for all scenarios, you going to need to modify the `auto_detect.sh` script to make it intentionally fail, like adding extra character to the EA download URL. You can modify it directly in the source if you're running Kibana locally, or you can modify it after it was downloaded and before running it. To test just one use-case without the script manipulations, you can modify the EA download URL inside the code snippet provided by Kibana. **Whatever way you choose, still go though the flow and make sure the successful path works as expected.** You can check the telemetry events emitted from the flow on [the staging telemetry cluster](https://telemetry-v2-staging.elastic.dev/s/observability-ux/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-24h%2Fh,to:now))&_a=(columns:!(),dataSource:(dataViewId:b4befdd3-9a15-40c2-be4d-3ac1084d6182,type:dataView),filters:!(),interval:auto,query:(language:kuery,query:'event_type%20:%20%22observability_onboarding%22'),sort:!(!('@timestamp',desc)))), but keep in mind that events delivered there with a delay of a couple of hours 😢 --------- Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>

…222948) (#225454) # Backport This will backport the following commits from `main` to `9.0`: - [[Oblt Onboarding] Add more errors logging for Auto Detect flow (#222948)](#222948)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Mykola Harmash <mykola.harmash@gmail.com> Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>

mykolaharmash · 2025-06-27T09:21:45Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…ic#222948) Closes elastic#218433 This change adds more descriptive error messages inside telemetry events in the auto detect onboarding flow. The extended error message be shown to the user as well. * New telemetry event for the case when user is not running the script as `root` * If Agent fails to download, bot the original error from `curl` and the download URL will be logged * If Agent fails to extract, error message from `tar` command will be logged * If Agent fails to install, error message from `elastic-agent install` command will be logged * Error response from the integrations install endpoint will be logged Example telemetry event: ![CleanShot 2025-06-06 at 11 29 20@2x](https://github.com/user-attachments/assets/700a3d21-8d1d-43e0-b084-198da7871dec) Example terminal output: ![CleanShot 2025-06-05 at 15 10 38@2x](https://github.com/user-attachments/assets/702e641e-ab4d-4820-8f4d-e551dde435e8) ## How to test In order to simulate errors for all scenarios, you going to need to modify the `auto_detect.sh` script to make it intentionally fail, like adding extra character to the EA download URL. You can modify it directly in the source if you're running Kibana locally, or you can modify it after it was downloaded and before running it. To test just one use-case without the script manipulations, you can modify the EA download URL inside the code snippet provided by Kibana. **Whatever way you choose, still go though the flow and make sure the successful path works as expected.** You can check the telemetry events emitted from the flow on [the staging telemetry cluster](https://telemetry-v2-staging.elastic.dev/s/observability-ux/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:now-24h%2Fh,to:now))&_a=(columns:!(),dataSource:(dataViewId:b4befdd3-9a15-40c2-be4d-3ac1084d6182,type:dataView),filters:!(),interval:auto,query:(language:kuery,query:'event_type%20:%20%22observability_onboarding%22'),sort:!(!('@timestamp',desc)))), but keep in mind that events delivered there with a delay of a couple of hours 😢 --------- Co-authored-by: Joe Reuter <johannes.reuter@elastic.co> (cherry picked from commit 6b452c7) # Conflicts: # x-pack/test/api_integration/deployment_agnostic/apis/observability/onboarding/update_progress.ts

kibanamachine · 2025-06-30T15:52:03Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @mykolaharmash

…#222948) (#225596) # Backport This will backport the following commits from `main` to `8.19`: - [[Oblt Onboarding] Add more errors logging for Auto Detect flow (#222948)](#222948)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

[Oblt Onboarding] Add more errors logging for Auto Detect flow

ee03807

mykolaharmash requested a review from a team as a code owner June 6, 2025 09:32

mykolaharmash added backport:version Backport to applied version labels v8.19.0 v9.0.3 release_note:skip Skip the PR/issue when compiling release notes labels Jun 6, 2025

mykolaharmash commented Jun 6, 2025

View reviewed changes

Merge branch 'main' into 218433-oblt-onboarding-add-more-logging-for-…

b0164b9

…errors-during-auto-detect-flow

mykolaharmash added 2 commits June 12, 2025 11:11

Merge branch 'main' into 218433-oblt-onboarding-add-more-logging-for-…

9700f56

…errors-during-auto-detect-flow

Merge branch 'main' into 218433-oblt-onboarding-add-more-logging-for-…

1746b25

…errors-during-auto-detect-flow

Merge branch 'main' into 218433-oblt-onboarding-add-more-logging-for-…

8d4cde8

…errors-during-auto-detect-flow

flash1293 reviewed Jun 18, 2025

View reviewed changes

mykolaharmash added 2 commits June 19, 2025 14:10

Merge remote-tracking branch 'upstream/main' into 218433-oblt-onboard…

3ae598a

…ing-add-more-logging-for-errors-during-auto-detect-flow

Add newline and backslash escaping to the message

11e2d69

mykolaharmash requested a review from flash1293 June 19, 2025 13:14

Merge branch 'main' into 218433-oblt-onboarding-add-more-logging-for-…

d2734c7

…errors-during-auto-detect-flow

mykolaharmash added 2 commits June 25, 2025 16:22

Add base64 encoding for massage

fa91b00

Merge remote-tracking branch 'upstream/main' into 218433-oblt-onboard…

8da76f1

…ing-add-more-logging-for-errors-during-auto-detect-flow

Merge branch 'main' into 218433-oblt-onboarding-add-more-logging-for-…

e0655c2

…errors-during-auto-detect-flow

flash1293 approved these changes Jun 26, 2025

View reviewed changes

Use base64 message in test

fc3af31

mykolaharmash added backport:prev-minor and removed v9.0.3 labels Jun 26, 2025

mykolaharmash enabled auto-merge (squash) June 26, 2025 13:12

mykolaharmash merged commit 6b452c7 into elastic:main Jun 26, 2025
13 checks passed

kibanamachine added the v9.1.0 label Jun 26, 2025

kibanamachine mentioned this pull request Jun 26, 2025

[9.0] [Oblt Onboarding] Add more errors logging for Auto Detect flow (#222948) #225454

Merged

kibanamachine added the v9.0.4 label Jun 26, 2025

mykolaharmash mentioned this pull request Jun 27, 2025

[8.19] [Oblt Onboarding] Add more errors logging for Auto Detect flow (#222948) #225596

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jun 30, 2025

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jul 1, 2025

Comments

Conversation

mykolaharmash commented Jun 6, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to test

Uh oh!

mykolaharmash commented Jun 6, 2025

Uh oh!

mykolaharmash Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

mykolaharmash commented Jun 10, 2025

Uh oh!

mykolaharmash commented Jun 16, 2025

Uh oh!

flash1293 left a comment

Choose a reason for hiding this comment

Uh oh!

flash1293 Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

mykolaharmash Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

flash1293 commented Jun 24, 2025

Uh oh!

mykolaharmash commented Jun 24, 2025

Uh oh!

mykolaharmash commented Jun 25, 2025

Uh oh!

flash1293 commented Jun 25, 2025

Uh oh!

flash1293 commented Jun 25, 2025

Uh oh!

flash1293 left a comment

Choose a reason for hiding this comment

Uh oh!

mykolaharmash commented Jun 26, 2025

Uh oh!

Uh oh!

kibanamachine commented Jun 26, 2025

Uh oh!

elasticmachine commented Jun 26, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

History

Uh oh!

kibanamachine commented Jun 26, 2025

💔 Some backports could not be created

Manual backport

Questions ?

Uh oh!

mykolaharmash commented Jun 27, 2025

💚 All backports created successfully

Questions ?

Uh oh!

kibanamachine commented Jun 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mykolaharmash commented Jun 6, 2025 •

edited by kibanamachine

Loading