Skip to content

[APM] Performance fix for 'cardinality' telemetry task#144061

Merged
basepi merged 3 commits intoelastic:mainfrom
basepi:fix_cardinality_telemetry
Oct 27, 2022
Merged

[APM] Performance fix for 'cardinality' telemetry task#144061
basepi merged 3 commits intoelastic:mainfrom
basepi:fix_cardinality_telemetry

Conversation

@basepi
Copy link
Contributor

@basepi basepi commented Oct 26, 2022 •
edited by kibanamachine
Loading

Summary

Someone found an instance where this telemetry task took 29 hours. We should figure out why the timeout argument didn't stop that earlier in that instance, but I also noticed that we're not narrowing the indices at all on these queries. We now only search the traces indices which should (hopefully) improve the performance on this dramatically.

/cc @felixbarny @dgieselaar

For maintainers

Closes #143819.

@basepi basepi requested review from a team and dgieselaar October 26, 2022 18:41
@botelastic botelastic bot added the Team:APM - DEPRECATED Use Team:obs-ux-infra_services. label Oct 26, 2022
@elasticmachine
Copy link
Contributor

elasticmachine commented Oct 26, 2022

Pinging @elastic/apm-ui (Team:APM)

@basepi basepi added the release_note:skip Skip the PR/issue when compiling release notes label Oct 26, 2022
@basepi
Copy link
Contributor Author

basepi commented Oct 26, 2022

I also wonder if it would be worthwhile to make this 1h instead of 1d? A lot less data to sift through.

sorenlouv
sorenlouv reviewed Oct 27, 2022
View reviewed changes
x-pack/plugins/apm/server/lib/apm_telemetry/collect_data_telemetry/tasks.ts
});

const rumAgentCardinalityResponse = await search({
index: [indices.transaction],
Copy link
Member

@sorenlouv sorenlouv Oct 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably should make index required

Copy link
Contributor

@dgieselaar dgieselaar Oct 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've got a fix for that in the works, just waiting on @basepi's permission to push it to this PR. That will also close #143819

@dgieselaar dgieselaar added v8.6.0 v8.5.1 bug Fixes for quality problems that affect the customer experience labels Oct 27, 2022
@kibana-ci
Copy link

kibana-ci commented Oct 27, 2022

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@dgieselaar
Copy link
Contributor

dgieselaar commented Oct 27, 2022

@basepi I think it's good to default to 24h in most cases, because e.g. a 1hr time range for rum data leads to very different results if the job runs at midnight.

@basepi basepi merged commit a5c8ebe into elastic:main Oct 27, 2022
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Oct 27, 2022
@dgieselaar
[APM] Performance fix for 'cardinality' telemetry task (elastic#144061)
d874d57 
* Performance fix for 'cardinality' telemetry task

* Make timeout/index required for telemetry searches

* Fix tests

Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
(cherry picked from commit a5c8ebe)
@kibanamachine kibanamachine mentioned this pull request Oct 27, 2022
Merged
@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 27, 2022

💚 All backports created successfully

Status Branch Result
✅ 8.5

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@basepi basepi mentioned this pull request Oct 27, 2022
Closed
1 task
kibanamachine added a commit that referenced this pull request Oct 27, 2022
@kibanamachine
[APM] Performance fix for 'cardinality' telemetry task (#144061) (#14…
d9d2438 
…4124)

* Performance fix for 'cardinality' telemetry task

* Make timeout/index required for telemetry searches

* Fix tests

Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
(cherry picked from commit a5c8ebe)

Co-authored-by: Colton Myers <colton@myers.fm>
jloleysens added a commit to jloleysens/kibana that referenced this pull request Oct 27, 2022
@jloleysens
Merge branch 'main' into files-move-to-src
1b2d913 
* main: (24 commits)
  [Files] Add file upload to file picker (elastic#143969)
  [Security solution] Guided onboarding, alerts & cases (elastic#143598)
  [APM] Critical path for a single trace (elastic#143735)
  skip failing test suite (elastic#143933)
  [Fleet] Update GH Projects automation (elastic#144123)
  [APM] Performance fix for 'cardinality' telemetry task (elastic#144061)
  [Enterprise Search] Attach ML Inference Pipeline - Pipeline re-use (elastic#143979)
  [8.5][DOCS] Add support for differential logs (elastic#143242)
  Bump nwsapi from v2.2.0 to v2.2.2 (elastic#144001)
  [APM] Add waterfall to dependency operations (elastic#143257)
  [Shared UX] Add deprecation message to kibana react Markdown (elastic#143766)
  [Security Solution][Endpoint] Adds RBAC API checks for Blocklist (elastic#144047)
  Improve `needs-team` auto labeling regex (elastic#143787)
  [Reporting/CSV Export] _id field can not be formatted (elastic#143807)
  Adds SavedObjectsWarning to analytics results pages. (elastic#144109)
  Bump chromedriver to 107 (elastic#144073)
  Update cypress (elastic#143755)
  [Maps] nest security layers in layer group (elastic#144055)
  [Lens][Agg based Heatmap] Navigate to Lens Agg based Heatmap. (elastic#143820)
  Added support of saved search (elastic#144095)
  ...
@mistic mistic removed the v8.5.0 label Nov 1, 2022
@mistic
Copy link
Contributor

mistic commented Nov 1, 2022

Added v8.5.1 tag has the backport for 8.5 didn't make the v8.5.0 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sorenlouv sorenlouv sorenlouv approved these changes

@dgieselaar dgieselaar Awaiting requested review from dgieselaar

Assignees

No one assigned

Labels

bug Fixes for quality problems that affect the customer experience release_note:skip Skip the PR/issue when compiling release notes Team:APM - DEPRECATED Use Team:obs-ux-infra_services. v8.5.1 v8.6.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[APM] Make sure telemetry queries are scoped and have a timeout

7 participants

@basepi @elasticmachine @kibana-ci @dgieselaar @kibanamachine @mistic @sorenlouv