-
Notifications
You must be signed in to change notification settings - Fork 25.7k
[Transform] Add telemetry support for transform features #71607
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Transform] Add telemetry support for transform features #71607
Conversation
0091d5a to
8c2a685
Compare
|
Pinging @elastic/ml-core (Team:ML) |
8c2a685 to
6faa44d
Compare
|
run elasticsearch-ci/2 |
d44160e to
3a3f42c
Compare
benwtrent
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks pretty good.
I wonder if we can use runtime_mappings to capture the configs from before the mapping was updated? Otherwise, we effectively lose all that data :/
I think there is a way to use runtime_mappings + the index mapping to handle this..
...ugin/core/src/main/java/org/elasticsearch/xpack/core/transform/TransformFeatureSetUsage.java
Outdated
Show resolved
Hide resolved
...in/core/src/main/java/org/elasticsearch/xpack/core/transform/transforms/TransformConfig.java
Outdated
Show resolved
Hide resolved
...transform/src/main/java/org/elasticsearch/xpack/transform/TransformUsageTransportAction.java
Outdated
Show resolved
Hide resolved
hendrikmuhs
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
The idea with the runtime mappings is good, but would require a lot of testing. E.g. on a mixed version cluster with old <7.11 nodes.
The other side-effect: if search.allow_expensive_queries is false it will break even on newer clusters.
I think it is not worth the potential trouble.
Note: A _update call moves the config to the latest internal index, so eventually (latest 8) configs will be stored in an index with proper mappings.
We could think of making this move easier, e.g. letting the UI suggest the update or do the update on _start if we detect an older config. Something to discuss as a follow up.
…al index and audit index"
b950802 to
71dae7c
Compare
przemekwitek
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @benwtrent for the suggestion to use runtime mappings. I was thinking about sth along these lines but by what @hendrikmuhs explained I also found it not worth the potential trouble.
This PR adds telemetry for transform features, i.e.: the number of existing transforms using each feature.
This PR adds support for 4 features:
pivot,latest,syncandretention_policybut the code is generic enough to handle other features too.Example response from
_xpack/usage:Relates #68461