[ML] Trained Models: Count tests run against trained models by rbrtj · Pull Request #212927 · elastic/kibana

rbrtj · 2025-03-03T14:30:47Z

Part of: #200725
This PR adds EBT for tests (success and failed) run against trained models.

elasticmachine · 2025-03-03T14:30:51Z

Pinging @elastic/ml-ui (:ml)

Copilot

PR Overview

This PR adds UI counters to track the number of tests run against trained models (both success and failure) by updating usage events and passing the MlUsageCollection dependency to various inference modules.

Added new ML usage event constants for test events in the constants file.
Updated the inference classes and their constructors to accept and use MlUsageCollection.
Modified UI components and service hooks to supply the new dependency to model inference instances.

Reviewed Changes

File	Description
x-pack/platform/plugins/shared/ml/common/constants/usage_collection.ts	Adds new usage event constants for test counters.
x-pack/platform/plugins/shared/ml/public/application/model_management/test_models/models/inference_base.ts	Updates runInfer method to count test events using mlUsageCollection.
x-pack/platform/plugins/shared/ml/public/application/services/usage_collection.ts	Exports the MlUsageCollection type to support dependency injection.
x-pack/platform/plugins/shared/ml/public/application/model_management/test_models/selected_model.tsx	Retrieves and passes mlUsageCollection to the inference model constructors.
Other inference modules (ner, text classification, lang ident, text embedding, question answering, zero shot, text expansion, fill mask)	Their constructors are updated to receive mlUsageCollection and pass it to the superclass.

Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (2)

x-pack/platform/plugins/shared/ml/common/constants/usage_collection.ts:15

[nitpick] The constant name is plural (TESTED_TRAINED_MODELS) but its value is singular ('tested_trained_model'); consider aligning these for consistency.

TESTED_TRAINED_MODELS: 'tested_trained_model',

x-pack/platform/plugins/shared/ml/public/application/model_management/test_models/models/inference_base.ts:317

Instead of using hardcoded event strings, consider using the defined constants (e.g., ML_USAGE_EVENT.TESTED_TRAINED_MODELS) to ensure consistency across the codebase.

this.mlUsageCollection.count('tested_trained_model');

Copilot

PR Overview

This PR integrates UI counters for tracking the number of tests run against trained models by adding new usage event constants and wiring up usage collection calls into the inference workflows.

Added two new usage event constants (“tested_trained_model” and “test_failed_trained_model”) to support tracking the tests.
Injected and utilized the mlUsageCollection dependency into various inference models to record UI counter events on success and error paths.
Updated the usage collection service to export a type for mlUsageCollection, ensuring type consistency across the plugin.

Reviewed Changes

File	Description
x-pack/platform/plugins/shared/ml/common/constants/usage_collection.ts	Added new usage event constants for tracking trained model tests.
x-pack/platform/plugins/shared/ml/public/application/model_management/test_models/models/inference_base.ts	Integrated mlUsageCollection counting in the runInfer method for both success and failure scenarios.
x-pack/platform/plugins/shared/ml/public/application/services/usage_collection.ts	Exported the MlUsageCollection type from the usage collection provider.
x-pack/platform/plugins/shared/ml/public/application/model_management/test_models/selected_model.tsx	Updated usage context extraction and passed mlUsageCollection to inference model constructors.
Other model files (text_embedding, ner, text_classification, lang_ident, question_answering, zero_shot_classification, text_expansion, fill_mask)	Updated constructors to accept the mlUsageCollection dependency and pass it on to their super class.

Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.

jgowdyelastic · 2025-03-04T08:52:09Z

...m/plugins/shared/ml/public/application/model_management/test_models/models/inference_base.ts

    protected readonly inputType: INPUT_TYPE,
-    protected readonly deploymentId: string
+    protected readonly deploymentId: string,
+    protected readonly mlUsageCollection: MlUsageCollection


This isn't used in any of the derived classes and so could be private

Done in 5704fb9

jgowdyelastic · 2025-03-04T08:58:59Z

...m/plugins/shared/ml/public/application/model_management/test_models/models/inference_base.ts

        DEFAULT_INFERENCE_TIME_OUT
      )) as unknown as TRawInferResponse;

+      this.mlUsageCollection.count('tested_trained_model');


Although it's good to have counts for when testing has been performed. I think it would be a lot more valuable to also track additional information, such as the model type and pytorch task type.

I assume we need event-based telemetry for that

After short discussion with @peteharverson, we agreed to collect EBT events with all relevant data, updated in: 5704fb9

jgowdyelastic

LGTM

elasticmachine · 2025-03-10T10:50:51Z

💚 Build Succeeded

Buildkite Build
Commit: 097b11f

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`ml`	5.3MB	5.3MB	+415.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`ml`	79.0KB	79.6KB	+588.0B

History

💛 Build #281452 was flaky 870de87
💚 Build #280835 succeeded e7ed1a4
💔 Build #280785 failed b58ae2b
💚 Build #280353 succeeded 255bdae

cc @rbrtj

darnautov

LGTM

kibanamachine · 2025-03-10T12:42:48Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/13764925902

…212927) Part of: elastic#200725 This PR adds UI Counters for tests (success and failed) run against trained models. (cherry picked from commit 6a184b4)

kibanamachine · 2025-03-10T12:48:29Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…12927) (#213727) # Backport This will backport the following commits from `main` to `8.x`: - [[ML] Trained Models: Count tests run against trained models (#212927)](#212927)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Robert Jaszczurek <92210485+rbrtj@users.noreply.github.com>

…212927) Part of: elastic#200725 This PR adds UI Counters for tests (success and failed) run against trained models.

ui counters for tests run against trained models

7a4643b

rbrtj added :ml release_note:skip Skip the PR/issue when compiling release notes Feature:3rd Party Models ML 3rd party models Team:ML Team label for ML (also use :ml) t// backport:version Backport to applied version labels v9.1.0 v8.19.0 labels Mar 3, 2025

rbrtj requested review from Copilot, darnautov and jgowdyelastic March 3, 2025 14:30

rbrtj self-assigned this Mar 3, 2025

rbrtj requested a review from a team as a code owner March 3, 2025 14:30

Copilot AI reviewed Mar 3, 2025

View reviewed changes

use singular in event names

255bdae

rbrtj requested a review from Copilot March 3, 2025 14:36

Copilot AI reviewed Mar 3, 2025

View reviewed changes

jgowdyelastic reviewed Mar 4, 2025

View reviewed changes

Merge branch 'main' into ml-count-tests-run-against-trained-models

b86796c

rbrtj marked this pull request as draft March 4, 2025 13:58

rbrtj added 3 commits March 4, 2025 15:42

use ebt for tested models telemetry

5704fb9

cleanup usage collection types

4945f6c

remove another unused usage collection type

5a45d76

rbrtj marked this pull request as ready for review March 4, 2025 14:44

add telemetry method

b58ae2b

rbrtj requested a review from jgowdyelastic March 4, 2025 15:10

update trained models service test

e7ed1a4

jgowdyelastic approved these changes Mar 6, 2025

View reviewed changes

Merge branch 'main' into ml-count-tests-run-against-trained-models

870de87

Merge branch 'main' into ml-count-tests-run-against-trained-models

097b11f

darnautov approved these changes Mar 10, 2025

View reviewed changes

rbrtj merged commit 6a184b4 into elastic:main Mar 10, 2025
9 checks passed

kibanamachine mentioned this pull request Mar 10, 2025

[8.x] [ML] Trained Models: Count tests run against trained models (#212927) #213727

Merged

kibanamachine mentioned this pull request Mar 10, 2025

docs(synthetics): add test plan #213535

Merged

Conversation

rbrtj commented Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Mar 3, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

PR Overview

Reviewed Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

PR Overview

Reviewed Changes

Uh oh!

jgowdyelastic Mar 4, 2025

Choose a reason for hiding this comment

Uh oh!

rbrtj Mar 4, 2025

Choose a reason for hiding this comment

Uh oh!

jgowdyelastic Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darnautov Mar 4, 2025

Choose a reason for hiding this comment

Uh oh!

rbrtj Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgowdyelastic left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 10, 2025

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

History

Uh oh!

darnautov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kibanamachine commented Mar 10, 2025

Uh oh!

kibanamachine commented Mar 10, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rbrtj commented Mar 3, 2025 •

edited

Loading

jgowdyelastic Mar 4, 2025 •

edited

Loading

rbrtj Mar 4, 2025 •

edited

Loading