🌊 Add task service and client to manage background tasks by miltonhultgren · Pull Request #245725 · elastic/kibana

miltonhultgren · 2025-12-09T20:05:06Z

Summary

This PR adds a way to manage the status and results of background tasks within the Streams plugin.

Via the TaskService we can get a TaskClient that has the following API:

get(id): Returns the status and result of a task
schedule({ task, params, request }): Schedules a task of the specified type with the taskManager as a single run task and writes a task document to an internal index to track the status and result
update(task): Update the stored task document, this should be called by the task run function

The TaskService also has a method registerTasks which takes a context object that is used to create the task definitions and registers them. Through the context, we are able to pass things like getScopedClients so that a background task can have access to this as well.
schedule requires a KibanaRequest to be passed, taskManager uses this request to create an API key for the task to run with and creates a fake KibanaRequest that we can use to pass to getScopedClients.
Within the task runner, we can get access to the taskClient also via getScopedClients.

See usage here

x-pack/platform/plugins/shared/streams/server/lib/tasks/task_client.ts

x-pack/platform/plugins/shared/streams/server/lib/tasks/storage.ts

x-pack/platform/plugins/shared/streams/server/lib/tasks/task_client.ts

x-pack/platform/plugins/shared/streams/server/lib/tasks/storage.ts

x-pack/platform/plugins/shared/streams/server/lib/tasks/task_client.ts

…und-task-service

… fields

x-pack/platform/plugins/shared/streams/server/lib/tasks/storage.ts

x-pack/platform/plugins/shared/streams/server/lib/tasks/task_client.ts

…und-task-service

klacabane

LGTM

…und-task-service

elasticmachine · 2025-12-16T14:37:15Z

💚 Build Succeeded

Buildkite Build
Commit: 40a6f76

Metrics [docs]

✅ unchanged

History

💔 Build #373717 failed 028ac21
💔 Build #373704 failed 0c7326f
💛 Build #372255 was flaky 5ec903f
💛 Build #371831 was flaky 65816b4

### Summary This PR: - Adds a task called `streams_feature_identification` via the newly added task service which calls the existing `identifyFeatures` function and stores the result on the task document - Updates the `POST /internal/streams/{name}/features/_identify` route to schedule this task and check for the results - Adds the `FeatureIdentificationControl` component which manages all of the API interaction around Feature identification - Moves related telemetry reporting to the server - Adds a way to type and store parameters on the task document - Adds a way to cancel tasks (wrap your run function in `cancellableTask`) - Adds another task state (`acknowledged`) to mark that the user has taken action on the result of the task - Adds a hook to poll for task updates for in progress tasks (and tasks being cancelled) https://github.com/user-attachments/assets/7c667112-e0a1-426d-a958-55cf4f2e26bb https://github.com/user-attachments/assets/61e2c079-53dd-4318-8075-fdce466de35d ### Route changes and flags The feature identification route now serves two roles: - Managing the task - Reporting the status of the task The route accepts three flags: `schedule`, `cancel` and `acknowledge` that all have a side effect. `schedule` tries to schedule the task with task manager (and is a `no-op` if the task is already running), this fails if the task is in the `being_cancelled` state. `cancel` moves the task document to `being_cancelled` state so that `cancellableTask` can engage the abort controller to stop on going work. `acknowledge` moves a `complete` task to the `acknowledged` state, indicating that the user has reviewed the results of this task and taken some follow up action, so it's safe to schedule this task again with losing results (this is not enforced) The route reports the following statuses: `'not_started' | 'in_progress' | 'stale' | 'being_canceled' | 'canceled' | 'failed' | 'completed' | 'acknowledged'` Most of them are the state of the task, but `stale` is a special route status that indicates that no updates were made to the task document for a while. The `failed` result includes an error message while `completed` and `acknowledged` include the payload found on `task.task.payload`. ### Task document schema Follow up to #245725 The stored documents have the following shape: ```typescript { id: string; type: string; status: TaskStatus; stream: string; space: string; created_at: string; task: { params: TaskParams; payload?: any // Only for completed and acknowledged tasks error?: string // Only for failed tasks }; } ``` All fields except `task` are indexed, and we store things under `task` to avoid indexing them because of #245974 The tasks are stored in `.kibana_streams_tasks` ### To do - Fix failing tests - Add test for `cancellableTask` - Manually test for robustness --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Shahzad <shahzad31comp@gmail.com> Co-authored-by: Mykola Harmash <mykola.harmash@gmail.com>

### Summary This PR: - Adds a task called `streams_feature_identification` via the newly added task service which calls the existing `identifyFeatures` function and stores the result on the task document - Updates the `POST /internal/streams/{name}/features/_identify` route to schedule this task and check for the results - Adds the `FeatureIdentificationControl` component which manages all of the API interaction around Feature identification - Moves related telemetry reporting to the server - Adds a way to type and store parameters on the task document - Adds a way to cancel tasks (wrap your run function in `cancellableTask`) - Adds another task state (`acknowledged`) to mark that the user has taken action on the result of the task - Adds a hook to poll for task updates for in progress tasks (and tasks being cancelled) https://github.com/user-attachments/assets/7c667112-e0a1-426d-a958-55cf4f2e26bb https://github.com/user-attachments/assets/61e2c079-53dd-4318-8075-fdce466de35d ### Route changes and flags The feature identification route now serves two roles: - Managing the task - Reporting the status of the task The route accepts three flags: `schedule`, `cancel` and `acknowledge` that all have a side effect. `schedule` tries to schedule the task with task manager (and is a `no-op` if the task is already running), this fails if the task is in the `being_cancelled` state. `cancel` moves the task document to `being_cancelled` state so that `cancellableTask` can engage the abort controller to stop on going work. `acknowledge` moves a `complete` task to the `acknowledged` state, indicating that the user has reviewed the results of this task and taken some follow up action, so it's safe to schedule this task again with losing results (this is not enforced) The route reports the following statuses: `'not_started' | 'in_progress' | 'stale' | 'being_canceled' | 'canceled' | 'failed' | 'completed' | 'acknowledged'` Most of them are the state of the task, but `stale` is a special route status that indicates that no updates were made to the task document for a while. The `failed` result includes an error message while `completed` and `acknowledged` include the payload found on `task.task.payload`. ### Task document schema Follow up to elastic#245725 The stored documents have the following shape: ```typescript { id: string; type: string; status: TaskStatus; stream: string; space: string; created_at: string; task: { params: TaskParams; payload?: any // Only for completed and acknowledged tasks error?: string // Only for failed tasks }; } ``` All fields except `task` are indexed, and we store things under `task` to avoid indexing them because of elastic#245974 The tasks are stored in `.kibana_streams_tasks` ### To do - Fix failing tests - Add test for `cancellableTask` - Manually test for robustness --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Shahzad <shahzad31comp@gmail.com> Co-authored-by: Mykola Harmash <mykola.harmash@gmail.com>

Add task service and client to manage background tasks

8171ee6

github-actions bot added the author:actionable-obs PRs authored by the actionable obs team label Dec 9, 2025

klacabane reviewed Dec 10, 2025

View reviewed changes

x-pack/platform/plugins/shared/streams/server/lib/tasks/task_client.ts Show resolved Hide resolved

klacabane reviewed Dec 10, 2025

View reviewed changes

x-pack/platform/plugins/shared/streams/server/lib/tasks/storage.ts Outdated Show resolved Hide resolved

klacabane reviewed Dec 10, 2025

View reviewed changes

x-pack/platform/plugins/shared/streams/server/lib/tasks/task_client.ts Show resolved Hide resolved

ruflin reviewed Dec 11, 2025

View reviewed changes

x-pack/platform/plugins/shared/streams/server/lib/tasks/storage.ts Outdated Show resolved Hide resolved

ruflin reviewed Dec 11, 2025

View reviewed changes

x-pack/platform/plugins/shared/streams/server/lib/tasks/task_client.ts Show resolved Hide resolved

ruflin reviewed Dec 11, 2025

View reviewed changes

x-pack/platform/plugins/shared/streams/server/lib/tasks/task_client.ts Outdated Show resolved Hide resolved

miltonhultgren added 4 commits December 11, 2025 10:11

Merge branch 'main' of github.com:elastic/kibana into streams-backgro…

608534b

…und-task-service

Don't map all fields, add way to pass typed parameters, add some more…

8a5dd5e

… fields

Fix race conditions, improve task params

52e562c

Merge branch 'main' into streams-background-task-service

65816b4

miltonhultgren marked this pull request as ready for review December 11, 2025 13:40

miltonhultgren requested review from a team as code owners December 11, 2025 13:40

miltonhultgren added release_note:skip Skip the PR/issue when compiling release notes backport:version Backport to applied version labels v9.3.0 Feature:SigEvents Significant events feature, related to streams and rules/alerts (RnA) labels Dec 11, 2025

klacabane reviewed Dec 11, 2025

View reviewed changes

x-pack/platform/plugins/shared/streams/server/lib/tasks/storage.ts Show resolved Hide resolved

klacabane reviewed Dec 11, 2025

View reviewed changes

x-pack/platform/plugins/shared/streams/server/lib/tasks/task_client.ts Outdated Show resolved Hide resolved

Merge branch 'main' of github.com:elastic/kibana into streams-backgro…

5ec903f

…und-task-service

klacabane approved these changes Dec 15, 2025

View reviewed changes

miltonhultgren and others added 3 commits December 16, 2025 10:56

Merge branch 'main' of github.com:elastic/kibana into streams-backgro…

1f142c1

…und-task-service

Use normal priority

0c7326f

Changes from node scripts/eslint_all_files --no-cache --fix

028ac21

elastic-vault-github-plugin-prod bot requested a review from a team as a code owner December 16, 2025 10:25

Merge branch 'main' into streams-background-task-service

40a6f76

miltonhultgren removed the request for review from a team December 16, 2025 12:53

miltonhultgren merged commit 7c2a857 into elastic:main Dec 16, 2025
13 checks passed

kibanamachine added backport:skip This PR does not require backporting and removed backport:version Backport to applied version labels labels Dec 16, 2025

miltonhultgren mentioned this pull request Dec 19, 2025

🌊 [Feature identification] Run as background task #245728

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌊 Add task service and client to manage background tasks#245725

🌊 Add task service and client to manage background tasks#245725
miltonhultgren merged 10 commits intoelastic:mainfrom
miltonhultgren:streams-background-task-service

miltonhultgren commented Dec 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

klacabane left a comment

Uh oh!

elasticmachine commented Dec 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

miltonhultgren commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

klacabane left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Dec 16, 2025

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

miltonhultgren commented Dec 9, 2025 •

edited

Loading