🌊 [Feature identification] Run as background task

[miltonhultgren]

Summary

This PR:

• Adds a task called streams_feature_identification via the newly added task service which calls the existing identifyFeatures function and stores the result on the task document
• Updates the POST /internal/streams/{name}/features/_identify route to schedule this task and check for the results
• Adds the FeatureIdentificationControl component which manages all of the API interaction around Feature identification
• Moves related telemetry reporting to the server
• Adds a way to type and store parameters on the task document
• Adds a way to cancel tasks (wrap your run function in cancellableTask)
• Adds another task state (acknowledged) to mark that the user has taken action on the result of the task
• Adds a hook to poll for task updates for in progress tasks (and tasks being cancelled)

Screen.Recording.2025-12-18.at.17.12.57.mov

Screen.Recording.2025-12-18.at.17.17.06.mov

Route changes and flags

The feature identification route now serves two roles:

• Managing the task
• Reporting the status of the task

The route accepts three flags: schedule, cancel and acknowledge that all have a side effect.
schedule tries to schedule the task with task manager (and is a no-op if the task is already running), this fails if the task is in the being_cancelled state.
cancel moves the task document to being_cancelled state so that cancellableTask can engage the abort controller to stop on going work.
acknowledge moves a complete task to the acknowledged state, indicating that the user has reviewed the results of this task and taken some follow up action, so it's safe to schedule this task again with losing results (this is not enforced)

The route reports the following statuses:
'not_started' | 'in_progress' | 'stale' | 'being_canceled' | 'canceled' | 'failed' | 'completed' | 'acknowledged'

Most of them are the state of the task, but stale is a special route status that indicates that no updates were made to the task document for a while.
The failed result includes an error message while completed and acknowledged include the payload found on task.task.payload.

Task document schema

Follow up to #245725

The stored documents have the following shape:

{
  id: string;
  type: string;
  status: TaskStatus;
  stream: string;
  space: string;
  created_at: string;
  task: {
    params: TaskParams;
    payload?: any // Only for completed and acknowledged tasks
    error?: string // Only for failed tasks
  };
}

All fields except task are indexed, and we store things under task to avoid indexing them because of #245974
The tasks are stored in .kibana_streams_tasks

[mykolaharmash]

Looks good overall, I added a few comments for a discussion.

I assume this would be the next iteration of this feature, but just in case, I think we need some kind of bulk variation of all endpoints to read, schedule, and update tasks considering the UI we have in mind where user selects all streams they want to analyze at the same time.

[mykolaharmash]

Any reason to not have it as a single parameter? This would let us use switch in the handler. Also, multiple boolean params sort of suggest that they can be used simultaneously within a single request which is not the case.

[shahzad31]

yeah for sure i thought the same, it's not super polished but i think we can move ahead while doing a refactor.

[miltonhultgren]

Yes, this was emergent code. I'm considering splitting this into two APIs, one for asking the current status (no side effects) and one for all side effects.

[mykolaharmash]

Having a single endpoint for reading and updating the task feels odd tbh, any particular reason to do that instead of having a separate GET endpoint?

[mykolaharmash]

Should we rely on the stale task status here instead? This way, if a task becomes stale we could provide some feedback about this in the UI and stop polling at the same time. Otherwise UI and the backend sort of go out of sync and there is no way to communicate the up-to-date status to the user.

[shahzad31]

LGTM !!

We can probably refine API format in a follow up.

[mykolaharmash]

Agree with @shahzad31, we can merge and iterate on this 👍

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 22f8921

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
datasetQuality	1092	1093	+1
streamsApp	1420	1423	+3
total			+4

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/streams-schema	226	229	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
streamsApp	1.5MB	1.5MB	+3.7KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
streams	24	25	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
streamsApp	24.1KB	23.5KB	-656.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/streams-schema	233	236	+3

History

• 💔 Build #377283 failed 253ca61
• 💔 Build #377148 failed c3e550e
• 💔 Build #376945 failed 46eb4ee
• 💔 Build #376705 failed a459e56
• 💔 Build #376702 failed c7c5db4

[pmuellr]

In the cancellable_task, I'm not seeing the task actually being cancelled via the passed in AbortController. Perhaps it's elsewhere and I missed it, but it seems like the call to abort the AC should be in this code.

[pmuellr]

It's not clear to me that the actual task is going to be cancelled, if the cancellation promise goes off. I weould expect the AbortController sent into the task to get signalled, to indicate to the task itself that it's cancelled. And then the task has to actually USE that AbortController, as needed, to check to see if the task has been cancelled.

[miltonhultgren]

As far as I understand, there are two sources that would call AbortController.abort:

1. The task manager itself (when cancelling a task run for any reason)
2. The cancellableTask wrapper (on line 32 here) if the task has been marked being_canceled by the Streams code

In either case, the actual tasks themselves should be using runContext.abortController to pass to their HTTP requests so that those can be cancelled in response to either of those two cases calling abort.

Does that address your concern?

[pmuellr]

Thanks for clearing that up. I think I must have missed the .abort() call in that code, somehow!

[pmuellr]

LGTM