[Streams][SigEvents] Add background task for significant events query generation by cesco-f · Pull Request #248608 · elastic/kibana

cesco-f · 2026-01-12T09:25:00Z

It closes https://github.com/elastic/streams-program/issues/708.

Summary

This PR introduces a background task-based approach for significant events query generation with dedicated _status and _task endpoints.

Changes

Backend

Task Definition

Added significant_events_queries_generation task definition that handles LLM-based generation of significant event queries as a background task
Integrated with telemetry for tracking generation events

API Endpoints

GET /api/streams/{name}/significant_events/_status - Check the status of the generation task (handles stale detection, returns task results)
POST /api/streams/{name}/significant_events/_task - Manage task lifecycle with a discriminated union body:
- action: 'schedule' - Schedule a new generation task with from, to, connectorId, sampleDocsSize, and systems parameters
- action: 'cancel' - Cancel an in-progress task
- action: 'acknowledge' - Acknowledge a completed task

Frontend

Added useSignificantEventsApi hook methods for task management (getGenerationTask, scheduleGenerationTask, cancelGenerationTask, acknowledgeGenerationTask)
Updated AddSignificantEventFlyout to work with the task-based generation flow

Screen.Recording.2026-01-09.at.16.53.35.mov

cesco-f · 2026-01-12T10:10:53Z

/ci

cesco-f · 2026-01-12T11:19:01Z

/ci

x-pack/platform/packages/shared/kbn-streams-schema/src/api/significant_events/index.ts

...detail_significant_events_view/add_significant_event_flyout/add_significant_event_flyout.tsx

x-pack/platform/plugins/shared/streams/server/routes/streams/significant_events/route.ts

...ns/shared/streams/server/lib/tasks/task_definitions/significant_events_queries_generation.ts

…vents from all data

miltonhultgren · 2026-01-12T15:10:19Z

There's two things I'd like to see in this PR:

When the task is in progress and I refresh the page, I would like to see the button have a spinner to show that generation is still happening
If the task has completed, when I refresh that page and click the button, I would like to see the results of the previous task, rather than schedule a new task that I have to wait for.

Do you think it will be a lot of effort to change the code to this?

cesco-f · 2026-01-12T15:22:31Z

There's two things I'd like to see in this PR:

When the task is in progress and I refresh the page, I would like to see the button have a spinner to show that generation is still happening

If the task has completed, when I refresh that page and click the button, I would like to see the results of the previous task, rather than schedule a new task that I have to wait for.

Do you think it will be a lot of effort to change the code to this?

What button are you referring to?
If the task is completed and you refresh the page you can see the results of the previous task, not sure if I'm understanding correctly.

In progress task:

Screen.Recording.2026-01-12.at.16.24.31.mov

Task completed:

Screen.Recording.2026-01-12.at.16.24.59.mov

miltonhultgren · 2026-01-12T15:30:23Z

@cesco-f Okay so:

I would expect the "+ Significant events" button to show loading state there
If you test this same flow from an empty state then it works differently (since it's different buttons being shown)

cesco-f · 2026-01-12T15:44:51Z

@cesco-f Okay so:

I would expect the "+ Significant events" button to show loading state there

If you test this same flow from an empty state then it works differently (since it's different buttons being shown)

Aren't we getting rid of this view? Does it make sense to invest more time to improve the UX?

miltonhultgren · 2026-01-13T09:42:58Z

@cesco-f You're right, ignore me, perfectionism is a hard habit to beat!

x-pack/platform/plugins/shared/streams/server/routes/streams/significant_events/route.ts

elasticmachine · 2026-01-13T12:07:38Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 2faf0b4

Failed CI Steps

FTR Configs #41

Test Failures

[job] [logs] FTR Configs #41 / Actions and Triggers app create alert should show KEEP command warning when creating a ES query rule with ESQL

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/streams-schema`	233	237	+4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`streamsApp`	1.5MB	1.5MB	+848.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/streams-schema`	240	244	+4

History

pmuellr

LGTM

… generation (elastic#248608) It closes elastic/streams-program#708. ## Summary This PR introduces a background task-based approach for significant events query generation with dedicated `_status` and `_task` endpoints. ## Changes ### Backend **Task Definition** - Added `significant_events_queries_generation` task definition that handles LLM-based generation of significant event queries as a background task - Integrated with telemetry for tracking generation events **API Endpoints** - `GET /api/streams/{name}/significant_events/_status` - Check the status of the generation task (handles stale detection, returns task results) - `POST /api/streams/{name}/significant_events/_task` - Manage task lifecycle with a discriminated union body: - `action: 'schedule'` - Schedule a new generation task with `from`, `to`, `connectorId`, `sampleDocsSize`, and `systems` parameters - `action: 'cancel'` - Cancel an in-progress task - `action: 'acknowledge'` - Acknowledge a completed task ### Frontend - Added `useSignificantEventsApi` hook methods for task management (`getGenerationTask`, `scheduleGenerationTask`, `cancelGenerationTask`, `acknowledgeGenerationTask`) - Updated `AddSignificantEventFlyout` to work with the task-based generation flow https://github.com/user-attachments/assets/64f82260-7108-4fcc-95e7-3330c644600a --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

cesco-f added 2 commits January 12, 2026 09:53

feat(sigevents queries): add sig events queries task

ef2e0bf

feat(split): split _task and _status endpoints

3cc5133

cesco-f added release_note:enhancement backport:skip This PR does not require backporting Feature:SigEvents Significant events feature, related to streams and rules/alerts (RnA) labels Jan 12, 2026

github-actions bot added the author:actionable-obs PRs authored by the actionable obs team label Jan 12, 2026

fix(cr): code review

6b49c6c

Changes from yarn openapi:bundle

f7f5149

cesco-f marked this pull request as ready for review January 12, 2026 10:46

cesco-f requested review from a team as code owners January 12, 2026 10:46

cesco-f and others added 2 commits January 12, 2026 11:46

Merge branch 'main' into sig-events-queries-task

66ae75c

Changes from make api-docs

4140fed

elastic-vault-github-plugin-prod bot requested a review from a team as a code owner January 12, 2026 11:00

kibanamachine and others added 3 commits January 12, 2026 11:29

Changes from node scripts/lint_ts_projects --fix

08e73b4

Changes from node scripts/regenerate_moon_projects.js --update

32a8351

fix(generate endpoint): bring back generate endpoint

93fb7d2

cesco-f force-pushed the sig-events-queries-task branch from 663ca3e to 93fb7d2 Compare January 12, 2026 13:45

kibanamachine added 2 commits January 12, 2026 14:01

Changes from yarn openapi:bundle

de2229b

Changes from make api-docs

b8810cb

miltonhultgren reviewed Jan 12, 2026

View reviewed changes

cesco-f added 4 commits January 12, 2026 15:52

fix(internal): new endpoints are internal

26e4bb8

fix(description): remove description generation when generating sig e…

c8864ac

…vents from all data

refactor(task status): add task status enum

8bebf19

refactor(plimit): use plimit for sigevents queries generation

61424af

refactor(internal endpoints): move internal endpoints

d54fcf3

kibanamachine and others added 3 commits January 12, 2026 15:32

Changes from yarn openapi:bundle

3b1cb4b

fix(tokens): remove total tokens used

a4978e1

refactor(task status): use task status

f2e01e2

kibanamachine added 2 commits January 12, 2026 16:05

Changes from make api-docs

8ca997f

Changes from node scripts/lint_ts_projects --fix

60a5ef7

florent-leborgne requested a review from mdbirnstiehl January 12, 2026 16:27

Changes from node scripts/regenerate_moon_projects.js --update

ce06b40

cesco-f removed request for a team and mdbirnstiehl January 12, 2026 16:38

cesco-f and others added 3 commits January 12, 2026 17:57

fix(types): fix types

0f4a9f5

Merge branch 'main' into sig-events-queries-task

69d71fc

fix(test): fix test

dc6b4c3

cesco-f requested a review from a team as a code owner January 12, 2026 19:02

Merge branch 'main' into sig-events-queries-task

4f1d0ad

cesco-f requested a review from miltonhultgren January 13, 2026 08:49

miltonhultgren approved these changes Jan 13, 2026

View reviewed changes

x-pack/platform/plugins/shared/streams/server/routes/streams/significant_events/route.ts Outdated Show resolved Hide resolved

fix(privilege): manage privilege for _task endpoint

2faf0b4

pmuellr approved these changes Jan 13, 2026

View reviewed changes

cesco-f merged commit 01ee4de into elastic:main Jan 13, 2026
13 checks passed

cesco-f deleted the sig-events-queries-task branch January 13, 2026 15:41

kibanamachine added the v9.4.0 label Jan 13, 2026

Conversation

cesco-f commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Backend

Frontend

Uh oh!

cesco-f commented Jan 12, 2026

Uh oh!

cesco-f commented Jan 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

miltonhultgren commented Jan 12, 2026

Uh oh!

cesco-f commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

miltonhultgren commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cesco-f commented Jan 12, 2026

Uh oh!

miltonhultgren commented Jan 13, 2026

Uh oh!

Uh oh!

elasticmachine commented Jan 13, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

Async chunks

API count

History

Uh oh!

pmuellr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cesco-f commented Jan 12, 2026 •

edited

Loading

cesco-f commented Jan 12, 2026 •

edited

Loading

miltonhultgren commented Jan 12, 2026 •

edited

Loading