[SLO] Bulk Purge SLI data by baileycash-elastic · Pull Request #218287 · elastic/kibana

baileycash-elastic · 2025-04-15T14:39:32Z

Summary

Introduces starter API for bulk purging of SLI data.
docs coming pending review
Users can submit a list of SLO ids for which they would like to purge rollup data, as well as a purge policy:

    "purgePolicy": {
        "purgeType": "fixed_age",
        "age": "1w"
    }

    "purgePolicy": {
        "purgeType": "fixed_time",
        "age": "2025-04-01T00:00:00Z"
    }

more tests

tweaks and tests

elasticmachine · 2025-04-16T16:33:17Z

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

kdelemme · 2025-04-16T16:52:14Z

x-pack/platform/packages/shared/kbn-slo-schema/src/rest_specs/routes/bulk_purge_rollup.ts

+const bulkPurgeRollupSchema = t.intersection([
+  t.partial({
+    query: t.partial({
+      force: t.string,
+    }),
+  }),
+  t.type({
+    body: t.type({
+      ids: t.array(t.string),
+      purgePolicy,
+    }),
+  }),
+]);


Might be simpler to move force into the body as well

kdelemme · 2025-04-16T16:54:41Z

x-pack/solutions/observability/plugins/slo/server/routes/slo/purge_rollup_data.ts

+    const repository = new KibanaSavedObjectsSLORepository(soClient, logger);
+    const purgeRollupData = new PurgeRollupData(esClient, repository);
+
+    await purgeRollupData.execute(params);


params contains body and query, which are considered implementation details of the route handler. The PurgeRollupData service should not know anything about a body or a query (http specific)

Also, if we move force into the body, you can then use a Params type like:

type BulkPurgeRollupDataParams = t.TypeOf<typeof bulkPurgeRollupSchema.props.body>;

kdelemme

Just a few naming convention, imports and schema/type to fix. And use the slo list from the repository for the delete by query ids

kdelemme · 2025-04-16T20:18:40Z

x-pack/solutions/observability/plugins/slo/server/services/purge_rollup_data.ts

+    if (purgePolicy.purgeType === 'fixed_age') {
+      return `now-${purgePolicy.age.format()}`;
+    } else {
+      return purgePolicy.timestamp.toISOString();
+    }


can you add an assertNever(purgeType);?

kdelemme · 2025-04-16T20:23:45Z

x-pack/solutions/observability/plugins/slo/server/services/purge_rollup_data.ts

+
+  private async validatePurgePolicy(params: PurgeRollupSchemaType) {
+    const { ids, purgePolicy } = params;
+    const slos = await this.repository.findAllByIds(ids);


I think we should fetch the slo list outside of the validation, and provide it to this function, and also use the sloList in the deleteByQuery: { 'slo.id': map(sloList, "id") }. so we don't delete slo that the user should not have access to.

kdelemme · 2025-04-16T20:24:34Z

x-pack/solutions/observability/plugins/slo/server/services/purge_rollup_data.ts

+        bool: {
+          filter: [
+            {
+              terms: { 'slo.id': params.ids },


So we don't delete SLOs that the user should not have access to.

Suggested change

terms: { 'slo.id': params.ids },

terms: { 'slo.id': map(sloList, "id") },

kdelemme · 2025-04-16T20:27:38Z

x-pack/solutions/observability/plugins/slo/server/services/purge_rollup_data.ts

+            return moment(timestamp).isAfter(
+              moment(Date.now()).startOf(slo.timeWindow.duration.unit)
+            );
+          } else {
+            return moment(timestamp).isAfter(
+              moment(Date.now()).subtract(slo.timeWindow.duration.asSeconds(), 's')
+            );


nitpick: create moment(timestamp) once outside the some predicate, so we don't recreate one every time.

kdelemme · 2025-04-16T20:28:15Z

x-pack/solutions/observability/plugins/slo/server/routes/slo/purge_rollup_data.ts

+import { KibanaSavedObjectsSLORepository } from '../../services';
+
+export const purgeRollupDataRoute = createSloServerRoute({
+  endpoint: 'POST /api/observability/slos/_purge_rollup 2023-10-31',


Suggested change

endpoint: 'POST /api/observability/slos/_purge_rollup 2023-10-31',

endpoint: 'POST /api/observability/slos/_bulk_purge_rollup 2023-10-31',

kdelemme · 2025-04-16T20:28:56Z

x-pack/platform/packages/shared/kbn-slo-schema/src/schema/slo.ts

 import { durationType } from './duration';
 import { indicatorSchema } from './indicators';
 import { timeWindowSchema } from './time_window';
+import { bulkPurgeRollupSchema } from '../rest_specs/routes/bulk_purge_rollup';


don't import the rest_specs here.

kdelemme · 2025-04-16T20:29:39Z

x-pack/platform/packages/shared/kbn-slo-schema/src/rest_specs/routes/bulk_purge_rollup.ts

+type PurgePolicyType = t.TypeOf<typeof purgePolicy>;
+type PurgeRollupSchemaType = t.TypeOf<typeof bulkPurgeRollupSchema.props.body>;
+
+export type { PurgePolicyType, PurgeRollupSchemaType };


add export * from './bulk_purge_rollup'; into rest_specs/routes/index.ts

kdelemme · 2025-04-16T20:33:20Z

x-pack/platform/packages/shared/kbn-slo-schema/src/rest_specs/routes/bulk_purge_rollup.ts

+});
+
+type PurgePolicyType = t.TypeOf<typeof purgePolicy>;
+type PurgeRollupSchemaType = t.TypeOf<typeof bulkPurgeRollupSchema.props.body>;


Can you name the schema and type like the other schemas to be consistent, and introduce the response type as well e.g.

const bulkPurgeRollupParamsSchema = t.type({ ... }) const bulkPurgeRollupResponseSchema = t.type({ taskId: t.string }); type BulkPurgeRollupInput = t.OutputOf<typeof bulkPurgeRollupParamsSchema.props.body>; type BulkPurgeRollupParams = t.TypeOf<typeof bulkPurgeRollupParamsSchema.props.body>; type BulkPurgeRollupResponse = t.TypeOf<typeof bulkPurgeRollupResponseSchema.props.body>;

kdelemme · 2025-04-16T20:34:07Z

x-pack/solutions/observability/plugins/slo/server/routes/slo/purge_rollup_data.ts

+import { PurgeRollupData } from '../../services/purge_rollup_data';
+import { KibanaSavedObjectsSLORepository } from '../../services';
+
+export const purgeRollupDataRoute = createSloServerRoute({


can you name that bulkPurgeRollupRoute to be consistent with the types and url

kdelemme · 2025-04-16T20:35:07Z

x-pack/solutions/observability/plugins/slo/server/services/purge_rollup_data.ts

+
+  public async execute(
+    params: PurgeRollupSchemaType
+  ): Promise<{ taskId: DeleteByQueryResponse['task'] }> {


Use the BulkPurgeRollupResponse here

kdelemme

LGTM thanks for the changes

kdelemme · 2025-04-22T13:21:31Z

@baileycash-elastic Can you merge main in, and use the new getScopedClient(request) from the route handler?

csr

x-pack/test/api_integration/deployment_agnostic/services/slo_api.ts purge endpoint change LGTM.

kibanamachine · 2025-04-22T21:58:48Z

Starting backport for target branches: 8.17, 8.18, 8.19

https://github.com/elastic/kibana/actions/runs/14605402192

elasticmachine · 2025-04-22T21:58:55Z

💚 Build Succeeded

Buildkite Build
Commit: 6fea364

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`observability`	1289	1290	+1
`slo`	1118	1119	+1
`synthetics`	1187	1188	+1
total			+3

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/slo-schema`	188	193	+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`observability`	1.3MB	1.3MB	+264.0B
`slo`	911.3KB	911.5KB	+224.0B
`synthetics`	954.6KB	954.9KB	+254.0B
total			+742.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`observability`	94.5KB	94.5KB	+1.0B
`synthetics`	27.1KB	27.1KB	-2.0B
total			-1.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/slo-schema`	188	193	+5

History

💔 Build #294949 failed 237a24c
💚 Build #294692 succeeded 39ab846
💚 Build #294591 succeeded e57f802
💚 Build #294349 succeeded 7e2bdf3
💔 Build #294334 failed 94d8f0a

kibanamachine · 2025-04-22T22:05:28Z

💔 All backports failed

Status	Branch	Result
❌	8.17	Backport failed because of merge conflicts
❌	8.18	Backport failed because of merge conflicts
❌	8.19	Backport failed because of merge conflicts You might need to backport the following PRs to 8.19: - [Cloud Connector] Add cloud_connectors config in Agentless API (#215421) - [streams][content pack] archive format and portable dashboards (#217288)

Manual backport

To create the backport manually run:

node scripts/backport --pr 218287

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2025-04-22T22:07:55Z

Starting backport for target branches: 8.17, 8.18, 8.19

https://github.com/elastic/kibana/actions/runs/14605533214

kdelemme · 2025-04-22T22:08:12Z

@baileycash-elastic I've removed the backport:prev-major. we should only backport to 9.1 and 8.19

kibanamachine · 2025-04-22T22:14:58Z

💔 All backports failed

Status	Branch	Result
❌	8.17	Backport failed because of merge conflicts
❌	8.18	Backport failed because of merge conflicts
❌	8.19	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 218287

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2025-04-23T22:49:03Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 218287 locally

baileycash-elastic · 2025-04-24T18:24:21Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

## Summary Resolves elastic#210025 Introduces starter API for bulk purging of SLI data. **docs coming pending review** Users can submit a list of SLO ids for which they would like to purge rollup data, as well as a purge policy: ``` "purgePolicy": { "purgeType": "fixed_age", "age": "1w" } ``` ``` "purgePolicy": { "purgeType": "fixed_time", "age": "2025-04-01T00:00:00Z" } ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 2df2e78) # Conflicts: # oas_docs/output/kibana.yaml # x-pack/solutions/observability/plugins/slo/docs/openapi/slo/bundled.json # x-pack/solutions/observability/plugins/slo/docs/openapi/slo/bundled.yaml # x-pack/solutions/observability/plugins/slo/docs/openapi/slo/entrypoint.yaml

kibanamachine · 2025-04-24T23:49:05Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @baileycash-elastic

# Backport This will backport the following commits from `main` to `8.19`: - [[SLO] Bulk Purge SLI data (#218287)](#218287)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kevin Delemme <kevin.delemme@elastic.co>

## Summary Resolves elastic#210025 Introduces starter API for bulk purging of SLI data. **docs coming pending review** Users can submit a list of SLO ids for which they would like to purge rollup data, as well as a purge policy: ``` "purgePolicy": { "purgeType": "fixed_age", "age": "1w" } ``` ``` "purgePolicy": { "purgeType": "fixed_time", "age": "2025-04-01T00:00:00Z" } ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

baileycash-elastic added 4 commits April 8, 2025 12:35

rolled up route start

0741439

purge policy

b440d48

add baseline test, cleanup api

09c531a

fix tests

4181771

baileycash-elastic changed the title ~~Slo 210025~~ [SLO] Bulk Purge SLI data Apr 15, 2025

baileycash-elastic added 3 commits April 15, 2025 13:54

fix logic

94fb94c

move code to service

2478f75

more tests

a88e2ba

more tests

baileycash-elastic force-pushed the slo-210025 branch from c428426 to a88e2ba Compare April 16, 2025 02:06

baileycash-elastic added Team:actionable-obs Formerly "obs-ux-management", responsible for SLO, o11y alerting, significant events, & synthetics. backport:prev-major v9.1.0 release_note:skip Skip the PR/issue when compiling release notes labels Apr 16, 2025

tweaks and tests

4e4a247

tweaks and tests

baileycash-elastic force-pushed the slo-210025 branch from f543477 to 4e4a247 Compare April 16, 2025 16:32

baileycash-elastic marked this pull request as ready for review April 16, 2025 16:33

baileycash-elastic requested review from a team as code owners April 16, 2025 16:33

baileycash-elastic added 3 commits April 16, 2025 12:39

add unit tests

0a9e8ad

update sp

7ce07fe

Merge branch 'main' into slo-210025

a3949db

kdelemme reviewed Apr 16, 2025

View reviewed changes

baileycash-elastic added 2 commits April 16, 2025 15:25

code improvements

d2884b0

Update index.ts

850b70e

kdelemme reviewed Apr 16, 2025

View reviewed changes

fixing linter issue(?)

b3cbd7b

baileycash-elastic marked this pull request as draft April 16, 2025 21:51

address comments

52e8535

baileycash-elastic marked this pull request as ready for review April 17, 2025 13:30

baileycash-elastic and others added 2 commits April 21, 2025 15:13

update docs

112f133

[CI] Auto-commit changed files from 'make api-docs'

39ab846

kdelemme approved these changes Apr 21, 2025

View reviewed changes

Merge branch 'main' into slo-210025

8e9d4c7

csr approved these changes Apr 22, 2025

View reviewed changes

uae scoped clients

237a24c

baileycash-elastic enabled auto-merge (squash) April 22, 2025 14:33

Merge branch 'main' into slo-210025

6fea364

baileycash-elastic merged commit 2df2e78 into elastic:main Apr 22, 2025
9 checks passed

kdelemme added backport:version Backport to applied version labels v8.19.0 and removed backport:prev-major labels Apr 22, 2025

kibanamachine mentioned this pull request Apr 23, 2025

[ObsUX][A11y] Add aria-label for elements announcements #218766

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Apr 23, 2025

baileycash-elastic mentioned this pull request Apr 24, 2025

[8.19] [SLO] Bulk Purge SLI data (#218287) #219147

Merged

kibanamachine mentioned this pull request Apr 25, 2025

feat(slo): bulk delete APIs #217405

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Apr 25, 2025

	terms: { 'slo.id': params.ids },
	terms: { 'slo.id': map(sloList, "id") },

	endpoint: 'POST /api/observability/slos/_purge_rollup 2023-10-31',
	endpoint: 'POST /api/observability/slos/_bulk_purge_rollup 2023-10-31',

Conversation

baileycash-elastic commented Apr 15, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

elasticmachine commented Apr 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kdelemme left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kdelemme left a comment

Choose a reason for hiding this comment

Uh oh!

kdelemme commented Apr 22, 2025

Uh oh!

csr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kibanamachine commented Apr 22, 2025

Uh oh!

elasticmachine commented Apr 22, 2025

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

History

Uh oh!

kibanamachine commented Apr 22, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

kibanamachine commented Apr 22, 2025

Uh oh!

kdelemme commented Apr 22, 2025

Uh oh!

kibanamachine commented Apr 22, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

kibanamachine commented Apr 23, 2025

Uh oh!

baileycash-elastic commented Apr 24, 2025

💚 All backports created successfully

Questions ?

Uh oh!

kibanamachine commented Apr 24, 2025

Uh oh!

Reviewers

baileycash-elastic commented Apr 15, 2025 •

edited by kibanamachine

Loading