-
Notifications
You must be signed in to change notification settings - Fork 8.5k
[Response Ops][Reporting] Scheduled Reports - Schedule API #219771
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Response Ops][Reporting] Scheduled Reports - Schedule API #219771
Conversation
…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling
… src/core/server/integration_tests/ci_checks'
…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling
…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling
… src/core/server/integration_tests/ci_checks'
…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling
|
Pinging @elastic/response-ops (Team:ResponseOps) |
| byhour?: number[]; | ||
| byminute?: number[]; | ||
| byweekday?: Weekday[]; | ||
| byweekday?: string[]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ersin-erdal Because the schema specifies this as a string (with validation), I had to make this change to the interface. WDYT
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, this makes sense.
As per the rrule lib interface it is byweekday?: Weekday[] | null; but the schema validation shows that it can be something like -1WE too. So using string[] here looks more convenient.
| ); | ||
| logger.debug(`Successfully created scheduled report: ${report.id}`); | ||
|
|
||
| // TODO - Schedule the report with Task Manager |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will be done in a followup PR
| }; | ||
|
|
||
| // Create a scheduled report saved object | ||
| const report = await soClient.create<ScheduledReportType>( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to add audit logging. Will do it in a followup PR
nreese
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
kibana-presentation changes LGTM
code review only
|
I ran the verification test, and was surprised to see so many I figured just SO viewed from Discover{
"_index": ".kibana_alerting_cases_9.1.0_001",
"_id": "scheduled_report:d87460b6-7843-47cd-9923-5afd5efe9203",
"_version": 1,
"_source": {
"scheduled_report": {
"createdAt": "2025-05-19T19:16:42.055Z",
"createdBy": "elastic",
"enabled": true,
"jobType": "printable_pdf_v2",
"meta": {
"isDeprecated": false,
"layout": "preserve_layout",
"objectType": "dashboard"
},
"migrationVersion": "9.1.0",
"title": "[Logs] Web Traffic",
"payload": "{\"browserTimezone\":\"America/New_York\",\"layout\":{\"dimensions\":{\"height\":2220,\"width\":1364},\"id\":\"preserve_layout\"},\"objectType\":\"dashboard\",\"title\":\"[Logs] Web Traffic\",\"version\":\"9.1.0\",\"locatorParams\":[{\"id\":\"DASHBOARD_APP_LOCATOR\",\"params\":{\"dashboardId\":\"edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b\",\"preserveSavedFilters\":true,\"timeRange\":{\"from\":\"now-7d/d\",\"to\":\"now\"},\"useHash\":false,\"viewMode\":\"view\"}}],\"isDeprecated\":false}",
"schedule": {
"rrule": {
"freq": 3,
"interval": 3,
"byhour": [
12
],
"byminute": [
0
],
"tzid": "UTC"
}
}
},
"type": "scheduled_report",
"references": [],
"managed": false,
"namespaces": [
"default"
],
"coreMigrationVersion": "8.8.0",
"typeMigrationVersion": "10.1.0",
"updated_at": "2025-05-19T19:16:42.055Z",
"created_at": "2025-05-19T19:16:42.055Z"
},
"fields": {
"scheduled_report.schedule.rrule.interval": [
3
],
"scheduled_report.jobType": [
"printable_pdf_v2"
],
"scheduled_report.schedule.rrule.byhour": [
12
],
"created_at": [
"2025-05-19T19:16:42.055Z"
],
"type": [
"scheduled_report"
],
"scheduled_report.title": [
"[Logs] Web Traffic"
],
"scheduled_report.schedule.rrule.byminute": [
0
],
"scheduled_report.meta.objectType": [
"dashboard"
],
"scheduled_report.createdAt": [
"2025-05-19T19:16:42.055Z"
],
"scheduled_report.meta.layout": [
"preserve_layout"
],
"updated_at": [
"2025-05-19T19:16:42.055Z"
],
"scheduled_report.createdBy": [
"elastic"
],
"scheduled_report.schedule.rrule.tzid": [
"UTC"
],
"managed": [
false
],
"scheduled_report.enabled": [
true
],
"typeMigrationVersion": [
"10.1.0"
],
"scheduled_report.meta.isDeprecated": [
false
],
"coreMigrationVersion": [
"8.8.0"
],
"scheduled_report.schedule.rrule.freq": [
3
],
"scheduled_report.migrationVersion": [
"9.1.0"
],
"scheduled_report.payload": [
"{\"browserTimezone\":\"America/New_York\",\"layout\":{\"dimensions\":{\"height\":2220,\"width\":1364},\"id\":\"preserve_layout\"},\"objectType\":\"dashboard\",\"title\":\"[Logs] Web Traffic\",\"version\":\"9.1.0\",\"locatorParams\":[{\"id\":\"DASHBOARD_APP_LOCATOR\",\"params\":{\"dashboardId\":\"edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b\",\"preserveSavedFilters\":true,\"timeRange\":{\"from\":\"now-7d/d\",\"to\":\"now\"},\"useHash\":false,\"viewMode\":\"view\"}}],\"isDeprecated\":false}"
],
"namespaces": [
"default"
]
}
}edit: Ying set me right, Kibana is being "helpful" here, showing "unmapped" fields :-) |
|
I tried scheduling a report with a bogus dashboard id, and it created it successfully. Seems like it should error. If so, we can defer to later ... |
Hmm...I wonder if it's something that's done by Discover? Here is a rule SO from Discover: rule SO viewed from Discover{
"_index": ".kibana_alerting_cases_9.1.0_001",
"_id": "alert:991eb5f6-7448-4702-8428-e5044632d4b3",
"_version": 5,
"_source": {
"alert": {
"name": "Always firing rule",
"tags": [],
"enabled": true,
"alertTypeId": "example.always-firing",
"consumer": "alerts",
"legacyId": null,
"schedule": {
"interval": "1m"
},
"actions": [],
"params": {},
"createdBy": "elastic",
"updatedBy": "elastic",
"createdAt": "2025-05-19T19:37:09.599Z",
"updatedAt": "2025-05-19T19:37:09.599Z",
"apiKey": "d2jZLZXjZK7tOB8WXwencU7Sceyws+mM0DRZCHPwldnB2D41Ndh+KDWyawu0yIY5Ppi72xUfSC1wKDl9udHCgZ1yvfIS/rpIIsDRNFwkw76LFu7FG7j0q/66NhQsMHSZTW7S45tWUam52FhMRiwIXl3UJlSRGgyYTTJqc2OSaZIEygeocRW6E+3t1+1XliqdAg4y/kjn+4kN8Q==",
"apiKeyOwner": "elastic",
"apiKeyCreatedByUser": false,
"throttle": null,
"notifyWhen": null,
"muteAll": false,
"mutedInstanceIds": [],
"executionStatus": {
"status": "active",
"lastExecutionDate": "2025-05-19T19:39:10.733Z",
"lastDuration": 871,
"warning": null,
"error": null
},
"monitoring": {
"run": {
"history": [
{
"duration": 502,
"success": true,
"timestamp": 1747683430717
},
{
"duration": 716,
"success": true,
"timestamp": 1747683490717
},
{
"duration": 871,
"success": true,
"timestamp": 1747683550733
}
],
"calculated_metrics": {
"success_ratio": 1,
"p99": 871,
"p50": 716,
"p95": 871
},
"last_run": {
"timestamp": "2025-05-19T19:39:10.733Z",
"metrics": {
"duration": 871,
"total_search_duration_ms": null,
"total_indexing_duration_ms": null,
"total_alerts_detected": null,
"total_alerts_created": null,
"gap_duration_s": null
}
}
}
},
"snoozeSchedule": [],
"revision": 0,
"running": false,
"alertDelay": {
"active": 1
},
"artifacts": {
"dashboards": [],
"investigation_guide": {
"blob": ""
}
},
"meta": {
"versionApiKeyLastmodified": "9.1.0"
},
"scheduledTaskId": "991eb5f6-7448-4702-8428-e5044632d4b3",
"lastRun": {
"outcomeOrder": 0,
"alertsCount": {
"new": 5,
"ignored": 0,
"recovered": 5,
"active": 5
},
"outcomeMsg": null,
"warning": null,
"outcome": "succeeded"
},
"nextRun": "2025-05-19T19:40:10.707Z"
},
"type": "alert",
"references": [],
"managed": false,
"namespaces": [
"default"
],
"coreMigrationVersion": "8.8.0",
"typeMigrationVersion": "10.6.0",
"updated_at": "2025-05-19T19:37:10.106Z",
"created_at": "2025-05-19T19:37:09.600Z"
},
"fields": {
"alert.running": [
false
],
"alert.monitoring.run.last_run.metrics.duration": [
871
],
"alert.apiKeyOwner": [
"elastic"
],
"alert.updatedBy": [
"elastic"
],
"alert.apiKeyCreatedByUser": [
false
],
"alert.muteAll": [
false
],
"created_at": [
"2025-05-19T19:37:09.600Z"
],
"alert.lastRun.alertsCount.active": [
5
],
"type": [
"alert"
],
"alert.alertDelay.active": [
1
],
"alert.executionStatus.lastExecutionDate": [
"2025-05-19T19:39:10.733Z"
],
"alert.lastRun.alertsCount.recovered": [
5
],
"alert.nextRun": [
"2025-05-19T19:40:10.707Z"
],
"updated_at": [
"2025-05-19T19:37:10.106Z"
],
"alert.scheduledTaskId": [
"991eb5f6-7448-4702-8428-e5044632d4b3"
],
"alert.createdAt": [
"2025-05-19T19:37:09.599Z"
],
"managed": [
false
],
"typeMigrationVersion": [
"10.6.0"
],
"alert.monitoring.run.calculated_metrics.success_ratio": [
1
],
"alert.updatedAt": [
"2025-05-19T19:37:09.599Z"
],
"alert.artifacts.investigation_guide.blob": [
""
],
"alert.name": [
"Always firing rule"
],
"alert.enabled": [
true
],
"alert.monitoring.run.history.duration": [
502,
716,
871
],
"alert.revision": [
0
],
"alert.alertTypeId": [
"example.always-firing"
],
"alert.monitoring.run.history.timestamp": [
1747683430717,
1747683490717,
1747683550733
],
"alert.lastRun.outcomeOrder": [
0
],
"alert.executionStatus.status": [
"active"
],
"alert.monitoring.run.last_run.timestamp": [
"2025-05-19T19:39:10.733Z"
],
"alert.schedule.interval": [
"1m"
],
"alert.lastRun.outcome": [
"succeeded"
],
"alert.consumer": [
"alerts"
],
"alert.name.keyword": [
"always firing rule"
],
"alert.apiKey": [
"d2jZLZXjZK7tOB8WXwencU7Sceyws+mM0DRZCHPwldnB2D41Ndh+KDWyawu0yIY5Ppi72xUfSC1wKDl9udHCgZ1yvfIS/rpIIsDRNFwkw76LFu7FG7j0q/66NhQsMHSZTW7S45tWUam52FhMRiwIXl3UJlSRGgyYTTJqc2OSaZIEygeocRW6E+3t1+1XliqdAg4y/kjn+4kN8Q=="
],
"alert.createdBy": [
"elastic"
],
"alert.lastRun.alertsCount.new": [
5
],
"alert.monitoring.run.calculated_metrics.p99": [
871
],
"alert.monitoring.run.calculated_metrics.p95": [
871
],
"alert.monitoring.run.history.success": [
true,
true,
true
],
"alert.monitoring.run.calculated_metrics.p50": [
716
],
"coreMigrationVersion": [
"8.8.0"
],
"alert.executionStatus.lastDuration": [
871
],
"alert.lastRun.alertsCount.ignored": [
0
],
"alert.meta.versionApiKeyLastmodified": [
"9.1.0"
],
"namespaces": [
"default"
]
},
"ignored_field_values": {
"alert.params": [
{}
]
}
}where |
pmuellr
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, left a few nit-ish comments.
Still wondering about the unexpected (to me) fields that I commented on in the PR comments (not in these review comments).
| "reporting" | ||
| ], | ||
| "requiredPlugins": [ | ||
| "actions", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
heh, could be a problem if we ever have "reporting" connectors, but we can factor the config bits out of the actions plugin later if we need to :-)
| export function setupSavedObjects(savedObjects: SavedObjectsServiceSetup) { | ||
| savedObjects.registerType({ | ||
| name: SCHEDULED_REPORT_SAVED_OBJECT_TYPE, | ||
| indexPattern: ALERTING_CASES_SAVED_OBJECT_INDEX, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
heh, not crazy about it, but seems survivable ...
| created_by: rawScheduledReport.attributes.createdBy as string | false, | ||
| payload: parsedPayload, | ||
| meta: rawScheduledReport.attributes.meta, | ||
| migration_version: rawScheduledReport.attributes.migrationVersion, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't tell what this is used for, and wonder how it differs from the base SO migration_version field. I guess the report docs have this, as some kind of "follow the SO pattern" since they aren't SO's - I assume we probably need this since the core reporting stuff does, but feels like it may get confusing ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AFAICT, this is always set to 7.14.0 so maybe it's something that's not needed anymore? I was trying to save everything that's currently saved and read for the report generation. I can make it optional in the schema so if we remove it in the future we're not forced to populate it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated to make migration version optional in SO schema: b6af770
| export const rawNotificationSchema = schema.object({ | ||
| email: schema.maybe( | ||
| schema.object({ | ||
| to: schema.arrayOf(schema.string(), { minSize: 1 }), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if someone will want "bcc only" or such. I think in the email connector, we make sure the count of all these is at least one. Certainly survivable for now, and "loosening" this in the future shouldn't cause any migration issues, if we want to do that ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated to require email in any of the fields: 55c7011
Ah, yea I think the current single report scheduling does not do any checking either. I can create a followup issue to check for both. I think maybe we can do it along with extracting and injecting references since I think both involve parsing out the job params to get the saved object ID to check |
Ah, ya, I see that now ... thanks, I figured I was holding it wrong, but scared there might be something in Core mixing up some fields somehow ... |
💚 Build Succeeded
Metrics [docs]Public APIs missing comments
Async chunks
Public APIs missing exports
Page load bundle
Unknown metric groupsAPI count
History
cc @ymao1 |
Resolves #216313 ## Summary This is a feature branch that contains the following commits. Each individual linked PR contains a summary and verification instructions. * Schedule API - #219771 * Scheduled report task runner - #219770 * List and disable API - #220922 * Audit logging - #221846 * Send scheduled report emails - #220539 * Commit to check license - f5f9d9d * Update to list API response format - #224262 --------- Co-authored-by: Ersin Erdal <[email protected]> Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Ersin Erdal <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Alexi Doak <[email protected]>
Resolves elastic#216313 ## Summary This is a feature branch that contains the following commits. Each individual linked PR contains a summary and verification instructions. * Schedule API - elastic#219771 * Scheduled report task runner - elastic#219770 * List and disable API - elastic#220922 * Audit logging - elastic#221846 * Send scheduled report emails - elastic#220539 * Commit to check license - elastic@f5f9d9d * Update to list API response format - elastic#224262 --------- Co-authored-by: Ersin Erdal <[email protected]> Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Ersin Erdal <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Alexi Doak <[email protected]> (cherry picked from commit a409627) # Conflicts: # src/platform/packages/private/kbn-reporting/common/routes.ts # x-pack/platform/plugins/private/canvas/server/feature.test.ts # x-pack/platform/plugins/private/reporting/server/core.ts # x-pack/platform/plugins/private/reporting/server/features.ts # x-pack/platform/plugins/shared/features/server/__snapshots__/oss_features.test.ts.snap # x-pack/platform/test/api_integration/apis/features/features/features.ts # x-pack/test_serverless/api_integration/test_suites/chat/platform_security/authorization.ts # x-pack/test_serverless/api_integration/test_suites/observability/platform_security/authorization.ts # x-pack/test_serverless/api_integration/test_suites/search/platform_security/authorization.ts # x-pack/test_serverless/api_integration/test_suites/security/platform_security/authorization.ts
Towards #216313
Note
This PR will be merged into a feature branch -
elastic:scheduled-reportsOnce this is merged, I will open a draft PR from the feature branch into
mainSummary
Schedule API
/internal/reporting/schedule/{exportTypeId}scheduled_reporttype saved object in the.kibana-alerting-casesindex.Verify
GET .kibana_alerting_cases/_search?q=type:scheduled_report