Skip to content

Conversation

@ymao1
Copy link
Contributor

@ymao1 ymao1 commented Apr 30, 2025

Towards #216313

Note

This PR will be merged into a feature branch - elastic:scheduled-reports
Once this is merged, I will open a draft PR from the feature branch into main

Summary

Schedule API

  • Added internal API /internal/reporting/schedule/{exportTypeId}
POST /internal/reporting/schedule/{exportTypeId}
{
  "jobParams": <same jobParams format as existing generate single report API>,
  "schedule": {
    "rrule": <modified Task Manager rRule schema from https://github.com/elastic/kibana/pull/217728>
  }
  "notification": {
    "email": {
      "to": <optional list of email addresses>,
      "cc": <optional list of email addresses>,
      "bcc": <optional list of email addresses>
    }
}
  • Scheduling a report creates a new scheduled_report type saved object in the .kibana-alerting-cases index.
  • Uses the same role privileges as the current generate report API, so user must have the reporting subfeature privilege for one of the Analytics feature privileges (Discover/Dashboard/Visualize)

Verify

  1. Run ES and Kibana with examples and install the sample web logs data set.
  2. In the DEV console, call the schedule API using
POST kbn:/internal/reporting/schedule/printablePdfV2
{
    "schedule": {
        "rrule": {
            "freq": 3,
            "interval": 3,
            "byhour": [12],
            "byminute": [0]
        }
    },
    "jobParams": "(browserTimezone:America/New_York,layout:(dimensions:(height:2220,width:1364),id:preserve_layout),locatorParams:!((id:DASHBOARD_APP_LOCATOR,params:(dashboardId:edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b,preserveSavedFilters:!t,timeRange:(from:now-7d/d,to:now),useHash:!f,viewMode:view))),objectType:dashboard,title:'[Logs] Web Traffic',version:'9.1.0')"
}
  1. Verify a saved object has been created using GET .kibana_alerting_cases/_search?q=type:scheduled_report
  2. Try different requests with different schedules and notifications. It should respect the email domain allowlist if configured and reject invalid rrule schedules and email addresses.

ersin-erdal and others added 30 commits April 9, 2025 18:38
…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling
… src/core/server/integration_tests/ci_checks'
…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling
…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling
… src/core/server/integration_tests/ci_checks'
…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling
@ymao1 ymao1 requested review from a team as code owners May 7, 2025 13:48
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@ymao1 ymao1 requested review from pmuellr and umbopepato May 7, 2025 13:48
byhour?: number[];
byminute?: number[];
byweekday?: Weekday[];
byweekday?: string[];
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ersin-erdal Because the schema specifies this as a string (with validation), I had to make this change to the interface. WDYT

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this makes sense.
As per the rrule lib interface it is byweekday?: Weekday[] | null; but the schema validation shows that it can be something like -1WE too. So using string[] here looks more convenient.

);
logger.debug(`Successfully created scheduled report: ${report.id}`);

// TODO - Schedule the report with Task Manager
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be done in a followup PR

};

// Create a scheduled report saved object
const report = await soClient.create<ScheduledReportType>(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to add audit logging. Will do it in a followup PR

Copy link
Contributor

@nreese nreese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kibana-presentation changes LGTM
code review only

@pmuellr
Copy link
Member

pmuellr commented May 19, 2025

I ran the verification test, and was surprised to see so many fields in Discover for these SO's:

I figured just scheduled_report.createdBy would be there as a field. I checked the mappings (in Kibana), and it shows only that field is mapped. Not like it's a problem, at least for scheduled reports, but I must be missing something, because it doesn't feel quite right ...

SO viewed from Discover
{
  "_index": ".kibana_alerting_cases_9.1.0_001",
  "_id": "scheduled_report:d87460b6-7843-47cd-9923-5afd5efe9203",
  "_version": 1,
  "_source": {
    "scheduled_report": {
      "createdAt": "2025-05-19T19:16:42.055Z",
      "createdBy": "elastic",
      "enabled": true,
      "jobType": "printable_pdf_v2",
      "meta": {
        "isDeprecated": false,
        "layout": "preserve_layout",
        "objectType": "dashboard"
      },
      "migrationVersion": "9.1.0",
      "title": "[Logs] Web Traffic",
      "payload": "{\"browserTimezone\":\"America/New_York\",\"layout\":{\"dimensions\":{\"height\":2220,\"width\":1364},\"id\":\"preserve_layout\"},\"objectType\":\"dashboard\",\"title\":\"[Logs] Web Traffic\",\"version\":\"9.1.0\",\"locatorParams\":[{\"id\":\"DASHBOARD_APP_LOCATOR\",\"params\":{\"dashboardId\":\"edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b\",\"preserveSavedFilters\":true,\"timeRange\":{\"from\":\"now-7d/d\",\"to\":\"now\"},\"useHash\":false,\"viewMode\":\"view\"}}],\"isDeprecated\":false}",
      "schedule": {
        "rrule": {
          "freq": 3,
          "interval": 3,
          "byhour": [
            12
          ],
          "byminute": [
            0
          ],
          "tzid": "UTC"
        }
      }
    },
    "type": "scheduled_report",
    "references": [],
    "managed": false,
    "namespaces": [
      "default"
    ],
    "coreMigrationVersion": "8.8.0",
    "typeMigrationVersion": "10.1.0",
    "updated_at": "2025-05-19T19:16:42.055Z",
    "created_at": "2025-05-19T19:16:42.055Z"
  },
  "fields": {
    "scheduled_report.schedule.rrule.interval": [
      3
    ],
    "scheduled_report.jobType": [
      "printable_pdf_v2"
    ],
    "scheduled_report.schedule.rrule.byhour": [
      12
    ],
    "created_at": [
      "2025-05-19T19:16:42.055Z"
    ],
    "type": [
      "scheduled_report"
    ],
    "scheduled_report.title": [
      "[Logs] Web Traffic"
    ],
    "scheduled_report.schedule.rrule.byminute": [
      0
    ],
    "scheduled_report.meta.objectType": [
      "dashboard"
    ],
    "scheduled_report.createdAt": [
      "2025-05-19T19:16:42.055Z"
    ],
    "scheduled_report.meta.layout": [
      "preserve_layout"
    ],
    "updated_at": [
      "2025-05-19T19:16:42.055Z"
    ],
    "scheduled_report.createdBy": [
      "elastic"
    ],
    "scheduled_report.schedule.rrule.tzid": [
      "UTC"
    ],
    "managed": [
      false
    ],
    "scheduled_report.enabled": [
      true
    ],
    "typeMigrationVersion": [
      "10.1.0"
    ],
    "scheduled_report.meta.isDeprecated": [
      false
    ],
    "coreMigrationVersion": [
      "8.8.0"
    ],
    "scheduled_report.schedule.rrule.freq": [
      3
    ],
    "scheduled_report.migrationVersion": [
      "9.1.0"
    ],
    "scheduled_report.payload": [
      "{\"browserTimezone\":\"America/New_York\",\"layout\":{\"dimensions\":{\"height\":2220,\"width\":1364},\"id\":\"preserve_layout\"},\"objectType\":\"dashboard\",\"title\":\"[Logs] Web Traffic\",\"version\":\"9.1.0\",\"locatorParams\":[{\"id\":\"DASHBOARD_APP_LOCATOR\",\"params\":{\"dashboardId\":\"edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b\",\"preserveSavedFilters\":true,\"timeRange\":{\"from\":\"now-7d/d\",\"to\":\"now\"},\"useHash\":false,\"viewMode\":\"view\"}}],\"isDeprecated\":false}"
    ],
    "namespaces": [
      "default"
    ]
  }
}

edit: Ying set me right, Kibana is being "helpful" here, showing "unmapped" fields :-)

@pmuellr
Copy link
Member

pmuellr commented May 19, 2025

I tried scheduling a report with a bogus dashboard id, and it created it successfully. Seems like it should error. If so, we can defer to later ...

@ymao1
Copy link
Contributor Author

ymao1 commented May 19, 2025

I ran the verification test, and was surprised to see so many fields in Discover for these SO's:

I figured just scheduled_report.createdBy would be there as a field. I checked the mappings (in Kibana), and it shows only that field is mapped. Not like it's a problem, at least for scheduled reports, but I must be missing something, because it doesn't feel quite right ...

SO viewed from Discover

Hmm...I wonder if it's something that's done by Discover? Here is a rule SO from Discover:

rule SO viewed from Discover
{
  "_index": ".kibana_alerting_cases_9.1.0_001",
  "_id": "alert:991eb5f6-7448-4702-8428-e5044632d4b3",
  "_version": 5,
  "_source": {
    "alert": {
      "name": "Always firing rule",
      "tags": [],
      "enabled": true,
      "alertTypeId": "example.always-firing",
      "consumer": "alerts",
      "legacyId": null,
      "schedule": {
        "interval": "1m"
      },
      "actions": [],
      "params": {},
      "createdBy": "elastic",
      "updatedBy": "elastic",
      "createdAt": "2025-05-19T19:37:09.599Z",
      "updatedAt": "2025-05-19T19:37:09.599Z",
      "apiKey": "d2jZLZXjZK7tOB8WXwencU7Sceyws+mM0DRZCHPwldnB2D41Ndh+KDWyawu0yIY5Ppi72xUfSC1wKDl9udHCgZ1yvfIS/rpIIsDRNFwkw76LFu7FG7j0q/66NhQsMHSZTW7S45tWUam52FhMRiwIXl3UJlSRGgyYTTJqc2OSaZIEygeocRW6E+3t1+1XliqdAg4y/kjn+4kN8Q==",
      "apiKeyOwner": "elastic",
      "apiKeyCreatedByUser": false,
      "throttle": null,
      "notifyWhen": null,
      "muteAll": false,
      "mutedInstanceIds": [],
      "executionStatus": {
        "status": "active",
        "lastExecutionDate": "2025-05-19T19:39:10.733Z",
        "lastDuration": 871,
        "warning": null,
        "error": null
      },
      "monitoring": {
        "run": {
          "history": [
            {
              "duration": 502,
              "success": true,
              "timestamp": 1747683430717
            },
            {
              "duration": 716,
              "success": true,
              "timestamp": 1747683490717
            },
            {
              "duration": 871,
              "success": true,
              "timestamp": 1747683550733
            }
          ],
          "calculated_metrics": {
            "success_ratio": 1,
            "p99": 871,
            "p50": 716,
            "p95": 871
          },
          "last_run": {
            "timestamp": "2025-05-19T19:39:10.733Z",
            "metrics": {
              "duration": 871,
              "total_search_duration_ms": null,
              "total_indexing_duration_ms": null,
              "total_alerts_detected": null,
              "total_alerts_created": null,
              "gap_duration_s": null
            }
          }
        }
      },
      "snoozeSchedule": [],
      "revision": 0,
      "running": false,
      "alertDelay": {
        "active": 1
      },
      "artifacts": {
        "dashboards": [],
        "investigation_guide": {
          "blob": ""
        }
      },
      "meta": {
        "versionApiKeyLastmodified": "9.1.0"
      },
      "scheduledTaskId": "991eb5f6-7448-4702-8428-e5044632d4b3",
      "lastRun": {
        "outcomeOrder": 0,
        "alertsCount": {
          "new": 5,
          "ignored": 0,
          "recovered": 5,
          "active": 5
        },
        "outcomeMsg": null,
        "warning": null,
        "outcome": "succeeded"
      },
      "nextRun": "2025-05-19T19:40:10.707Z"
    },
    "type": "alert",
    "references": [],
    "managed": false,
    "namespaces": [
      "default"
    ],
    "coreMigrationVersion": "8.8.0",
    "typeMigrationVersion": "10.6.0",
    "updated_at": "2025-05-19T19:37:10.106Z",
    "created_at": "2025-05-19T19:37:09.600Z"
  },
  "fields": {
    "alert.running": [
      false
    ],
    "alert.monitoring.run.last_run.metrics.duration": [
      871
    ],
    "alert.apiKeyOwner": [
      "elastic"
    ],
    "alert.updatedBy": [
      "elastic"
    ],
    "alert.apiKeyCreatedByUser": [
      false
    ],
    "alert.muteAll": [
      false
    ],
    "created_at": [
      "2025-05-19T19:37:09.600Z"
    ],
    "alert.lastRun.alertsCount.active": [
      5
    ],
    "type": [
      "alert"
    ],
    "alert.alertDelay.active": [
      1
    ],
    "alert.executionStatus.lastExecutionDate": [
      "2025-05-19T19:39:10.733Z"
    ],
    "alert.lastRun.alertsCount.recovered": [
      5
    ],
    "alert.nextRun": [
      "2025-05-19T19:40:10.707Z"
    ],
    "updated_at": [
      "2025-05-19T19:37:10.106Z"
    ],
    "alert.scheduledTaskId": [
      "991eb5f6-7448-4702-8428-e5044632d4b3"
    ],
    "alert.createdAt": [
      "2025-05-19T19:37:09.599Z"
    ],
    "managed": [
      false
    ],
    "typeMigrationVersion": [
      "10.6.0"
    ],
    "alert.monitoring.run.calculated_metrics.success_ratio": [
      1
    ],
    "alert.updatedAt": [
      "2025-05-19T19:37:09.599Z"
    ],
    "alert.artifacts.investigation_guide.blob": [
      ""
    ],
    "alert.name": [
      "Always firing rule"
    ],
    "alert.enabled": [
      true
    ],
    "alert.monitoring.run.history.duration": [
      502,
      716,
      871
    ],
    "alert.revision": [
      0
    ],
    "alert.alertTypeId": [
      "example.always-firing"
    ],
    "alert.monitoring.run.history.timestamp": [
      1747683430717,
      1747683490717,
      1747683550733
    ],
    "alert.lastRun.outcomeOrder": [
      0
    ],
    "alert.executionStatus.status": [
      "active"
    ],
    "alert.monitoring.run.last_run.timestamp": [
      "2025-05-19T19:39:10.733Z"
    ],
    "alert.schedule.interval": [
      "1m"
    ],
    "alert.lastRun.outcome": [
      "succeeded"
    ],
    "alert.consumer": [
      "alerts"
    ],
    "alert.name.keyword": [
      "always firing rule"
    ],
    "alert.apiKey": [
      "d2jZLZXjZK7tOB8WXwencU7Sceyws+mM0DRZCHPwldnB2D41Ndh+KDWyawu0yIY5Ppi72xUfSC1wKDl9udHCgZ1yvfIS/rpIIsDRNFwkw76LFu7FG7j0q/66NhQsMHSZTW7S45tWUam52FhMRiwIXl3UJlSRGgyYTTJqc2OSaZIEygeocRW6E+3t1+1XliqdAg4y/kjn+4kN8Q=="
    ],
    "alert.createdBy": [
      "elastic"
    ],
    "alert.lastRun.alertsCount.new": [
      5
    ],
    "alert.monitoring.run.calculated_metrics.p99": [
      871
    ],
    "alert.monitoring.run.calculated_metrics.p95": [
      871
    ],
    "alert.monitoring.run.history.success": [
      true,
      true,
      true
    ],
    "alert.monitoring.run.calculated_metrics.p50": [
      716
    ],
    "coreMigrationVersion": [
      "8.8.0"
    ],
    "alert.executionStatus.lastDuration": [
      871
    ],
    "alert.lastRun.alertsCount.ignored": [
      0
    ],
    "alert.meta.versionApiKeyLastmodified": [
      "9.1.0"
    ],
    "namespaces": [
      "default"
    ]
  },
  "ignored_field_values": {
    "alert.params": [
      {}
    ]
  }
}

where alert.apiKeyOwner, alert.apiKeyCreatedByUser, alert.alertDelay.active, alert.nextRun, alert.monitoring.run.history.duration, alert.monitoring.run.history.timestamp, alert.apiKey, show up in the fields list but they are not mapped

Copy link
Member

@pmuellr pmuellr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, left a few nit-ish comments.

Still wondering about the unexpected (to me) fields that I commented on in the PR comments (not in these review comments).

"reporting"
],
"requiredPlugins": [
"actions",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

heh, could be a problem if we ever have "reporting" connectors, but we can factor the config bits out of the actions plugin later if we need to :-)

export function setupSavedObjects(savedObjects: SavedObjectsServiceSetup) {
savedObjects.registerType({
name: SCHEDULED_REPORT_SAVED_OBJECT_TYPE,
indexPattern: ALERTING_CASES_SAVED_OBJECT_INDEX,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

heh, not crazy about it, but seems survivable ...

created_by: rawScheduledReport.attributes.createdBy as string | false,
payload: parsedPayload,
meta: rawScheduledReport.attributes.meta,
migration_version: rawScheduledReport.attributes.migrationVersion,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't tell what this is used for, and wonder how it differs from the base SO migration_version field. I guess the report docs have this, as some kind of "follow the SO pattern" since they aren't SO's - I assume we probably need this since the core reporting stuff does, but feels like it may get confusing ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICT, this is always set to 7.14.0 so maybe it's something that's not needed anymore? I was trying to save everything that's currently saved and read for the report generation. I can make it optional in the schema so if we remove it in the future we're not forced to populate it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to make migration version optional in SO schema: b6af770

export const rawNotificationSchema = schema.object({
email: schema.maybe(
schema.object({
to: schema.arrayOf(schema.string(), { minSize: 1 }),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if someone will want "bcc only" or such. I think in the email connector, we make sure the count of all these is at least one. Certainly survivable for now, and "loosening" this in the future shouldn't cause any migration issues, if we want to do that ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to require email in any of the fields: 55c7011

@ymao1
Copy link
Contributor Author

ymao1 commented May 19, 2025

I tried scheduling a report with a bogus dashboard id, and it created it successfully. Seems like it should error. If so, we can defer to later ...

Ah, yea I think the current single report scheduling does not do any checking either. I can create a followup issue to check for both. I think maybe we can do it along with extracting and injecting references since I think both involve parsing out the job params to get the saved object ID to check

@pmuellr
Copy link
Member

pmuellr commented May 19, 2025

Hmm...I wonder if it's something that's done by Discover? Here is a rule SO from Discover:
....

Ah, ya, I see that now ... thanks, I figured I was holding it wrong, but scared there might be something in Core mixing up some fields somehow ...

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/reporting-common 89 90 +1
taskManager 67 71 +4
total +5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
fleet 1.7MB 1.7MB +30.0B

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id before after diff
taskManager 9 8 -1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
reporting 49.5KB 49.5KB +30.0B
Unknown metric groups

API count

id before after diff
@kbn/reporting-common 99 100 +1
taskManager 113 117 +4
total +5

History

cc @ymao1

@ymao1 ymao1 merged commit fd0a43d into elastic:scheduled-reports May 20, 2025
7 checks passed
@ymao1 ymao1 deleted the scheduled-reports-api branch May 29, 2025 13:18
ymao1 added a commit that referenced this pull request Jun 19, 2025
Resolves #216313

## Summary

This is a feature branch that contains the following commits. Each
individual linked PR contains a summary and verification instructions.

* Schedule API - #219771
* Scheduled report task runner -
#219770
* List and disable API - #220922
* Audit logging - #221846
* Send scheduled report emails -
#220539
* Commit to check license -
f5f9d9d
* Update to list API response format -
#224262

---------

Co-authored-by: Ersin Erdal <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Ersin Erdal <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: Alexi Doak <[email protected]>
ymao1 added a commit to ymao1/kibana that referenced this pull request Jun 20, 2025
Resolves elastic#216313

## Summary

This is a feature branch that contains the following commits. Each
individual linked PR contains a summary and verification instructions.

* Schedule API - elastic#219771
* Scheduled report task runner -
elastic#219770
* List and disable API - elastic#220922
* Audit logging - elastic#221846
* Send scheduled report emails -
elastic#220539
* Commit to check license -
elastic@f5f9d9d
* Update to list API response format -
elastic#224262

---------

Co-authored-by: Ersin Erdal <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Ersin Erdal <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: Alexi Doak <[email protected]>
(cherry picked from commit a409627)

# Conflicts:
#	src/platform/packages/private/kbn-reporting/common/routes.ts
#	x-pack/platform/plugins/private/canvas/server/feature.test.ts
#	x-pack/platform/plugins/private/reporting/server/core.ts
#	x-pack/platform/plugins/private/reporting/server/features.ts
#	x-pack/platform/plugins/shared/features/server/__snapshots__/oss_features.test.ts.snap
#	x-pack/platform/test/api_integration/apis/features/features/features.ts
#	x-pack/test_serverless/api_integration/test_suites/chat/platform_security/authorization.ts
#	x-pack/test_serverless/api_integration/test_suites/observability/platform_security/authorization.ts
#	x-pack/test_serverless/api_integration/test_suites/search/platform_security/authorization.ts
#	x-pack/test_serverless/api_integration/test_suites/security/platform_security/authorization.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Feature:Reporting:Framework Reporting issues pertaining to the overall framework Team:ResponseOps Platform ResponseOps team (formerly the Cases and Alerting teams) t// v9.1.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants