-
Notifications
You must be signed in to change notification settings - Fork 8.5k
[Response Ops][Reporting] Scheduled Reports #221028
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Towards #216313 > [!Note] > This PR will be merged into a feature branch - `elastic:scheduled-reports` > Once this is merged, I will open a draft PR from the feature branch into `main` ## Summary ### Schedule API - Added internal API `/internal/reporting/schedule/{exportTypeId}` ``` POST /internal/reporting/schedule/{exportTypeId} { "jobParams": <same jobParams format as existing generate single report API>, "schedule": { "rrule": <modified Task Manager rRule schema from #217728> } "notification": { "email": { "to": <optional list of email addresses>, "cc": <optional list of email addresses>, "bcc": <optional list of email addresses> } } ``` - Scheduling a report creates a new `scheduled_report` type saved object in the `.kibana-alerting-cases` index. - Uses the same role privileges as the current generate report API, so user must have the reporting subfeature privilege for one of the Analytics feature privileges (Discover/Dashboard/Visualize) ## Verify 1. Run ES and Kibana with examples and install the sample web logs data set. 2. In the DEV console, call the schedule API using ``` POST kbn:/internal/reporting/schedule/printablePdfV2 { "schedule": { "rrule": { "freq": 3, "interval": 3, "byhour": [12], "byminute": [0] } }, "jobParams": "(browserTimezone:America/New_York,layout:(dimensions:(height:2220,width:1364),id:preserve_layout),locatorParams:!((id:DASHBOARD_APP_LOCATOR,params:(dashboardId:edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b,preserveSavedFilters:!t,timeRange:(from:now-7d/d,to:now),useHash:!f,viewMode:view))),objectType:dashboard,title:'[Logs] Web Traffic',version:'9.1.0')" } ``` 3. Verify a saved object has been created using `GET .kibana_alerting_cases/_search?q=type:scheduled_report` 4. Try different requests with different schedules and notifications. It should respect the email domain allowlist if configured and reject invalid rrule schedules and email addresses. --------- Co-authored-by: Ersin Erdal <[email protected]> Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Ersin Erdal <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
…ure branch) (#219770) Towards #216313 > [!Note] > This PR will be merged into a [feature branch](#221028) ## Summary This PR adds the `scheduled_report` task, which generates reports on a recurring cadence. * maxConcurrency of this task is 1 so only one can run at a time * this task shares concurrency with the existing `report:execute` task, so only one of either task type can run at a time * this task requires an API key to run so it will only run when ES security and API keys are enabled and a permanent encryption key for Kibana is set * when the task runs, it creates a new `ReportSource` document in the `.kibana-reporting` index that stores the final base64 encoded generated output. This `ReportSource` matches the data model of existing reports with the addition of a `scheduled_report_id` field which links to the scheduled report saved object ID. ## Verify * Use the dev console to schedule some reports. * Verify the the reports get generated at the expected cadence * Verify that the correct time range is used for each scheduled report Note, the UI has not been updated to differentiate between scheduled and single run reports. They will look the same from in the report listing, except the scheduled report will have a timestamp appended to the report name. In the cloud deployment for this PR, we have an example scheduled report that is running every day at 4:45 EDT: https://kibana-pr-219770.kb.us-west2.gcp.elastic-cloud.com/app/management/insightsAndAlerting/reporting --------- Co-authored-by: Ersin Erdal <[email protected]> Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Ersin Erdal <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
…220922) Towards #216313 > [!Note] > This PR will be merged into a [feature branch](#221028) ## Summary Adds internal APIs for listing and disabling scheduled reports. For this initial iteration, we are not adding an `enable` API. This adds a `Manage Scheduled Reports` feature privilege that, if the user has it, allows them to view and disable the scheduled reports of any user in the current space. ## To Verify 1. Create a user (`user1`) with the ability to generate reports but not manage reports. Use the schedule API to schedule some reports as this user. 2. Create another user (`user2`) with the ability to manage reports. Use the schedule API to schedule some reports as this user. 3. As `user1`, use the Dev Console to list scheduled reports. You should only see those created by `user1`. `user1` should also be able to disable their own reports. 4. As `user2`, use the Dev Console to list scheduled reports. You should see reports from both `user1` and `user2`. `user2` should be able to disable a report created by `user1`. --------- Co-authored-by: Ersin Erdal <[email protected]> Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Ersin Erdal <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
tsullivan
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thank you so much!
I'm really looking forward to this new feature!
⏳ Build in-progress
History
cc @ymao1 |
elena-shostak
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Left a couple of nits
| // round up from ms to the nearest second | ||
| const queueTimeout = | ||
| Math.ceil(numberToDuration(this.opts.config.queue.timeout).asSeconds()) + 's'; | ||
| const maxConcurrency = this.opts.config.queue.pollEnabled ? 1 : 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: we can also add function similar to getMaxAttempts
protected getMaxConcurrency() {
return this.opts.config.queue.pollEnabled ? 1 : 0;
}since both RunScheduledReportTask and RunSingleReportTask extend RunReportTask, we can even move it there. Logic for getting config options seems to be the same
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated in 48829a9
| if (req.body) { | ||
| const { jobParams: jobParamsPayload } = req.body; | ||
| jobParamsRison = jobParamsPayload ? jobParamsPayload : null; | ||
| } else if (req.query?.jobParams) { | ||
| const { jobParams: queryJobParams } = req.query; | ||
| if (queryJobParams) { | ||
| jobParamsRison = queryJobParams; | ||
| } else { | ||
| jobParamsRison = null; | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| if (req.body) { | |
| const { jobParams: jobParamsPayload } = req.body; | |
| jobParamsRison = jobParamsPayload ? jobParamsPayload : null; | |
| } else if (req.query?.jobParams) { | |
| const { jobParams: queryJobParams } = req.query; | |
| if (queryJobParams) { | |
| jobParamsRison = queryJobParams; | |
| } else { | |
| jobParamsRison = null; | |
| } | |
| const jobParamsRison = | |
| req.body?.jobParams ?? req.query?.jobParams ?? null; |
| auditLogger.log( | ||
| scheduledReportAuditEvent({ | ||
| action: ScheduledReportAuditAction.LIST, | ||
| savedObject: { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we have added support for the name field in audit events in #206644
would you mind to add it here and other corresponding audit logs?
SavedObjectsUtils.getName(registry.getNameAttribute(type), savedObject)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated in 48829a9. Thanks for catching! I had the name field in some audit events but not others for some reason 🤦♀️
|
update: I wasn't running the branch with the latest commits 🤦🏻♂️ |
|
@pmuellr I set up a scheduled report on the cloud deployment to email us with a dashboard every hour on the Also, could you share your kibana config? |
Sorry, I was running old code. I can see it working now! \o/ And yes, I'm getting spammed by the scheduled report you set up :-) |
pmuellr
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, tested with multiple users with different management feature setting, and with the notify. Quite nice! Can't wait to get a UX on here!
| export interface AttachmentEmail extends PlainTextEmail { | ||
| export interface AttachmentEmail extends Omit<PlainTextEmail, 'to'> { | ||
| attachments: Attachment[]; | ||
| to?: string[]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feels like this is something we shouldn't have had to do. The notifications service seems like it could use some polish - it should support at least bcc and cc as well as optional to, and probably attachments, not sure about spaceId :-). Can we open an issue to get this updated?
js-jankisalvi
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Verified locally, works as expected 👍
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]Public APIs missing comments
Async chunks
Public APIs missing exports
Page load bundle
Unknown metric groupsAPI count
ESLint disabled line counts
References to deprecated APIs
Total ESLint disabled count
History
cc @ymao1 |
|
Starting backport for target branches: 8.19 https://github.com/elastic/kibana/actions/runs/15758885164 |
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
Resolves elastic#216313 ## Summary This is a feature branch that contains the following commits. Each individual linked PR contains a summary and verification instructions. * Schedule API - elastic#219771 * Scheduled report task runner - elastic#219770 * List and disable API - elastic#220922 * Audit logging - elastic#221846 * Send scheduled report emails - elastic#220539 * Commit to check license - elastic@f5f9d9d * Update to list API response format - elastic#224262 --------- Co-authored-by: Ersin Erdal <[email protected]> Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Ersin Erdal <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Alexi Doak <[email protected]> (cherry picked from commit a409627) # Conflicts: # src/platform/packages/private/kbn-reporting/common/routes.ts # x-pack/platform/plugins/private/canvas/server/feature.test.ts # x-pack/platform/plugins/private/reporting/server/core.ts # x-pack/platform/plugins/private/reporting/server/features.ts # x-pack/platform/plugins/shared/features/server/__snapshots__/oss_features.test.ts.snap # x-pack/platform/test/api_integration/apis/features/features/features.ts # x-pack/test_serverless/api_integration/test_suites/chat/platform_security/authorization.ts # x-pack/test_serverless/api_integration/test_suites/observability/platform_security/authorization.ts # x-pack/test_serverless/api_integration/test_suites/search/platform_security/authorization.ts # x-pack/test_serverless/api_integration/test_suites/security/platform_security/authorization.ts
## Summary > [!IMPORTANT] > This PR is targeting the `scheduled-reports-ui` feature branch, where the backend changes from the `scheduled-reports` branch are temporarily integrated while waiting for #221028 to be merged (see the squashed `[TMP] ...` commit message). Implements the flyout UI for the creation and read-only viewing of scheduled reports (exports). <img height="500" alt="image" src="https://github.com/user-attachments/assets/6bbd274b-1bbb-481f-ac40-4d7131be6d85" /> <details> <summary> ## Known issues </summary> - Console error due to `compressed` attribute wrongly forwarded by form-hook-lib to DOM element (this is likely a form lib issue): <img width="916" alt="image" src="https://github.com/user-attachments/assets/09d20ba9-8781-46d6-bcfa-862d8a4cbf90" /> - Email validation errors accumulate instead of replacing the previous one (again looks like a fom lib issue): https://github.com/user-attachments/assets/f2dc7a46-a3a9-465d-b8a1-3187b200f9b9 </details> <details> <summary> ## Screenshots </summary> Health API error: <img height="500" alt="Screenshot 2025-05-31 at 10 48 40" src="https://github.com/user-attachments/assets/dd069597-971c-489f-9c07-eb5edfd7bede" /> Health API loading state: <img height="500" alt="Screenshot 2025-05-31 at 10 49 04" src="https://github.com/user-attachments/assets/27d95bf3-bf7d-42c7-9a40-2826f38aa837" /> Health API success with some missing prerequisites: <img width="449" alt="Screenshot 2025-06-17 at 16 59 57" src="https://github.com/user-attachments/assets/c44afa97-70ff-4618-8b73-41b816514459" /> Form validation: <img height="500" alt="image" src="https://github.com/user-attachments/assets/a8d4cae1-2819-4f71-a911-9300a6cf81f8" /> Success toast: <img width="480" alt="image" src="https://github.com/user-attachments/assets/a87c3af5-dbb0-40e8-915a-fc9d7e1d97f2" /> Failure toast: <img width="518" alt="image" src="https://github.com/user-attachments/assets/908f9dea-b5cb-4da9-b4a5-76e313837f18" /> Print format toggle: <img width="502" alt="image" src="https://github.com/user-attachments/assets/602f3ab9-07ef-4689-a305-dc1b2b5495cd" /> Missing notifications email connector callout: <img width="499" alt="image" src="https://github.com/user-attachments/assets/fe4997a5-75e6-4450-85e5-7d853049e085" /> </details> ## References Closes #216321 Stacked on #220535 --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Eyo O. Eyo <[email protected]>
…224354) ## Summary Resolves #216322 > [!IMPORTANT] > This PR is targeting the `scheduled-reports-ui` feature branch, where the backend changes from the `scheduled-reports` branch are temporarily integrated while waiting for #221028 to be merged (see the squashed `[TMP] ...` commit message). > This PR has few commits from #222135 to support view schedule config table action. Please ignore code changes in folders if you are already reviewing other PR. `src/platform/packages/private/kbn-reporting/public/share/share_context_menu`, `src/platform/packages/shared/response-ops/recurring-schedule-form`, `x-pack/platform/plugins/private/reporting/public/management/schemas`, `x-pack/platform/plugins/private/reporting/public/management/validators` and `xpack/platform/plugins/shared/alerting/public/pages/maintenance_windows/components/create_maintenance_windows_form.tsx`. This PR adds new tabs `Exports` and `Schedules` in Reporting section. `Exports` tab shows the list of all reports. Allows to open dashboard, download report and view report information. `Schedules` tab shows list of scheduled reports. Allows to disable schedule. ### Checklist Check the PR satisfies following conditions. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### What to test - Verify tabs - Verify exports table shows list of all reports - Verify schedules table shows list of scheduled report - Verify you can disable scheduled report - Verify pagination in both tables --------- Co-authored-by: kibanamachine <[email protected]>
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
# Backport This will backport the following commits from `main` to `8.19`: - [[Response Ops][Reporting] Scheduled Reports (#221028)](#221028) <!--- Backport version: 10.0.1 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Ying Mao","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-06-19T13:20:18Z","message":"[Response Ops][Reporting] Scheduled Reports (#221028)\n\nResolves https://github.com/elastic/kibana/issues/216313\n\n## Summary\n\nThis is a feature branch that contains the following commits. Each\nindividual linked PR contains a summary and verification instructions.\n\n* Schedule API - https://github.com/elastic/kibana/pull/219771\n* Scheduled report task runner -\nhttps://github.com//pull/219770\n* List and disable API - https://github.com/elastic/kibana/pull/220922\n* Audit logging - https://github.com/elastic/kibana/pull/221846\n* Send scheduled report emails -\nhttps://github.com//pull/220539\n* Commit to check license -\nhttps://github.com//pull/221028/commits/f5f9d9daedcd18447b6a02335a53631a44413788\n* Update to list API response format -\nhttps://github.com//pull/224262\n\n---------\n\nCo-authored-by: Ersin Erdal <[email protected]>\nCo-authored-by: kibanamachine <[email protected]>\nCo-authored-by: Ersin Erdal <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>\nCo-authored-by: Alexi Doak <[email protected]>","sha":"a409627765dfaf3d588c35a0d510b8d1857cd266","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:ResponseOps","release_note:feature","ci:cloud-deploy","ci:cloud-persist-deployment","Feature:Reporting:Framework","backport:version","v9.1.0","v8.19.0"],"title":"[Response Ops][Reporting] Scheduled Reports","number":221028,"url":"https://github.com/elastic/kibana/pull/221028","mergeCommit":{"message":"[Response Ops][Reporting] Scheduled Reports (#221028)\n\nResolves https://github.com/elastic/kibana/issues/216313\n\n## Summary\n\nThis is a feature branch that contains the following commits. Each\nindividual linked PR contains a summary and verification instructions.\n\n* Schedule API - https://github.com/elastic/kibana/pull/219771\n* Scheduled report task runner -\nhttps://github.com//pull/219770\n* List and disable API - https://github.com/elastic/kibana/pull/220922\n* Audit logging - https://github.com/elastic/kibana/pull/221846\n* Send scheduled report emails -\nhttps://github.com//pull/220539\n* Commit to check license -\nhttps://github.com//pull/221028/commits/f5f9d9daedcd18447b6a02335a53631a44413788\n* Update to list API response format -\nhttps://github.com//pull/224262\n\n---------\n\nCo-authored-by: Ersin Erdal <[email protected]>\nCo-authored-by: kibanamachine <[email protected]>\nCo-authored-by: Ersin Erdal <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>\nCo-authored-by: Alexi Doak <[email protected]>","sha":"a409627765dfaf3d588c35a0d510b8d1857cd266"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/221028","number":221028,"mergeCommit":{"message":"[Response Ops][Reporting] Scheduled Reports (#221028)\n\nResolves https://github.com/elastic/kibana/issues/216313\n\n## Summary\n\nThis is a feature branch that contains the following commits. Each\nindividual linked PR contains a summary and verification instructions.\n\n* Schedule API - https://github.com/elastic/kibana/pull/219771\n* Scheduled report task runner -\nhttps://github.com//pull/219770\n* List and disable API - https://github.com/elastic/kibana/pull/220922\n* Audit logging - https://github.com/elastic/kibana/pull/221846\n* Send scheduled report emails -\nhttps://github.com//pull/220539\n* Commit to check license -\nhttps://github.com//pull/221028/commits/f5f9d9daedcd18447b6a02335a53631a44413788\n* Update to list API response format -\nhttps://github.com//pull/224262\n\n---------\n\nCo-authored-by: Ersin Erdal <[email protected]>\nCo-authored-by: kibanamachine <[email protected]>\nCo-authored-by: Ersin Erdal <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>\nCo-authored-by: Alexi Doak <[email protected]>","sha":"a409627765dfaf3d588c35a0d510b8d1857cd266"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <[email protected]>
Resolves #216313
Summary
This is a feature branch that contains the following commits. Each individual linked PR contains a summary and verification instructions.