PPL Alerting: Execute Monitor and Monitor Stats by toepkerd · Pull Request #1960 · opensearch-project/alerting

toepkerd · 2025-10-27T21:42:13Z

Description

This PR is mostly the Execute Monitor and Monitor Stats logic, as well as various miscellaneous changes that were made after the previous PRs were merged.

The most important file in this PR is PPLSQLMonitorRunner, this contains the monitor execution logic

Rough Survey of PR:

Execute Monitor logic: large portion
Monitor Stats logic: small to moderate portion
Misc changes and gap bridging: small portion
Tests for changes in previous PRs (can be ignored): small to moderate

Related Issues

#1880

Check List

[Y] New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
[Y] Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

… request and response Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

lezzago · 2025-11-12T19:23:47Z

            )

-        return indices
+        // remove any backticks that might have been read in


Do we need to add back in back ticks when sending the requests to PPL?

No, backticks are optional when submitting the request to PPL plugin. They are trimmed here for the call to Get Mappings on these indices when checking if given timestamp field is valid.

The query with trimmed index names isn't actually what gets passed into PPL execute API, the original query retains its backticks for that call and is untouched by this logic.

lezzago · 2025-11-12T21:24:33Z

+/**
+ * RestScheduledJobStatsHandler is handler for getting ScheduledJob Stats for Alerting V2 Scheduled Jobs.
+ */
+class RestScheduledJobStatsV2Handler : BaseRestHandler() {


Cant we reuse the same job stats api and take in a param for v2? These would do the same thing

Will need to get a product confirmation on this question

Product alignment reached. Monitor V2 stats will be eliminated. Existing Monitor Stats will accept an optional version endpoint parameter. If version is not specified, all monitor stats are returned, v1 and v2 together. If version == v1, only v1 monitor stats are returned. If version == v2, only v2 monitor stats are returned.

lezzago

Please remember to add the experimental tags

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

…itor stats v2 api in favor of version parameter in existing monitor stats api Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

Signed-off-by: Dennis Toepker <toepkerd@amazon.com>

Lgtm
Thanks for the changed Dennis.

) * PPL Alerting initial commit Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * removing unused get alerts params and adding serde tests for alertsv2 request and response Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding get alerts rbac IT Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * misc changes Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * removing actioned upon TODO Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding execute monitor request/response serde tests that are currently failing Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding request response serde tests and passing in explicit readers and writers for PPLSQLMonitorRunResult serde Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * finishing out request/response serde tests Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * minor changes in tests Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding debug logs for playground Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding expires null check in alert expiration Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * removing null check as expires duration should never be null Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * cleanup Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * updating timestamp field check to include any date type that includes the string date Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * addressing PR comments Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * misc logs changes Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * changing default max query results to 1000 and replacing get alerts filters with constants Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * putting query index parsing logic in try catch Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding backtick trimming when reading index names in ppl query Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding time measurements and logs Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * refactoring PPL query execution call to use transport service with timeout instead of client Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding monitor run timeout test Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * removing return in timeout catch block to allow for monitors to get updated with last triggered times Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * addressing pr comments Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * addressing pr comments Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * update monitor now ignores seqNo and primaryTerm, and eliminating monitor stats v2 api in favor of version parameter in existing monitor stats api Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding experimental tags Signed-off-by: Dennis Toepker <toepkerd@amazon.com> --------- Signed-off-by: Dennis Toepker <toepkerd@amazon.com> Co-authored-by: Dennis Toepker <toepkerd@amazon.com>

* PPL Alerting initial commit * removing unused get alerts params and adding serde tests for alertsv2 request and response * adding get alerts rbac IT * misc changes * removing actioned upon TODO * adding execute monitor request/response serde tests that are currently failing * adding request response serde tests and passing in explicit readers and writers for PPLSQLMonitorRunResult serde * finishing out request/response serde tests * minor changes in tests * adding debug logs for playground * adding expires null check in alert expiration * removing null check as expires duration should never be null * cleanup * updating timestamp field check to include any date type that includes the string date * addressing PR comments * misc logs changes * changing default max query results to 1000 and replacing get alerts filters with constants * putting query index parsing logic in try catch * adding backtick trimming when reading index names in ppl query * adding time measurements and logs * refactoring PPL query execution call to use transport service with timeout instead of client * adding monitor run timeout test * removing return in timeout catch block to allow for monitors to get updated with last triggered times * addressing pr comments * addressing pr comments * update monitor now ignores seqNo and primaryTerm, and eliminating monitor stats v2 api in favor of version parameter in existing monitor stats api * adding experimental tags --------- Signed-off-by: Dennis Toepker <toepkerd@amazon.com> Co-authored-by: Dennis Toepker <toepkerd@amazon.com>

* PPL Alerting initial commit Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * removing unused get alerts params and adding serde tests for alertsv2 request and response Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding get alerts rbac IT Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * misc changes Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * removing actioned upon TODO Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding execute monitor request/response serde tests that are currently failing Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding request response serde tests and passing in explicit readers and writers for PPLSQLMonitorRunResult serde Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * finishing out request/response serde tests Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * minor changes in tests Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding debug logs for playground Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding expires null check in alert expiration Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * removing null check as expires duration should never be null Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * cleanup Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * updating timestamp field check to include any date type that includes the string date Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * addressing PR comments Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * misc logs changes Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * changing default max query results to 1000 and replacing get alerts filters with constants Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * putting query index parsing logic in try catch Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding backtick trimming when reading index names in ppl query Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding time measurements and logs Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * refactoring PPL query execution call to use transport service with timeout instead of client Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding monitor run timeout test Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * removing return in timeout catch block to allow for monitors to get updated with last triggered times Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * addressing pr comments Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * addressing pr comments Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * update monitor now ignores seqNo and primaryTerm, and eliminating monitor stats v2 api in favor of version parameter in existing monitor stats api Signed-off-by: Dennis Toepker <toepkerd@amazon.com> * adding experimental tags Signed-off-by: Dennis Toepker <toepkerd@amazon.com> --------- Signed-off-by: Dennis Toepker <toepkerd@amazon.com> Co-authored-by: Dennis Toepker <toepkerd@amazon.com> (cherry picked from commit 74aed08) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…roject#1960)" This reverts commit 74aed08. Signed-off-by: Thomas Hurney <hurneyt@amazon.com>

* [AUTO] Add release notes for 3.5.0 (#2014) * Add release notes for 3.5.0 Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> * Add release notes for 3.5.0 Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> --------- Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Execute Monitor and Monitor Stats (#1960)" This reverts commit 74aed08. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Get Alerts and Alert Lifecycle (#1972)" This reverts commit d9614ba Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Delete Monitor, More V1/V2 Separation (#1968)" This reverts commit fd151de. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Get and Search Monitors (#1966)" This reverts commit 819599f. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Create and Update Monitor V2 (#1961)" This reverts commit c911cfa Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Models (#1955)" This reverts commit 497dea8. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Fixed test role creation. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> --------- Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: Thomas Hurney <hurneyt@amazon.com> Co-authored-by: opensearch-ci <83309141+opensearch-ci-bot@users.noreply.github.com>

* [AUTO] Add release notes for 3.5.0 (#2014) * Add release notes for 3.5.0 Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> * Add release notes for 3.5.0 Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> --------- Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Execute Monitor and Monitor Stats (#1960)" This reverts commit 74aed08. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Get Alerts and Alert Lifecycle (#1972)" This reverts commit d9614ba Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Delete Monitor, More V1/V2 Separation (#1968)" This reverts commit fd151de. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Get and Search Monitors (#1966)" This reverts commit 819599f. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Create and Update Monitor V2 (#1961)" This reverts commit c911cfa Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Models (#1955)" This reverts commit 497dea8. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Fixed test role creation. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> --------- Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: Thomas Hurney <hurneyt@amazon.com> Co-authored-by: opensearch-ci <83309141+opensearch-ci-bot@users.noreply.github.com> (cherry picked from commit c39caca) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…roject#1960) (opensearch-project#1988)" This reverts commit 90dddfb. Signed-off-by: Thomas Hurney <hurneyt@amazon.com>

…roject#1960)" This reverts commit 74aed08. Signed-off-by: Thomas Hurney <hurneyt@amazon.com>

* Revert "PPL Alerting: Execute Monitor and Monitor Stats (#1960) (#1988)" This reverts commit 90dddfb. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Get Alerts and Alert Lifecycle (#1972) (#1985)" This reverts commit 0677625. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Delete Monitor, More V1/V2 Separation (#1968) (#1984)" This reverts commit 154fcb2. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Get and Search Monitors (#1966) (#1983)" This reverts commit 852eeee. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Create and Update Monitor V2 (#1961) (#1982)" This reverts commit f7b9b50. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Models (#1955) (#1981)" This reverts commit 844954a. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> --------- Signed-off-by: Thomas Hurney <hurneyt@amazon.com>

* Update logback dependencies to version 1.5.19 (#1996) (#2002) (cherry picked from commit 72f9fcf) Signed-off-by: Daniel Widdis <widdis@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Include alerting spi as part of the mavenlocal (#2003) (#2004) (cherry picked from commit 4550b44) Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Add release notes for 3.4.0 (#1999) Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Publish snapshots to Staging (#2005) (#2006) (cherry picked from commit 8a6bad3) Signed-off-by: Divya Madala <divyaasm@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Execute Monitor and Monitor Stats (#1960)" This reverts commit 74aed08. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Get Alerts and Alert Lifecycle (#1972)" This reverts commit d9614ba Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Delete Monitor, More V1/V2 Separation (#1968)" This reverts commit fd151de. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Get and Search Monitors (#1966)" This reverts commit 819599f. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Create and Update Monitor V2 (#1961)" This reverts commit c911cfa Signed-off-by: Thomas Hurney <hurneyt@amazon.com> * Revert "PPL Alerting: Models (#1955)" This reverts commit 497dea8. Signed-off-by: Thomas Hurney <hurneyt@amazon.com> --------- Signed-off-by: Daniel Widdis <widdis@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Thomas Hurney <hurneyt@amazon.com> Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: Divya Madala <divyaasm@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci <83309141+opensearch-ci-bot@users.noreply.github.com>

* [AUTO] Add release notes for 3.5.0 (#2014) * Add release notes for 3.5.0 * Add release notes for 3.5.0 --------- * Revert "PPL Alerting: Execute Monitor and Monitor Stats (#1960)" This reverts commit 74aed08. * Revert "PPL Alerting: Get Alerts and Alert Lifecycle (#1972)" This reverts commit d9614ba * Revert "PPL Alerting: Delete Monitor, More V1/V2 Separation (#1968)" This reverts commit fd151de. * Revert "PPL Alerting: Get and Search Monitors (#1966)" This reverts commit 819599f. * Revert "PPL Alerting: Create and Update Monitor V2 (#1961)" This reverts commit c911cfa * Revert "PPL Alerting: Models (#1955)" This reverts commit 497dea8. * Fixed test role creation. --------- (cherry picked from commit c39caca) Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: Thomas Hurney <hurneyt@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci <83309141+opensearch-ci-bot@users.noreply.github.com>

toepkerd force-pushed the ppl-main branch from f313e57 to 8f7a1d7 Compare October 30, 2025 15:40