[Response Ops][Alerting] Update framework alerts client to write flattened alerts docs by ymao1 · Pull Request #167439 · elastic/kibana

ymao1 · 2023-09-27T18:28:27Z

Summary

The rule registry has traditionally written out AAD docs with flattened keys, like

{
  "kibana.alert.rule.name": "test"
}

The framework alerts client has been writing out AAD docs as objects, like

{
  "kibana": {
    "alert": {
      "rule": {
        "name": "test"
      }
    }
  }
}

We've identified a few places where we're updating the docs where having this divergence makes things more difficult, so this is to switch the framework to writing flattened alert docs before onboarding more rule types.

This PR is targeted for 8.11, which is also when we onboarded the index threshold rule type to FAAD. The only other rule type using FAAD to write docs is ES query, which landed in 8.10 so there will be a followup issue to handle the case of updating unflattened ES query AAD docs from 8.10

To Verify

ES Query and Index Threshold AaD

Create these rules that trigger alerts and verify that their AaD docs are written out as flattened. For the ES Query rule type, select a Metrics/Logs consumer and verify that they appear on the O11y alerts table.

ML alerts

ML alerts added in #166349 looked like:

Unflattened

{
	"kibana": {
		"alert": {
			"url": "/app/ml/explorer/?_g=(ml%3A(jobIds%3A!(rt-anomaly-mean-value))%2Ctime%3A(from%3A'2023-09-28T14%3A57%3A00.000Z'%2Cmode%3Aabsolute%2Cto%3A'2023-09-28T15%3A17%3A00.000Z'))&_a=(explorer%3A(mlExplorerFilter%3A(filterActive%3A!t%2CfilteredFields%3A!(key%2Cthird-key)%2CinfluencersFilterQuery%3A(bool%3A(minimum_should_match%3A1%2Cshould%3A!((match_phrase%3A(key%3Athird-key)))))%2CqueryString%3A'key%3A%22third-key%22')%2CmlExplorerSwimlane%3A()))",
			"reason": "Alerts are raised based on real-time scores. Remember that scores may be adjusted over time as data continues to be analyzed.",
			"job_id": "rt-anomaly-mean-value",
			"anomaly_score": 73.63508175828011,
			"is_interim": false,
			"anomaly_timestamp": 1695913620000,
			"top_records": [{
				"job_id": "rt-anomaly-mean-value",
				"record_score": 73.63516446528412,
				"initial_record_score": 73.63516446528412,
				"detector_index": 0,
				"is_interim": false,
				"timestamp": 1695913620000,
				"partition_field_name": "key",
				"partition_field_value": "third-key",
				"function": "mean",
				"actual": [
					3
				],
				"typical": [
					4.187715468532429
				]
			}],
			"top_influencers": [{
				"job_id": "rt-anomaly-mean-value",
				"influencer_field_name": "key",
				"influencer_field_value": "third-key",
				"influencer_score": 73.63508175828011,
				"initial_influencer_score": 73.63508175828011,
				"is_interim": false,
				"timestamp": 1695913620000
			}],
			"action_group": "anomaly_score_match",
			"flapping": false,
			"flapping_history": [
				true,
				false,
				false,
				false
			],
			"instance": {
				"id": "rt-anomaly-mean-value"
			},
			"maintenance_window_ids": [],
			"rule": {
				"category": "Anomaly detection alert",
				"consumer": "alerts",
				"execution": {
					"uuid": "e9e681d4-c8e4-43eb-82e5-a58bdf7ffe12"
				},
				"name": "rt-ad-alert-influencer",
				"parameters": {
					"severity": 5,
					"resultType": "influencer",
					"includeInterim": false,
					"jobSelection": {
						"jobIds": [
							"rt-anomaly-mean-value"
						],
						"groupIds": []
					},
					"lookbackInterval": null,
					"topNBuckets": null
				},
				"producer": "ml",
				"revision": 0,
				"rule_type_id": "xpack.ml.anomaly_detection_alert",
				"tags": [],
				"uuid": "9e1d6bc0-5e10-11ee-8416-3bf48cca0922"
			},
			"status": "active",
			"uuid": "c9c1f075-9985-4c55-8ff8-22349cb30269",
			"workflow_status": "open",
			"duration": {
				"us": "99021000000"
			},
			"start": "2023-09-28T15:07:12.868Z",
			"time_range": {
				"gte": "2023-09-28T15:07:12.868Z"
			}
		},
		"space_ids": [
			"default"
		],
		"version": "8.11.0"
	},
	"@timestamp": "2023-09-28T15:08:51.889Z",
	"event": {
		"action": "active",
		"kind": "signal"
	},
	"tags": []
}

Now they look like:

Flattened

{
	"kibana.alert.url": "/app/ml/explorer/?_g=(ml%3A(jobIds%3A!(rt-anomaly-mean-value))%2Ctime%3A(from%3A'2023-09-28T15%3A03%3A00.000Z'%2Cmode%3Aabsolute%2Cto%3A'2023-09-28T15%3A23%3A00.000Z'))&_a=(explorer%3A(mlExplorerFilter%3A(filterActive%3A!t%2CfilteredFields%3A!(key%2Cthird-key)%2CinfluencersFilterQuery%3A(bool%3A(minimum_should_match%3A1%2Cshould%3A!((match_phrase%3A(key%3Athird-key)))))%2CqueryString%3A'key%3A%22third-key%22')%2CmlExplorerSwimlane%3A()))",
	"kibana.alert.reason": "Alerts are raised based on real-time scores. Remember that scores may be adjusted over time as data continues to be analyzed.",
	"kibana.alert.job_id": "rt-anomaly-mean-value",
	"kibana.alert.anomaly_score": 72.75515452061356,
	"kibana.alert.is_interim": false,
	"kibana.alert.anomaly_timestamp": 1695913980000,
	"kibana.alert.top_records": [{
		"job_id": "rt-anomaly-mean-value",
		"record_score": 72.75515452061356,
		"initial_record_score": 72.75515452061356,
		"detector_index": 0,
		"is_interim": false,
		"timestamp": 1695913980000,
		"partition_field_name": "key",
		"partition_field_value": "third-key",
		"function": "mean",
		"actual": [
			0.5
		],
		"typical": [
			4.138745343296527
		]
	}],
	"kibana.alert.top_influencers": [{
		"job_id": "rt-anomaly-mean-value",
		"influencer_field_name": "key",
		"influencer_field_value": "third-key",
		"influencer_score": 72.75515452061356,
		"initial_influencer_score": 72.75515452061356,
		"is_interim": false,
		"timestamp": 1695913980000
	}],
	"kibana.alert.rule.category": "Anomaly detection alert",
	"kibana.alert.rule.consumer": "alerts",
	"kibana.alert.rule.execution.uuid": "17fef3d3-d595-4362-837e-b2a73650169e",
	"kibana.alert.rule.name": "rt-ad-alert-influencer",
	"kibana.alert.rule.parameters": {
		"severity": 5,
		"resultType": "influencer",
		"includeInterim": false,
		"jobSelection": {
			"jobIds": [
				"rt-anomaly-mean-value"
			],
			"groupIds": []
		},
		"lookbackInterval": null,
		"topNBuckets": null
	},
	"kibana.alert.rule.producer": "ml",
	"kibana.alert.rule.revision": 0,
	"kibana.alert.rule.rule_type_id": "xpack.ml.anomaly_detection_alert",
	"kibana.alert.rule.tags": [],
	"kibana.alert.rule.uuid": "757c7610-5e11-11ee-8bc6-a95c3ced4757",
	"kibana.space_ids": [
		"default"
	],
	"@timestamp": "2023-09-28T15:14:52.057Z",
	"event.action": "active",
	"event.kind": "signal",
	"kibana.alert.action_group": "anomaly_score_match",
	"kibana.alert.flapping": false,
	"kibana.alert.flapping_history": [
		true,
		false,
		false,
		false
	],
	"kibana.alert.instance.id": "rt-anomaly-mean-value",
	"kibana.alert.maintenance_window_ids": [],
	"kibana.alert.status": "active",
	"kibana.alert.uuid": "ac1f0d7c-461b-4fc6-b4c3-04416ac876d3",
	"kibana.alert.workflow_status": "open",
	"kibana.alert.duration.us": "99028000000",
	"kibana.alert.start": "2023-09-28T15:13:13.028Z",
	"kibana.alert.time_range": {
		"gte": "2023-09-28T15:13:13.028Z"
	},
	"kibana.version": "8.11.0",
	"tags": []
}

ymao1 · 2023-09-28T21:02:33Z

@elasticmachine merge upstream

…n-aad

ymao1 · 2023-09-29T12:50:43Z

@elasticmachine run elasticsearch-ci/docs

mikecote

Changes LGTM! Left some questions and nitpicks but was able to test locally and see everything populate as flattened.

mikecote · 2023-09-29T12:57:54Z

packages/kbn-alerts-as-data-utils/src/schemas/create_schema_from_field_map.ts

  useEcs = false,
  useLegacyAlerts = false,
-  flattened = false,
+  flattened = true,


question: is flattened always true now? If so, at a later time / PR, sounds like we may be able to clean this up.

Yea, I think we could clean it up in the future. As long as we don't switch back to unflattened 🙈

x-pack/plugins/alerting/server/alert/alert.test.ts

x-pack/plugins/alerting/server/alerts_client/alerts_client.ts

x-pack/plugins/alerting/server/alerts_client/lib/build_ongoing_alert.ts

x-pack/plugins/alerting/server/alerts_client/lib/build_ongoing_alert.test.ts

mikecote · 2023-09-29T16:20:58Z

x-pack/plugins/alerting/server/alerts_client/lib/build_new_alert.ts

-            ...((cleanedPayload?.tags as string[]) ?? []),
-            ...(rule.kibana?.alert.rule.tags ?? []),
-          ])
+        ...rule,


question: Is copying the rule new? as in we should of done that all along?

I think we have been doing that

mikecote · 2023-09-29T16:21:52Z

x-pack/plugins/stack_alerts/server/rule_types/es_query/executor.ts

        [ALERT_EVALUATION_CONDITIONS]: actionContext.conditions,
-        [ALERT_EVALUATION_VALUE]: actionContext.value,
-      }),
+        [ALERT_EVALUATION_VALUE]: `${actionContext.value}`,


question: what caused string to be the required type for es_query and index_threshold? Is it a float which works mainly with strings to support long values?

This is because when the field map was defined for the evaluation value, it specified keyword mapping type and in our schema generation script, it converts keyword to string schema. I actually think the mapping should have been numerical, it was an oversight :(

However, I think O11y rules are using ALERT_EVALUATION_VALUES (plural not singular) so we should probably update to using that as well and stop populating the singular version.

kibana-ci · 2023-09-29T17:05:17Z

💔 Build Failed

Failed CI Steps

Test Failures

[job] [logs] x-pack/test/accessibility/config.ts / Search Profiler Editor Accessibility "before all" hook for "input the JSON in the aceeditor"

Metrics [docs]

✅ unchanged

History

💛 Build #163787 was flaky e8925c7
💛 Build #163747 was flaky 5752c3f3b6a6d7778b0db5c7d001b7adae7780f1
💔 Build #163682 failed 1d21183
💔 Build #163534 failed 02764e2bf383e491cbf0ae42b99d4dbbba9d1292

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ymao1

@timestamp

…tened alerts docs (#167691) Resolves #166946 ## PRs to this feature branch * #167439 * #167583 ## Summary The rule registry has traditionally written out AAD docs with flattened keys, like ``` { "kibana.alert.rule.name": "test" } ``` The framework alerts client has been writing out AAD docs as objects, like ``` { "kibana": { "alert": { "rule": { "name": "test" } } } } ``` We've identified a few places where we're updating the docs where having this divergence makes things more difficult, so this is to switch the framework to writing flattened alert docs before onboarding more rule types. This PR is targeted for 8.11, which is also when we onboarded the index threshold rule type and the ML anomaly detection rule type to FAAD. For the ES query rule, which started writing unflattened AaD docs in 8.10, this PR adds special handling to ensure that those unflattened docs are correctly updated with flattened fields. ## To Verify ### ES Query and Index Threshold AaD Create these rules that trigger alerts and verify that their AaD docs are written out as flattened. For the ES Query rule type, select a Metrics/Logs consumer and verify that they appear on the O11y alerts table. ### ML alerts ML alerts added in #166349 looked like: <details> <summary>Unflattened</summary> ``` { "kibana": { "alert": { "url": "/app/ml/explorer/?_g=(ml%3A(jobIds%3A!(rt-anomaly-mean-value))%2Ctime%3A(from%3A'2023-09-28T14%3A57%3A00.000Z'%2Cmode%3Aabsolute%2Cto%3A'2023-09-28T15%3A17%3A00.000Z'))&_a=(explorer%3A(mlExplorerFilter%3A(filterActive%3A!t%2CfilteredFields%3A!(key%2Cthird-key)%2CinfluencersFilterQuery%3A(bool%3A(minimum_should_match%3A1%2Cshould%3A!((match_phrase%3A(key%3Athird-key)))))%2CqueryString%3A'key%3A%22third-key%22')%2CmlExplorerSwimlane%3A()))", "reason": "Alerts are raised based on real-time scores. Remember that scores may be adjusted over time as data continues to be analyzed.", "job_id": "rt-anomaly-mean-value", "anomaly_score": 73.63508175828011, "is_interim": false, "anomaly_timestamp": 1695913620000, "top_records": [{ "job_id": "rt-anomaly-mean-value", "record_score": 73.63516446528412, "initial_record_score": 73.63516446528412, "detector_index": 0, "is_interim": false, "timestamp": 1695913620000, "partition_field_name": "key", "partition_field_value": "third-key", "function": "mean", "actual": [ 3 ], "typical": [ 4.187715468532429 ] }], "top_influencers": [{ "job_id": "rt-anomaly-mean-value", "influencer_field_name": "key", "influencer_field_value": "third-key", "influencer_score": 73.63508175828011, "initial_influencer_score": 73.63508175828011, "is_interim": false, "timestamp": 1695913620000 }], "action_group": "anomaly_score_match", "flapping": false, "flapping_history": [ true, false, false, false ], "instance": { "id": "rt-anomaly-mean-value" }, "maintenance_window_ids": [], "rule": { "category": "Anomaly detection alert", "consumer": "alerts", "execution": { "uuid": "e9e681d4-c8e4-43eb-82e5-a58bdf7ffe12" }, "name": "rt-ad-alert-influencer", "parameters": { "severity": 5, "resultType": "influencer", "includeInterim": false, "jobSelection": { "jobIds": [ "rt-anomaly-mean-value" ], "groupIds": [] }, "lookbackInterval": null, "topNBuckets": null }, "producer": "ml", "revision": 0, "rule_type_id": "xpack.ml.anomaly_detection_alert", "tags": [], "uuid": "9e1d6bc0-5e10-11ee-8416-3bf48cca0922" }, "status": "active", "uuid": "c9c1f075-9985-4c55-8ff8-22349cb30269", "workflow_status": "open", "duration": { "us": "99021000000" }, "start": "2023-09-28T15:07:12.868Z", "time_range": { "gte": "2023-09-28T15:07:12.868Z" } }, "space_ids": [ "default" ], "version": "8.11.0" }, "@timestamp": "2023-09-28T15:08:51.889Z", "event": { "action": "active", "kind": "signal" }, "tags": [] } ``` </details> Now they look like: <details> <summary>Flattened</summary> ``` { "kibana.alert.url": "/app/ml/explorer/?_g=(ml%3A(jobIds%3A!(rt-anomaly-mean-value))%2Ctime%3A(from%3A'2023-09-28T15%3A03%3A00.000Z'%2Cmode%3Aabsolute%2Cto%3A'2023-09-28T15%3A23%3A00.000Z'))&_a=(explorer%3A(mlExplorerFilter%3A(filterActive%3A!t%2CfilteredFields%3A!(key%2Cthird-key)%2CinfluencersFilterQuery%3A(bool%3A(minimum_should_match%3A1%2Cshould%3A!((match_phrase%3A(key%3Athird-key)))))%2CqueryString%3A'key%3A%22third-key%22')%2CmlExplorerSwimlane%3A()))", "kibana.alert.reason": "Alerts are raised based on real-time scores. Remember that scores may be adjusted over time as data continues to be analyzed.", "kibana.alert.job_id": "rt-anomaly-mean-value", "kibana.alert.anomaly_score": 72.75515452061356, "kibana.alert.is_interim": false, "kibana.alert.anomaly_timestamp": 1695913980000, "kibana.alert.top_records": [{ "job_id": "rt-anomaly-mean-value", "record_score": 72.75515452061356, "initial_record_score": 72.75515452061356, "detector_index": 0, "is_interim": false, "timestamp": 1695913980000, "partition_field_name": "key", "partition_field_value": "third-key", "function": "mean", "actual": [ 0.5 ], "typical": [ 4.138745343296527 ] }], "kibana.alert.top_influencers": [{ "job_id": "rt-anomaly-mean-value", "influencer_field_name": "key", "influencer_field_value": "third-key", "influencer_score": 72.75515452061356, "initial_influencer_score": 72.75515452061356, "is_interim": false, "timestamp": 1695913980000 }], "kibana.alert.rule.category": "Anomaly detection alert", "kibana.alert.rule.consumer": "alerts", "kibana.alert.rule.execution.uuid": "17fef3d3-d595-4362-837e-b2a73650169e", "kibana.alert.rule.name": "rt-ad-alert-influencer", "kibana.alert.rule.parameters": { "severity": 5, "resultType": "influencer", "includeInterim": false, "jobSelection": { "jobIds": [ "rt-anomaly-mean-value" ], "groupIds": [] }, "lookbackInterval": null, "topNBuckets": null }, "kibana.alert.rule.producer": "ml", "kibana.alert.rule.revision": 0, "kibana.alert.rule.rule_type_id": "xpack.ml.anomaly_detection_alert", "kibana.alert.rule.tags": [], "kibana.alert.rule.uuid": "757c7610-5e11-11ee-8bc6-a95c3ced4757", "kibana.space_ids": [ "default" ], "@timestamp": "2023-09-28T15:14:52.057Z", "event.action": "active", "event.kind": "signal", "kibana.alert.action_group": "anomaly_score_match", "kibana.alert.flapping": false, "kibana.alert.flapping_history": [ true, false, false, false ], "kibana.alert.instance.id": "rt-anomaly-mean-value", "kibana.alert.maintenance_window_ids": [], "kibana.alert.status": "active", "kibana.alert.uuid": "ac1f0d7c-461b-4fc6-b4c3-04416ac876d3", "kibana.alert.workflow_status": "open", "kibana.alert.duration.us": "99028000000", "kibana.alert.start": "2023-09-28T15:13:13.028Z", "kibana.alert.time_range": { "gte": "2023-09-28T15:13:13.028Z" }, "kibana.version": "8.11.0", "tags": [] } ``` </details> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

ymao1 mentioned this pull request Sep 27, 2023

[Response Ops][Alerting] Should framework alerts as data write out data flattened keys #166946

Closed

ymao1 changed the title ~~Alerting/flatten aad~~ [Response Ops][Alerting] Update framework alerts client to write flattened alerts docs Sep 27, 2023

ymao1 force-pushed the alerting/flatten-aad branch 2 times, most recently from e5b5a1a to 1d21183 Compare September 28, 2023 18:15

Writing flattened fields in framework AaD

7255e9a

ymao1 force-pushed the alerting/flatten-aad branch from 5752c3f to 7255e9a Compare September 28, 2023 23:45

Merge branch 'main' of github.com:elastic/kibana into alerting/flatte…

e8925c7

…n-aad

ymao1 self-assigned this Sep 29, 2023

ymao1 changed the base branch from main to alerting/flattened-faad-docs September 29, 2023 12:48

ymao1 marked this pull request as ready for review September 29, 2023 12:49

ymao1 requested review from a team as code owners September 29, 2023 12:49

mikecote self-requested a review September 29, 2023 14:20

mikecote approved these changes Sep 29, 2023

View reviewed changes

PR feedback

90b9b76

ymao1 merged commit 8dec3f6 into elastic:alerting/flattened-faad-docs Sep 29, 2023

ymao1 mentioned this pull request Sep 29, 2023

[Response Ops][Alerting] Update framework alerts client to write flattened alerts docs #167691

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Response Ops][Alerting] Update framework alerts client to write flattened alerts docs#167439

[Response Ops][Alerting] Update framework alerts client to write flattened alerts docs#167439
ymao1 merged 3 commits intoelastic:alerting/flattened-faad-docsfrom
ymao1:alerting/flatten-aad

ymao1 commented Sep 27, 2023 •

edited

Loading

Uh oh!

ymao1 commented Sep 28, 2023

Uh oh!

ymao1 commented Sep 29, 2023

Uh oh!

mikecote left a comment

Uh oh!

mikecote Sep 29, 2023

Uh oh!

ymao1 Sep 29, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mikecote Sep 29, 2023

Uh oh!

ymao1 Sep 29, 2023

Uh oh!

mikecote Sep 29, 2023

Uh oh!

ymao1 Sep 29, 2023

Uh oh!

kibana-ci commented Sep 29, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ymao1 commented Sep 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

To Verify

ES Query and Index Threshold AaD

ML alerts

Uh oh!

ymao1 commented Sep 28, 2023

Uh oh!

ymao1 commented Sep 29, 2023

Uh oh!

mikecote left a comment

Choose a reason for hiding this comment

Uh oh!

mikecote Sep 29, 2023

Choose a reason for hiding this comment

Uh oh!

ymao1 Sep 29, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mikecote Sep 29, 2023

Choose a reason for hiding this comment

Uh oh!

ymao1 Sep 29, 2023

Choose a reason for hiding this comment

Uh oh!

mikecote Sep 29, 2023

Choose a reason for hiding this comment

Uh oh!

ymao1 Sep 29, 2023

Choose a reason for hiding this comment

Uh oh!

kibana-ci commented Sep 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ymao1 commented Sep 27, 2023 •

edited

Loading

kibana-ci commented Sep 29, 2023 •

edited

Loading