[Alerting] Provision UIAM APIKeys for alerting rules by ersin-erdal · Pull Request #254211 · elastic/kibana

ersin-erdal · 2026-02-20T13:52:41Z

This PR adds a background task that migrates alerting rules from legacy API keys to UIAM (Unified Identity and Access Management) API keys on Serverless. The task runs only when the alerting.rules.provisionUiamApiKeys feature flag is enabled and only in Serverless mode.

What's in scope

New task alerting:api_key_provisioning
- Runs every 1 minute (configurable interval).
- Timeout: 5 minutes.
- Only registered and started when isServerless is true.
- Scheduled when alerting.rules.provisionUiamApiKeys is enabled; removed when the flag is disabled.
Provisioning flow
1. Find rules that have apiKey but no uiamApiKey, and where the key was not user-created (apiKeyCreatedByUser !== true).
2. Exclude rules that already have a provisioning status of completed or skipped (via new saved object type).
3. Call the core UIAM convert API to convert legacy API keys to UIAM format.
4. Bulk-update rules with the new uiamApiKey.
5. Write provisioning status per rule (completed / failed / skipped). THe SO is in the Alerting/Cases scope (Maybe should be independent since we want to use it for TM provisioning too)
6. If a rule update fails after a successful conversion, the newly created UIAM API key is marked for invalidation (orphan handling).
New saved object type
uiam-api-keys-provisioning-status — one document per rule with entityId, entityType, status (completed | failed | skipped), optional message, and @timestamp. Used to avoid re-processing and to report status.
Skipped rules
Rules are skipped (and status recorded) when:
- No apiKey, or
- Already have uiamApiKey, or
- apiKeyCreatedByUser === true.
Batching
Rules are read in batches of 300. If more rules remain, the task reschedules itself 1 minute later.

Out of scope / behavior

Non-Serverless: Task is not registered; no behavior change.
Flag off: Task is not scheduled; no conversion runs.
License: If the UIAM convert API is unavailable (e.g. license), the task throws and Task Manager will retry.

To verify:

Add below configs to your serverless.dev.yml

xpack.alerting.rules.apiKeyType: 'es'
xpack.alerting.invalidateApiKeysTask.removalDelay: '10s'
xpack.alerting.invalidateApiKeysTask.interval: '1m'
feature_flags.overrides:
  alerting.rules.provisionUiamApiKeys: false

so we can create some rules with the ES API Keys.

Run Kibana and Elasticsearch with the below commands:
yarn es serverless --projectType observability
yarn start --serverless oblt --run-examples
Login with the system_indices_superuser user.
Create 4-5 rules and let them run at least for one time.
Stop your ES and Kibana
Update the below FF in your serverless.yml

feature_flags.overrides:
  alerting.rules.provisionUiamApiKeys: true

Update GET_RULES_BATCH_SIZE to 2 in uiam_api_key_provisioning_task.ts
Run Kibana and Elasticsearch with the below commands:
yarn es serverless --projectType observability --uiam
yarn start --serverless oblt --run-examples --uiam
The task should run and update the first 2 rules with uiamApiKey

GET /.kibana_alerting_cases_*/_search
{
  "query": {
    "match": {
      "type": {
        "query": "alert"
      }
    }
  }
}

Wait for 1m, the task should run again and update the next 2 rules.
You can observe the provisioning status with:

GET /.kibana_alerting_cases_*/_search
{
  "query": {
    "match": {
      "type": {
        "query": "uiam_api_keys_provisioning_status"
      }
    }
  }
}

If your rules run often (every 1m) Rule update may fail. In this case we set the provisioning status to failed and fetch the rule in the next batch again. We also set the orphaned converted apiKeys to be invalidated.
You can see them by using the below query:

GET /.kibana_alerting_cases_*/_search
{
  "query": {
    "match": {
      "type": {
        "query": "api_key_pending_invalidation"
      }
    }
  }
}

…tion_tests/ci_checks

# Conflicts: # x-pack/platform/plugins/shared/alerting/server/plugin.ts

…tion_tests/ci_checks

# Conflicts: # src/core/packages/saved-objects/server-internal/src/object_types/index.ts # x-pack/platform/plugins/shared/alerting/server/plugin.ts

elasticmachine · 2026-03-10T21:53:32Z

Pinging @elastic/response-ops (Team:ResponseOps)

azasypkin

Changes in x-pack/platform/plugins/shared/alerting/server/saved_objects/index.ts LGTM: a newly defined uiam_api_keys_provisioning_status SO type doesn't have any encrypted fields.

darnautov

LGTM

darnautov · 2026-03-17T16:25:20Z

x-pack/platform/plugins/shared/alerting/server/provisioning/uiam_api_key_provisioning_task.ts

+    core.featureFlags
+      .getBooleanValue$(PROVISION_UIAM_API_KEYS_FLAG, false)
+      .subscribe((enabled: boolean) => {
+        this.applyProvisioningFlag(enabled, taskManager).catch(() => {});
+      });


we need to keep the reference to this subscription, and unsubscribe on plugin stop

darnautov · 2026-03-17T16:43:10Z

x-pack/platform/plugins/shared/alerting/server/provisioning/uiam_api_key_provisioning_task.ts

+      const keys = apiKeysToConvert.map(({ attributes }) => attributes.apiKey!);
+      const convertResponse = await context.uiamConvert(keys);
+      if (convertResponse === null) {
+        throw new Error('License required for the UIAM convert API is not enabled');


does it happen with a missing license only?

Yes uiam.convert returns null only when license is missing

darnautov · 2026-03-17T17:08:47Z

x-pack/platform/plugins/shared/alerting/server/provisioning/lib/get_exclude_rules_filter.ts

+    page += 1;
+  }
+  if (ruleIds.size === 0) return undefined;
+  return nodeTypes.function.buildNode('not', convertRuleIdsToKueryNode(Array.from(ruleIds)));


It's unlikely we hit the clause limit, buy it's worth adding a safeguard here?

Done, Claude recommended me to use chunks there.
But I don't think we would hit the limit.
Default limit is 4096 and the project has the most rules has 3600 rules.

elasticmachine · 2026-03-17T21:29:15Z

⏳ Build in-progress, with failures

Buildkite Build
Commit: db65860

Failed CI Steps

History

💛 Build #410542 was flaky a459a2e
💔 Build #410452 failed 7796386
💚 Build #407755 succeeded 7aefb99
💔 Build #407604 failed 111ba60
💔 Build #405489 failed 8d2db7b

Merge timing conflict between #257678 and #254211

Merge timing conflict between elastic#257678 and elastic#254211

This PR adds a background task that migrates alerting rules from legacy API keys to UIAM (Unified Identity and Access Management) API keys on Serverless. The task runs only when the `alerting.rules.provisionUiamApiKeys` feature flag is enabled and only in Serverless mode. ### What's in scope - **New task** `alerting:api_key_provisioning` - Runs every 1 minute (configurable interval). - Timeout: 5 minutes. - Only registered and started when `isServerless` is true. - Scheduled when `alerting.rules.provisionUiamApiKeys` is enabled; removed when the flag is disabled. - **Provisioning flow** 1. Find rules that have `apiKey` but no `uiamApiKey`, and where the key was not user-created (`apiKeyCreatedByUser !== true`). 2. Exclude rules that already have a provisioning status of `completed` or `skipped` (via new saved object type). 3. Call the core UIAM convert API to convert legacy API keys to UIAM format. 4. Bulk-update rules with the new `uiamApiKey`. 5. Write provisioning status per rule (completed / failed / skipped). THe SO is in the Alerting/Cases scope (Maybe should be independent since we want to use it for TM provisioning too) 6. If a rule update fails after a successful conversion, the newly created UIAM API key is marked for invalidation (orphan handling). - **New saved object type** `uiam-api-keys-provisioning-status` — one document per rule with `entityId`, `entityType`, `status` (completed | failed | skipped), optional `message`, and `@timestamp`. Used to avoid re-processing and to report status. - **Skipped rules** Rules are skipped (and status recorded) when: - No `apiKey`, or - Already have `uiamApiKey`, or - `apiKeyCreatedByUser === true`. - **Batching** Rules are read in batches of 300. If more rules remain, the task reschedules itself 1 minute later. ### Out of scope / behavior - **Non-Serverless**: Task is not registered; no behavior change. - **Flag off**: Task is not scheduled; no conversion runs. - **License**: If the UIAM convert API is unavailable (e.g. license), the task throws and Task Manager will retry. ## To verify: 1. Add below configs to your serverless.dev.yml ``` xpack.alerting.rules.apiKeyType: 'es' xpack.alerting.invalidateApiKeysTask.removalDelay: '10s' xpack.alerting.invalidateApiKeysTask.interval: '1m' feature_flags.overrides: alerting.rules.provisionUiamApiKeys: false ``` so we can create some rules with the ES API Keys. 2. Run Kibana and Elasticsearch with the below commands: `yarn es serverless --projectType observability` `yarn start --serverless oblt --run-examples` 3. Login with the `system_indices_superuser` user. 4. Create 4-5 rules and let them run at least for one time. 5. Stop your ES and Kibana 6. Update the below FF in your serverless.yml ``` feature_flags.overrides: alerting.rules.provisionUiamApiKeys: true ``` 7. Update `GET_RULES_BATCH_SIZE` to `2` in `uiam_api_key_provisioning_task.ts` 8. Run Kibana and Elasticsearch with the below commands: `yarn es serverless --projectType observability --uiam` `yarn start --serverless oblt --run-examples --uiam` 9. The task should run and update the first 2 rules with `uiamApiKey` ``` GET /.kibana_alerting_cases_*/_search { "query": { "match": { "type": { "query": "alert" } } } } ``` 10. Wait for 1m, the task should run again and update the next 2 rules. 11. You can observe the provisioning status with: ``` GET /.kibana_alerting_cases_*/_search { "query": { "match": { "type": { "query": "uiam_api_keys_provisioning_status" } } } } ``` 12. If your rules run often (every 1m) Rule update may fail. In this case we set the provisioning status to failed and fetch the rule in the next batch again. We also set the orphaned converted apiKeys to be invalidated. You can see them by using the below query: ``` GET /.kibana_alerting_cases_*/_search { "query": { "match": { "type": { "query": "api_key_pending_invalidation" } } } } ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Tyler Smalley <tyler.smalley@elastic.co>

Merge timing conflict between elastic#257678 and elastic#254211

This PR adds a background task that migrates alerting rules from legacy API keys to UIAM (Unified Identity and Access Management) API keys on Serverless. The task runs only when the `alerting.rules.provisionUiamApiKeys` feature flag is enabled and only in Serverless mode. ### What's in scope - **New task** `alerting:api_key_provisioning` - Runs every 1 minute (configurable interval). - Timeout: 5 minutes. - Only registered and started when `isServerless` is true. - Scheduled when `alerting.rules.provisionUiamApiKeys` is enabled; removed when the flag is disabled. - **Provisioning flow** 1. Find rules that have `apiKey` but no `uiamApiKey`, and where the key was not user-created (`apiKeyCreatedByUser !== true`). 2. Exclude rules that already have a provisioning status of `completed` or `skipped` (via new saved object type). 3. Call the core UIAM convert API to convert legacy API keys to UIAM format. 4. Bulk-update rules with the new `uiamApiKey`. 5. Write provisioning status per rule (completed / failed / skipped). THe SO is in the Alerting/Cases scope (Maybe should be independent since we want to use it for TM provisioning too) 6. If a rule update fails after a successful conversion, the newly created UIAM API key is marked for invalidation (orphan handling). - **New saved object type** `uiam-api-keys-provisioning-status` — one document per rule with `entityId`, `entityType`, `status` (completed | failed | skipped), optional `message`, and `@timestamp`. Used to avoid re-processing and to report status. - **Skipped rules** Rules are skipped (and status recorded) when: - No `apiKey`, or - Already have `uiamApiKey`, or - `apiKeyCreatedByUser === true`. - **Batching** Rules are read in batches of 300. If more rules remain, the task reschedules itself 1 minute later. ### Out of scope / behavior - **Non-Serverless**: Task is not registered; no behavior change. - **Flag off**: Task is not scheduled; no conversion runs. - **License**: If the UIAM convert API is unavailable (e.g. license), the task throws and Task Manager will retry. ## To verify: 1. Add below configs to your serverless.dev.yml ``` xpack.alerting.rules.apiKeyType: 'es' xpack.alerting.invalidateApiKeysTask.removalDelay: '10s' xpack.alerting.invalidateApiKeysTask.interval: '1m' feature_flags.overrides: alerting.rules.provisionUiamApiKeys: false ``` so we can create some rules with the ES API Keys. 2. Run Kibana and Elasticsearch with the below commands: `yarn es serverless --projectType observability` `yarn start --serverless oblt --run-examples` 3. Login with the `system_indices_superuser` user. 4. Create 4-5 rules and let them run at least for one time. 5. Stop your ES and Kibana 6. Update the below FF in your serverless.yml ``` feature_flags.overrides: alerting.rules.provisionUiamApiKeys: true ``` 7. Update `GET_RULES_BATCH_SIZE` to `2` in `uiam_api_key_provisioning_task.ts` 8. Run Kibana and Elasticsearch with the below commands: `yarn es serverless --projectType observability --uiam` `yarn start --serverless oblt --run-examples --uiam` 9. The task should run and update the first 2 rules with `uiamApiKey` ``` GET /.kibana_alerting_cases_*/_search { "query": { "match": { "type": { "query": "alert" } } } } ``` 10. Wait for 1m, the task should run again and update the next 2 rules. 11. You can observe the provisioning status with: ``` GET /.kibana_alerting_cases_*/_search { "query": { "match": { "type": { "query": "uiam_api_keys_provisioning_status" } } } } ``` 12. If your rules run often (every 1m) Rule update may fail. In this case we set the provisioning status to failed and fetch the rule in the next batch again. We also set the orphaned converted apiKeys to be invalidated. You can see them by using the below query: ``` GET /.kibana_alerting_cases_*/_search { "query": { "match": { "type": { "query": "api_key_pending_invalidation" } } } } ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Tyler Smalley <tyler.smalley@elastic.co>

Merge timing conflict between elastic#257678 and elastic#254211

ersin-erdal and others added 20 commits February 5, 2026 23:55

Migrate APIKeys

7d023d5

re-schedule the task to convert the rest

634abc4

add provisioning index

3c17a4a

Merge branch 'main' into 507-uiam-migration

ee446f5

Changes from node scripts/check_mappings_update --fix

f437fb6

clear types

cf7a717

bump number of SO

ce02b64

Changes from node scripts/jest_integration -u src/core/server/integra…

24b7f3e

…tion_tests/ci_checks

fix check types

d598f8f

add the new task type to the check lists

e30c9db

add feature flag check

7ab6554

fix runAt interval

02ef875

use constants rather than magic numbers

26b4e1f

fix check task types

9a30274

add message field to the provisioning index

ab13d41

refactoring

41bf90f

prettify

a54606e

add unit tests

a7d03f4

fix linting

bd7270b

fix mapping

2d7f0a9

ersin-erdal changed the title ~~507 uiam migration~~ Provision UIAM APIKeys for alerting rules Feb 20, 2026

ersin-erdal mentioned this pull request Feb 20, 2026

Provision UIAM APIKeys for alerting rules #251990

Closed

ersin-erdal and others added 4 commits February 20, 2026 15:18

fix current_fields

25f9582

Merge branch 'main' into 507-uiam-migration

5545e31

# Conflicts: # x-pack/platform/plugins/shared/alerting/server/plugin.ts

Changes from node scripts/check_mappings_update --fix

3d8f36f

Changes from node scripts/jest_integration -u src/core/server/integra…

af9e1e8

…tion_tests/ci_checks

ersin-erdal changed the title ~~Provision UIAM APIKeys for alerting rules~~ [Alerting] Provision UIAM APIKeys for alerting rules Feb 25, 2026

ersin-erdal added the Feature:CPS label Feb 27, 2026

ersin-erdal added 2 commits March 3, 2026 23:53

register the task but don't start for non-serverless

3ab7493

Merge branch 'main' into 507-uiam-migration

d806fae

# Conflicts: # src/core/packages/saved-objects/server-internal/src/object_types/index.ts # x-pack/platform/plugins/shared/alerting/server/plugin.ts

ersin-erdal added 2 commits March 10, 2026 19:44

add total

5e619ef

refactor rule update status handling

7aefb99

ersin-erdal marked this pull request as ready for review March 10, 2026 21:37

ersin-erdal requested review from a team as code owners March 10, 2026 21:37

ersin-erdal requested a review from azasypkin March 10, 2026 21:37

ersin-erdal added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Team:ResponseOps Platform ResponseOps team (formerly the Cases and Alerting teams) t// labels Mar 10, 2026

ersin-erdal requested a review from darnautov March 11, 2026 16:49

Merge branch 'main' into 507-uiam-migration

40b08d2

azasypkin approved these changes Mar 12, 2026

View reviewed changes

TinaHeiligers approved these changes Mar 13, 2026

View reviewed changes

ersin-erdal and others added 3 commits March 16, 2026 11:45

Merge branch 'main' into 507-uiam-migration

7796386

Merge branch 'main' into 507-uiam-migration

883effc

bump SO count

a459a2e

darnautov approved these changes Mar 17, 2026

View reviewed changes

use chunks to get excluded rules

db65860

ersin-erdal merged commit af7972c into elastic:main Mar 18, 2026
18 checks passed

kibanamachine added the v9.4.0 label Mar 18, 2026

jbudz mentioned this pull request Mar 18, 2026

Bump saved objects count #258257

Merged

jbudz added a commit that referenced this pull request Mar 18, 2026

Bump saved objects count (#258257)

4891220

Merge timing conflict between #257678 and #254211

szwarckonrad pushed a commit to szwarckonrad/kibana that referenced this pull request Mar 18, 2026

Bump saved objects count (elastic#258257)

103c0ec

Merge timing conflict between elastic#257678 and elastic#254211

qn895 pushed a commit to qn895/kibana that referenced this pull request Mar 18, 2026

Bump saved objects count (elastic#258257)

1b2109e

Merge timing conflict between elastic#257678 and elastic#254211

jeramysoucy pushed a commit to jeramysoucy/kibana that referenced this pull request Mar 26, 2026

Bump saved objects count (elastic#258257)

eddd6ea

Merge timing conflict between elastic#257678 and elastic#254211

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Alerting] Provision UIAM APIKeys for alerting rules#254211

[Alerting] Provision UIAM APIKeys for alerting rules#254211
ersin-erdal merged 63 commits intoelastic:mainfrom
ersin-erdal:507-uiam-migration

ersin-erdal commented Feb 20, 2026 •

edited

Loading

Uh oh!

elasticmachine commented Mar 10, 2026

Uh oh!

azasypkin left a comment

Uh oh!

darnautov left a comment

Uh oh!

darnautov Mar 17, 2026

Uh oh!

darnautov Mar 17, 2026

Uh oh!

ersin-erdal Mar 17, 2026

Uh oh!

darnautov Mar 17, 2026

Uh oh!

ersin-erdal Mar 17, 2026

Uh oh!

elasticmachine commented Mar 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

ersin-erdal commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's in scope

Out of scope / behavior

To verify:

Uh oh!

elasticmachine commented Mar 10, 2026

Uh oh!

azasypkin left a comment

Choose a reason for hiding this comment

Uh oh!

darnautov left a comment

Choose a reason for hiding this comment

Uh oh!

darnautov Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

darnautov Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

ersin-erdal Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

darnautov Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

ersin-erdal Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⏳ Build in-progress, with failures

Failed CI Steps

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ersin-erdal commented Feb 20, 2026 •

edited

Loading

elasticmachine commented Mar 17, 2026 •

edited

Loading