[Cloud-Security] Adding var-groups to support authentication vars - related to cloud connector feature by seanrathier · Pull Request #16985 · elastic/integrations

seanrathier · 2026-01-16T16:05:42Z

Summary

Adds var_groups configuration to the AWS integration package to enable a credential type selector in Fleet UI, improving the user experience when configuring AWS authentication methods.

Related PRs

related [Cloud Security] [Fleet] Add var_group components and validation kibana#249449
depends on Adding var_groups spec package-spec#1059

Changes

Added var_groups section to package manifest with credential_type group containing:
- Direct Access Keys (access_key_id, secret_access_key)
- Temporary Access Keys (access_key_id, secret_access_key, session_token)
- Cloud Connector (role_arn, external_id) - for agentless deployments
- Assume Role (role_arn)
- Assume Role with External ID (role_arn, external_id)
- Shared Credentials (shared_credential_file, credential_profile_name)
Added hide_in_var_group_options to GuardDuty aws-s3 input to hide Cloud Connector option (not supported for S3 input)
Updated credential variables (role_arn, external_id, shared_credential_file, credential_profile_name) to show_user: true so that users can see the vars in var group options

Screenshots

Screen.Recording.2026-01-23.at.10.06.07.AM.mov

Related Issues

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

elasticmachine · 2026-01-23T16:48:34Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2026-01-23T16:48:35Z

Pinging @elastic/fleet (Team:Fleet)

MichaelKatsoulis · 2026-01-27T08:44:53Z

+        vars: [role_arn, external_id]
+        hide_in_deployment_modes: [default]
+        provider: aws
+        iac_template_url: https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://elastic-cspm-cft.s3.eu-central-1.amazonaws.com/cloudformation-cloud-connectors-ACCOUNT_TYPE-9.2.0.yml&param_ElasticResourceId=RESOURCE_ID


Is the point of the template to be public ?

User: arn:aws:sts::627286350134:assumed-role/okta-dev/michail.katsoulis@elastic.co is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::elastic-cspm-cft" because no resource-based policy allows the s3:ListBucket action

It is public but the Cloud Connector UI need to replace the ACCOUNT_TYPE and RESOURCE_ID in the URL. We will be adding a Cloud Connector Account Type selection in the Cloud Connector UI component to select it.

https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://elastic-cspm-cft.s3.eu-central-1.amazonaws.com/cloudformation-cloud-connectors-organization-account-9.2.0.yml&param_ElasticResourceId=1234567890ABCDE

https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://elastic-cspm-cft.s3.eu-central-1.amazonaws.com/cloudformation-cloud-connectors-single-account-9.2.0.yml&param_ElasticResourceId=1234567890ABCDE

This is what we use in the CSPM and Cloud Asset Discovery Fleet extension to control the template URL.

MichaelKatsoulis · 2026-01-27T08:57:29Z

        title: Collect Amazon GuardDuty logs via AWS S3 or SQS
        description: Collecting Amazon GuardDuty logs via AWS S3 or SQS input.
+        hide_in_var_group_options:
+          credential_type: [cloud_connectors]


Is there an end goal for this?

@MichaelKatsoulis thanks for the question.

The goal here

We currently have no way to identify group of vars represent, specifically for the use case when we refer to cloud provider credentials. This is why we are adding the var_groups.

hide_in_var_group_options are for inputs in the policy that don't support a var_group selection, in this case Cloud Connectors don't currently support S3 data collection.

This specific input you are referring does not support agentless either and cloud connectors can only work ATM with agentless agent deployments

The end goal

The end goals for all of this is to have credential vars when grouped together be required, allow us to know when are user to use the Cloud Connector feature so we can add some policy effects and the Cloud Connector UI

Thank you @seanrathier for your detailed reply!

elasticmachine · 2026-02-05T17:01:07Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

botelastic · 2026-03-19T16:15:23Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

agithomas

LGTM from obs packages. Kindly address the suggested changelog changes

seanrathier · 2026-04-01T16:58:51Z

/test

andrewkroh

Please make sure you perform a manual test to validate the existing users on an earlier version of the aws guardduty package and seamlessly upgrade to the new version.

I think the big challenge with this change is going to be figuring out how we continue to support aws package users running Elastic stack 8.19.

andrewkroh · 2026-04-01T21:11:40Z

@@ -1,7 +1,7 @@
-format_version: 3.4.0
+format_version: 3.6.0


The package now declares format_version: 3.6.0 while still allowing ^8.19.4 || ^9.2.1.

Per package-spec support guidance, 8.19 supports format 3.4 and 9.2 supports 3.5; 3.6 is not listed for those versions. (I'm not sure if those docs accurate, maybe 8.19 was updated?)

So some aspect of this is going to have to change.

I bumped the version to ^9.3 but I think this needs a wider team discussion.

CC: @Omolola-Akinleye @olegsu

I ran a test with 8.19.15-SNAPSHOT, and this new version of the aws package isn't visible on Kibana 8.19 because Kibana queries the elastic package-registry (EPR) with spec.max so it filters this release out.

https://epr.elastic.co/search?package=aws&spec.max=3.4&kibana.version=8.19.15

So even though we tested this version does work on 8.19 via directly upload, it won't work because the Kibana and EPR filter it out.

@kpollich I think this is going to be a very common problem that we run into where we want to be able to continue delivering fixes for integrations to users that are on 8.19.x (while it is still officially under support) while also being able to adopt some newer package-spec features.

One solution is to bump the major version and drop support for 8.19 from the AWS package, but then we are signing up for maintaining an aws-6.x branch where we backport changes to the 8.19 users (where those changes are compatible with format_version: 3.4).

Providing maintenance to 8.19 users for all of the packages that want/need to move past format_version 3.4 could be a lot of work, especially since we have to do it for about ~14 months until July 2027.

Maybe you have some other ideas on how to handle this?

andrewkroh · 2026-04-01T21:14:07Z

  elastic:
    subscription: basic
  kibana:
    version: "^8.19.4 || ^9.2.1"


This PR switches GuardDuty httpjson auth to auth.aws, but package constraints still allow ^8.19.4 || ^9.2.1. But Beats httpjson only supports auth.aws from 9.3 onward.

I bumped the version to ^9.3 but I think this needs a wider team discussion.

CC: @Omolola-Akinleye @olegsu

elasticmachine · 2026-04-02T17:36:21Z

💚 Build Succeeded

Buildkite Build
Commit: 8a3761b

History

💚 Build #40763 succeeded e19db9e
💔 Build #40752 failed b5d1063
💚 Build #40729 succeeded 20a1f30
💔 Build #40712 failed af4efcd
💔 Build #40647 failed af4efcd
💔 Build #40646 failed f1877f8

cc @seanrathier

andrewkroh · 2026-04-02T18:56:00Z

/test stack 8.19.14-SNAPSHOT

olegsu · 2026-04-22T20:21:53Z

Quick update:
@qcorporation and I discussed the option of merging this before of elastic/package-spec#1148, given that the AWS package is under active maintenance (10 commits since April) and has been receiving multiple updates, the risk of merging now feels too high. To stay on the safe side, we will wait for stack 8.19 to fully work with 3.6 spec
cc @andrewkroh @kpollich @jlind23

elasticmachine · 2026-04-30T20:11:05Z

💔 Build Failed

Buildkite Build
Commit: 2bf8aab

Failed CI Steps

Check integrations aws

History

💚 Build #41547 succeeded c684fd0
💚 Build #41506 succeeded c38c4e0
💚 Build #41357 succeeded 45ae177

cc @seanrathier

seanrathier · 2026-05-01T15:39:44Z

Continued in #18762 — this PR could not be reopened (GitHub returned "no new commits on the seanrathier:var_groups branch" for both API and web reopen after the head branch was reset). The new PR uses the same head branch (seanrathier:var_groups) with the work restored.

andrewkroh added the Integration:aws AWS label Jan 16, 2026

seanrathier mentioned this pull request Jan 19, 2026

Adding var_groups spec elastic/package-spec#1059

Merged

2 tasks

seanrathier added Team:Integrations Label for the Integrations team Team:Fleet Fleet team [elastic/fleet] Team:Cloud Security Cloud Security team [elastic/cloud-security-posture] labels Jan 19, 2026

seanrathier self-assigned this Jan 19, 2026

seanrathier requested review from Copilot and removed request for Copilot January 20, 2026 00:59

seanrathier force-pushed the var_groups branch from 38a8b03 to 14f60c0 Compare January 23, 2026 15:25

seanrathier marked this pull request as ready for review January 23, 2026 16:48

seanrathier requested review from a team as code owners January 23, 2026 16:48

andrewkroh added the Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] label Jan 23, 2026

efd6 reviewed Jan 25, 2026

View reviewed changes

Comment thread packages/aws/changelog.yml Outdated

Comment thread packages/aws/changelog.yml Outdated

Comment thread packages/aws/manifest.yml Outdated

Comment thread packages/aws/manifest.yml Outdated

seanrathier requested a review from efd6 January 26, 2026 20:29

MichaelKatsoulis reviewed Jan 27, 2026

View reviewed changes

seanrathier requested a review from MichaelKatsoulis January 27, 2026 18:10

MichaelKatsoulis approved these changes Jan 28, 2026

View reviewed changes

seanrathier requested a review from agithomas January 28, 2026 15:35

seanrathier mentioned this pull request Feb 5, 2026

[Cloud Security][Fleet] Add cloud connector var group policy effects and UI hookup changes elastic/kibana#251488

Merged

6 tasks

andrewkroh added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Feb 5, 2026

seanrathier force-pushed the var_groups branch from c5396d6 to a42cf8f Compare February 10, 2026 19:21

jsoriano mentioned this pull request Feb 17, 2026

Release 3.6 elastic/package-spec#1060

Open

botelastic Bot added the Stalled label Mar 19, 2026

agithomas approved these changes Apr 1, 2026

View reviewed changes

seanrathier enabled auto-merge (squash) April 1, 2026 16:25

seanrathier removed request for a team and efd6 April 1, 2026 16:27

seanrathier requested a review from efd6 April 1, 2026 19:29

andrewkroh reviewed Apr 1, 2026

View reviewed changes

olegsu mentioned this pull request Apr 7, 2026

[8.19] Backport AWS cloud connectors credential helper (#48956) elastic/beats#49956

Closed

andrewkroh mentioned this pull request Apr 8, 2026

[aws]: inspector2/guarduty/securityhub - no role_arn or session_token #10784

Open

kcreddy mentioned this pull request Apr 10, 2026

[aws] AWS Config: improve error when credentials are missing and clarify Role ARN limitation in docs #18332

Closed

seanrathier force-pushed the var_groups branch from 8a3761b to 721b1e3 Compare April 14, 2026 19:31

kpollich mentioned this pull request Apr 21, 2026

Discuss: stack version support policy for integration packages elastic/package-spec#1148

Open

github-actions Bot mentioned this pull request Apr 27, 2026

[box_events] Add Agentless Deployment Support #18656

Open

5 tasks

seanrathier force-pushed the var_groups branch from c684fd0 to d51d88f Compare April 30, 2026 19:44

macroscopeapp Bot reviewed Apr 30, 2026

View reviewed changes

Comment thread packages/aws/manifest.yml

seanrathier force-pushed the var_groups branch 2 times, most recently from d963f5d to 2bf8aab Compare April 30, 2026 19:51

macroscopeapp Bot reviewed Apr 30, 2026

View reviewed changes

Comment thread packages/aws/manifest.yml

seanrathier force-pushed the var_groups branch 2 times, most recently from 4e911c2 to 47bc590 Compare May 1, 2026 15:28

seanrathier closed this May 1, 2026

auto-merge was automatically disabled May 1, 2026 15:28
Pull request was closed

seanrathier force-pushed the var_groups branch from 47bc590 to b828f39 Compare May 1, 2026 15:28

seanrathier mentioned this pull request May 1, 2026

[Cloud-Security] Adding var-groups to support authentication vars - related to cloud connector feature #18762

Open

5 tasks

github-actions Bot mentioned this pull request May 4, 2026

[claroty_ctd] Add Agentless Deployment Support #18781

Open

5 tasks

Conversation

seanrathier commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related PRs

Changes

Screenshots

Related Issues

Checklist

Author's Checklist

How to test this PR locally

Uh oh!

elasticmachine commented Jan 23, 2026

Uh oh!

elasticmachine commented Jan 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

The goal here

The end goal

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Feb 5, 2026

Uh oh!

botelastic Bot commented Mar 19, 2026

Uh oh!

agithomas left a comment

Choose a reason for hiding this comment

Uh oh!

seanrathier commented Apr 1, 2026

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

andrewkroh Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seanrathier Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Apr 2, 2026

💚 Build Succeeded

History

Uh oh!

andrewkroh commented Apr 2, 2026

Uh oh!

olegsu commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

seanrathier commented Jan 16, 2026 •

edited

Loading

andrewkroh Apr 1, 2026 •

edited

Loading

seanrathier Apr 2, 2026 •

edited

Loading

olegsu commented Apr 22, 2026 •

edited

Loading

elasticmachine commented Apr 30, 2026 •

edited

Loading