[8.19] Backport AWS cloud connectors credential helper (#48956) by olegsu · Pull Request #49956 · elastic/beats

olegsu · 2026-04-07T00:35:58Z

Context

We want to migrate integrations to use var_groups - elastic/integrations#16985.
The backport allow us to continue supporting 8.19 version and any future changes as well.

Summary

Manual backport of #48956 to the 8.19 branch.

Adds AWS cloud connectors credential helper support — a two-step STS role-chaining flow for OIDC/JWT-based federation used by the agentless controller:

AssumeRoleWithWebIdentity using an ID token file (Elastic global role)
AssumeRole using the intermediate credentials (customer role)

Changes

x-pack/libbeat/common/aws/cloud_connectors.go — new credential helper implementing the chain
x-pack/libbeat/common/aws/cloud_connectors_test.go — unit tests for the helper
x-pack/libbeat/common/aws/credentials.go — wire in cloud connectors path via UseCloudConnectors config flag
x-pack/libbeat/common/aws/credentials_test.go — integration smoke test

🤖 Generated with Claude Code

github-actions · 2026-04-07T00:36:07Z

🤖 GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

olegsu · 2026-04-07T00:46:16Z

cc @andrewkroh need you help to add team label pls

elasticmachine · 2026-04-07T14:16:08Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

olegsu · 2026-04-09T15:46:36Z

Update: using custom agentless image (with this changes) seems to work on 8.19. I will continue to test it today and update, please do not merge yet

Backports the cloud connectors AWS implementation and the combined External ID improvement from main to the 8.19 branch. The cloud connectors flow enables agentless AWS integrations via a 2-step role chaining: 1. Assume the Elastic Global Role using a web identity token (OIDC) 2. Assume the customer's remote role using the combined External ID (resourceID-externalIDPart) instead of separate ExternalID and SourceIdentity fields, allowing remote trust policies to scope access by specific cloud resources. Adds UseCloudConnectors bool to ConfigAWS, the cloud_connectors.go implementation, and full test coverage. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Cherry-picks x-pack/libbeat/common/aws/signer.go (new), x-pack/libbeat/common/aws/signer_test.go (new), x-pack/filebeat/input/httpjson/config_auth.go (auth.aws field added), x-pack/filebeat/input/httpjson/input.go (AWS SigV4 transport wiring), x-pack/filebeat/input/httpjson/input_test.go from main commit c1883b5. CEL input changes are excluded (httpjson-only backport). Original PR: elastic#47260

…tic#47712) Cherry-picks signer.go and signer_test.go fixes from main commit 183016c. Original PR: elastic#47712

Fixes 11 lint issues triggered by the auth.aws backport: - signer_test.go: EqualValues -> Equal (testifylint) - input.go: %v -> %w in errorf (errorlint); //nolint for pre-existing net.DialTimeout, paths.Resolve, net.Dial (noctx/forbidigo) - input_test.go: _, _ = for unchecked io.ReadAll/w.Write (errcheck); //nolint for pre-existing logp.TestingSetup (staticcheck), logp.NewLogger (forbidigo), val.(string) type assertion (errcheck) - config_auth.go: //nolint for pre-existing google.CredentialsFromJSON deprecation (staticcheck) Pre-existing issues use //nolint directives to avoid churn; only newly introduced code is fixed substantively.

Log auth config state (nil/enabled/fields) in run() and newHTTPClient() to diagnose 403 Missing Authentication Token errors. Log credential resolution path and per-request signing in SignerTransport.

New changes

andrewkroh · 2026-04-10T11:59:13Z

+
+# REQUIRED for all kinds
+# Change summary; a 80ish characters long description of the change.
+summary: Add AWS auth method for CEL and HTTP JSON inputs.


We are missing the CEL changes from #47260 (on my first review I assumed those changes were going to be back-ported on a separate PR).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Cherry-picks CEL input changes excluded from the original httpjson-only backport: x-pack/filebeat/input/cel/config_auth.go, x-pack/filebeat/input/cel/input.go, x-pack/filebeat/input/cel/input_test.go from main commit c1883b5. Original PR: elastic#47260 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add required explanations to all //nolint directives added by the auth.aws backport, and suppress the pre-existing staticcheck deprecation warning in cel/config_auth.go. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

CI uses --whole-files so touching these files surfaces all pre-existing issues. Add nolint directives with explanations for: - errcheck: type assertions src.(*source) (check-type-assertions:true) - errcheck: unchecked w.Write/io.ReadAll/enc.Encode in test handlers - forbidigo: paths.Resolve and logp.NewLogger - noctx: net.Dial (unix socket) and net.DialTimeout - nolintlint: remove unused directive before tokenAuthHandler Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

olegsu · 2026-04-14T21:13:07Z

I run a complete test today on 2 integrations with many changes to verify the expected behavior
Official image - the image that agentless platform decide to set when creating an agentless integration
custom image - is an image I build locally and pushed to ECR and used it as project overrides for specific policies

I also used a local package of the changes in the aws integration , thanks @seanrathier!

CSPM integration with official and custom images
AWS Guard duty with official and custom images and previous (6.4.3 and custom pacakge of AWS)

Test Matrix

ID	Test Name	Package Name	Image	Package Version	Expected Result	Status
TC-CSPM-01	CSPM — baseline	cloud_security_posture	Official	2.0.2	✅ PASS	✅ PASS
TC-CSPM-02	CSPM — regression test - new image wont break previous behavior	cloud_security_posture	Custom	2.0.2	✅ PASS	✅ PASS
TC-GD-01	GuardDuty — baseline	aws	Official	6.4.3	✅ PASS (baseline)	✅ PASS
TC-GD-02	GuardDuty — new integration PR breaks the current behavior - introduce regression	aws	Official	Custom	❌ FAIL	❌ FAIL (expected)
TC-GD-03	GuardDuty — regression test - new image wont break previous behavior	aws	Custom	6.4.3	✅ PASS	✅ PASS
TC-GD-04	GuardDuty — new image works well with the new integration	aws	Custom	Custom	✅ SUCCESS	✅ PASS

andrewkroh

Thanks for doing the end-to-end test with all the commits together, that's really valuable for validating that the pieces work as a whole on 8.19. I compared every file in this PR against the corresponding file on main and found that the core AWS code (cloud_connectors.go, cloud_connectors_test.go, signer.go, signer_test.go, credentials_test.go) is identical to main, which is great.

However, the input files (cel and httpjson) have accumulated 10s of nolint directives, a %v vs %w error wrapping fix, _, _ = errcheck suppressions, and a new package comment on cel/input.go that don't exist on main. These were needed because golangci-lint's --whole-files mode flagged pre-existing issues when the backport touched these files.

The lint CI checks are not mandatory for merging, so these extra changes shouldn't be included in a backport, they make it difficult to verify that the already-reviewed code from main is still functionally the same.

I'd recommend merging these as separate backport PRs, one per source PR, using the original merge commits:

c1883b58ff  # #47260 - AWS auth method
183016ce92  # #47712 - signer fix
f909d4fb70  # #48956 - combined External ID

The standard way to do this is the Mergify backport command (@mergifyio backport 8.19) on each original PR. This produces one backport PR per source PR, making each independently verifiable. If any cherry-pick has a merge conflict, the resolution is scoped to that PR and easy to review.

Any pre-existing lint issues surfaced by --whole-files should be left alone, the lint CI is advisory and won't block the merge. If there are real lint issues worth fixing, those should go to main first as an isolated PR and then be backported independently. This preserves the invariant that main is always a superset of release branches.

olegsu · 2026-04-15T16:44:03Z

Thanks for the review @andrewkroh I will start with the suggested approach and will close this pr at the end

olegsu · 2026-04-17T14:23:01Z

Closing now as all cherry-picks backports merged

olegsu requested a review from a team as a code owner April 7, 2026 00:35

olegsu requested review from khushijain21 and leehinman and removed request for a team April 7, 2026 00:36

botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 7, 2026

mergify Bot assigned olegsu Apr 7, 2026

olegsu requested a review from andrewkroh April 7, 2026 00:45

orestisfl added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 7, 2026

andrewkroh added the Team:Security-Cloud Services Label for the Security Data Experience - Cloud Services team. label Apr 8, 2026

andrewkroh previously approved these changes Apr 8, 2026

View reviewed changes

andrewkroh mentioned this pull request Apr 9, 2026

[aws]: inspector2/guarduty/securityhub - no role_arn or session_token elastic/integrations#10784

Open

olegsu and others added 3 commits April 9, 2026 16:04

Backport: fix aws auth method service name and region overwrite (elas…

9d7af01

…tic#47712) Cherry-picks signer.go and signer_test.go fixes from main commit 183016c. Original PR: elastic#47712

olegsu force-pushed the backport/aws-cloud-connectors-8.19 branch from 973bab3 to 9d7af01 Compare April 9, 2026 21:05

olegsu requested a review from a team as a code owner April 9, 2026 21:05

olegsu added 2 commits April 9, 2026 17:46

Add debug logging to trace auth.aws config and SigV4 signing path

5b52ef8

Log auth config state (nil/enabled/fields) in run() and newHTTPClient() to diagnose 403 Missing Authentication Token errors. Log credential resolution path and per-request signing in SignerTransport.

andrewkroh reviewed Apr 10, 2026

View reviewed changes

kcreddy mentioned this pull request Apr 10, 2026

[aws] AWS Config: improve error when credentials are missing and clarify Role ARN limitation in docs elastic/integrations#18332

Closed

olegsu and others added 4 commits April 10, 2026 09:21

build tag: pr-49956 2026-04-10T14:21:47Z

39c3ac1

fix build tag: use init() to prevent dead-code elimination

e639ab6

Merge ea-8.19-with-pr-49956: add build tag for PR-49956

d70543c

Remove debug logging added for AWS SigV4 auth tracing

db4738c

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

olegsu and others added 3 commits April 10, 2026 14:43

olegsu requested a review from andrewkroh April 14, 2026 21:13

andrewkroh reviewed Apr 15, 2026

View reviewed changes

olegsu closed this Apr 17, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[8.19] Backport AWS cloud connectors credential helper (#48956)#49956

[8.19] Backport AWS cloud connectors credential helper (#48956)#49956
olegsu wants to merge 12 commits intoelastic:8.19from
olegsu:backport/aws-cloud-connectors-8.19

olegsu commented Apr 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 7, 2026

Uh oh!

olegsu commented Apr 7, 2026 •

edited

Loading

Uh oh!

elasticmachine commented Apr 7, 2026

Uh oh!

olegsu commented Apr 9, 2026 •

edited

Loading

Uh oh!

andrewkroh Apr 10, 2026 •

edited

Loading

Uh oh!

olegsu commented Apr 14, 2026

Uh oh!

andrewkroh left a comment •

edited

Loading

Uh oh!

olegsu commented Apr 15, 2026

Uh oh!

olegsu commented Apr 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

olegsu commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Summary

Changes

Uh oh!

github-actions Bot commented Apr 7, 2026

🤖 GitHub comments

Uh oh!

olegsu commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Apr 7, 2026

Uh oh!

olegsu commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewkroh Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olegsu commented Apr 14, 2026

Test Matrix

Uh oh!

andrewkroh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olegsu commented Apr 15, 2026

Uh oh!

olegsu commented Apr 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

olegsu commented Apr 7, 2026 •

edited

Loading

olegsu commented Apr 7, 2026 •

edited

Loading

olegsu commented Apr 9, 2026 •

edited

Loading

andrewkroh Apr 10, 2026 •

edited

Loading

andrewkroh left a comment •

edited

Loading