Implementing basic authentication for webhook auditlog sink

[sandervandegeijn]

Description

This change adds HTTP Basic Authentication support to the webhook audit log sink, enabling webhook endpoints to require authentication when receiving audit logs from OpenSearch.

• Category: Enhancement
• Why these changes are required?

In OpenSearch 2.x, users could embed credentials in webhook URLs (e.g., https://user:[email protected]), but this approach was deprecated in 3.x due to the Apache HttpComponents Client5 upgrade. The configuration parameters
plugins.security.audit.config.username and plugins.security.audit.config.password were available but only functional for external_opensearch sinks. Webhook sinks had no way to authenticate with secured endpoints, limiting their
use in production environments where webhook endpoints require authentication.

• What is the old behavior before changes and new behavior after changes?

Old Behavior:

• Webhook audit log sinks sent HTTP requests without any authentication headers
• The username and password configuration settings were ignored for webhook sinks
• Users could not use webhooks with endpoints that require Basic Authentication
• Only workaround was embedding credentials in URLs (deprecated in 3.x)

New Behavior:

• Webhook sinks read username and password from configuration settings
• When credentials are configured, a properly formatted HTTP Basic Authentication header is automatically generated and included in all webhook requests
• The Authorization header contains Base64-encoded credentials in the standard format: Authorization: Basic <base64_encoded_credentials>
• Fully backward compatible - webhooks without configured credentials continue to work as before without any Authorization header

Issues Resolved

Closes #5738

Is this a backport? No

Do these changes introduce new permission(s) to be displayed in the static dropdown on the front-end? No

Testing

Unit Testing:

• Added basicAuthPostTest() - Verifies that when username and password are configured, the webhook sink correctly generates a Base64-encoded Basic Auth header and includes it in POST requests. The test also decodes and validates
  the credentials match the configured values.
• Added webhookWithoutAuthTest() - Ensures backward compatibility by verifying that when no credentials are configured, webhook requests are sent without an Authorization header.
• Enhanced TestHttpHandler to capture and expose HTTP request headers for validation in tests.
• All existing webhook audit log tests pass without modification, confirming backward compatibility.

Integration Testing:

• Created test webhook server that captures and displays incoming requests with their headers
• Manually verified that:
  • Basic Auth header is correctly formatted and Base64-encoded
  • Credentials can be successfully decoded from the header
  • Requests without credentials do not include Authorization header
  • Both POST and GET request methods properly include the header when configured

Manual Testing:

• Built the modified security plugin successfully
• Verified the plugin JAR contains the updated WebhookSink class
• Tested configuration with username/password settings
• Confirmed existing configurations without credentials continue to work

Test Results:

• All existing WebhookAuditLogTest tests pass (8 tests)
• New authentication tests pass
• Full test suite: BUILD SUCCESSFUL

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request https://github.com/opensearch-project/opensearch-api-specification/blob/main/DEVELOPER_GUIDE.md
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check https://github.com/opensearch-project/security/blob/main/CONTRIBUTING.md#developer-certificate-of-origin.

---

Configuration Example:

plugins.security.audit.type: webhook
plugins.security.audit.config.webhook.url: "https://your-webhook-endpoint.com/audit"
plugins.security.audit.config.webhook.format: "json"
plugins.security.audit.config.username: "audit_user"
plugins.security.audit.config.password: "secure_password"

[nibix]

Looks quite good!

I think we also need to update the docs at https://docs.opensearch.org/latest/security/audit-logs/storage-types/ ... would you also be willing to file a PR for that?

[codecov]

Codecov Report

❌ Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 73.51%. Comparing base (190b7c7) to head (3c66c9e).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...opensearch/security/auditlog/sink/WebhookSink.java	91.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5792   +/-   ##
=======================================
  Coverage   73.51%   73.51%           
=======================================
  Files         438      438           
  Lines       26710    26722   +12     
  Branches     3956     3959    +3     
=======================================
+ Hits        19637    19646    +9     
- Misses       5195     5197    +2     
- Partials     1878     1879    +1     

Files with missing lines	Coverage Δ
.../opensearch/security/OpenSearchSecurityPlugin.java	85.38% <ø> (ø)
...security/auditlog/sink/ExternalOpenSearchSink.java	62.36% <100.00%> (ø)
...g/opensearch/security/support/ConfigConstants.java	77.27% <ø> (ø)
...opensearch/security/auditlog/sink/WebhookSink.java	77.29% <91.66%> (+0.99%)	⬆️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sandervandegeijn]

@nibix are the CI jobs stuck by any chance? Was waiting to see if my fixes were ok :)

[DarshitChanpura]

@sandervandegeijn thank you for adding this feature!

Would you please run ./gradlew spotlessApply to fix errors from spotless scan?

[sandervandegeijn]

The test suite doesn't seen entirely stable, look like the changes are ok?

[cwperks]

@nibix are the CI jobs stuck by any chance? Was waiting to see if my fixes were ok :)

A maintainer needs to approve them to run for first time contributors. After your first PR is merged, they then run automatically on push.

[cwperks]

Thank you for the PR @sandervandegeijn ! The changes LGTM. Claude code knows this code base quite well at this point..one of the giveaways that a code block (particularly a test) may have been autogenerated is the use of full qualified classnames inline rather than top of file imports. In this case, it added some good tests that cover all of Webhook POST, GET and when auth is not configured at all which is what I would expect for a contribution like this.

[DarshitChanpura]

CI fails dues to changes from here: opensearch-project/OpenSearch#19803