Skip to content

Implement verifier integration #17386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jeniawhite merged 23 commits into from
Mar 24, 2026
Merged

Implement verifier integration #17386

Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
4a65219
Implement verifier integration
jeniawhite Feb 12, 2026
caf957c
Update manifest.yml
jeniawhite Feb 12, 2026
1f7be75
integration version support
jeniawhite Feb 12, 2026
4b0439a
Update CODEOWNERS
jeniawhite Feb 17, 2026
38bdb0c
Merge branch 'main' into evgb-VerifierIntegration
jeniawhite Feb 17, 2026
6cf4b97
Update manifest.yml
jeniawhite Feb 17, 2026
6fd47eb
Merge branch 'evgb-VerifierIntegration' of github.com:elastic/integra…
jeniawhite Feb 17, 2026
8748334
Update README.md
jeniawhite Feb 18, 2026
7026b5c
Update test-eprcheck.expected
jeniawhite Feb 18, 2026
6e4c4b9
Align policy and update manifest
jeniawhite Feb 18, 2026
c62a34c
Your meaningful commit message
jeniawhite Feb 18, 2026
7513af7
Update test-local-config.yml
jeniawhite Feb 18, 2026
d883e02
Update test-local-config.yml
jeniawhite Feb 18, 2026
a3f587f
Merge branch 'main' into evgb-VerifierIntegration
jeniawhite Feb 18, 2026
47431b3
CR comments and namespace
jeniawhite Mar 3, 2026
449fe98
Update README.md
jeniawhite Mar 5, 2026
c41bd90
Align name
jeniawhite Mar 5, 2026
187c61f
Rename datastream
jeniawhite Mar 6, 2026
c6a3938
Update manifest.yml
jeniawhite Mar 10, 2026
d9dc915
Update manifest.yml
jeniawhite Mar 10, 2026
0a5cb5e
Rollback default and align auth with docs
jeniawhite Mar 13, 2026
a5accb8
Align templates
jeniawhite Mar 24, 2026
bc1fd6d
Merge branch 'main' into evgb-VerifierIntegration
jeniawhite Mar 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/CODEOWNERS
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,6 +519,7 @@
/packages/varonis @elastic/security-service-integrations
/packages/vectra_detect @elastic/security-service-integrations
/packages/vectra_rux @elastic/security-service-integrations
/packages/verifier_otel @elastic/cloud-services
/packages/vsphere @elastic/obs-infraobs-integrations
/packages/watchguard_firebox @elastic/integration-experience
/packages/websocket @elastic/security-service-integrations
Expand Down Expand Up @@ -547,4 +548,4 @@
/packages/zoom @elastic/security-service-integrations
/packages/zscaler_zia @elastic/security-service-integrations
/packages/zscaler_zpa @elastic/security-service-integrations
/packages/otel_rum_dashboards @elastic/apm-agent-rum
/packages/otel_rum_dashboards @elastic/apm-agent-rum
93 changes: 93 additions & 0 deletions packages/verifier_otel/LICENSE.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
Elastic License 2.0

URL: https://www.elastic.co/licensing/elastic-license

## Acceptance

By using the software, you agree to all of the terms and conditions below.

## Copyright License

The licensor grants you a non-exclusive, royalty-free, worldwide,
non-sublicensable, non-transferable license to use, copy, distribute, make
available, and prepare derivative works of the software, in each case subject to
the limitations and conditions below.

## Limitations

You may not provide the software to third parties as a hosted or managed
service, where the service provides users with access to any substantial set of
the features or functionality of the software.

You may not move, change, disable, or circumvent the license key functionality
in the software, and you may not remove or obscure any functionality in the
software that is protected by the license key.

You may not alter, remove, or obscure any licensing, copyright, or other notices
of the licensor in the software. Any use of the licensor's trademarks is subject
to applicable law.

## Patents

The licensor grants you a license, under any patent claims the licensor can
license, or becomes able to license, to make, have made, use, sell, offer for
sale, import and have imported the software, in each case subject to the
limitations and conditions in this license. This license does not cover any
patent claims that you cause to be infringed by modifications or additions to
the software. If you or your company make any written claim that the software
infringes or contributes to infringement of any patent, your patent license for
the software granted under these terms ends immediately. If your company makes
such a claim, your patent license ends immediately for work on behalf of your
company.

## Notices

You must ensure that anyone who gets a copy of any part of the software from you
also gets a copy of these terms.

If you modify the software, you must include in any modified copies of the
software prominent notices stating that you have modified the software.

## No Other Rights

These terms do not imply any licenses other than those expressly granted in
these terms.

## Termination

If you use the software in violation of these terms, such use is not licensed,
and your licenses will automatically terminate. If the licensor provides you
with a notice of your violation, and you cease all violation of this license no
later than 30 days after you receive that notice, your licenses will be
reinstated retroactively. However, if you violate these terms after such
reinstatement, any additional violation of these terms will cause your licenses
to terminate automatically and permanently.

## No Liability

*As far as the law allows, the software comes as is, without any warranty or
condition, and the licensor will not be liable to you for any damages arising
out of these terms or the use or nature of the software, under any kind of
legal claim.*

## Definitions

The **licensor** is the entity offering these terms, and the **software** is the
software the licensor makes available under these terms, including any portion
of it.

**you** refers to the individual or entity agreeing to these terms.

**your company** is any legal entity, sole proprietorship, or other kind of
organization that you work for, plus all organizations that have control over,
are under the control of, or are under common control with that
organization. **control** means ownership of substantially all the assets of an
entity, or the power to direct its management and policies by vote, contract, or
otherwise. Control can be direct or indirect.

**your licenses** are all the licenses granted to you for the software under
these terms.

**use** means anything you do with the software requiring one of your licenses.

**trademark** means trademarks, service marks, and similar rights.
256 changes: 256 additions & 0 deletions packages/verifier_otel/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,256 @@
# Permission Verifier Integration

## Overview

The Permission Verifier integration uses the OpenTelemetry Collector's Verifier receiver to verify cloud connector based integration permissions and report results to Elasticsearch.

This integration is designed for Cloud Connectors to proactively check that all necessary permissions are available for attached integrations.

## Supported Providers

| Provider | Status | Description |
|----------|--------|-------------|
| AWS | Active | CloudTrail, GuardDuty, Security Hub, S3, EC2, VPC Flow Logs, WAF, Route53, ELB, CloudFront, CSPM, Asset Inventory |
| Azure | Active | Activity Logs, Audit Logs, Blob Storage, CSPM, Asset Inventory |
| GCP | Active | Audit Logs, Cloud Storage, Pub/Sub, CSPM, Asset Inventory |
| Okta | Planned | System Logs, User Events |

## Configuration

### Cloud Connector Identification

| Field | Required | Description |
|-------|----------|-------------|
| Cloud Connector ID | Yes | Unique identifier for the Cloud Connector being verified |
| Cloud Connector Name | No | Human-readable name of the Cloud Connector |
| Verification ID | Yes | Unique identifier for this verification session |
| Verification Type | No | Type of verification: `on_demand` (default) or `scheduled` |

### Provider and Account Configuration

| Field | Required | Description |
|-------|----------|-------------|
| `provider` | Yes | Cloud provider type (`aws`, `azure`, `gcp`, `okta`) |
| `account_type` | No | Whether the target is a `single_account` (default) or `organization` (management account). Affects which permissions are verified since assuming a role behaves differently for single accounts vs organization management accounts. |

Check notice on line 34 in packages/verifier_otel/_dev/build/docs/README.md

View workflow job for this annotation

GitHub Actions / Lint Documentation 

Elastic.Wordiness: Consider using 'because' instead of 'since'.

Check warning on line 34 in packages/verifier_otel/_dev/build/docs/README.md

View workflow job for this annotation

GitHub Actions / Lint Documentation 

Elastic.Latinisms: Latin terms and abbreviations are a common source of confusion. Use 'versus' instead of 'vs'.

### Credentials

Credential fields use a flat, normalized naming convention to stay consistent with Fleet and avoid translation layers across packages. Only set the fields relevant to your `provider`.

#### AWS Credentials

| Field | Required | Description |
|-------|----------|-------------|
| `credentials_role_arn` | Yes | ARN of the IAM role to assume in the customer's AWS account |
| `credentials_external_id` | Yes | External ID to prevent confused deputy attacks |
| `default_region` | No | Default AWS region for API calls (default: `us-east-1`) |

#### Azure Credentials

| Field | Required | Description |
|-------|----------|-------------|
| `credentials_tenant_id` | Yes | Azure AD tenant ID |
| `credentials_client_id` | Yes | Azure application (client) ID |

#### GCP Credentials

| Field | Required | Description |
|-------|----------|-------------|
| `credentials_project_id` | Yes | GCP project ID to scope verification to |
Comment thread
Omolola-Akinleye marked this conversation as resolved.
| `credentials_workload_identity_provider` | No | Full Workload Identity Federation resource name |
| `credentials_service_account_email` | No | GCP service account email for impersonation |

### Policy Configuration

| Field | Required | Description |
|-------|----------|-------------|
| Policy ID | Yes | The agent policy ID for this set of integrations |
| Policy Name | No | Human-readable name of the policy |

### Integration Configuration (Package Metadata)

Integration identification uses `policy_template` + `package_name` as the composite unique key, aligning with Fleet's package policy API vocabulary. This keeps the Agentless API request self-describing with no lookups needed.

| Field | Required | Description |
|-------|----------|-------------|
| `policy_template` | Yes | Policy template name from the integration package (for example, `cloudtrail`, `guardduty`, `activitylogs`). Not globally unique on its own; must be combined with `package_name`. |
| `package_name` | Yes | Integration package name (for example, `aws`, `azure`, `gcp`, `okta`) |
| `package_title` | No | Human-readable title of the integration package (for example, `AWS`, `Azure`) |
| `package_version` | No | Semantic version of the integration package (for example, `2.17.0`). Different versions can require different permissions. When empty, the latest permission set is used. |
| `package_policy_id` | No | Unique identifier for the package policy instance |
| `namespace` | No | Namespace for the integration (default: `default`) |

## Supported Policy Templates

Each `policy_template` is scoped per integration following the least-privilege principle. Only the permissions required by that specific policy template are verified, rather than checking global permissions shared across the entire integration package. This ensures that each Cloud Connector only needs the exact IAM permissions its attached integrations require.

### AWS Integrations (`package_name: aws`)

| Policy Template | Permissions Verified |
|-----------------|---------------------|
| `cloudtrail` | cloudtrail:LookupEvents, DescribeTrails, s3:GetObject, ListBucket, sqs:ReceiveMessage |
| `guardduty` | guardduty:ListDetectors, GetFindings, ListFindings |
| `securityhub` | securityhub:GetFindings, DescribeHub |
| `s3` | s3:ListBucket, GetObject, GetBucketLocation |
| `ec2` | ec2:DescribeInstances, DescribeRegions, cloudwatch:GetMetricData |
| `vpcflow` | logs:FilterLogEvents, DescribeLogGroups, ec2:DescribeFlowLogs |
| `waf` | wafv2:GetWebACL, ListWebACLs, s3:GetObject |
| `route53` | logs:FilterLogEvents, DescribeLogGroups, route53:ListHostedZones |
| `elb` | s3:GetObject, ListBucket, elasticloadbalancing:DescribeLoadBalancers |
| `cloudfront` | s3:GetObject, ListBucket, cloudfront:ListDistributions |
| `cspm` | SecurityAudit managed policy attachment (policy_attachment_check) |
| `asset_inventory` | SecurityAudit managed policy attachment (policy_attachment_check) |

### Azure Integrations (`package_name: azure`)

| Policy Template | Permissions Verified |
|-----------------|---------------------|
| `activitylogs` | Microsoft.Insights/eventtypes/values/Read |
| `auditlogs` | Microsoft.Insights/eventtypes/values/Read |
| `blob_storage` | Microsoft.Storage/storageAccounts/blobServices/containers/read |
| `cspm` | Reader built-in role assignment (policy_attachment_check) |
| `asset_inventory` | Reader built-in role assignment (policy_attachment_check) |

### GCP Integrations (`package_name: gcp`)

| Policy Template | Permissions Verified |
|-----------------|---------------------|
| `audit` | logging.logEntries.list |
| `storage` | storage.objects.get, storage.objects.list |
| `pubsub` | pubsub.subscriptions.consume |
| `cspm` | roles/cloudasset.viewer, roles/browser IAM bindings (policy_attachment_check) |
| `asset_inventory` | roles/cloudasset.viewer, roles/browser IAM bindings (policy_attachment_check) |

### Okta Integrations (`package_name: okta`) — Planned

| Policy Template | Permissions Verified |
|-----------------|---------------------|
| `system` | okta.logs.read |
| `users` | okta.users.read |

## Output

The integration emits OTEL logs with the following structure:

### Resource Attributes

| Attribute | Description |
|-----------|-------------|
| `cloud_connector.id` | Cloud Connector identifier |
| `cloud_connector.name` | Cloud Connector name |
| `cloud_connector.namespace` | Kibana Space the Cloud Connector belongs to (default: `default`) |
| `data_stream.type` | Always `logs` |
| `data_stream.dataset` | Always `verifier_otel.verification` |
| `data_stream.namespace` | Data stream namespace, matches `cloud_connector.namespace` |
| `verification.id` | Verification session ID |
| `verification.timestamp` | When verification started |
| `verification.type` | `on_demand` or `scheduled` |
| `service.name` | Always `permission-verifier` |
| `service.version` | Service version (for example, `1.0.0`) |

### Log Record Attributes

| Attribute | Description |
|-----------|-------------|
| `policy.id` | Policy identifier |
| `policy.name` | Policy name |
| `policy_template` | Policy template name (for example, `cloudtrail`) |
| `package.name` | Integration package name (for example, `aws`) |
| `package.title` | Integration package title (for example, `AWS`) |
| `package.version` | Integration package version (for example, `2.17.0`) or `unspecified` |
| `package_policy.id` | Package policy instance identifier |
| `provider.type` | Provider type (`aws`, `azure`, `gcp`, `okta`) |
| `provider.account` | Provider account identifier |
| `provider.region` | Provider region |
| `provider.project_id` | GCP project ID (when applicable) |
| `account_type` | `single_account` or `organization` |
| `permission.action` | Permission being checked (for example, `cloudtrail:LookupEvents`) |
| `permission.category` | Permission category (for example, `data_access`) |
| `permission.status` | Result: `granted`, `denied`, `error`, or `skipped` |
| `permission.required` | Whether this permission is required |

@Omolola-Akinleye Omolola-Akinleye Mar 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jeniawhite just one more thought when I sync'd with Oleg. He mentioned there can be IAM Policy with multiple permissions which I didn't think about. What is the relationship between verification and handling IAM Policy/permissions.

For instance,if user could update IAM policy include multiple permissions. Will each verification doc check multiple permissions?

IAM Policy AmazonS3ReadOnlyAccess for Amazon S3.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*",
        "s3:Describe*",
        "s3-object-lambda:Get*",
        "s3-object-lambda:List*"
      ],
      "Resource": "*"
    }
  ]
}

@jeniawhite jeniawhite Mar 5, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current state of the verifier maps the required actions per integration and these are the actions that are being verified and send as events. If we'd like we can verify the IAM roles as well by attempting to assume them (without triggering actions). This would require changes to the verifier by adding an additional type of verification (non API...). We'll need to understand what we want to verify for the integration and have it as part of the registry. If we decide to only assume then there are cons like SCP rules that might block requests and misaligned action configurations. Should I create discussion around this topic? @Omolola-Akinleye @olegsu

@olegsu olegsu Mar 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we ask customers in documentation page that in order for an integration to work properly we need an access to builtit, support by aws role, this is how I expect it to be reflected in case the trust was revoked.

| `permission.error_code` | Error code from provider (if denied/error) |
| `permission.error_message` | Error message from provider (if denied/error) |
| `verification.method` | Method used: `api_call`, `dry_run`, or `policy_attachment_check` |
| `verification.endpoint` | The API endpoint used for verification |
| `verification.duration_ms` | Time taken for verification in milliseconds |
| `verification.verified_at` | ISO 8601 timestamp of when this individual permission check was performed |

## Example Configurations

### AWS Example

```yaml
cloud_connector_id: "cc-12345"
cloud_connector_name: "Production Connector"
verification_id: "verify-abc123"
verification_type: "on_demand"

provider: "aws"
account_type: "single_account"

credentials_role_arn: "arn:aws:iam::123456789012:role/ElasticAgentRole"
credentials_external_id: "elastic-external-id-from-setup"
default_region: "us-east-1"

policy_id: "policy-aws-security"
policy_name: "AWS Security Monitoring"

policy_template: "cloudtrail"
package_name: "aws"
package_title: "AWS"
package_version: "2.17.0"
namespace: "default"
```

### Azure Example

```yaml
cloud_connector_id: "cc-67890"
cloud_connector_name: "Azure Connector"
verification_id: "verify-def456"
verification_type: "on_demand"

provider: "azure"
account_type: "single_account"

credentials_tenant_id: "00000000-0000-0000-0000-000000000000"
credentials_client_id: "11111111-1111-1111-1111-111111111111"

policy_id: "policy-azure-monitoring"
policy_name: "Azure Activity Monitoring"

policy_template: "activitylogs"
package_name: "azure"
package_title: "Azure"
package_version: "1.5.0"
namespace: "default"
```

### GCP Example

```yaml
cloud_connector_id: "cc-gcp-01"
cloud_connector_name: "GCP Connector"
verification_id: "verify-ghi789"
verification_type: "on_demand"

provider: "gcp"
account_type: "single_account"

credentials_project_id: "my-gcp-project-123"
credentials_workload_identity_provider: "//iam.googleapis.com/projects/123/locations/global/workloadIdentityPools/pool/providers/provider"

@Omolola-Akinleye Omolola-Akinleye Mar 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

credentials_workload_identity_provider makes sense but I notice our integrations use term audience can we keep it consistent here as well? I think GCP Docs treats the url as audience as well.

credentials_service_account_email: "verifier@my-gcp-project-123.iam.gserviceaccount.com"

policy_id: "policy-gcp-audit"
policy_name: "GCP Audit Monitoring"

policy_template: "audit"
package_name: "gcp"
package_title: "GCP"
package_version: "1.2.0"
namespace: "default"
```

## Related

- [Verifier Receiver Documentation](https://github.com/elastic/opentelemetry-collector-components/tree/main/receiver/verifierreceiver)
Loading
Loading