Introduce verifier receiver to EDOT #12728
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| # Kind can be one of: | ||
| # - breaking-change: a change to previously-documented behavior | ||
| # - deprecation: functionality that is being removed in a later release | ||
| # - bug-fix: fixes a problem in a previous version | ||
| # - enhancement: extends functionality but does not break or fix existing behavior | ||
| # - feature: new functionality | ||
| # - known-issue: problems that we are aware of in a given version | ||
| # - security: impacts on the security of a product or a user's deployment. | ||
| # - upgrade: important information for someone upgrading from a prior version | ||
| # - other: does not fit into any of the other categories | ||
| kind: feature | ||
|
|
||
| # Change summary; a 80ish characters long description of the change. | ||
| summary: Add verifierreceiver for cloud integration permission verification | ||
|
|
||
| # Long description; in case the summary is not enough to describe the change | ||
| # this field accommodate a description without length limits. | ||
| # NOTE: This field will be rendered only for breaking-change and known-issue kinds at the moment. | ||
| description: | | ||
| Add verifierreceiver to EDOT collector. This receiver verifies integration | ||
| permissions (Cloud Connector integrations) and emits structured logs | ||
| with permission verification results. | ||
|
|
||
| # Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc. | ||
| component: "elastic-agent" | ||
|
|
||
| # PR URL; optional; the PR number that added the changeset. | ||
| # If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added. | ||
| # NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number. | ||
| # Please provide it if you are adding a fragment for a different PR. | ||
| #pr: https://github.com/elastic/elastic-agent/pull/XXXX | ||
|
|
||
| # Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of). | ||
| # If not present is automatically filled by the tooling with the issue linked to the PR number. | ||
| #issue: https://github.com/elastic/opentelemetry-dev/issues/XXX |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,209 @@ | ||
| --- | ||
| navigation_title: Verifier receiver | ||
| description: The Verifier receiver is an OpenTelemetry Collector component that verifies permissions for integrations and reports results as OTEL logs. | ||
| applies_to: | ||
| stack: | ||
| serverless: | ||
| observability: | ||
| product: | ||
| edot_collector: | ||
| products: | ||
| - id: elastic-agent | ||
| - id: observability | ||
| - id: edot-collector | ||
| --- | ||
|
|
||
| # Verifier receiver | ||
|
|
||
| The Verifier receiver is an OpenTelemetry Collector component that verifies permissions for integrations (such as AWS CloudTrail, GuardDuty, S3 and others) and reports verification results as OTEL logs. It is used with Cloud Connectors to confirm that the configured IAM roles and credentials have the permissions required by each integration. | ||
|
|
||
| :::{important} | ||
| This receiver is in **development** stability. AWS is supported; Azure, GCP, and Okta are planned. | ||
| ::: | ||
|
|
||
| ## Get started | ||
|
|
||
| To use the Verifier receiver, include it in the receiver definitions of the [Collector configuration](/reference/edot-collector/config/index.md) and wire it into a logs pipeline: | ||
|
|
||
| ```yaml | ||
| receivers: | ||
| verifier: | ||
| cloud_connector_id: "cc-12345" | ||
| cloud_connector_name: "Production Connector" | ||
| verification_id: "verify-abc123" | ||
| verification_type: "on_demand" | ||
| providers: | ||
| aws: | ||
| credentials: | ||
| use_default_credentials: true | ||
| default_region: "us-east-1" | ||
| policies: | ||
| - policy_id: "policy-1" | ||
| policy_name: "AWS Security Monitoring" | ||
| integrations: | ||
| - integration_id: "int-cloudtrail-001" | ||
| integration_type: "aws_cloudtrail" | ||
| integration_name: "AWS CloudTrail" | ||
| integration_version: "2.17.0" | ||
| config: | ||
| account_id: "123456789012" | ||
| region: "us-east-1" | ||
|
|
||
| service: | ||
| pipelines: | ||
| logs: | ||
| receivers: [verifier] | ||
| exporters: [elasticsearch] | ||
| ``` | ||
|
|
||
| ## Configuration | ||
|
|
||
| The receiver uses a Cloud Connector-oriented configuration: you specify the connector identity, a verification session, provider credentials, and a list of policies with their integrations. The receiver looks up required permissions per integration type and version, then performs the checks. | ||
|
|
||
| ### Required fields | ||
|
|
||
| | Option | Type | Description | | ||
| |--------|------|-------------| | ||
| | `cloud_connector_id` | `string` | Unique identifier for the Cloud Connector. | | ||
| | `verification_id` | `string` | Unique identifier for this verification session. | | ||
| | `policies` | `[]PolicyConfig` | List of policies; each policy must have at least one integration with `integration_type` set. | | ||
|
|
||
| ### Optional fields | ||
|
|
||
| | Option | Type | Default | Description | | ||
| |--------|------|---------|-------------| | ||
| | `cloud_connector_name` | `string` | - | Human-readable name of the Cloud Connector. | | ||
| | `verification_type` | `string` | `on_demand` | `on_demand` or `scheduled`. | | ||
| | `providers` | `ProvidersConfig` | - | Credentials for AWS, and (when available) Azure, GCP, Okta. | | ||
|
|
||
| ### AWS credentials (`providers.aws.credentials`) | ||
|
|
||
| For production use with a Cloud Connector, use STS AssumeRole: | ||
|
|
||
| ```yaml | ||
| receivers: | ||
| verifier: | ||
| cloud_connector_id: "${CLOUD_CONNECTOR_ID}" | ||
| verification_id: "${VERIFICATION_ID}" | ||
| providers: | ||
| aws: | ||
| credentials: | ||
| role_arn: "arn:aws:iam::123456789012:role/ElasticAgentRole" | ||
| external_id: "elastic-external-id-from-setup" | ||
| default_region: "us-east-1" | ||
| policies: [] | ||
| ``` | ||
|
|
||
| For local testing, you can use the default credential chain: | ||
|
|
||
| ```yaml | ||
| providers: | ||
| aws: | ||
| credentials: | ||
| use_default_credentials: true | ||
| default_region: "us-east-1" | ||
| ``` | ||
|
|
||
| | Option | Type | Required | Description | | ||
| |--------|------|----------|-------------| | ||
| | `role_arn` | `string` | Yes* | IAM role ARN to assume. | | ||
| | `external_id` | `string` | Yes* | External ID for assume-role. | | ||
| | `default_region` | `string` | No | Default AWS region (for example `us-east-1`). | | ||
| | `use_default_credentials` | `bool` | No | Use default credential chain (for testing). | | ||
|
|
||
| *Required when not using `use_default_credentials`. | ||
|
|
||
| ### Policy and integration structure | ||
|
|
||
| Each policy must have `policy_id` and at least one integration. Each integration must specify `integration_type` (for example `aws_cloudtrail`, `aws_guardduty`, `aws_s3`). Optional `integration_id`, `integration_name`, `integration_version`, and `config` provide context and provider-specific settings. | ||
|
|
||
| The `integration_version` field accepts a semantic version string (for example `2.17.0`). Different versions of an integration may require different permissions. When omitted, the latest registered permission set is used. | ||
|
|
||
| ```yaml | ||
| policies: | ||
| - policy_id: "policy-1" | ||
| policy_name: "AWS Security Monitoring" | ||
| integrations: | ||
| - integration_id: "int-cloudtrail-001" | ||
| integration_type: "aws_cloudtrail" | ||
| integration_name: "AWS CloudTrail" | ||
| integration_version: "2.17.0" | ||
| config: | ||
| account_id: "123456789012" | ||
| region: "us-east-1" | ||
| - integration_id: "int-guardduty-001" | ||
| integration_type: "aws_guardduty" | ||
| integration_name: "AWS GuardDuty" | ||
| integration_version: "1.5.0" | ||
| config: | ||
| account_id: "123456789012" | ||
| region: "us-east-1" | ||
| ``` | ||
|
|
||
| ## Supported integration types (AWS) | ||
|
|
||
| | Integration type | Permissions verified | | ||
| |------------------|----------------------| | ||
| | `aws_cloudtrail` | CloudTrail, S3, SQS (for example `LookupEvents`, `GetObject`, `ReceiveMessage`) | | ||
| | `aws_guardduty` | GuardDuty (for example `ListDetectors`, `GetFindings`, `ListFindings`) | | ||
| | `aws_securityhub` | Security Hub (for example `GetFindings`, `DescribeHub`) | | ||
| | `aws_s3` | S3 (for example `ListBucket`, `GetObject`, `GetBucketLocation`) | | ||
| | `aws_ec2` | EC2 and CloudWatch (for example `DescribeInstances`, `DescribeRegions`, `GetMetricData`) | | ||
| | `aws_vpcflow` | VPC Flow Logs (for example `FilterLogEvents`, `DescribeLogGroups`, `DescribeFlowLogs`) | | ||
| | `aws_waf` | WAFv2 and S3 | | ||
| | `aws_route53` | Route53 and CloudWatch Logs | | ||
| | `aws_elb` | ELB and S3 | | ||
| | `aws_cloudfront` | CloudFront and S3 | | ||
|
|
||
| Azure, GCP, and Okta integration types are planned. | ||
|
|
||
| ## Output | ||
|
|
||
| The receiver emits OTEL logs. Each log record represents a permission verification result. Resource and log attributes include: | ||
|
|
||
| - **Resource**: `cloud_connector.id`, `cloud_connector.name`, `verification.id`, `verification.type`, `service.name` (`permission-verifier`) | ||
| - **Log attributes**: `policy.id`, `policy.name`, `integration.id`, `integration.type`, `integration.version`, `provider.type`, `permission.action`, `permission.status` (`granted` / `denied` / `error` / `skipped`), `permission.error_code`, `permission.error_message`, `verification.method`, `verification.duration_ms` | ||
|
|
||
| Export these logs to {{es}} (for example with the Elasticsearch exporter) and use the `logs-cloud_connector.permission_verification-*` data stream or a custom index for dashboards and alerts. | ||
|
|
||
| ## Example pipeline | ||
|
|
||
| ```yaml | ||
| receivers: | ||
| verifier: | ||
| cloud_connector_id: "${CLOUD_CONNECTOR_ID}" | ||
| verification_id: "${VERIFICATION_ID}" | ||
| providers: | ||
| aws: | ||
| credentials: | ||
| role_arn: "${AWS_ROLE_ARN}" | ||
| external_id: "${AWS_EXTERNAL_ID}" | ||
| default_region: "us-east-1" | ||
| policies: | ||
| - policy_id: "policy-1" | ||
| policy_name: "AWS Security Monitoring" | ||
| integrations: | ||
| - integration_id: "int-cloudtrail-001" | ||
| integration_type: "aws_cloudtrail" | ||
| integration_name: "AWS CloudTrail" | ||
| integration_version: "2.17.0" | ||
| config: | ||
| region: "us-east-1" | ||
|
|
||
| exporters: | ||
| elasticsearch: | ||
| endpoints: ["${ES_ENDPOINT}"] | ||
| api_key: "${ES_API_KEY}" | ||
| logs_index: "logs-cloud_connector.permission_verification-default" | ||
|
|
||
| service: | ||
| pipelines: | ||
| logs: | ||
| receivers: [verifier] | ||
| exporters: [elasticsearch] | ||
| ``` | ||
|
|
||
| ## Resources | ||
|
|
||
| * [Verifier receiver source](https://github.com/elastic/opentelemetry-collector-components/tree/main/receiver/verifierreceiver) | ||
| * [Configure logs collection in EDOT](../config/configure-logs-collection.md) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will be changed once verifier will get merged