Implement verifier integration #17386
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| Elastic License 2.0 | ||
|
|
||
| URL: https://www.elastic.co/licensing/elastic-license | ||
|
|
||
| ## Acceptance | ||
|
|
||
| By using the software, you agree to all of the terms and conditions below. | ||
|
|
||
| ## Copyright License | ||
|
|
||
| The licensor grants you a non-exclusive, royalty-free, worldwide, | ||
| non-sublicensable, non-transferable license to use, copy, distribute, make | ||
| available, and prepare derivative works of the software, in each case subject to | ||
| the limitations and conditions below. | ||
|
|
||
| ## Limitations | ||
|
|
||
| You may not provide the software to third parties as a hosted or managed | ||
| service, where the service provides users with access to any substantial set of | ||
| the features or functionality of the software. | ||
|
|
||
| You may not move, change, disable, or circumvent the license key functionality | ||
| in the software, and you may not remove or obscure any functionality in the | ||
| software that is protected by the license key. | ||
|
|
||
| You may not alter, remove, or obscure any licensing, copyright, or other notices | ||
| of the licensor in the software. Any use of the licensor's trademarks is subject | ||
| to applicable law. | ||
|
|
||
| ## Patents | ||
|
|
||
| The licensor grants you a license, under any patent claims the licensor can | ||
| license, or becomes able to license, to make, have made, use, sell, offer for | ||
| sale, import and have imported the software, in each case subject to the | ||
| limitations and conditions in this license. This license does not cover any | ||
| patent claims that you cause to be infringed by modifications or additions to | ||
| the software. If you or your company make any written claim that the software | ||
| infringes or contributes to infringement of any patent, your patent license for | ||
| the software granted under these terms ends immediately. If your company makes | ||
| such a claim, your patent license ends immediately for work on behalf of your | ||
| company. | ||
|
|
||
| ## Notices | ||
|
|
||
| You must ensure that anyone who gets a copy of any part of the software from you | ||
| also gets a copy of these terms. | ||
|
|
||
| If you modify the software, you must include in any modified copies of the | ||
| software prominent notices stating that you have modified the software. | ||
|
|
||
| ## No Other Rights | ||
|
|
||
| These terms do not imply any licenses other than those expressly granted in | ||
| these terms. | ||
|
|
||
| ## Termination | ||
|
|
||
| If you use the software in violation of these terms, such use is not licensed, | ||
| and your licenses will automatically terminate. If the licensor provides you | ||
| with a notice of your violation, and you cease all violation of this license no | ||
| later than 30 days after you receive that notice, your licenses will be | ||
| reinstated retroactively. However, if you violate these terms after such | ||
| reinstatement, any additional violation of these terms will cause your licenses | ||
| to terminate automatically and permanently. | ||
|
|
||
| ## No Liability | ||
|
|
||
| *As far as the law allows, the software comes as is, without any warranty or | ||
| condition, and the licensor will not be liable to you for any damages arising | ||
| out of these terms or the use or nature of the software, under any kind of | ||
| legal claim.* | ||
|
|
||
| ## Definitions | ||
|
|
||
| The **licensor** is the entity offering these terms, and the **software** is the | ||
| software the licensor makes available under these terms, including any portion | ||
| of it. | ||
|
|
||
| **you** refers to the individual or entity agreeing to these terms. | ||
|
|
||
| **your company** is any legal entity, sole proprietorship, or other kind of | ||
| organization that you work for, plus all organizations that have control over, | ||
| are under the control of, or are under common control with that | ||
| organization. **control** means ownership of substantially all the assets of an | ||
| entity, or the power to direct its management and policies by vote, contract, or | ||
| otherwise. Control can be direct or indirect. | ||
|
|
||
| **your licenses** are all the licenses granted to you for the software under | ||
| these terms. | ||
|
|
||
| **use** means anything you do with the software requiring one of your licenses. | ||
|
|
||
| **trademark** means trademarks, service marks, and similar rights. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,156 @@ | ||
| # Permission Verifier Integration | ||
|
|
||
| ## Overview | ||
|
|
||
| The Permission Verifier integration uses the OpenTelemetry Collector's Verifier receiver to verify cloud connector based integration permissions and report results to Elasticsearch. | ||
|
|
||
| This integration is designed for Cloud Connectors to proactively check that all necessary permissions are available for attached integrations. | ||
|
|
||
| ## Supported Providers | ||
|
|
||
| | Provider | Status | Description | | ||
| |----------|--------|-------------| | ||
| | AWS | Active | CloudTrail, GuardDuty, Security Hub, S3, EC2, VPC Flow Logs, WAF, Route53, ELB, CloudFront | | ||
| | Azure | Planned | Activity Logs, Audit Logs, Blob Storage | | ||
| | GCP | Planned | Audit Logs, Cloud Storage, Pub/Sub | | ||
| | Okta | Planned | System Logs, User Events | | ||
|
|
||
| ## Configuration | ||
|
|
||
| ### Cloud Connector Identification | ||
|
|
||
| | Field | Required | Description | | ||
| |-------|----------|-------------| | ||
| | Cloud Connector ID | Yes | Unique identifier for the Cloud Connector being verified | | ||
| | Cloud Connector Name | No | Human-readable name of the Cloud Connector | | ||
| | Verification ID | Yes | Unique identifier for this verification session | | ||
| | Verification Type | No | Type of verification: `on_demand` (default) or `scheduled` | | ||
|
|
||
| ### AWS Provider Authentication | ||
|
|
||
| | Field | Required | Description | | ||
| |-------|----------|-------------| | ||
| | `providers.aws.credentials.role_arn` | Yes (for AWS) | ARN of the IAM role to assume in the customer's AWS account | | ||
| | `providers.aws.credentials.external_id` | Yes (for AWS) | External ID to prevent confused deputy attacks | | ||
| | `providers.aws.credentials.default_region` | No | Default AWS region for API calls (default: `us-east-1`) | | ||
|
|
||
| ### Policy Configuration | ||
|
|
||
| | Field | Required | Description | | ||
| |-------|----------|-------------| | ||
| | Policy ID | Yes | The agent policy ID for this set of integrations | | ||
| | Policy Name | No | Human-readable name of the policy | | ||
|
|
||
| ### Integration Configuration | ||
|
|
||
| | Field | Required | Description | | ||
| |-------|----------|-------------| | ||
| | Integration Type | Yes | Package type (for example, `aws_cloudtrail`, `azure_activitylogs`, `gcp_audit`, `okta_system`) | | ||
| | Integration ID | No | Unique identifier for the integration instance | | ||
| | Integration Name | No | Human-readable name of the integration | | ||
| | Integration Version | No | Semantic version of the integration package (for example, `2.17.0`). Different versions can require different permissions. When empty, the latest permission set is used. | | ||
|
|
||
| ## Supported Integration Types | ||
|
|
||
| ### AWS Integrations | ||
|
|
||
| | Integration Type | Permissions Verified | | ||
| |-----------------|---------------------| | ||
| | `aws_cloudtrail` | cloudtrail:LookupEvents, DescribeTrails, s3:GetObject, ListBucket, sqs:ReceiveMessage | | ||
| | `aws_guardduty` | guardduty:ListDetectors, GetFindings, ListFindings | | ||
| | `aws_securityhub` | securityhub:GetFindings, DescribeHub | | ||
| | `aws_s3` | s3:ListBucket, GetObject, GetBucketLocation | | ||
| | `aws_ec2` | ec2:DescribeInstances, DescribeRegions, cloudwatch:GetMetricData | | ||
| | `aws_vpcflow` | logs:FilterLogEvents, DescribeLogGroups, ec2:DescribeFlowLogs | | ||
| | `aws_waf` | wafv2:GetWebACL, ListWebACLs, s3:GetObject | | ||
| | `aws_route53` | logs:FilterLogEvents, DescribeLogGroups, route53:ListHostedZones | | ||
| | `aws_elb` | s3:GetObject, ListBucket, elasticloadbalancing:DescribeLoadBalancers | | ||
| | `aws_cloudfront` | s3:GetObject, ListBucket, cloudfront:ListDistributions | | ||
|
|
||
| ### Azure Integrations (Planned) | ||
|
|
||
| | Integration Type | Permissions Verified | | ||
| |-----------------|---------------------| | ||
| | `azure_activitylogs` | Microsoft.Insights/eventtypes/values/Read | | ||
| | `azure_auditlogs` | Microsoft.Insights/eventtypes/values/Read | | ||
| | `azure_blob_storage` | Microsoft.Storage/storageAccounts/blobServices/containers/read | | ||
|
|
||
| ### GCP Integrations (Planned) | ||
|
|
||
| | Integration Type | Permissions Verified | | ||
| |-----------------|---------------------| | ||
| | `gcp_audit` | logging.logEntries.list | | ||
| | `gcp_storage` | storage.objects.get, storage.objects.list | | ||
| | `gcp_pubsub` | pubsub.subscriptions.consume | | ||
|
|
||
| ### Okta Integrations (Planned) | ||
|
|
||
| | Integration Type | Permissions Verified | | ||
| |-----------------|---------------------| | ||
| | `okta_system` | okta.logs.read | | ||
| | `okta_users` | okta.users.read | | ||
|
|
||
| ## Output | ||
|
|
||
| The integration emits OTEL logs with the following structure: | ||
|
|
||
| ### Resource Attributes | ||
|
|
||
| | Attribute | Description | | ||
| |-----------|-------------| | ||
| | `cloud_connector.id` | Cloud Connector identifier | | ||
| | `cloud_connector.name` | Cloud Connector name | | ||
| | `verification.id` | Verification session ID | | ||
| | `verification.timestamp` | When verification started | | ||
| | `verification.type` | `on_demand` or `scheduled` | | ||
| | `service.name` | Always `permission-verifier` | | ||
|
|
||
| ### Log Record Attributes | ||
|
|
||
| | Attribute | Description | | ||
| |-----------|-------------| | ||
| | `policy.id` | Policy identifier | | ||
| | `policy.name` | Policy name | | ||
| | `integration.id` | Integration instance identifier | | ||
| | `integration.name` | Integration name | | ||
| | `integration.type` | Integration type (for example, `aws_cloudtrail`) | | ||
| | `integration.version` | Integration package version (for example, `2.17.0`) or `unspecified` | | ||
| | `provider.type` | Provider type (`aws`, `azure`, `gcp`, `okta`) | | ||
| | `provider.account` | Provider account identifier | | ||
| | `provider.region` | Provider region | | ||
| | `permission.action` | Permission being checked (for example, `cloudtrail:LookupEvents`) | | ||
| | `permission.category` | Permission category (for example, `data_access`) | | ||
| | `permission.status` | Result: `granted`, `denied`, `error`, or `skipped` | | ||
| | `permission.required` | Whether this permission is required | | ||
|
Contributor
There was a problem hiding this comment. @jeniawhite just one more thought when I sync'd with Oleg. He mentioned there can be IAM Policy with multiple permissions which I didn't think about. What is the relationship between verification and handling IAM Policy/permissions. For instance,if user could update IAM policy include multiple permissions. Will each verification doc check multiple permissions? IAM Policy
Contributor
Author
There was a problem hiding this comment. Current state of the verifier maps the required actions per integration and these are the actions that are being verified and send as events. If we'd like we can verify the IAM roles as well by attempting to assume them (without triggering actions). This would require changes to the verifier by adding an additional type of verification (non API...). We'll need to understand what we want to verify for the integration and have it as part of the registry. If we decide to only assume then there are cons like SCP rules that might block requests and misaligned action configurations. Should I create discussion around this topic? @Omolola-Akinleye @olegsu
Contributor
There was a problem hiding this comment. When we ask customers in documentation page that in order for an integration to work properly we need an access to builtit, support by aws role, this is how I expect it to be reflected in case the trust was revoked. |
||
| | `permission.error_code` | Error code from provider (if denied/error) | | ||
| | `permission.error_message` | Error message from provider (if denied/error) | | ||
| | `verification.method` | Method used: `api_call` or `dry_run` | | ||
| | `verification.endpoint` | The API endpoint used for verification | | ||
| | `verification.duration_ms` | Time taken for verification in milliseconds | | ||
|
|
||
| ## Example Configuration | ||
|
|
||
| ```yaml | ||
| cloud_connector_id: "cc-12345" | ||
| cloud_connector_name: "Production Connector" | ||
| verification_id: "verify-abc123" | ||
| verification_type: "on_demand" | ||
|
|
||
| # AWS Provider Authentication | ||
| providers.aws.credentials.role_arn: "arn:aws:iam::123456789012:role/ElasticAgentRole" | ||
| providers.aws.credentials.external_id: "elastic-external-id-from-setup" | ||
| providers.aws.credentials.default_region: "us-east-1" | ||
|
|
||
| # Policy | ||
| policy_id: "policy-aws-security" | ||
| policy_name: "AWS Security Monitoring" | ||
|
|
||
| # Integration | ||
| integration_type: "aws_cloudtrail" | ||
| integration_name: "AWS CloudTrail" | ||
| integration_version: "2.17.0" | ||
| ``` | ||
|
|
||
| ## Related | ||
|
|
||
| - [Verifier Receiver Documentation](https://github.com/elastic/opentelemetry-collector-components/tree/main/receiver/verifierreceiver) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| connectors: | ||
| forward: {} | ||
| exporters: | ||
| elasticsearch/componentid-0: | ||
| endpoints: | ||
| - https://elasticsearch:9200 | ||
| inputs: [] | ||
| output_permissions: | ||
| default: | ||
| _elastic_agent_checks: | ||
| cluster: | ||
| - monitor | ||
| _elastic_agent_monitoring: | ||
| indices: [] | ||
| uuid-for-permissions-on-related-indices: | ||
| indices: | ||
| - names: | ||
| - logs-*-* | ||
| privileges: | ||
| - auto_configure | ||
| - create_doc | ||
| processors: | ||
| transform/componentid-0: | ||
| log_statements: | ||
| - context: log | ||
| statements: | ||
| - set(attributes["data_stream.type"], "logs") | ||
| - set(attributes["data_stream.dataset"], "verifier_otel.verifier") | ||
| - set(attributes["data_stream.namespace"], "ep") | ||
| receivers: | ||
| verifier/componentid-0: | ||
| cloud_connector_id: test-cc-123 | ||
| cloud_connector_name: Test Connector | ||
| policies: | ||
| - integrations: | ||
| - integration_id: test-int-789 | ||
| integration_name: Test CloudTrail | ||
| integration_type: aws_cloudtrail | ||
| integration_version: "2.17.0" | ||
| policy_id: test-policy-123 | ||
| policy_name: Test Policy | ||
| providers: | ||
| aws: | ||
| credentials: | ||
| default_region: us-east-1 | ||
| external_id: ${SECRET_0} | ||
| role_arn: arn:aws:iam::123456789012:role/TestRole | ||
| verification_id: test-verify-456 | ||
| verification_type: on_demand | ||
| secret_references: | ||
| - {} | ||
| service: | ||
| pipelines: | ||
| logs: | ||
| exporters: | ||
| - elasticsearch/componentid-0 | ||
| receivers: | ||
| - forward | ||
| logs/componentid-0: | ||
| exporters: | ||
| - forward | ||
| processors: | ||
| - transform/componentid-0 | ||
| receivers: | ||
| - verifier/componentid-0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| vars: | ||
| cloud_connector_id: test-cc-123 | ||
| cloud_connector_name: Test Connector | ||
| verification_id: test-verify-456 | ||
| verification_type: on_demand | ||
| # AWS Provider Credentials | ||
| providers.aws.credentials.role_arn: arn:aws:iam::123456789012:role/TestRole | ||
| providers.aws.credentials.external_id: test-external-id | ||
|
Contributor
There was a problem hiding this comment. Assuming a role behaves differently depending on whether you're targeting a single account vs an organization (management) account in cloudbeat. How about when verifying permissions? Do you need additional |
||
| providers.aws.credentials.default_region: us-east-1 | ||
| # Policy | ||
| policy_id: test-policy-123 | ||
| policy_name: Test Policy | ||
| # Integration | ||
| integration_type: aws_cloudtrail | ||
| integration_id: test-int-789 | ||
| integration_name: Test CloudTrail | ||
| integration_version: "2.17.0" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| skip: | ||
| reason: "Requires real AWS credentials to verify integration permissions" | ||
| link: https://github.com/elastic/security-team/issues/16017 | ||
| vars: | ||
| cloud_connector_id: system-test-cc | ||
| cloud_connector_name: System Test Connector | ||
| verification_id: system-test-verify | ||
| verification_type: on_demand | ||
| providers.aws.credentials.role_arn: arn:aws:iam::123456789012:role/SystemTestRole | ||
| providers.aws.credentials.external_id: system-test-external-id | ||
| providers.aws.credentials.default_region: us-east-1 | ||
| policy_id: system-test-policy | ||
| policy_name: System Test Policy | ||
| integration_type: aws_cloudtrail | ||
| integration_id: system-test-int | ||
| integration_name: System Test CloudTrail |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So is the permission matrix base on least-privilege principle so each permission will be scope to integration type as opposed global permissions shared by integration package?