-
Notifications
You must be signed in to change notification settings - Fork 643
docs: Edit ECS tutorial to add new steps for the otelcol.receiver.awsecscontainermetrics component
#5347
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: Edit ECS tutorial to add new steps for the otelcol.receiver.awsecscontainermetrics component
#5347
Changes from 4 commits
d73ce59
2cdc3a0
35dd2b4
04b47ee
5f454c0
b6cb793
570c2a3
006d85f
f353a36
683b74e
dc838ae
7d27954
80ca81a
59ea9de
840b322
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -22,20 +22,20 @@ You can also collect logs and traces from your applications instrumented for Pro | |||||
|
|
||||||
| ## Before you begin | ||||||
|
|
||||||
| * Ensure that you have basic familiarity with instrumenting applications with OpenTelemetry. | ||||||
| * Have an available Amazon ECS or AWS Fargate deployment. | ||||||
| * Identify where {{< param "PRODUCT_NAME" >}} writes received telemetry data. | ||||||
| * Be familiar with the concept of [Components][] in {{< param "PRODUCT_NAME" >}}. | ||||||
| - Ensure that you have basic familiarity with instrumenting applications with OpenTelemetry. | ||||||
| - Have an available Amazon ECS or AWS Fargate deployment. | ||||||
| - Identify where {{< param "PRODUCT_NAME" >}} writes received telemetry data. | ||||||
| - Familiarize yourself with the concept of [Components][] in {{< param "PRODUCT_NAME" >}}. | ||||||
|
|
||||||
| ## Collect task and container metrics | ||||||
|
|
||||||
| In this configuration, you add an OpenTelemetry Collector to the task running your application, and it uses the ECS Metadata Endpoint to gather task and container metrics in your cluster. | ||||||
|
|
||||||
| You can choose between two collector implementations: | ||||||
|
|
||||||
| - You can use ADOT, the AWS OpenTelemetry collector. ADOT has native support for scraping task and container metrics. ADOT comes with default configurations that can be selected in the task definition. | ||||||
| - You can use ADOT, the AWS OpenTelemetry collector. ADOT has native support for scraping task and container metrics. ADOT comes with default configurations that you can select in the task definition. | ||||||
|
|
||||||
| - Alternatively, you can use {{< param "PRODUCT_NAME" >}} as a collector alongside the [Prometheus ECS exporter](https://github.com/prometheus-community/ecs_exporter) which exposes the ECS metadata endpoint metrics in Prometheus format. | ||||||
| - You can use {{< param "PRODUCT_NAME" >}} with either the [`otelcol.receiver.awsecscontainermetrics`][awsecscontainermetrics] component or the [Prometheus ECS exporter](https://github.com/prometheus-community/ecs_exporter) sidecar. | ||||||
|
|
||||||
| ### Configure ADOT | ||||||
|
|
||||||
|
|
@@ -44,10 +44,10 @@ If you use ADOT as a collector, add a container to your task definition and use | |||||
| You can find sample OpenTelemetry configuration files in the [AWS Observability repository][otel-templates]. | ||||||
| You can use these samples as a starting point and add the appropriate exporter configuration to send metrics to a Prometheus or OpenTelemetry endpoint. | ||||||
|
|
||||||
| * Use [`ecs-default-config`][ecs-default-config] to consume StatsD metrics, OTLP metrics and traces, and AWS X-Ray SDK traces. | ||||||
| * Use [`otel-task-metrics-config`][otel-task-metrics-config] to consume StatsD, OTLP, AWS X-Ray, and Container Resource utilization metrics. | ||||||
| - Use [`ecs-default-config`][ecs-default-config] to consume StatsD metrics, OTLP metrics and traces, and AWS X-Ray SDK traces. | ||||||
| - Use [`otel-task-metrics-config`][otel-task-metrics-config] to consume StatsD, OTLP, AWS X-Ray, and Container Resource utilization metrics. | ||||||
|
|
||||||
| Read [`otel-prometheus`][otel-prometheus] to find out how to set the Prometheus remote write (AWS managed Prometheus in the example). | ||||||
| Read [`otel-prometheus`][otel-prometheus] to find out how to set the Prometheus remote write. The example uses AWS managed Prometheus. | ||||||
|
|
||||||
| Complete the following steps to create a sample task. Refer to the [ADOT doc][adot-doc] for more information. | ||||||
|
|
||||||
|
|
@@ -58,55 +58,108 @@ Complete the following steps to create a sample task. Refer to the [ADOT doc][ad | |||||
| 1. Choose **Create parameter**. | ||||||
| 1. Create a parameter with the following values: | ||||||
|
|
||||||
| * **Name:** `collector-config` | ||||||
| * **Tier:** Standard | ||||||
| * **Type:** String | ||||||
| * **Data type:** Text | ||||||
| * **Value:** Copy and paste your custom OpenTelemetry configuration file. | ||||||
| - **Name:** `collector-config` | ||||||
| - **Tier:** Standard | ||||||
| - **Type:** String | ||||||
| - **Data type:** Text | ||||||
| - **Value:** Copy and paste your custom OpenTelemetry configuration file. | ||||||
|
|
||||||
| 1. Download the [ECS Fargate][fargate-template] or [ECS EC2][ec2-template] task definition template from GitHub. | ||||||
| 1. Edit the task definition template and add the following parameters. | ||||||
|
|
||||||
| * _`{{region}}`_: The region to send the data to. | ||||||
| * _`{{ecsTaskRoleArn}}`_: The AWSOTTaskRole ARN. | ||||||
| * _`{{ecsTaskExecutionRoleArn}}`_: The AWSOTTaskExecutionRole ARN. | ||||||
| * Add an environment variable named `AOT_CONFIG_CONTENT`. Select **ValueFrom** to tell ECS to get the value from the Parameter Store, and set the value to `collector-config`. | ||||||
| - _`{{region}}`_: The region to send the data to. | ||||||
| - _`{{ecsTaskRoleArn}}`_: The AWSOTTaskRole ARN. | ||||||
| - _`{{ecsTaskExecutionRoleArn}}`_: The AWSOTTaskExecutionRole ARN. | ||||||
| - Add an environment variable named `AOT_CONFIG_CONTENT`. Select **ValueFrom** to tell ECS to get the value from the Parameter Store, and set the value to `collector-config`. | ||||||
|
|
||||||
| 1. Follow the ECS Fargate setup instructions to [create a task definition][task] using the template. | ||||||
|
|
||||||
| ### Configure {{% param "PRODUCT_NAME" %}} | ||||||
|
|
||||||
| {{< param "PRODUCT_NAME" >}} offers two approaches for collecting ECS task and container metrics: | ||||||
|
|
||||||
| - **Native receiver:** Use the [`otelcol.receiver.awsecscontainermetrics`][awsecscontainermetrics] component to read the ECS Task Metadata Endpoint V4 directly. This approach is simpler but requires enabling experimental features. | ||||||
|
|
||||||
| - **Prometheus exporter:** Use the [Prometheus ECS exporter](https://github.com/prometheus-community/ecs_exporter) as a sidecar container and scrape metrics with `prometheus.scrape`. This approach requires an additional container. | ||||||
|
|
||||||
| #### Native receiver | ||||||
|
|
||||||
| The `otelcol.receiver.awsecscontainermetrics` component reads the [ECS Task Metadata Endpoint V4][ecs-metadata-v4] and provides access to 52 task and container-level metrics including CPU, memory, network, and disk usage. | ||||||
|
|
||||||
| {{< admonition type="note" >}} | ||||||
| The `otelcol.receiver.awsecscontainermetrics` component is experimental. You must set the `--stability.level=experimental` flag to use it. Refer to [Permitted stability levels][stability-levels] for more information. | ||||||
|
|
||||||
| [stability-levels]: https://grafana.com/docs/alloy/<ALLOY_VERSION>/reference/cli/run/#permitted-stability-levels | ||||||
| {{< /admonition >}} | ||||||
|
|
||||||
| Use the following as a starting point for your {{< param "PRODUCT_NAME" >}} configuration: | ||||||
|
|
||||||
| ```alloy | ||||||
| prometheus.scrape "stats" { | ||||||
| targets = [ | ||||||
| { "__address__" = "localhost:9779" }, | ||||||
| ] | ||||||
| metrics_path = "/metrics" | ||||||
| scheme = "http" | ||||||
| forward_to = [prometheus.remote_write.default.receiver] | ||||||
| otelcol.receiver.awsecscontainermetrics "default" { | ||||||
| collection_interval = "60s" | ||||||
|
|
||||||
| output { | ||||||
| metrics = [otelcol.exporter.otlphttp.default.input] | ||||||
| } | ||||||
| } | ||||||
|
|
||||||
| otelcol.exporter.otlphttp "default" { | ||||||
| client { | ||||||
| endpoint = env("OTLP_ENDPOINT") | ||||||
|
|
||||||
| auth = otelcol.auth.basic.default.handler | ||||||
| } | ||||||
| } | ||||||
|
clayton-cornell marked this conversation as resolved.
|
||||||
|
|
||||||
| otelcol.auth.basic "default" { | ||||||
| client_auth { | ||||||
| username = env("OTLP_USERNAME") | ||||||
| password = env("OTLP_PASSWORD") | ||||||
| } | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| This configuration collects ECS task and container metrics every 60 seconds and exports them to an OTLP endpoint. | ||||||
|
|
||||||
| Replace the following: | ||||||
|
|
||||||
| // Additional OpenTelemetry configuration as in [ecs-default-config] | ||||||
| // OTLP receiver | ||||||
| // statsd | ||||||
| // Use the alloy convert command to use one of the AWS ADOT files | ||||||
| // https://grafana.com/docs/alloy/latest/reference/cli/convert/ | ||||||
| ... | ||||||
| - _`<OTLP_ENDPOINT>`_: The URL of your OTLP-compatible endpoint, for example `https://otlp-gateway-prod-us-central-0.grafana.net/otlp`. | ||||||
|
clayton-cornell marked this conversation as resolved.
Outdated
|
||||||
| - _`<OTLP_USERNAME>`_: The username for authentication. | ||||||
| - _`<OTLP_PASSWORD>`_: The password or API token for authentication. | ||||||
|
|
||||||
| #### Prometheus exporter | ||||||
|
|
||||||
| If you prefer a stable approach, use the Prometheus ECS exporter as a sidecar container. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. IMO this segue implies that the receiver isn't a great choice, I would lean more towards something like "Running the ECS exporter in a sidecar container does not require running Alloy with experimental stability" |
||||||
|
|
||||||
| Use the following as a starting point for your {{< param "PRODUCT_NAME" >}} configuration: | ||||||
|
|
||||||
| ```alloy | ||||||
| prometheus.scrape "ecs" { | ||||||
| targets = [{"__address__" = "localhost:9779"}] | ||||||
| forward_to = [prometheus.remote_write.default.receiver] | ||||||
| } | ||||||
|
|
||||||
| prometheus.remote_write "default" { | ||||||
| endpoint { | ||||||
| url = sys.env("PROMETHEUS_REMOTE_WRITE_URL") | ||||||
| basic_auth { | ||||||
| username = sys.env("PROMETHEUS_USERNAME") | ||||||
| password = sys.env("PROMETHEUS_PASSWORD") | ||||||
| } | ||||||
| url = env("PROMETHEUS_REMOTE_WRITE_URL") | ||||||
|
|
||||||
| basic_auth { | ||||||
| username = env("PROMETHEUS_USERNAME") | ||||||
| password = env("PROMETHEUS_PASSWORD") | ||||||
|
clayton-cornell marked this conversation as resolved.
Outdated
|
||||||
| } | ||||||
| } | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| This configuration sets up a scrape job for the container metrics and exports them to a Prometheus endpoint. | ||||||
| This configuration scrapes metrics from the ECS exporter running on port 9779 and exports them to a Prometheus endpoint. | ||||||
|
|
||||||
| Replace the following: | ||||||
|
|
||||||
| - _`<PROMETHEUS_REMOTE_WRITE_URL>`_: The URL of your Prometheus remote write endpoint. | ||||||
|
clayton-cornell marked this conversation as resolved.
Outdated
|
||||||
| - _`<PROMETHEUS_USERNAME>`_: The username for authentication. | ||||||
| - _`<PROMETHEUS_PASSWORD>`_: The password or API token for authentication. | ||||||
|
|
||||||
| #### Deploy the task | ||||||
|
|
||||||
| Complete the following steps to create a sample task. | ||||||
|
|
||||||
|
|
@@ -117,48 +170,58 @@ Complete the following steps to create a sample task. | |||||
| 1. Choose **Create parameter**. | ||||||
| 1. Create a parameter with the following values: | ||||||
|
|
||||||
| * **Name:** `collector-config` | ||||||
| * **Tier:** Standard | ||||||
| * **Type:** String | ||||||
| * **Data type:** Text | ||||||
| * **Value:** Copy and paste your custom {{< param "PRODUCT_NAME" >}} configuration file. | ||||||
| - **Name:** `collector-config` | ||||||
| - **Tier:** Standard | ||||||
| - **Type:** String | ||||||
| - **Data type:** Text | ||||||
| - **Value:** Copy and paste your custom {{< param "PRODUCT_NAME" >}} configuration file. | ||||||
|
|
||||||
| 1. Download the [ECS Fargate][fargate-template] or [ECS EC2][ec2-template] task definition template from GitHub. | ||||||
| 1. Edit the task definition template and add the following parameters. | ||||||
|
|
||||||
| * _`{{region}}`_: The region to send the data to. | ||||||
| * _`{{ecsTaskRoleArn}}`_: The AWSOTTaskRole ARN. | ||||||
| * _`{{ecsTaskExecutionRoleArn}}`_: The AWSOTTaskExecutionRole ARN. | ||||||
| * Set the container image to `grafana/alloy:<VERSION>`, for example `grafana/alloy:latest` or a specific version such as `grafana/alloy:v1.5.0`. | ||||||
| * Add a custom environment variable named `ALLOY_CONFIG_CONTENT`. Select **ValueFrom** to tell ECS to get the value from the Parameter Store, and set the value to `collector-config`. {{< param "PRODUCT_NAME" >}} doesn't read this variable directly, but you use it with the command below to pass the configuration. | ||||||
| * Add environment variables for Prometheus remote write: | ||||||
| * `PROMETHEUS_REMOTE_WRITE_URL` | ||||||
| * `PROMETHEUS_USERNAME` | ||||||
| * `PROMETHEUS_PASSWORD` - For increased security, create a password in AWS Secrets Manager and reference the ARN of the secret in the **ValueFrom** field. | ||||||
| * In the Docker configuration, change the **Entrypoint** to `bash,-c`. | ||||||
| * _`{{command}}`_: `"echo \"$ALLOY_CONFIG_CONTENT\" > /tmp/config_file && exec alloy run --server.http.listen-addr=0.0.0.0:12345 /tmp/config_file"`. This command writes the configuration from the environment variable to a file and then runs {{< param "PRODUCT_NAME" >}} with that configuration. Make sure you don't omit the double quotes around the command. | ||||||
| * {{< param "PRODUCT_NAME" >}} doesn't support collecting container metrics from the ECS metadata endpoint directly, so you need to add a second container for the [Prometheus exporter](https://github.com/prometheus-community/ecs_exporter) if needed: | ||||||
|
|
||||||
| 1. Add a container to the task. | ||||||
| 1. Set the container name to `ecs-exporter`. | ||||||
| 1. Set the image to `quay.io/prometheuscommunity/ecs-exporter:latest`. | ||||||
| 1. Add `tcp/9779` as a port mapping. | ||||||
| - _`{{region}}`_: The region to send the data to. | ||||||
| - _`{{ecsTaskRoleArn}}`_: The AWSOTTaskRole ARN. | ||||||
| - _`{{ecsTaskExecutionRoleArn}}`_: The AWSOTTaskExecutionRole ARN. | ||||||
| - Set the container image to `grafana/alloy:<VERSION>`, for example `grafana/alloy:latest` or a specific version such as `grafana/alloy:v1.8.0`. | ||||||
| - Add a custom environment variable named `ALLOY_CONFIG_CONTENT`. Select **ValueFrom** to tell ECS to get the value from the Parameter Store, and set the value to `collector-config`. {{< param "PRODUCT_NAME" >}} doesn't read this variable directly, but you use it with the command below to pass the configuration. | ||||||
| - Add environment variables for your endpoint. | ||||||
| - For the native receiver: | ||||||
| - `OTLP_ENDPOINT` | ||||||
| - `OTLP_USERNAME` | ||||||
| - `OTLP_PASSWORD` - For increased security, create a password in AWS Secrets Manager and reference the ARN of the secret in the **ValueFrom** field. | ||||||
| - For the Prometheus exporter: | ||||||
| - `PROMETHEUS_REMOTE_WRITE_URL` | ||||||
| - `PROMETHEUS_USERNAME` | ||||||
|
clayton-cornell marked this conversation as resolved.
|
||||||
| - `PROMETHEUS_PASSWORD` - For increased security, create a password in AWS Secrets Manager and reference the ARN of the secret in the **ValueFrom** field. | ||||||
| - In the Docker configuration, change the **Entrypoint** to `/bin/bash,-c`. | ||||||
| - _`{{command}}`_: For the native receiver, use `"echo \"$ALLOY_CONFIG_CONTENT\" > /tmp/config_file && exec /bin/alloy run --stability.level=experimental --server.http.listen-addr=0.0.0.0:12345 /tmp/config_file"`. The `--stability.level=experimental` flag enables the use of the `otelcol.receiver.awsecscontainermetrics` component. | ||||||
|
clayton-cornell marked this conversation as resolved.
Outdated
|
||||||
|
|
||||||
| For the Prometheus exporter, use `"echo \"$ALLOY_CONFIG_CONTENT\" > /tmp/config_file && exec /bin/alloy run --server.http.listen-addr=0.0.0.0:12345 /tmp/config_file"`. | ||||||
|
|
||||||
| Make sure you don't omit the double quotes around the command. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Double negative? "Make sure you include..." |
||||||
| - **Prometheus exporter only:** Add the Prometheus ECS exporter as a sidecar container: | ||||||
|
clayton-cornell marked this conversation as resolved.
Outdated
|
||||||
|
|
||||||
| 1. Add a container to the task. | ||||||
| 1. Set the container name to `ecs-exporter`. | ||||||
| 1. Set the image to `quay.io/prometheuscommunity/ecs-exporter:latest`. | ||||||
|
clayton-cornell marked this conversation as resolved.
Outdated
|
||||||
| 1. Add `tcp/9779` as a port mapping. | ||||||
|
|
||||||
| 1. Follow the ECS Fargate setup instructions to [create a task definition][task] using the template. | ||||||
|
|
||||||
| ## Collect EC2 instance metrics | ||||||
|
|
||||||
| For ECS Clusters running on EC2, you can collect instance metrics by using AWS ADOT or {{< param "PRODUCT_NAME" >}} in a separate ECS task deployed as a daemon. | ||||||
|
|
||||||
| ### {{% param "PRODUCT_NAME" %}} | ||||||
| ### Collect metrics with {{% param "PRODUCT_NAME" %}} | ||||||
|
|
||||||
| You can follow the steps described in [Configure {{< param "PRODUCT_NAME" >}}](#configure-alloy) to create another task, with the following changes: | ||||||
|
|
||||||
| * Only add the {{< param "PRODUCT_NAME" >}} container, not the Prometheus exporter, and run the task as a daemon so it automatically runs one instance per node in your cluster. | ||||||
| * Update your {{< param "PRODUCT_NAME" >}} configuration to collect metrics from the instance. | ||||||
| - If you're using the Prometheus exporter, you don't need to add the ecs-exporter sidecar for EC2 instance metrics. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| - Run the task as a daemon so it automatically runs one instance per node in your cluster. | ||||||
| - Update your {{< param "PRODUCT_NAME" >}} configuration to collect metrics from the instance. | ||||||
| The configuration varies depending on the type of EC2 node. Refer to the [`collect`](https://grafana.com/docs/alloy/latest/collect/) documentation for details. | ||||||
|
|
||||||
| ### ADOT | ||||||
| ### Collect metrics with ADOT | ||||||
|
|
||||||
| The approach described in [the AWS OpenTelemetry documentation](https://aws-otel.github.io/docs/setup/ecs#3-setup-the-adot-collector-for-ecs-ec2-instance-metrics) uses the `awscontainerinsightreceiver` receiver from OpenTelemetry. ADOT includes this receiver. | ||||||
|
|
||||||
|
|
@@ -185,11 +248,8 @@ You can also send everything directly to your final destination. | |||||
| [adot-doc]: https://aws-otel.github.io/docs/setup/ecs | ||||||
| [otel-task-metrics-config]: https://github.com/aws-observability/aws-otel-collector/blob/main/config/ecs/container-insights/otel-task-metrics-config.yaml | ||||||
| [ecs-default-config]: https://github.com/aws-observability/aws-otel-collector/blob/main/config/ecs/ecs-default-config.yaml | ||||||
| [awsecscontainermetrics]: https://grafana.com/docs/alloy/<ALLOY_VERSION>/reference/components/otelcol/otelcol.receiver.awsecscontainermetrics/ | ||||||
| [ecs-metadata-v4]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html | ||||||
| [fargate-template]: https://github.com/aws-observability/aws-otel-collector/blob/main/examples/ecs/aws-cloudwatch/ecs-fargate-sidecar.json | ||||||
| [ec2-template]: https://github.com/aws-observability/aws-otel-collector/blob/main/examples/ecs/aws-prometheus/ecs-ec2-task-def.json | ||||||
| [configure]: https://grafana.com/docs/alloy/<ALLOY_VERSION>/configure/ | ||||||
| [steps]: https://medium.com/ci-t/9-steps-to-ssh-into-an-aws-fargate-managed-container-46c1d5f834e2 | ||||||
| [install]: https://grafana.com/docs/alloy/<ALLOY_VERSION>/set-up/install/linux/ | ||||||
| [deploy]: https://grafana.com/docs/alloy/<ALLOY_VERSION>/set-up/deploy/ | ||||||
| [task]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html | ||||||
| [run]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/standalone-task-create.html | ||||||
Uh oh!
There was an error while loading. Please reload this page.