Skip to content

Commit

Permalink
remove suggestion to process internal telemetry through collector (#5749
Browse files Browse the repository at this point in the history
)

Signed-off-by: Alex Boten <[email protected]>
  • Loading branch information
codeboten authored Dec 11, 2024
1 parent 9eb8429 commit f01201c
Showing 1 changed file with 20 additions and 66 deletions.
86 changes: 20 additions & 66 deletions content/en/docs/collector/internal-telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,26 @@ You can configure how internal metrics are generated and exposed by the
Collector. By default, the Collector generates basic metrics about itself and
exposes them using the OpenTelemetry Go
[Prometheus exporter](https://github.com/open-telemetry/opentelemetry-go/tree/main/exporters/prometheus)
for scraping at `http://127.0.0.1:8888/metrics`. You can expose the endpoint to
one specific or all network interfaces when needed. For containerized
environments, you might want to expose this port on a public interface.
for scraping at `http://127.0.0.1:8888/metrics`.

The Collector can push its internal metrics to an OTLP backend via the following
configuration:

```yaml
service:
telemetry:
metrics:
readers:
- periodic:
exporter:
otlp:
protocol: grpc/protobuf
endpoint: http://localhost:14317
```
Alternatively, you can expose the Prometheus endpoint to one specific or all
network interfaces when needed. For containerized environments, you might want
to expose this port on a public interface.
Set the Prometheus config under `service::telemetry::metrics`:

Expand Down Expand Up @@ -151,69 +168,6 @@ Note that the `tracer_provider` section there corresponds to `traces` here.
[kitchen-sink-config]:
https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/kitchen-sink.yaml

### Self-monitoring

The Collector can be configured to push its own telemetry to an
[OTLP receiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver)
and send the data through configured pipelines. In the following example, the
Collector is configured to push metrics, traces, and logs every 10s using OTLP
gRPC to `localhost:14317`:

```yaml
receivers:
otlp/internal:
protocols:
grpc:
endpoint: localhost:14317
exporters:
debug:
service:
pipelines:
metrics:
receivers: [otlp/internal]
exporters: [debug]
traces:
receivers: [otlp/internal]
exporters: [debug]
telemetry:
metrics:
readers:
- periodic:
interval: 10000
exporter:
otlp:
protocol: grpc/protobuf
endpoint: http://localhost:14317
traces:
processors:
- batch:
exporter:
otlp:
protocol: grpc/protobuf
endpoint: http://localhost:14317
logs:
processors:
- batch:
exporter:
otlp:
protocol: grpc/protobuf
endpoint: http://localhost:14317
```

{{% alert title="Caution" color="warning" %}}

When self-monitoring, the Collector collects its own telemetry and sends it to
the desired backend for analysis. This can be a risky practice. If the Collector
is underperforming, its self-monitoring capability could be impacted. As a
result, the self-monitored telemetry might not reach the backend in time for
critical analysis.

Moreover, sending internal telemetry through the Collector's own pipelines can
create a continuous loop of spans, metric points, or logs, putting undue strain
on the Collector's performance. This setup should not be used in production.

{{% /alert %}}

## Types of internal telemetry

The OpenTelemetry Collector aims to be a model of observable service by clearly
Expand Down

0 comments on commit f01201c

Please sign in to comment.