Skip to content

README.md

Latest commit

 

History

History
712 lines (580 loc) · 28.3 KB

README.md

File metadata and controls

712 lines (580 loc) · 28.3 KB
Raw

Target Allocator

Target Allocator is an optional component of the OpenTelemetry Collector Custom Resource (CR). The release version matches the operator's most recent release as well.

🚨 Note: the TargetAllocator currently supports the statefulset and daemonset deployment modes of the OpenTelemetryCollector CR.

In a nutshell, the TA is a mechanism for decoupling the service discovery and metric collection functions of Prometheus such that they can be scaled independently. The Collector manages Prometheus metrics without needing to install Prometheus. The TA manages the configuration of the Collector's Prometheus Receiver.

The TA serves two functions:

  • Even distribution of Prometheus targets among a pool of Collectors
  • Discovery of Prometheus Custom Resources

Configuration

The Target Allocator uses a configuration file (by default under /conf/targetallocator.yaml). It accepts the following elements:

Name Description Default Value Environment variable
collector_namespace (required) Namespace to watch for collector deployments for job assignments OTELCOL_NAMESPACE
collector_selector Kubernetes selector to select collectors for job assignments
listen_addr Endpoint on which the target allocator exposes job definitions for collectors :8080 or :8443 if https is set to true
kube_config_file_path Path to the file on the pod containing the Kube config. "~/.kube/config" KUBECONFIG
config Prometheus configuration block
allocation_strategy Allocation strategy to apply to job assignments consistent-hashing
allocation_fallback_strategy Fallback allocation strategy for job assignments
filter_strategy Filter strategy to apply to metrics relabel-config
prometheus_cr Whether to watch Prometheus Custom Resources
https Whether to expose the target allocator endpoint over https
allow_insecure_auth_secrets Serve auth secret values over plain HTTP without mTLS false ALLOW_INSECURE_AUTH_SECRETS
collector_not_ready_grace_period Wait time before assigning jobs to a new collector. 30s

Additional configuration options are present under ./internal/config/config.go.

Even Distribution of Prometheus Targets

The Target Allocator’s first job is to discover targets to scrape and OTel Collectors to allocate targets to. Then it can distribute the targets it discovers among the Collectors. The Collectors in turn query the Target Allocator for Metrics endpoints to scrape, and then the Collectors’ Prometheus Receivers scrape the Metrics targets.

This means that the OTel Collectors collect the metrics instead of a Prometheus scraper.

sequenceDiagram
  participant Target Allocator
  participant Metrics Targets
  participant OTel Collectors
  Target Allocator ->>Metrics Targets: 1. Discover Metrics targets
  Target Allocator ->>OTel Collectors: 2. Discover available Collectors
  Target Allocator ->>Target Allocator: 3. Assign Metrics targets
  OTel Collectors ->>Target Allocator: 4. Query TA for Metrics endpoints scrape
  OTel Collectors ->>Metrics Targets: 5. Scrape Metrics target
Loading

Allocation strategies

Several target allocation strategies are available. Some strategies may only make sense for a given Collector deployment mode. For example, the per-node strategy only works correctly with a Collector deployed as a DaemonSet.

consistent-hashing

A consistent hashing strategy implementing the following algorithm. Only the target url is hashed to prevent label changes from causing targets to be moved between collectors. This strategy consistently assigns targets to the same collectors, but will experience rebalancing when the collector count changes.

This is the default.

least-weighted

A strategy that simply assigns the target to the collector with the least number of targets. It achieves more stability in target assignment when collector count changes, but at the cost of less even distribution of targets.

per-node

This strategy assigns each target to the collector running on the same Node the target is. As such, it only makes sense to use it with a collector running as a DaemonSet.

Warning

The per-node strategy ignores targets not assigned to a Node, like for example control plane components.

Discovery of Prometheus Custom Resources

The Target Allocator also provides for the discovery of Prometheus Operator CRs, namely the ServiceMonitor and PodMonitor. The ServiceMonitors and the PodMonitors purpose is to inform the Target Allocator (or PrometheusOperator) to add a new job to their scrape configuration. The Target Allocator then provides the jobs to the OTel Collector Prometheus Receiver.

flowchart RL
  pm(PodMonitor)
  sm(ServiceMonitor)
  ta(Target Allocator)
  oc1(OTel Collector)
  oc2(OTel Collector)
  oc3(OTel Collector)
  ta --> pm
  ta --> sm
  oc1 --> ta
  oc2 --> ta
  oc3 --> ta

  %% Labels positioned correctly using text nodes
  crs@{shape: text, label: "Discover Prometheus Operator CRs"}
  ta_scrape@{shape: text, label: "Add job to TA scrape configuration"}
  oc_scrape@{shape: text, label: "Add job to OTel Collector scrape configuration"}

  oc_scrape ~~~ ta
  oc_scrape ~~~ ta_scrape
  ta ~~~ crs

  %% Apply grey background to text labels
  style crs fill:#e0e0e0,stroke:#cccccc
  style ta_scrape fill:#e0e0e0,stroke:#cccccc
  style oc_scrape fill:#e0e0e0,stroke:#cccccc
Loading

Even though Prometheus is not required to be installed in your Kubernetes cluster to use the Target Allocator for Prometheus CR discovery, the TA does require that the ServiceMonitor and PodMonitor be installed. These CRs are bundled with Prometheus Operator; however, they can be installed standalone as well.

The easiest way to do this is to grab a copy of the individual PodMonitor YAML and ServiceMonitor YAML custom resource definitions (CRDs) from the Kube Prometheus Operator’s Helm chart.

✨ For more information on configuring the PodMonitor and ServiceMonitor, check out the PodMonitor API and the ServiceMonitor API.

Usage

The spec.targetAllocator: controls the TargetAllocator general properties. Full API spec can be found here: api/opentelemetrycollectors.md#opentelemetrycollectorspectargetallocator

A basic example that deploys.

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: collector-with-ta
spec:
  mode: statefulset
  targetAllocator:
    enabled: true
  config:
    receivers:
      prometheus:
        config:
          scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 10s
            static_configs:
            - targets: [ '0.0.0.0:8888' ]

    exporters:
      debug: {}

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          exporters: [debug]

In essence, Prometheus Receiver configs are overridden with a http_sd_config directive that points to the Allocator, these are then loadbalanced/sharded to the Collectors. The Prometheus Receiver configs that are overridden are what will be distributed with the same name.

Using the Target Allocator from the Collector CR

The OpenTelemetry Operator comes with an optional component, the Target Allocator (TA). When creating an OpenTelemetryCollector Custom Resource (CR) and setting the TA as enabled, the Operator will create a new deployment and service to serve specific http_sd_config directives for each Collector pod as part of that CR. It will also rewrite the Prometheus receiver configuration in the CR, so that it uses the deployed target allocator. The following example shows how to get started with the Target Allocator:

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: collector-with-ta
spec:
  mode: statefulset
  targetAllocator:
    enabled: true
  config:
    receivers:
      prometheus:
        config:
          scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 10s
            static_configs:
            - targets: [ '0.0.0.0:8888' ]
            metric_relabel_configs:
            - action: labeldrop
              regex: (id|name)
            - action: labelmap
              regex: label_(.+)
              replacement: $$1

    exporters:
      debug: {}

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          exporters: [debug]
EOF

The usage of $$ in the replacement keys in the example above is based on the information provided in the Prometheus receiver README documentation, which states: Note: Since the collector configuration supports env variable substitution $ characters in your prometheus configuration are interpreted as environment variables. If you want to use $ characters in your prometheus configuration, you must escape them using $$.

Behind the scenes, the OpenTelemetry Operator will convert the Collector’s configuration after the reconciliation into the following:

receivers:
  prometheus:
    target_allocator:
      endpoint: http://collector-with-ta-targetallocator:80
      interval: 30s
      collector_id: $POD_NAME

exporters:
  debug:

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [debug]

The OpenTelemetry Operator will also convert the Target Allocator's Prometheus configuration after the reconciliation into the following:

config:
  scrape_configs:
    - job_name: otel-collector
      scrape_interval: 10s
      static_configs:
        - targets: ["0.0.0.0:8888"]
      metric_relabel_configs:
        - action: labeldrop
          regex: (id|name)
        - action: labelmap
          regex: label_(.+)
          replacement: $1

Note that in this case, the Operator replaces "$$" with a single "$" in the replacement keys. This is because the collector supports environment variable substitution, whereas the TA (Target Allocator) does not. Therefore, to ensure compatibility, the TA configuration should only contain a single "$" symbol.

TargetAllocator CRD

The spec.targetAllocator attribute allows very limited control over the target allocator resources. More customization is possible by using the TargetAllocator CRD. We create the TargetAllocator CR, and then add its name in the opentelemetry.io/target-allocator label on the respective OpenTelemetryCollector CR.

The basic example from above looks as follows with this setup:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: collector-with-ta
  labels:
    opentelemetry.io/target-allocator: ta
spec:
  mode: statefulset
  config:
    receivers:
      prometheus:
        config:
          scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 10s
            static_configs:
            - targets: [ '0.0.0.0:8888' ]

    exporters:
      debug: {}

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          exporters: [debug]
---
apiVersion: opentelemetry.io/v1alpha1
kind: TargetAllocator
metadata:
  name: ta
spec:

Note that the scrape configs can be specified either in the prometheus receiver configuration, or directly in the TargetAllocator CRD. The resultant target allocator will use both.

PrometheusCR specifics

TargetAllocator discovery of PrometheusCRs can be turned on by setting .spec.targetAllocator.prometheusCR.enabled to true, which it presents as scrape configs and jobs on the /scrape_configs and /jobs endpoints respectively.

The CRs can be filtered by labels as documented here: api/opentelemetrycollectors.md#opentelemetrycollectorspectargetallocatorprometheuscr

Upstream documentation here: PrometheusReceiver

Pod/Service Monitor Selectors

As of v1beta1 of the OpenTelemetryOperator, a serviceMonitorSelector and podMonitorSelector must be included, even if you don’t intend to use it, like this:

prometheusCR:
  enabled: true
  podMonitorSelector: {}
  serviceMonitorSelector: {}

This will make the TargetAllocator scrape all the Service and Pod Monitors inside of the cluster, in every namespace. On a cluster where you do not trust everyone who can create a monitor, read Security: arbitrary file access through Service/Pod Monitors before using {} — a monitor is enough to steal the collector's credentials.

If you need something more specific, you can also add a label filter:

prometheusCR:
  enabled: true
  serviceMonitorSelector:
    matchLabels:
      app: my-app

By setting the value of spec.targetAllocator.prometheusCR.serviceMonitorSelector.matchLabels to app: my-app, it means that your ServiceMonitor resource must in turn have that same value in metadata.labels.

See Security: arbitrary file access through Service/Pod Monitors for the risks of enabling prometheusCR on a cluster with untrusted tenants, and how to mitigate them.

RBAC

Before the TargetAllocator can start scraping, you need to set up Kubernetes RBAC (role-based access controls) resources. This means that you need to have a ServiceAccount and corresponding ClusterRoles/Roles so that the TargetAllocator has access to all the necessary resources to pull metrics from.

You can create your own ServiceAccount, and reference it in spec.targetAllocator.serviceAccount in your OpenTelemetryCollector CR. You’ll then need to configure the ClusterRole and ClusterRoleBinding or Role and RoleBinding for this ServiceAccount, as per below.

Cluster-scoped RBAC

  targetAllocator:
    enabled: true
    serviceAccount: opentelemetry-targetallocator-sa
    prometheusCR:
      enabled: true

🚨 Note: The Collector part of this same CR also has a serviceAccount key which only affects the collector and not the TargetAllocator.

If you omit the ServiceAccount name, the TargetAllocator creates a ServiceAccount for you. The ServiceAccount’s default name is a concatenation of the Collector name and the -targetallocator suffix. By default, this ServiceAccount has no defined policy, so you’ll need to create your own ClusterRole and ClusterRoleBinding or Role and RoleBinding for it, as per below.

The ClusterRole below will provide the minimum access required for the Target Allocator to query all the targets it needs based on any Prometheus configurations:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: opentelemetry-targetallocator-role
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs: ["get", "list", "watch"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

If you enable the prometheusCR (set spec.targetAllocator.prometheusCR.enabled to true) in the OpenTelemetryCollector CR, you will also need to define the following ClusterRoles. These give the TargetAllocator access to the PodMonitor and ServiceMonitor CRs. It also gives namespace access to the PodMonitor and ServiceMonitor.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: opentelemetry-targetallocator-cr-role
rules:
- apiGroups:
  - monitoring.coreos.com
  resources:
  - servicemonitors
  - podmonitors
  verbs:
  - '*'
- apiGroups: [""]
  resources:
  - namespaces
  verbs: ["get", "list", "watch"]

✨ The above ClusterRoles can be combined into a single ClusterRole.

Namespace-scoped RBAC

If you want to have the TargetAllocator watch a specific namespace, you can set the allowNamespaces field in the TargetAllocator's prometheusCR configuration. This is useful if you want to restrict the TargetAllocator to only watch Prometheus CRs in a specific namespace, and not have cluster-wide access.

  targetAllocator:
    enabled: true
    serviceAccount: opentelemetry-targetallocator-sa
    prometheusCR:
      enabled: true
      allowNamespaces: 
      - foo

In this case, you will need to create a Role and RoleBinding instead of a ClusterRole and ClusterRoleBinding. The Role and RoleBinding should be created in the namespace specified by the allowNamespaces field.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: opentelemetry-targetallocator-role
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - services
      - endpoints
      - configmaps
      - secrets
      - namespaces
    verbs:
      - get
      - watch
      - list
  - apiGroups:
      - apps
    resources:
      - statefulsets
    verbs:
      - get
      - watch
      - list
  - apiGroups:
      - discovery.k8s.io
    resources:
      - endpointslices
    verbs:
      - get
      - watch
      - list
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - get
      - watch
      - list
  - apiGroups:
      - monitoring.coreos.com
    resources:
      - servicemonitors
      - podmonitors
      - scrapeconfigs
      - probes
    verbs:
      - get
      - watch
      - list

Service / Pod monitor endpoint credentials

If your service or pod monitor endpoints require authentication (such as bearer tokens, basic auth, OAuth2, etc.), you must ensure that the collector has access to these credentials.

To secure the connection between the target allocator and the collector so that the secrets can be retrieved, mTLS is used. You can either let cert-manager manage the certificates (the default) or provide your own.

With cert-manager (default)

cert-manager provisions the CA, server, and client certificates automatically.

  • Prerequisites:

    • Ensure cert-manager is installed in your Kubernetes cluster.

    • Grant the target allocator RBAC permission to get the secrets referenced in the Service / Pod monitor.

    • Grant the operator RBAC permission to manage cert-manager resources. The following ClusterRole can be used:

      apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: opentelemetry-operator-controller-manager-cert-manager-role
rules:
- apiGroups:
  - cert-manager.io
  resources:
  - issuers
  - certificaterequests
  - certificates
  verbs:
  - create
  - get
  - list
  - watch
  - update
  - patch
  - delete

  • Configuration:

    • Enable mTLS by setting spec.mtls.enabled: true on the TargetAllocator CR.
    • If you create the allocator from an OpenTelemetryCollector, set spec.targetAllocator.mtls.enabled: true there instead; the operator forwards it into the generated TargetAllocator resource.
    • useCertManager defaults to true, so cert-manager provisions the serving and client certificates unless you explicitly disable it.

Without cert-manager (user-provided certificates)

Set useCertManager: false and reference your own certificates through the mtls.tls block. cert-manager is not required; the operator mounts the referenced Secrets and ConfigMaps into the target allocator and collector pods.

  • The mtls.tls block has three references:
    • certificateAuthorityCertificate: the CA used to verify the peer. Set exactly one of secret or configMap. Because CA bundles are public, distributing them via a ConfigMap is common.
    • serverCertificate: the target allocator's HTTPS server certificate and private key.
    • clientCertificate: the collector's client certificate and private key.
  • serverCertificate and clientCertificate each reference their certificateSecret (the certificate) and keySecret (the private key) independently, so the certificate and key may live in different Secrets.
  • Every reference has a required name and an optional key. When key is omitted it defaults to tls.crt for a certificate, tls.key for a private key, and ca.crt for a CA.
  • The referenced keys are projected into the pods at /tls via subPath mounts, so rotating a certificate requires the pods to be restarted.
  • Prerequisites:
    • Create the Secrets/ConfigMaps holding the CA, server, and client material in the same namespace as the workload. A standard kubernetes.io/tls Secret (which uses the tls.crt/tls.key keys) works out of the box.
    • Grant the target allocator RBAC permission to get the secrets referenced in the Service / Pod monitor (same as the cert-manager path). No cert-manager RBAC is needed.

Example on the standalone TargetAllocator CR:

apiVersion: opentelemetry.io/v1alpha1
kind: TargetAllocator
metadata:
  name: ta
spec:
  mtls:
    enabled: true
    useCertManager: false
    tls:
      # CA used to verify the peer. Required. Exactly one of secret or configMap.
      certificateAuthorityCertificate:
        configMap:
          name: my-ca            # key defaults to ca.crt
        # ...or from a Secret instead:
        # secret:
        #   name: my-ca
        #   key: ca.crt
      # Target allocator's HTTPS server cert + key (may be in different Secrets).
      serverCertificate:
        certificateSecret:
          name: my-server        # key defaults to tls.crt
        keySecret:
          name: my-server        # key defaults to tls.key
      # Collector's client cert + key.
      clientCertificate:
        certificateSecret:
          name: my-client
        keySecret:
          name: my-client

On an OpenTelemetryCollector with an embedded target allocator, place the same block under spec.targetAllocator.mtls.

Alternative: allow insecure auth secrets

If transport security is already handled by a service mesh or equivalent, you can skip the mTLS setup and serve auth secret values over plain HTTP.

With the Operator (CRD):

targetAllocator:
  enabled: true
  allowInsecureAuthSecrets: true

This works on both the OpenTelemetryCollector CR (embedded target allocator) and the standalone TargetAllocator CR.

Standalone Target Allocator (without Operator):

Set allow_insecure_auth_secrets: true in the target allocator config file, or set the ALLOW_INSECURE_AUTH_SECRETS=true environment variable.

Warning: Only enable this when transport-level security is guaranteed by other means.

Design

If the Allocator is activated, all Prometheus configurations will be transferred in a separate ConfigMap which get in turn mounted to the Allocator. This configuration will be resolved to target configurations and then split across all OpenTelemetryCollector instances.

TargetAllocators expose the results as HTTP_SD endpoints split by collector.

Currently, the Target Allocator handles the sharding of targets. The operator sets the $SHARD variable to 0 to allow collectors to keep targets generated by a Prometheus CRD. Using Prometheus sharding and target allocator sharding is not recommended currently and may lead to unknown results. See this thread for more information

Endpoints

/scrape_configs:

{
  "job1": {
    "follow_redirects": true,
    "honor_timestamps": true,
    "job_name": "job1",
    "metric_relabel_configs": [],
    "metrics_path": "/metrics",
    "scheme": "http",
    "scrape_interval": "1m",
    "scrape_timeout": "10s",
    "static_configs": []
  },
  "job2": {
    "follow_redirects": true,
    "honor_timestamps": true,
    "job_name": "job2",
    "metric_relabel_configs": [],
    "metrics_path": "/metrics",
    "relabel_configs": [],
    "scheme": "http",
    "scrape_interval": "1m",
    "scrape_timeout": "10s",
    "kubernetes_sd_configs": []
  }
}

/jobs:

{
  "job1": {
    "_link": "/jobs/job1/targets"
  },
  "job2": {
    "_link": "/jobs/job1/targets"
  }
}

/jobs/{jobID}/targets:

{
  "collector-1": {
    "_link": "/jobs/job1/targets?collector_id=collector-1",
    "targets": [
      {
        "Targets": [
          "10.100.100.100",
          "10.100.100.101",
          "10.100.100.102"
        ],
        "Labels": {
          "namespace": "a_namespace",
          "pod": "a_pod"
        }
      }
    ]
  }
}

/jobs/{jobID}/targets?collector_id={collectorID}:

[
  {
    "targets": [
      "10.100.100.100",
      "10.100.100.101",
      "10.100.100.102"
    ],
    "labels": {
      "namespace": "a_namespace",
      "pod": "a_pod"
    }
  }
]

Packages

Watchers

Watchers are responsible for the translation of external sources into Prometheus readable scrape configurations and triggers updates to the DiscoveryManager

DiscoveryManager

Watches the Prometheus service discovery for new targets and sets targets to the Allocator

Allocator

Shards the received targets based on the discovered Collector instances

Collector

Client to watch for deployed Collector instances which will then provided to the Allocator.

Troubleshooting

For troubleshooting tips, please visit: https://opentelemetry.io/docs/platforms/kubernetes/operator/troubleshooting/target-allocator/

More