feat(base-cluster/tracing): add gateway to enable tail sampling

[cwrau]

That way traces with errors are always stored, traces that
are >200ms and with a random chance of 0.1%.

Adjust your clients to just sample 100% of the spans/traces.

Summary by CodeRabbit

• New Features

  • Added a telemetry gateway for distributed tracing with tail-sampling policies and OTLP endpoints.
  • Enabled JSON-formatted logs in the telemetry collector.
  • Added a load‑balanced OTLP exporter with Kubernetes service discovery and zstd compression.

• Refactor

  • Simplified telemetry enablement across components (no longer tied to Prometheus).
  • Standardized OTLP endpoint handling to explicit host:port.
  • Moved descheduler tracing options to a unified telemetry command-options block.
  • Enriched Kubernetes metadata on traces and increased default trace sampling to 100%.

[coderabbitai]

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

Decouples telemetry tracing enablement from Prometheus, moves descheduler tracing options to a top-level cmdOptions block, standardizes OTLP host:port endpoint formatting, adds a telemetry-gateway HelmRelease and load‑balancing exporter, enables JSON logging, expands k8sattributes extraction, and raises Prometheus tracing sampling to 1.0.

Changes

Cohort / File(s)	Summary
Telemetry condition simplification charts/base-cluster/templates/descheduler/descheduler.yaml, charts/base-cluster/templates/ingress/nginx.yaml, charts/base-cluster/templates/kyverno/kyverno.yaml	Removed dependency on monitoring.prometheus.enabled for rendering telemetry/tracing blocks; tracing now gated solely by telemetryConf.enabled. Descheduler telemetry fields moved into a new top-level cmdOptions subsection.
Traefik OTLP endpoint formatting charts/base-cluster/templates/ingress/traefik.yaml	OTLP gRPC endpoint changed from a direct endpoint ref to a formatted host:port string (quoted) with explicit grpc.enabled: true; insecure handling remains gated by telemetry config.
Alloy collector pipeline overhaul charts/base-cluster/templates/monitoring/alloy-collector.yaml	Added JSON logging; added tracing block (sampling_fraction=1.0); expanded k8sattributes metadata extraction; replaced distinct tempo exporter with a loadbalancing.gateway exporter using a Kubernetes resolver targeting telemetry-gateway, OTLP + zstd compression and tls.insecure.
New telemetry gateway HelmRelease charts/base-cluster/templates/monitoring/alloy-gateway.yaml	Added HelmRelease telemetry-gateway (namespace monitoring) deploying Grafana Alloy collector with OTLP receiver, tail/batch sampling policies, Tempo exporter endpoint, extra OTLP/metrics ports, autoscaling, and a ServiceMonitor; rendering gated by both monitoring.prometheus.enabled and monitoring.tracing.enabled.
Prometheus tracing sampling charts/base-cluster/templates/monitoring/kube-prometheus-stack/_prometheus_config.yaml	Increased tracingConfig.clientType.grpc.samplingFraction from 0.1 to 1.0.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant W as Instrumented Workload
  participant C as Alloy Collector (node/agent)
  participant G as Telemetry Gateway (Alloy HelmRelease)
  participant T as Tempo Distributor

  Note over W,C: OTLP gRPC export uses formatted host:port
  W->>C: Export spans (OTLP gRPC)
  C->>C: k8sattributes extraction, JSON logging, batch processing
  C->>G: OTLP gRPC (load‑balancing resolver, zstd, tls.insecure)
  G->>G: Tail sampling & batch processing (sampling_fraction=1.0)
  G->>T: Export traces to tempo distributor:4317 (OTLP)
  Note right of T: ServiceMonitor exposes metrics if Prometheus enabled

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• feat(base-cluster/monitoring): rename alloy to be a generic name #1433 — Overlaps changes to Alloy telemetry HelmRelease and routing (telemetry-gateway / alloy values).
• feat(base-cluster/monitoring): configure service graph for grafana #1422 — Similar telemetry templating changes affecting endpoint formatting and tracing condition logic.
• feat(base-cluster/ingress)!: add option traefik for ingress controller and make it default #1420 — Related edits in ingress telemetry templates (nginx/traefik) and OTLP configuration.

Suggested reviewers

• tasches
• marvinWolff
• teutonet-bot

Poem

A rabbit scurries, code in paw,
Traces hop along the raw.
Gateway hums and spans take flight,
JSON logs glow through the night.
Zstd crunches, sampling beams — carrots for observability dreams! 🥕

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title "feat(base-cluster/tracing): add gateway to enable tail sampling" directly corresponds to the primary change in this pull request: the introduction of a new telemetry-gateway HelmRelease component with tail sampling capabilities (as evidenced by the new alloy-gateway.yaml file). The title is concise, specific, and clearly communicates the main feature being added, which aligns with the PR objectives of adding a gateway to enable tail sampling with policies for error traces, long traces, and random sampling. While the changeset includes refactoring of telemetry configuration across multiple other components (descheduler, nginx, traefik, kyverno, collector), these are supporting changes that enable the core gateway feature rather than the main point of the PR.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feat/base-cluster/add-telemetry-gateway-for-tail-sampling

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b6606b5 and 74da747.

📒 Files selected for processing (7)

• charts/base-cluster/templates/descheduler/descheduler.yaml (1 hunks)
• charts/base-cluster/templates/ingress/nginx.yaml (1 hunks)
• charts/base-cluster/templates/ingress/traefik.yaml (1 hunks)
• charts/base-cluster/templates/kyverno/kyverno.yaml (1 hunks)
• charts/base-cluster/templates/monitoring/alloy-collector.yaml (3 hunks)
• charts/base-cluster/templates/monitoring/alloy-gateway.yaml (1 hunks)
• charts/base-cluster/templates/monitoring/kube-prometheus-stack/_prometheus_config.yaml (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (3)

• charts/base-cluster/templates/kyverno/kyverno.yaml
• charts/base-cluster/templates/ingress/nginx.yaml
• charts/base-cluster/templates/monitoring/kube-prometheus-stack/_prometheus_config.yaml

🧰 Additional context used

🪛 YAMLlint (1.37.1)

charts/base-cluster/templates/monitoring/alloy-gateway.yaml

[error] 1-1: syntax error: expected the node content, but found '-'

(syntax)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: check licenses
• GitHub Check: lint helm chart (base-cluster)

🔇 Additional comments (10)

charts/base-cluster/templates/ingress/traefik.yaml (1)

80-91: LGTM!

The telemetry endpoint construction and OTLP gRPC configuration looks correct. The host:port formatting with proper int64 type conversion is sound.

charts/base-cluster/templates/monitoring/alloy-collector.yaml (4)

44-46: LGTM!

JSON logging format is consistent with broader observability improvements across ingress components.

---

196-199: LGTM!

The tracing block with 100% sampling fraction correctly implements the PR objective of instructing the collector to sample 100% of spans, allowing downstream tail sampling to make decisions.

---

201-215: Comprehensive metadata extraction enhancement.

The expanded metadata extraction (replicaset, statefulset, daemonset, cronjob, job, pod.start_time, container details) significantly improves trace context and debuggability. Well done.

---

243-265: k8sattributes processor authentication is not broken—remove this review comment.

The k8sattributes processor defaults to "serviceAccount" authentication when auth_type is omitted, using the Pod's service account token to access the Kubernetes API. The removal of the explicit auth_type = "serviceAccount" line does not introduce a failure; the processor will authenticate correctly by default. The current configuration (line 201 onward) will function as intended for pod discovery and resource attribute enrichment.

Likely an incorrect or invalid review comment.

charts/base-cluster/templates/monitoring/alloy-gateway.yaml (4)

1-1: LGTM!

The conditional gating correctly uses and with 2+ arguments (unlike the earlier descheduler issue). Both prometheus.enabled and tracing.enabled conditions are required.

---

44-91: Tail sampling policies correctly implement PR objectives.

The three sampling policies (error traces, 200ms latency threshold, probabilistic 0.1%) align well with stated requirements and ensure critical traces are retained while reducing overall volume.

---

84-84: Clarify sampling_percentage semantics in comments.

The sampling_percentage = 0.1 value is ambiguous without documentation. Based on PR description mentioning "0.1% random sampling," confirm this is the correct interpretation:

• If sampling_percentage expects a percentage value (0–100 range): 0.1 = 0.1% ✓ correct
• If sampling_percentage expects a fraction (0–1 range): 0.1 = 10% ✗ incorrect

Add a clarifying comment above line 84 to document the intended semantics:

             policy {
               name = "sample-random"
               type = "probabilistic"
               probabilistic {
+                # sampling_percentage = 0.1 means 0.1%, not 10% (percentage range is 0-100)
                 sampling_percentage = 0.1
               }
             }

Also verify against the Alloy/OpenTelemetry documentation for the tail sampling processor to confirm the value range interpretation.

---

99-107: LGTM!

The OTLP Tempo exporter endpoint and TLS insecure configuration are appropriate for in-cluster communication. The endpoint targets the Grafana Tempo distributor service correctly.

charts/base-cluster/templates/descheduler/descheduler.yaml (1)

41-41: Consider passing global context for consistency with traefik, though autodiscovery will still function.

Line 41 doesn't pass the "global" .Values.global parameter that traefik.yaml uses (line 80). Unlike the review comment implies, the template won't malfunction—it has a fallback (line 44 in _telemetry.tpl uses | default (dict)) that triggers autodiscovery mode. However, this means descheduler relies on service autodiscovery rather than explicit telemetry config from .Values.global.telemetry.otlp. If explicit configuration is needed (e.g., for non-standard collector endpoints), add the parameter:

-    {{- $telemetryConf := include "common.telemetry.conf" (dict "protocol" "otlp") | fromYaml }}
+    {{- $telemetryConf := include "common.telemetry.conf" (dict "protocol" "otlp" "global" .Values.global) | fromYaml }}

Note: Other components (prometheus, kyverno, nginx) omit global and use autodiscovery, so this is not a blocking issue.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull Request Overview

This PR implements tail sampling for distributed tracing by introducing a telemetry gateway. The gateway filters traces based on errors, latency (>200ms), and random sampling (0.1%), while upstream clients now sample 100% of traces.

• Introduces a new Alloy-based telemetry gateway with tail sampling policies
• Updates all tracing clients to use 100% sampling and route through the gateway
• Removes the requirement for Prometheus to be enabled when tracing is enabled

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
charts/base-cluster/templates/monitoring/alloy-gateway.yaml	New telemetry gateway deployment with tail sampling configuration
charts/base-cluster/templates/monitoring/alloy-collector.yaml	Routes traces through load-balanced gateway instead of directly to Tempo
charts/base-cluster/templates/monitoring/kube-prometheus-stack/_prometheus_config.yaml	Increases sampling fraction from 0.1 to 1.0
charts/base-cluster/templates/kyverno/kyverno.yaml	Removes Prometheus dependency from telemetry enablement
charts/base-cluster/templates/ingress/traefik.yaml	Adds explicit gRPC enablement for OTLP endpoint
charts/base-cluster/templates/ingress/nginx.yaml	Removes Prometheus dependency from telemetry enablement
charts/base-cluster/templates/descheduler/descheduler.yaml	Migrates from deschedulerPolicy.tracing to cmdOptions for OTLP configuration

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

charts/base-cluster/templates/kyverno/kyverno.yaml (1)

70-77: Fix Helm template: and needs at least two args.

and in Go templates requires ≥2 operands; calling and $telemetryConf.enabled raises wrong number of args for and at render time. Assign the value directly instead.

-      {{- $telemetryEnabled := and $telemetryConf.enabled -}}
+      {{- $telemetryEnabled := $telemetryConf.enabled -}}

charts/base-cluster/templates/ingress/nginx.yaml (1)

32-39: Remove the single-operand and to avoid template failure.

{{- if and $telemetryConf.enabled }} triggers wrong number of args for and during Helm rendering. Use the flag directly.

-      {{- if and $telemetryConf.enabled }}
+      {{- if $telemetryConf.enabled }}

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 51a42a2 and 8b44a8e.

📒 Files selected for processing (7)

• charts/base-cluster/templates/descheduler/descheduler.yaml (1 hunks)
• charts/base-cluster/templates/ingress/nginx.yaml (1 hunks)
• charts/base-cluster/templates/ingress/traefik.yaml (1 hunks)
• charts/base-cluster/templates/kyverno/kyverno.yaml (1 hunks)
• charts/base-cluster/templates/monitoring/alloy-collector.yaml (3 hunks)
• charts/base-cluster/templates/monitoring/alloy-gateway.yaml (1 hunks)
• charts/base-cluster/templates/monitoring/kube-prometheus-stack/_prometheus_config.yaml (1 hunks)

🧰 Additional context used

🪛 YAMLlint (1.37.1)

charts/base-cluster/templates/monitoring/alloy-gateway.yaml

[error] 1-1: syntax error: expected the node content, but found '-'

(syntax)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: check licenses
• GitHub Check: lint helm chart (base-cluster)