Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 5 additions & 11 deletions .trivyignore
Original file line number Diff line number Diff line change
@@ -1,17 +1,11 @@
# This file contains CVEs to be ignored by Trivy
# Auto-generated on Tue Jul 22 22:44:41 UTC 2025

# CRITICAL

# HIGH
# kube-state-metrics
CVE-2024-33599
CVE-2024-33601

# MEDIUM
# prometheusui
CVE-2025-22872 # golang.org/x/net
GHSA-fv92-fjc5-jj9h
# kube-state-metrics
CVE-2023-4527
CVE-2023-4806
CVE-2024-33600

# LOW
# LOW

2 changes: 1 addition & 1 deletion OPENTELEMETRY_VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
v0.127.0
v0.129.0
2 changes: 1 addition & 1 deletion TARGETALLOCATOR_VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
v0.127.0
v0.129.1
2 changes: 1 addition & 1 deletion otelcollector/configuration-reader-builder/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ require (
github.com/onsi/ginkgo/v2 v2.22.2 // indirect
github.com/onsi/gomega v1.36.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.64.0 // indirect
github.com/prometheus/common v0.65.0 // indirect
github.com/x448/float16 v0.8.4 // indirect
golang.org/x/net v0.40.0 // indirect
golang.org/x/oauth2 v0.30.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions otelcollector/configuration-reader-builder/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@ github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRI
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.64.0 h1:pdZeA+g617P7oGv1CzdTzyeShxAGrTBsolKNOLQPGO4=
github.com/prometheus/common v0.64.0/go.mod h1:0gZns+BLRQ3V6NdaerOhMbwwRbNh9hkGINtQAsP5GS8=
github.com/prometheus/common v0.65.0 h1:QDwzd+G1twt//Kwj/Ww6E9FQq1iVMmODnILtW1t2VzE=
github.com/prometheus/common v0.65.0/go.mod h1:0gZns+BLRQ3V6NdaerOhMbwwRbNh9hkGINtQAsP5GS8=
github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8=
github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
Expand Down
298 changes: 150 additions & 148 deletions otelcollector/opentelemetry-collector-builder/go.mod

Large diffs are not rendered by default.

653 changes: 333 additions & 320 deletions otelcollector/opentelemetry-collector-builder/go.sum

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion otelcollector/opentelemetry-collector-builder/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ func main() {
info := component.BuildInfo{
Command: "custom-collector-distro",
Description: "Custom OpenTelemetry Collector distribution",
Version: "0.127.0",
Version: "0.129.0",
}

set := otelcol.CollectorSettings{
Expand Down
20 changes: 20 additions & 0 deletions otelcollector/otel-allocator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,26 @@ The TA serves two functions:
* Even distribution of Prometheus targets among a pool of Collectors
* Discovery of Prometheus Custom Resources

## Configuration

The Target Allocator uses a configuration file (by default under `/conf/targetallocator.yaml`). It accepts the following elements:

| Name | Description | Default Value | Environment variable |
|------------------------------------|-------------------------------------------------------------------------------|-----------------------------------------------|----------------------|
| `collector_namespace` (required) | Namespace to watch for collector deployments for job assignments | | `OTELCOL_NAMESPACE` |
| `collector_selector` | Kubernetes selector to select collectors for job assignments | | |
| `listen_addr` | Endpoint on which the target allocator exposes job definitions for collectors | `:8080` or `:8443` if `https` is set to true | |
| `kube_config_file_path` | Path to the file on the pod containing the Kube config. | "~/.kube/config" | `KUBECONFIG` |
| `config` | Prometheus configuration block | | |
| `allocation_strategy` | Allocation strategy to apply to job assignments | `consistent-hashing` | |
| `allocation_fallback_strategy` | Fallback allocation strategy for job assignments | | |
| `filter_strategy` | Filter strategy to apply to metrics | `relabel-config` | |
| `prometheus_cr` | Whether to watch Prometheus Custom Resources | | |
| `https` | Whether to expose the target allocator endpoint over https | | |
| `collector_not_ready_grace_period` | Wait time before assigning jobs to a new collector. | 30s | |

Additional configuration options are present under [./internal/config/config.go](./internal/config/config.go).

## Even Distribution of Prometheus Targets

The Target Allocator’s first job is to discover targets to scrape and OTel Collectors to allocate targets to. Then it can distribute the targets it discovers among the Collectors. The Collectors in turn query the Target Allocator for Metrics endpoints to scrape, and then the Collectors’ [Prometheus Receivers](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md) scrape the Metrics targets.
Expand Down
24 changes: 15 additions & 9 deletions otelcollector/otel-allocator/benchmark_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ package main
import (
"context"
"fmt"
"os"
"strconv"
"strings"
"testing"
Expand Down Expand Up @@ -41,7 +40,8 @@ func BenchmarkProcessTargets(b *testing.B) {
tsets := prepareBenchmarkData(numTargets, targetsPerGroup, groupsPerJob)
for _, strategy := range allocation.GetRegisteredAllocatorNames() {
b.Run(fmt.Sprintf("%s/%d", strategy, numTargets), func(b *testing.B) {
targetDiscoverer := createTestDiscoverer(strategy, map[string][]*relabel.Config{})
targetDiscoverer, err := createTestDiscoverer(strategy, map[string][]*relabel.Config{})
require.NoError(b, err)
targetDiscoverer.UpdateTsets(tsets)
b.ResetTimer()
for i := 0; i < b.N; i++ {
Expand Down Expand Up @@ -82,7 +82,8 @@ func BenchmarkProcessTargetsWithRelabelConfig(b *testing.B) {

for _, strategy := range allocation.GetRegisteredAllocatorNames() {
b.Run(fmt.Sprintf("%s/%d", strategy, numTargets), func(b *testing.B) {
targetDiscoverer := createTestDiscoverer(strategy, prehookConfig)
targetDiscoverer, err := createTestDiscoverer(strategy, prehookConfig)
require.NoError(b, err)
targetDiscoverer.UpdateTsets(tsets)
b.ResetTimer()
for i := 0; i < b.N; i++ {
Expand Down Expand Up @@ -167,21 +168,26 @@ func prepareBenchmarkData(numTargets, targetsPerGroup, groupsPerJob int) map[str
return tsets
}

func createTestDiscoverer(allocationStrategy string, prehookConfig map[string][]*relabel.Config) *target.Discoverer {
func createTestDiscoverer(allocationStrategy string, prehookConfig map[string][]*relabel.Config) (*target.Discoverer, error) {
ctx := context.Background()
logger := ctrl.Log.WithName(fmt.Sprintf("bench-%s", allocationStrategy))
ctrl.SetLogger(logr.New(log.NullLogSink{}))
allocatorPrehook := prehook.New("relabel-config", logger)
allocatorPrehook.SetConfig(prehookConfig)
allocator, err := allocation.New(allocationStrategy, logger, allocation.WithFilter(allocatorPrehook))
srv := server.NewServer(logger, allocator, "localhost:0")
if err != nil {
setupLog.Error(err, "Unable to initialize allocation strategy")
os.Exit(1)
return nil, err
}
srv, err := server.NewServer(logger, allocator, "localhost:0")
if err != nil {
return nil, err
}
registry := prometheus.NewRegistry()
sdMetrics, _ := discovery.CreateAndRegisterSDMetrics(registry)
discoveryManager := discovery.NewManager(ctx, config.NopLogger, registry, sdMetrics)
targetDiscoverer := target.NewDiscoverer(logger, discoveryManager, allocatorPrehook, srv, allocator.SetTargets)
return targetDiscoverer
targetDiscoverer, err := target.NewDiscoverer(logger, discoveryManager, allocatorPrehook, srv, allocator.SetTargets)
if err != nil {
return nil, err
}
return targetDiscoverer, nil
}
55 changes: 28 additions & 27 deletions otelcollector/otel-allocator/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,20 +6,15 @@ toolchain go1.24.1

retract v1.51.0

// pointing to this fork for prometheus-operator since we need fixes for asset store which is only available from v0.84.0 of prometheus-operator
// targetallocator cannot upgrade to v0.84.0 because of this issue - https://github.com/open-telemetry/opentelemetry-operator/issues/4196
// this commit is from this repository -https://github.com/rashmichandrashekar/prometheus-operator/tree/rashmi/v0.81.0-patch-assetstore - which only has the asset store fixes on top of v0.81.0 of prometheus-operator
replace github.com/prometheus-operator/prometheus-operator => github.com/rashmichandrashekar/prometheus-operator v0.0.0-20250715221118-b55ea6d3c138

require (
github.com/blang/semver/v4 v4.0.0
github.com/buraksezer/consistent v0.10.0
github.com/cespare/xxhash/v2 v2.3.0
github.com/ghodss/yaml v1.0.0
github.com/gin-gonic/gin v1.10.0
github.com/gin-gonic/gin v1.10.1
github.com/go-kit/log v0.2.1 // indirect
github.com/go-logr/logr v1.4.2
github.com/go-viper/mapstructure/v2 v2.2.1
github.com/go-logr/logr v1.4.3
github.com/go-viper/mapstructure/v2 v2.3.0
github.com/google/uuid v1.6.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
Expand All @@ -28,12 +23,15 @@ require (
github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.81.0
github.com/prometheus-operator/prometheus-operator/pkg/client v0.81.0
github.com/prometheus/client_golang v1.22.0
github.com/prometheus/common v0.64.0
github.com/prometheus/common v0.65.0
github.com/prometheus/prometheus v0.301.0
github.com/spf13/pflag v1.0.6
github.com/stretchr/testify v1.10.0
go.opentelemetry.io/otel v1.36.0 // indirect
go.opentelemetry.io/otel/metric v1.36.0 // indirect
go.opentelemetry.io/otel v1.37.0
go.opentelemetry.io/otel/exporters/prometheus v0.59.0
go.opentelemetry.io/otel/metric v1.37.0
go.opentelemetry.io/otel/sdk v1.37.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.37.0
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0
gopkg.in/yaml.v2 v2.4.0
Expand Down Expand Up @@ -70,7 +68,7 @@ require (
github.com/bytedance/sonic/loader v0.1.1 // indirect
github.com/cloudwego/base64x v0.1.4 // indirect
github.com/cloudwego/iasm v0.2.0 // indirect
github.com/cncf/xds/go v0.0.0-20250121191232-2f005788dc42 // indirect
github.com/cncf/xds/go v0.0.0-20250326154945-ae57f3c0d45f // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dennwc/varint v1.0.0 // indirect
github.com/digitalocean/godo v1.132.0 // indirect
Expand Down Expand Up @@ -122,6 +120,7 @@ require (
github.com/gophercloud/gophercloud v1.14.1 // indirect
github.com/gorilla/websocket v1.5.3 // indirect
github.com/grafana/regexp v0.0.0-20240518133315-a468a5bfb3bc // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.1 // indirect
github.com/hashicorp/consul/api v1.30.0 // indirect
github.com/hashicorp/cronexpr v1.1.2 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
Expand Down Expand Up @@ -179,27 +178,27 @@ require (
github.com/x448/float16 v0.8.4 // indirect
go.mongodb.org/mongo-driver v1.17.2 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 // indirect
go.opentelemetry.io/otel/trace v1.36.0 // indirect
go.opentelemetry.io/proto/otlp v1.6.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0 // indirect
go.opentelemetry.io/otel/trace v1.37.0 // indirect
go.opentelemetry.io/proto/otlp v1.7.0 // indirect
go.uber.org/atomic v1.11.0
golang.org/x/arch v0.8.0 // indirect
golang.org/x/crypto v0.38.0 // indirect
golang.org/x/crypto v0.39.0 // indirect
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 // indirect
golang.org/x/mod v0.24.0 // indirect
golang.org/x/net v0.40.0 // indirect
golang.org/x/mod v0.25.0 // indirect
golang.org/x/net v0.41.0 // indirect
golang.org/x/oauth2 v0.30.0 // indirect
golang.org/x/sync v0.14.0 // indirect
golang.org/x/sync v0.15.0 // indirect
golang.org/x/sys v0.33.0 // indirect
golang.org/x/term v0.32.0 // indirect
golang.org/x/text v0.25.0 // indirect
golang.org/x/text v0.26.0 // indirect
golang.org/x/time v0.9.0 // indirect
golang.org/x/tools v0.31.0 // indirect
golang.org/x/tools v0.33.0 // indirect
gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
google.golang.org/api v0.213.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250519155744-55703ea1f237 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250519155744-55703ea1f237 // indirect
google.golang.org/grpc v1.72.1 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250603155806-513f23925822 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250603155806-513f23925822 // indirect
google.golang.org/grpc v1.73.0 // indirect
google.golang.org/protobuf v1.36.6 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
Expand All @@ -209,7 +208,9 @@ require (
sigs.k8s.io/structured-merge-diff/v4 v4.6.0 // indirect
)

require github.com/goccy/go-yaml v1.17.1
require gotest.tools/v3 v3.5.2

require github.com/goccy/go-yaml v1.18.0

require (
github.com/envoyproxy/go-control-plane/envoy v1.32.4 // indirect
Expand All @@ -219,5 +220,5 @@ require (
github.com/onsi/ginkgo/v2 v2.23.4 // indirect
github.com/onsi/gomega v1.37.0 // indirect
github.com/prometheus/sigv4 v0.1.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.35.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.36.0 // indirect
)
Loading
Loading