CORE-554: Add TLS settings to Clickhouse destination + Update collector/otel to 141 + Remove deprecated components + Bump k8s min version to 1.21 by damemi · Pull Request #4111 · odigos-io/odigos

damemi · 2026-01-20T15:24:35Z

The clickhouse exporter supports TLS settings similar to otlp: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/945e5a71ef31793ff3280b28c1425086ea5332b6/exporter/clickhouseexporter/README.md#tls

Some users need this to connect to clickhouse, adding them as options in the destination here

This adds:

insecure_skip_verify
ca_file (using the k8sconfig interface to mount the secret as a file, similar to how the GCP exporter supports application default credentials)

The direct string fields (such as CAPem, CertPem, KeyPem) aren't yet supported in the clickhouse exporter, so it has to be a mounted file. See open-telemetry/opentelemetry-collector-contrib#43911 (comment)

To do this, it required bumping the collector/otel deps to 136 when TLS config support was added to clickhouse. This required the following changes:

This actually needs collector v0.136.0 for these settings from open-telemetry/opentelemetry-collector-contrib#42581 (open-telemetry/opentelemetry-collector-contrib@d9769f7)

Also needs to remove loki exporter (removed in 131) for 136 🙃 open-telemetry/opentelemetry-collector-contrib#41413, see https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.130.0/exporter/lokiexporter#deprecation-notice it's replaced with just otlp. The only destination that actually looks like it's using the loki exporter is OpsVerse

As well as the opencensus exporter, removed in 133 upstream by open-telemetry/opentelemetry-collector-contrib#42239

Also routing processor open-telemetry/opentelemetry-collector-contrib#36616

See previous attempt #3669 (reverted in #3734)

Then, it turns out that 136 was bugged and did not have full support for TLS settings like insecure_skip_verify. This was fixed in 141, which required the following extra changes:

Actually needs collector v141 due to this bug in clickhouse not handling all tls settings: open-telemetry/opentelemetry-collector-contrib#43911 fixed in open-telemetry/opentelemetry-collector-contrib#44093

Remove deprecated carbon exporter support (unmaintained upstream) open-telemetry/opentelemetry-collector-contrib#44532

another upstream breaking change giving go mod trouble open-telemetry/opentelemetry-collector#13948

configgrpc update: open-telemetry/opentelemetry-collector#13996

and now metadata.yaml metrics require stablity levels open-telemetry/opentelemetry-collector#13756

Error: failed loading /app/collector/receivers/odigosebpfreceiver/metadata.yaml: decoding failed due to the following error(s):

'telemetry.metrics[ebpf_memory_pressure_wait_time_total]' missing required field: `stability`
'telemetry.metrics[ebpf_total_bytes_read]' missing required field: `stability`
'telemetry.metrics[ebpf_lost_samples]' missing required field: `stability`
Error: failed loading /app/collector/receivers/odigosebpfreceiver/metadata.yaml: decoding failed due to the following error(s):

'telemetry.metrics[ebpf_memory_pressure_wait_time_total]' missing required field: `stability`
'telemetry.metrics[ebpf_total_bytes_read]' missing required field: `stability`
'telemetry.metrics[ebpf_lost_samples]' missing required field: `stability`
Error: metadata.yaml ordering check failed: [telemetry metrics] keys are not sorted: [odigos_log_data_size odigos_metric_data_size odigos_trace_data_size odigos_accepted_spans odigos_accepted_metric_points odigos_accepted_log_records]
Error: metadata.yaml ordering check failed: [telemetry metrics] keys are not sorted: [odigos_log_data_size odigos_metric_data_size odigos_trace_data_size odigos_accepted_spans odigos_accepted_metric_points odigos_accepted_log_records]

This bump also required adding the endpointslices permission to the odiglet service account for the data-collection collector

Finally, endpointslices was not GA in k8s 1.20. This PR bumps our minimum supported k8s version to 1.21. Enterprise update in https://github.com/odigos-io/odigos-enterprise/pull/2117

natyharary

Tiny thing.

natyharary · 2026-01-20T15:27:19Z

 	return common.ClickhouseDestinationType
 }

+func clickhouseTlsConfig(dest ExporterConfigurer) GenericMap {


I see we check if clickhouseTlsEnabled is true, but we don't check that all keys exist. Isn't it prone to misconfiguration?

Good point you don't need to set all of the keys (just having tls=true is enough for the minimum) but from what I could find it sounds like certPem and keyPem always need to be provided together if they are used for mTLS. updated

damemi · 2026-01-20T17:35:30Z

This actually needs collector v0.136.0 for these settings from open-telemetry/opentelemetry-collector-contrib#42581 (open-telemetry/opentelemetry-collector-contrib@d9769f7)

Also needs to remove loki exporter (removed in 131) for 136 🙃 open-telemetry/opentelemetry-collector-contrib#41413, see https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.130.0/exporter/lokiexporter#deprecation-notice it's replaced with just otlp. The only destination that actually looks like it's using the loki exporter is OpsVerse

As well as the opencensus exporter, removed in 133 upstream by open-telemetry/opentelemetry-collector-contrib#42239

Also routing processor open-telemetry/opentelemetry-collector-contrib#36616

See previous attempt #3669 (reverted in #3734)

RonFed · 2026-01-21T07:21:45Z

If we are bumping the collector I'd consider bumping to a newer version.

damemi · 2026-01-21T14:02:21Z

If we are bumping the collector I'd consider bumping to a newer version.

@RonFed for this I'm just trying to bump it to the minimum for the feature I need. We can do more incremental bumps this way and have fewer huge PRs with lots of changes. wdyt about that approach?

RonFed · 2026-01-21T16:53:57Z

If we are bumping the collector I'd consider bumping to a newer version.

@RonFed for this I'm just trying to bump it to the minimum for the feature I need. We can do more incremental bumps this way and have fewer huge PRs with lots of changes. wdyt about that approach?

That makes sense. On the other side, since this is quite a pain to do, if we have the opportunity - maybe it is something worth to consider. There are also performance improvements that we can gain that are not necessarily related to new features.

RonFed · 2026-01-21T17:19:29Z

+      componentProps:
+        required: false
+        tooltip: 'Skip TLS certificate verification'
+      renderCondition: ['CLICKHOUSE_TLS_ENABLED', '==', 'true']


should be rendered only when there is a certificate? or should be the opposite?

@RonFed wdym? This is the same as the otlphttp skip verify setting

odigos/destinations/data/otlphttp.yaml

Lines 183 to 190 in 0c278ae

- name: OTLP_HTTP_INSECURE_SKIP_VERIFY

displayName: Insecure Skip Verify

componentType: checkbox

initialValue: false

componentProps:

required: false

tooltip: 'Skip TLS certificate verification'

renderCondition: ['OTLP_HTTP_TLS_ENABLED', '==', 'true']

(they actually both use the upstream commmon TLS config structs)

@RonFed ah I think I get what you mean. Only show the insecure skip verify if no cert is provided? Yeah I would like to do that, but from what I can tell right now the render conditions don't allow multiple conditionals which would be nice. Talked to Ben about that and we could use that in some other destinations too, but for now I think it's fine

RonFed · 2026-01-21T17:27:22Z

+	tlsConfig := GenericMap{
+		"insecure": false,
+	}
+	if caPem, ok := dest.GetConfig()[clickhouseCaPem]; ok && caPem != "" {


should we validate the ca format here to be a valid one?

added a function to validate the ca format

damemi · 2026-01-21T21:36:10Z

Actually needs collector v141 due to this bug in clickhouse not handling all tls settings: open-telemetry/opentelemetry-collector-contrib#43911 fixed in open-telemetry/opentelemetry-collector-contrib#44093

Remove deprecated carbon exporter support (unmaintained upstream) open-telemetry/opentelemetry-collector-contrib#44532

another upstream breaking change giving go mod trouble open-telemetry/opentelemetry-collector#13948

configgrpc update: open-telemetry/opentelemetry-collector#13996

and now metadata.yaml metrics require stablity levels open-telemetry/opentelemetry-collector#13756

Error: failed loading /app/collector/receivers/odigosebpfreceiver/metadata.yaml: decoding failed due to the following error(s):

'telemetry.metrics[ebpf_memory_pressure_wait_time_total]' missing required field: `stability`
'telemetry.metrics[ebpf_total_bytes_read]' missing required field: `stability`
'telemetry.metrics[ebpf_lost_samples]' missing required field: `stability`
Error: failed loading /app/collector/receivers/odigosebpfreceiver/metadata.yaml: decoding failed due to the following error(s):

'telemetry.metrics[ebpf_memory_pressure_wait_time_total]' missing required field: `stability`
'telemetry.metrics[ebpf_total_bytes_read]' missing required field: `stability`
'telemetry.metrics[ebpf_lost_samples]' missing required field: `stability`
Error: metadata.yaml ordering check failed: [telemetry metrics] keys are not sorted: [odigos_log_data_size odigos_metric_data_size odigos_trace_data_size odigos_accepted_spans odigos_accepted_metric_points odigos_accepted_log_records]
Error: metadata.yaml ordering check failed: [telemetry metrics] keys are not sorted: [odigos_log_data_size odigos_metric_data_size odigos_trace_data_size odigos_accepted_spans odigos_accepted_metric_points odigos_accepted_log_records]

RonFed

Since all the k8s v1.20 tests are failing I guess something in the 141 bump caused issues in that version.
Looking at the EndpointSlice docs it seems it became stable on v1.21 - I guess this might have something to do with the tests failures.

@damemi Do you know which component has this requirement? this open-telemetry/opentelemetry-collector-contrib#44079 seems like one case that might cause this

damemi · 2026-01-26T13:14:45Z

E2E Tests / kubernetes-test (1.20.15, source) (pull_request)

Looking at the parent issue (open-telemetry/opentelemetry-collector-contrib#43891) it seems like

loadbalancing exporter
k8sattributes processor
k8sobjects receiver
and internal/aws

if that's the case then it might be a problem either blocking us from updating otel or forcing us to drop 1.20. looking into it

if endpointslices were stable in 1.21 then they should have at least been beta (on by default) in 1.20 🤔

damemi · 2026-01-26T13:57:59Z

@RonFed it does seem to be the issue (ran 1.20 test locally):

$ kubectl logs pod/odiglet-rxlpd -c data-collection -n odigos-test
2026-01-26T13:55:27.058Z	info	odigostrafficmetrics@v0.141.0/processor.go:38	Odigos traffic metrics processor is enabled with the following configuration	{"resource": {"k8s.node.name": "kind-control-plane", "k8s.pod.name": "odiglet-rxlpd", "service.instance.id": "aac4957b-c6a6-49c2-970d-cc6abe000a25"}, "otelcol.component.id": "odigostrafficmetrics", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "traces", "otelcol.signal": "traces", "sampling_ratio": "1.000000", "inverse_sampling_ratio": "1"}
2026-01-26T13:55:27.059Z	info	memorylimiter@v0.141.0/memorylimiter.go:71	Memory limiter configured	{"resource": {"k8s.node.name": "kind-control-plane", "k8s.pod.name": "odiglet-rxlpd", "service.instance.id": "aac4957b-c6a6-49c2-970d-cc6abe000a25"}, "otelcol.component.kind": "processor", "limit_mib": 450, "spike_limit_mib": 90, "check_interval": 0.5}
2026-01-26T13:55:27.041Z	info	odigosk8scmprovider@v0.141.0/provider.go:152	configuration retrieved from ConfigMap	{"resource": {"k8s.node.name": "kind-control-plane", "k8s.pod.name": "odiglet-rxlpd", "service.instance.id": "aac4957b-c6a6-49c2-970d-cc6abe000a25"}, "name": "odigos-data-collection", "namespace": "odigos-test"}
2026-01-26T13:55:27.062Z	info	service@v0.141.0/service.go:224	Starting odigosotelcol...	{"resource": {"k8s.node.name": "kind-control-plane", "k8s.pod.name": "odiglet-rxlpd", "service.instance.id": "aac4957b-c6a6-49c2-970d-cc6abe000a25"}, "Version": "0.141.0", "NumCPU": 12}
2026-01-26T13:55:27.062Z	info	extensions/extensions.go:40	Starting extensions...	{"resource": {"k8s.node.name": "kind-control-plane", "k8s.pod.name": "odiglet-rxlpd", "service.instance.id": "aac4957b-c6a6-49c2-970d-cc6abe000a25"}}
2026-01-26T13:55:27.062Z	info	extensions/extensions.go:44	Extension is starting...	{"resource": {"k8s.node.name": "kind-control-plane", "k8s.pod.name": "odiglet-rxlpd", "service.instance.id": "aac4957b-c6a6-49c2-970d-cc6abe000a25"}, "otelcol.component.id": "pprof", "otelcol.component.kind": "extension"}
2026-01-26T13:55:27.062Z	info	pprofextension@v0.141.0/pprofextension.go:61	Starting net/http/pprof server	{"resource": {"k8s.node.name": "kind-control-plane", "k8s.pod.name": "odiglet-rxlpd", "service.instance.id": "aac4957b-c6a6-49c2-970d-cc6abe000a25"}, "otelcol.component.id": "pprof", "otelcol.component.kind": "extension", "config": {"TCPAddr":{"Endpoint":"0.0.0.0:1777","DialerConfig":{"Timeout":0}},"BlockProfileFraction":0,"MutexProfileFraction":0,"SaveToFile":""}}
2026-01-26T13:55:27.062Z	info	extensions/extensions.go:61	Extension started.	{"resource": {"k8s.node.name": "kind-control-plane", "k8s.pod.name": "odiglet-rxlpd", "service.instance.id": "aac4957b-c6a6-49c2-970d-cc6abe000a25"}, "otelcol.component.id": "pprof", "otelcol.component.kind": "extension"}
2026-01-26T13:55:27.062Z	info	extensions/extensions.go:44	Extension is starting...	{"resource": {"k8s.node.name": "kind-control-plane", "k8s.pod.name": "odiglet-rxlpd", "service.instance.id": "aac4957b-c6a6-49c2-970d-cc6abe000a25"}, "otelcol.component.id": "health_check", "otelcol.component.kind": "extension"}
2026-01-26T13:55:27.062Z	info	healthcheckextension@v0.141.0/healthcheckextension.go:32	Starting health_check extension	{"resource": {"k8s.node.name": "kind-control-plane", "k8s.pod.name": "odiglet-rxlpd", "service.instance.id": "aac4957b-c6a6-49c2-970d-cc6abe000a25"}, "otelcol.component.id": "health_check", "otelcol.component.kind": "extension", "config": {"Endpoint":"0.0.0.0:13133","TLS":{},"CORS":{},"Auth":{},"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"CompressionAlgorithms":null,"ReadTimeout":0,"ReadHeaderTimeout":0,"WriteTimeout":0,"IdleTimeout":0,"Middlewares":null,"KeepAlivesEnabled":false,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2026-01-26T13:55:27.063Z	info	extensions/extensions.go:61	Extension started.	{"resource": {"k8s.node.name": "kind-control-plane", "k8s.pod.name": "odiglet-rxlpd", "service.instance.id": "aac4957b-c6a6-49c2-970d-cc6abe000a25"}, "otelcol.component.id": "health_check", "otelcol.component.kind": "extension"}
E0126 13:55:27.068275    5151 reflector.go:205] "Failed to watch" err="failed to list *v1.EndpointSlice: the server could not find the requested resource" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.EndpointSlice"
E0126 13:55:28.248412    5151 reflector.go:205] "Failed to watch" err="failed to list *v1.EndpointSlice: the server could not find the requested resource" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.EndpointSlice"
E0126 13:55:30.755185    5151 reflector.go:205] "Failed to watch" err="failed to list *v1.EndpointSlice: the server could not find the requested resource" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.EndpointSlice"
E0126 13:55:35.209142    5151 reflector.go:205] "Failed to watch" err="failed to list *v1.EndpointSlice: the server could not find the requested resource" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.EndpointSlice"
E0126 13:55:42.914484    5151 reflector.go:205] "Failed to watch" err="failed to list *v1.EndpointSlice: the server could not find the requested resource" logger="UnhandledError" reflector="k8s.io/client-go@v0.34.2/tools/cache/reflector.go:290" type="*v1.EndpointSlice"

RonFed · 2026-02-04T08:11:59Z

the changes lgtm, seems like all the 1.21 tests are failing though

damemi force-pushed the mike/core-554-add-tls-support-for-clickhouse-destination branch from 8567006 to e135186 Compare January 20, 2026 15:26

natyharary reviewed Jan 20, 2026

View reviewed changes

damemi force-pushed the mike/core-554-add-tls-support-for-clickhouse-destination branch from e135186 to 6e08959 Compare January 20, 2026 15:34

damemi changed the title ~~CORE-554: Add TLS settings to Clickhouse destination~~ CORE-554: Add TLS settings to Clickhouse destination + Update collector/otel to 136 + Remove deprecated components Jan 20, 2026

damemi force-pushed the mike/core-554-add-tls-support-for-clickhouse-destination branch 2 times, most recently from 7fe94ae to ec0ccf8 Compare January 21, 2026 16:53

damemi force-pushed the mike/core-554-add-tls-support-for-clickhouse-destination branch from ec0ccf8 to c08e16e Compare January 21, 2026 17:05

RonFed approved these changes Jan 21, 2026

View reviewed changes

damemi force-pushed the mike/core-554-add-tls-support-for-clickhouse-destination branch from c9b8b16 to cc62af6 Compare January 21, 2026 20:29

damemi force-pushed the mike/core-554-add-tls-support-for-clickhouse-destination branch 7 times, most recently from f4d2833 to 4422338 Compare January 22, 2026 21:24

tamirdavid1 approved these changes Jan 25, 2026

View reviewed changes

RonFed requested changes Jan 25, 2026

View reviewed changes

damemi changed the title ~~CORE-554: Add TLS settings to Clickhouse destination + Update collector/otel to 136 + Remove deprecated components~~ CORE-554: Add TLS settings to Clickhouse destination + Update collector/otel to 141 + Remove deprecated components Jan 26, 2026

damemi force-pushed the mike/core-554-add-tls-support-for-clickhouse-destination branch from 65eb76a to 391fde6 Compare February 2, 2026 13:30

damemi force-pushed the mike/core-554-add-tls-support-for-clickhouse-destination branch 2 times, most recently from 536273c to 34704f9 Compare February 2, 2026 19:32

damemi force-pushed the mike/core-554-add-tls-support-for-clickhouse-destination branch from a8b73ea to 7e43290 Compare February 4, 2026 15:43

damemi added 25 commits February 4, 2026 11:44

CORE-554: Add TLS settings to Clickhouse destination

d0a4ae4

Remove Loki exporter and OpsVerse destination

c385285

remove opencensus exporter

1ebc9af

Remove routing processor

c08c0fe

Update collector to 136

0f392e9

Update otel dependencies to match collector 136

d4d6627

make go-mod-tidy

09ea6e3

make genodigoscol generate

8141385

Add grpc netaddr default server config

9a63c23

Update metric naming

43bb8ef

Verify clickhouse pem formats

386abb3

Make TLS fields more clearer

8120a6c

Ignore temp go work files so I stop committing them

7e3ab7a

Update collector versions to 141 in otel deps

f7faaee

Update otel+collector to 141

5f8f8ef

make go-mod-tidy

36092b1

Add endpointslices permission

f031305

Use file-mounted ca pem

bb9cf58

Regenerate after rebase

50707bb

Update e2e workflow for 1.20.15->1.21.14

5edca6d

make go-modo-tidy after rebase

a247423

Move endpointslices to clusterrole for odiglet

0a06dbb

make rbac-docs

82a2e07

Switch to 1.21.1

1aee686

Switch to 1.21.12

a8bde6b

damemi force-pushed the mike/core-554-add-tls-support-for-clickhouse-destination branch from 7e43290 to a8bde6b Compare February 4, 2026 16:44

RonFed approved these changes Feb 4, 2026

View reviewed changes

damemi merged commit ea4f3cb into odigos-io:main Feb 4, 2026
173 of 178 checks passed

	- name: OTLP_HTTP_INSECURE_SKIP_VERIFY
	displayName: Insecure Skip Verify
	componentType: checkbox
	initialValue: false
	componentProps:
	required: false
	tooltip: 'Skip TLS certificate verification'
	renderCondition: ['OTLP_HTTP_TLS_ENABLED', '==', 'true']

Conversation

damemi commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

natyharary left a comment

Choose a reason for hiding this comment

Uh oh!

natyharary Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

damemi Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

damemi commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RonFed commented Jan 21, 2026

Uh oh!

damemi commented Jan 21, 2026

Uh oh!

RonFed commented Jan 21, 2026

Uh oh!

RonFed Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

damemi Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

damemi Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

RonFed Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

damemi Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

damemi commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RonFed left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

damemi commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

damemi commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RonFed commented Feb 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

damemi commented Jan 20, 2026 •

edited

Loading

damemi commented Jan 20, 2026 •

edited

Loading

damemi commented Jan 21, 2026 •

edited

Loading

RonFed left a comment •

edited

Loading

damemi commented Jan 26, 2026 •

edited

Loading

damemi commented Jan 26, 2026 •

edited

Loading