Sampling helm chart docs do not represent correct, desired setup with k8s-monitoring-helm chart or provide any docs on how to link k8s-monitoring-helm to sampling.

[ab-andresc]

#2918 (comment)

This is outlined here, it is clear the docs in the chart do not specify this and it is extremely confusing to newcomers how this should be set up.

It would be advisable to tell users to use the following in their setup:

note: grafana-sampling-deployment would be the service pointing to the load balancing deployment and grafana-cloud being the namespace.

externalServices:
tempo:
host: http://grafana-sampling-deployment.grafana-cloud.svc.cluster.local:4318
authmode: none
secret:
create: false
tls:
insecure: true
protocol: "otlphttp"

[jorsmatthys]

I was looking for pointers on this as well. We have been using the k8s-monitoring helm (v1) for a while and we now want to implement tail-based based sampling, but there is no clear recommendation of how to use the two together.

Is the recommendation to gather traces using the k8s-monitoring (as suggested in this issue) and forward them to the loadbalancer of the grafana-sampling setup or is the recommendation to bypass the k8s-monitoring helm and send traces directly to the grafana-sampling setup?

... Or is there an option to send the traces to the sampling-deployment first and to keep the connection with Grafana Cloud in the k8s-monitoring deployment. (I suspect this is not an option since the values.yaml only shows how to connect with Grafana Cloud using specific env vars:

helm-charts/charts/grafana-sampling/values.yaml

Line 138 in aa76d2b

extraEnv:

)