[tempo-distributed] - Unable to change s3 endpoint

[jseiser]

{"caller":"main.go:124","err":"failed to init module services: error initialising module: store: failed to create store: unexpected error from ListObjects on xxxx-tempo-01-ops-dev: Get \"https://xxxx-tempo-01-ops-dev.s3.us
-gov-west-1.amazonaws.com/?delimiter=%2F&encoding-type=url&prefix=\": net/http: TLS handshake timeout","level":"error","msg":"error running Tempo","ts":"2025-02-06T14:33:09.300557188Z"}   

The first problem is, Tempo 2.7.0 is not able to operate behind AWS Network Firewall, the new Go 1.23 TLS changes break.

So to mitigate this, we have an AWS S3 endpoint in our VPC. Before upgrading tempo, we changed the values to reference this new endpoint, and things seem to work, but when we move to the 2.7.0 version, it all breaks with the above error. This leads me to believe tempo is not actually using the new S3 endpoint.

Our values file used to look like this

storage:
  trace:
    backend: s3
    s3:
      bucket: ${bucket_name}
      endpoint: s3.us-gov-west-1.amazonaws.com
      region: us-gov-west-1

We updated it to be this

storage:
  trace:
    backend: s3
    s3:
      bucket: ${bucket_name}
      endpoint: s3.vpce-xxxx-3tpx7dkq.s3.us-gov-west-1.vpce.amazonaws.com
      region: us-gov-west-1

We then re-deployed an older working version. Then we upgrade, and you get the error above, which def. references the old value.

The config map, is definitely updated.

kubectl get configmap -n tempo tempo-distributed-config -o yaml 

    storage:
      trace:
        backend: s3
        blocklist_poll: 5m
        local:
          path: /var/tempo/traces
        pool:
          max_workers: 400
          queue_depth: 20000
        s3:
          bucket: xxxx-tempo-01-ops-dev
          endpoint: s3.vpce-xxxx-3tpx7dkq.s3.us-gov-west-1.vpce.amazonaws.com
          region: us-gov-west-1

If I check a running pod, on the pre Go lang 1.23 release, the /conf/tempo.yml also shows the correct S3 endpoint.

[joe-elliott]

If you're setting the s3 endpoint and it does not seem to be working that might be a tempo bug and not a helm chart issue. We pass the endpoint here:

https://github.com/grafana/tempo/blob/main/tempodb/backend/s3/s3.go#L681

but we use this endpoint config internally and I think it's pretty widely used so I believe it works fine.

[jseiser]

I fail to find an explanation for what we see then. The endpoint is clearly showing in the logs to not match the endpoint being provided, or matching what's running on the previous version