Application controller memory leak

[tomjohnburton]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Just observing the resource usage for the application controller reveals a potential memory leak. At some point, the logs just stop.

To Reproduce

Deploy ArgoCD latest version.

Expected behavior

No memory leak.

Screenshots

Version

{
    "Version": "v2.8.0+804d4b8",
    "BuildDate": "2023-08-07T14:25:33Z",
    "GitCommit": "804d4b8ca6bc4c2cf02c5c971aa923ec5b8623f0",
    "GitTreeState": "clean",
    "GoVersion": "go1.20.6",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.1.0 2023-06-19T16:58:18Z",
    "HelmVersion": "v3.12.1+gf32a527",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.20.0"
}

Logs
Seeing a lot of the following logs

argocd-application-controller-0 application-controller {"level":"info","msg":"Alloc=218457 TotalAlloc=7342845364 Sys=669006 NumGC=64417 Goroutines=697","time":"2023-08-22T08:45:02Z"}
argocd-application-controller-0 application-controller {"level":"info","msg":"Alloc=222369 TotalAlloc=7345913723 Sys=669006 NumGC=64436 Goroutines=697","time":"2023-08-22T08:55:02Z"}
argocd-application-controller-0 application-controller {"level":"info","msg":"Alloc=321144 TotalAlloc=7348923090 Sys=669006 NumGC=64454 Goroutines=697","time":"2023-08-22T09:05:02Z"}
argocd-application-controller-0 application-controller {"level":"info","msg":"Alloc=167841 TotalAlloc=7351995234 Sys=669262 NumGC=64474 Goroutines=697","time":"2023-08-22T09:15:02Z"}
argocd-application-controller-0 application-controller {"level":"info","msg":"Alloc=311410 TotalAlloc=7355039331 Sys=669262 NumGC=64492 Goroutines=697","time":"2023-08-22T09:25:02Z"}

[alexmt]

Seeing it in prod as well. Compared two memory dumps and found this:

It is very similar to this issue: open-telemetry/opentelemetry-go-contrib#2413

Probably we are incorrectly using open-telemetry interceptor.

[alexmt]

We did upgrade otel libraries in 2.8 #13798

[alexmt]

It looks like this is the culprit: open-telemetry/opentelemetry-go-contrib#4226 . Preparing PR to re-use interceptors

[tomjohnburton]

Thanks for the quick fix!