feat(helm): Add health probes to long-lived containers

[lindhe]

What this PR does / why we need it:

This PR attempts to add configurable health probes to all long-lived containers where they are currently missing, to improve Kubernetes' lifecycle management of Loki's pods.

Which issue(s) this PR fixes:

Fixes #13967
Partially addresses #13678

Special notes for your reviewer:

• This PR grew significantly from what I imagined initially. Tell me if this should be split into multiple PRs instead!
• There are more components with missing health probes than I can test. Some I can test, because they are used in my current setup. Others I can only visually inspected that the rendered templates looks reasonable.
• I think we should create a follow-up PR that handles the remaining parts of Add support for liveness and startup probes in loki helm chart #13678 and generally aims to optimize all health probes.
• This PR does not add probes to init containers. While sometimes a bit tricker to write good probes for them, it's every bit as important as for long-lived containers. But I think that belongs in a separate PR. Same goes for containers in Jobs.
• I am trying to keep the code style consistent within each file. But I do notice that there are many places where the same thing (e.g. inserting a readinessProbe) is done in slightly different ways. I think it might be a good thing to eventually streamline the code style, but this too belongs in another PR.
• I've noticed that there are many unused values in values.yaml (e.g. .Values.bloomGateway.livenessProbe before this PR). It's probably worth auditing the values and attempt to catch any such cases (in another PR).

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[lindhe]

The failing test worked before I rebased. I'll see if I can understand why it's failing, but I'd guess it's not my fault.

[AndreZiviani]

visually it looks ok (did not actually test it) but you are changing some behaviors that I'm not sure you should be doing, for example: readiness and liveness only run after startup (not in parallel) so now instead of loki-gateway (or any other you configured) taking 15 seconds to be available (readinessProbe initial delay) it will take 20 seconds (startupProbe initial delay + readinessProbe initial delay). Maybe leave them commented out on values.yaml in order to not change current behavior

I'm not a maintainer, just some feedback from our discussion on Slack

[lindhe]

Thank you, that is great input. I agree, we should set the new probes to {} so the behavior is unchanged! But if the user chooses to, or if a future PR wants to make improvements on the default values, it's now possible to configure!

I'll fix that.

[lindhe]

I have now updated the default values so that I introduce no new probes via the values set in values.yaml. I think there may be some changes where probes used to not exist but now they use the shared .loki.* probes, for example. But this variant should be a lot more careful than the last revision I pushed.

Looking forward to more input!

[lindhe]

Apparently, empty probes for Deployments and DaemonSets are no good. I'll fix it.

[lindhe]

Pretty sure this error is not my fault:

[github-actions]

Kubernetes Manifest Diff Summary

Scenario: default-single-binary-values (Added: 0, Modified: 8, Removed: 0)

Summary:

• Added: 0

• Modified: 8

• Removed: 0

Added Files

No added files

Modified Files

loki/templates/compactor/statefulset-compactor.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/compactor/statefulset-compactor.yaml	2024-12-14 15:44:10.919877504 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/compactor/statefulset-compactor.yaml	2024-12-14 15:44:13.991894339 +0000
***************
*** 74,79 ****
--- 74,80 ----
 initialDelaySeconds: 30
 timeoutSeconds: 1
 
+ 
 volumeMounts:
 - name: temp
 mountPath: /tmp

loki/templates/distributor/deployment-distributor.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/distributor/deployment-distributor.yaml	2024-12-14 15:44:10.918877499 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/distributor/deployment-distributor.yaml	2024-12-14 15:44:13.991894339 +0000
***************
*** 72,79 ****
 port: http-metrics
 initialDelaySeconds: 30
 timeoutSeconds: 1
- livenessProbe:
- null
 volumeMounts:
 - name: config
 mountPath: /etc/loki/config
--- 72,77 ----

loki/templates/querier/deployment-querier.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/querier/deployment-querier.yaml	2024-12-14 15:44:10.918877499 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/querier/deployment-querier.yaml	2024-12-14 15:44:13.991894339 +0000
***************
*** 79,86 ****
 port: http-metrics
 initialDelaySeconds: 30
 timeoutSeconds: 1
- livenessProbe:
- null
 volumeMounts:
 - name: config
 mountPath: /etc/loki/config
--- 79,84 ----

loki/templates/index-gateway/statefulset-index-gateway.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/index-gateway/statefulset-index-gateway.yaml	2024-12-14 15:44:10.919877504 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/index-gateway/statefulset-index-gateway.yaml	2024-12-14 15:44:13.991894339 +0000
***************
*** 69,76 ****
 port: http-metrics
 initialDelaySeconds: 30
 timeoutSeconds: 1
- livenessProbe:
- null
 volumeMounts:
 - name: config
 mountPath: /etc/loki/config
--- 69,74 ----

loki/templates/ingester/statefulset-ingester-zone-b.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/ingester/statefulset-ingester-zone-b.yaml	2024-12-14 15:44:10.919877504 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/ingester/statefulset-ingester-zone-b.yaml	2024-12-14 15:44:13.991894339 +0000
***************
*** 91,96 ****
--- 91,97 ----
 initialDelaySeconds: 30
 timeoutSeconds: 1
 
+ 
 volumeMounts:
 - name: config
 mountPath: /etc/loki/config

loki/templates/ingester/statefulset-ingester-zone-c.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/ingester/statefulset-ingester-zone-c.yaml	2024-12-14 15:44:10.919877504 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/ingester/statefulset-ingester-zone-c.yaml	2024-12-14 15:44:13.991894339 +0000
***************
*** 91,96 ****
--- 91,97 ----
 initialDelaySeconds: 30
 timeoutSeconds: 1
 
+ 
 volumeMounts:
 - name: config
 mountPath: /etc/loki/config

loki/templates/ingester/statefulset-ingester-zone-a.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/ingester/statefulset-ingester-zone-a.yaml	2024-12-14 15:44:10.919877504 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/ingester/statefulset-ingester-zone-a.yaml	2024-12-14 15:44:13.991894339 +0000
***************
*** 91,96 ****
--- 91,97 ----
 initialDelaySeconds: 30
 timeoutSeconds: 1
 
+ 
 volumeMounts:
 - name: config
 mountPath: /etc/loki/config

loki/templates/query-scheduler/deployment-query-scheduler.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/query-scheduler/deployment-query-scheduler.yaml	2024-12-14 15:44:10.919877504 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/query-scheduler/deployment-query-scheduler.yaml	2024-12-14 15:44:13.991894339 +0000
***************
*** 70,77 ****
 port: http-metrics
 initialDelaySeconds: 30
 timeoutSeconds: 1
- livenessProbe:
- null
 volumeMounts:
 - name: config
 mountPath: /etc/loki/config
--- 70,75 ----

Removed Files

No removed files

Scenario: default-values (Added: 0, Modified: 0, Removed: 0)

Summary:

• Added: 0

• Modified: 0

• Removed: 0

Added Files

No added files

Modified Files

No modified files

Removed Files

No removed files

Scenario: ingress-values (Added: 0, Modified: 0, Removed: 0)

Summary:

• Added: 0

• Modified: 0

• Removed: 0

Added Files

No added files

Modified Files

No modified files

Removed Files

No removed files

Scenario: legacy-monitoring-values (Added: 0, Modified: 0, Removed: 0)

Summary:

• Added: 0

• Modified: 0

• Removed: 0

Added Files

No added files

Modified Files

No modified files

Removed Files

No removed files

Scenario: simple-scalable-aws-kube-irsa-values (Added: 0, Modified: 1, Removed: 0)

Summary:

• Added: 0

• Modified: 1

• Removed: 0

Added Files

No added files

Modified Files

loki/templates/gateway/deployment-gateway-enterprise.yaml

*** /home/runner/work/loki/loki/output/base/simple-scalable-aws-kube-irsa-values/loki/templates/gateway/deployment-gateway-enterprise.yaml	2024-12-14 15:44:11.325879722 +0000
--- /home/runner/work/loki/loki/output/pr/simple-scalable-aws-kube-irsa-values/loki/templates/gateway/deployment-gateway-enterprise.yaml	2024-12-14 15:44:14.403896602 +0000
***************
*** 28,34 ****
 app.kubernetes.io/component: gateway
 annotations:
 checksum/config: 8cfa6bbdb9a4dbb7b195fc7162e2fddac7836db36a03aa494881c1fc01f7c3ac
! spec: 
 serviceAccountName: enterprise-logs
 securityContext:
 fsGroup: 10001
--- 28,34 ----
 app.kubernetes.io/component: gateway
 annotations:
 checksum/config: 8cfa6bbdb9a4dbb7b195fc7162e2fddac7836db36a03aa494881c1fc01f7c3ac
! spec:
 serviceAccountName: enterprise-logs
 securityContext:
 fsGroup: 10001

Removed Files

No removed files

[lindhe]

I've rebased on main a couple of times now, still getting this error:

I cannot image that is caused by my changes. So I have to wait until someone fixes main, I guess. 🤷