[Bug] Persist irrelevant Kubernetes Secrets

[kevin85421]

Describe the bug

When you create a Kind cluster, the Kubernetes Secret bootstrap-token-abcdef in the kube-system namespace will also be created. However, NGF watches all Kubernetes Secrets in the Kubernetes cluster. Is it necessary to persist these irrelevant Kubernetes Secrets?

{"level":"info","ts":"2023-12-25T08:15:03Z","msg":"Upserted the resource","controller":"secret","controllerGroup":"","controllerKind":"Secret","Secret":{"name":"bootstrap-token-abcdef","namespace":"kube-system"},"namespace":"kube-system","name":"bootstrap-token-abcdef","reconcileID":"acf04978-8167-4300-9249-d67caa5c19a5"}

To Reproduce

• Create a Kind cluster
• Deploy NGF
• Check logs

Expected behavior

The irrelevant secret should not be reconciled and persisted.

[sjberman]

The controller will see all Secrets because of our current RBAC rules. We can't exclude specific Secrets, it's all or nothing. However, we don't actually do any processing if the Secret is irrelevant. We see it, log a message, and ignore it.

There could be an opportunity to improve the logging, though. I see a couple of options for improvement:

1. In the reconciler (where we log "Upserted the resource"), we could move that log message to the debug level instead of info to reduce the noise.
2. If we ignore a resource, add a log message for that (probably also at the debug level).
3. If we actually process and use a resource, ensure that we log that at the info level.

[pleshakov]

There are few possible improvements:

(1) For the controller for Secrets, we can make it watch for metadata changes, instead of the whole object changes. This should prevent NGF from caching all cluster Secrets in its memory. This was implemented for the controller which watches for cluster CRDs -- d6bbdba In this case, only when NGF needs some secret(s), it will fetch it from the API.

(2) Restrict access to Secrets, so that fewer secrets will be watched -- #1321

We also have plans to split control plane from data plane into separate deployments, so that the data plane pods will not have direct access to Kubernetes APIs and thus any cluster Secrets, but only the ones used in generated NGINX configuration, fetched from the control plane pod(s).

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.