Enable consumers of gitops-engine API to specify an alternative temp directory to '/dev/shm', for generated manifests

[jgwest]

At present, gitops-engine (and consumers of gitops-engine API) use /dev/shm to store temporary files that are passed to kubectl, such as manifests and cluster credentials. The temp directory logic can be found within pkg/utils/io/io.go.

One disadvantage of this is that since /dev/shm is backed by RAM, it has limited size. For many use cases this is not an issue, however, when concurrently deploying to a large number of Kubernetes clusters, or deploying large k8s manifests, /dev/shm can run out of disk space.

One option is that we could just move to using /tmp, rather than /dev/shm. But, it appears that /dev/shm was chosen intentionally to ensure that sensitive data (such as credentials) are kept off disk (absent disk caching of memory).

Thus, rather than moving to use /tmp, I propose instead allowing API consumers to optionally use a different path, which can then by enabled by API consumers (for example, Argo CD configmap value or env var)

I've also been tackling this problem by reducing the leaking of files into /dev/shm. You can see that work below.

Related issues:

• In 'ManageResources' and 'runResourceCommand', temporary files written to '/dev/shm' may not be cleaned up, in rare cases
  • In 'ManageResources' and 'runResourceCommand', temporary files written to '/dev/shm' may not be cleaned up, in rare cases #568
  • fix: temporary files written to '/dev/shm' not cleaned up (#568) #569
• 'Apply/ReplaceResource' in resource_ops.go may leak files to '/dev/shm' since the kubectl 'apply/replace' commands never time out
  • 'Apply/ReplaceResource' in resource_ops.go may leak files to '/dev/shm' since the kubectl 'apply/replace' commands never time out #572
• 'util/git/creds.go' will leak temporary files to '/dev/shm' if proxy URL cannot be parsed, or if write fails
  • 'util/git/creds.go' will leak temporary files to '/dev/shm' if proxy URL cannot be parsed, or if write fails argo-cd#17658
  • fix: prevent leak of files to /dev/shm in corner cases (#17658) argo-cd#17659