Attempts to make Drone play nicer with Docker Hub

[tcsc]

The Story so far....

1. Modelled after the AWS credential handling, I've created temporary volumes to store the docker credentials for each workflow run.
2. I've added references to the above volume top any step that also mounts the dind dockersock volume. (At least all that I can find, anyway).
3. I've added a dockerhub login command to the wait-for-docker step that logs into dockerhub and writes the login credentials to the volume
4. Drone itself still downloads heaps of images itself - all of the steps that use a container to isolate themselves (i.e. everything in a kubernetes pipeline, for example) need an image in order to spawn a container to run in. So,
   1. I've added Dockerhub credentials to each non-exec workflow as per this document
   2. Docker will cache images on its own, but only if that image is pinned to a version. If it's pinned to latest Drone will re-download it every time it is used. I've added a bunch of pull: if-not-exists properties in places that are referencing weakly-pinned images in order to get Drone to try and cache them. The Docs only say "Drone" will cache them, but it doesn't specify the scale... is it per executor, the whole cluster, or just inside one piepline? Who knows? Maybe this will help.
5. I've added new secrets to the GCP Drone and the Drone SOPS file yo supply the credentials used above (See gravitational/ops#485)

[wadells]

Drone itself still downloads heaps of images itself - all of the steps that use a container to isolate themselves (i.e. everything in a kubernetes pipeline, for example) need an image in order to spawn a container to run in. So,

https://github.com/gravitational/cloud-terraform/pull/1839/files#diff-3f6a82b01dffe1fec2af511f783801a51bf4a4074f74a51b0e712f29f537ea8fR75-R81 should hopefully help with this.

[wadells]

Update -- I'm working off the https://github.com/gravitational/teleport-private/tree/walt/dockerhub-login branch.

I'm currently chasing two issues:

1. I think teleport buildboxes pull docker images as part of the build here:

teleport/build.assets/Dockerfile

Lines 253 to 263 in 99c5002

	RUN helm plugin install https://github.com/quintush/helm-unittest --version 0.2.11 && \
	mkdir -p /home/ci/.local/share/helm && \
	cp -r /root/.local/share/helm/plugins /home/ci/.local/share/helm/plugins-new && \
	chown -R ci /home/ci/.local/share/helm && \
	helm plugin uninstall unittest && \
	HELM_PLUGINS=/home/ci/.local/share/helm/plugins-new helm plugin list
	RUN helm plugin install https://github.com/vbehar/helm3-unittest && \
	mkdir -p /home/ci/.local/share/helm && \
	cp -r /root/.local/share/helm/plugins /home/ci/.local/share/helm && \
	chown -R ci /home/ci/.local/share/helm && \
	HELM_PLUGINS=/home/ci/.local/share/helm/plugins helm plugin list

I'm still debugging / trying to get the credentials properly plumbed through.

2. I've started encontering a lot of Get : <internal url> remote error: tls: internal error errors. These intermittently present on the old cluster (as slack search will show), but they're causing noise on the testing I'm doing above. I suspect this is either teleport -> drone or drone -> kube-runner related -- not related to the login creds afaik.

See https://drone.platform.teleport.sh/gravitational/teleport-private/528/11/2 for an example.

[wadells]

I've got a cleaned up version of this PR at #23956