Consider switching to Docker for all tasks

[eu9ene]

It would be easier to maintain the docker images for all tasks in this repo compared to updating the generic worker image elsewhere every time we need to add something.

Also it would easier to reproduce running the tasks in a different environment, for example on a local machine or on premises cluster.

It's very common to run GPU workloads under Nvidia Docker toolkit https://docs.nvidia.com/ai-enterprise/deployment-guide-vmware/0.1.0/docker.html

[bhearsum]

We're already running docker tasks on generic-worker machine for CPU tasks. I did a quick test to see if its trivial to do this, but alas, #619 ran into some odd issues - it looks like generic-worker is not interpreting the payload as a docker-worker one. My read is that the "insecure" generic-worker engine that the GPU workers run doesn't support d2g, while the CPU workers use the "multi" engine that does support it. (I'm confirming this theory.)

Assuming this is true, we'll need to switch the GPU workers to the "multi" engine before we can proceed here. That will probably require some Release SRE involvement; I don't know how much work it is.

[bhearsum]

Taskcluster added support for d2g in the insecure engine: taskcluster/taskcluster#7031. It looks like they may have a follow-up to deal with there, but once it's stablized we'll need to roll out a new GPU worker image with an updated version of generic-worker, and then we can give this a try again.