CI: vagrant takes too much time uploading cache.

[VannTen]

Which jobs are failing ?

The vagrant jobs (in particular molecule_no_container_engine) takes a disproportionate restoring and uploading cache :

Running with gitlab-runner 17.4.1 (32fe5502)
  on gitlab-runner-kubevirt-med-6784f978cb-f8fck Psf_DMUdQ, system ID: r_Li8WmMCMcgNn
Resolving secrets
Preparing the "custom" executor 00:26
Preparing environment 00:02
Getting source from Git repository 00:06
Restoring cache 08:46
W0207 14:40:58.329368  194225 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
Checking cache for molecule-no-container-engines-4-non_protected...
Runtime platform                                    arch=amd64 os=linux pid=1836 revision=c6eae8d7 version=17.5.2
Downloading cache from http://minio.minio.svc.cluster.local:9000/runner-cache/project/9659330/molecule-no-container-engines-4-non_protected  ETag="b7336f58c90ffec6fa6de74fabc37972"
Successfully extracted cache
Executing "step_script" stage of the job script 07:46
Running after_script 00:01
W0207 14:57:30.932030  198456 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
Running after script...
$ ./tests/scripts/molecule_logs.sh
Saving cache for successful job 11:50
W0207 14:57:31.178361  198471 warnings.go:70] kubevirt.io/v1alpha3 is now deprecated and will be removed in a future release.
Creating cache molecule-no-container-engines-4-non_protected...
Runtime platform                                    arch=amd64 os=linux pid=5781 revision=c6eae8d7 version=17.5.2
.vagrant.d/boxes: found 49 matching artifact files and directories 
.cache/pip: found 23 matching artifact files and directories 
Uploading cache.zip to http://minio.minio.svc.cluster.local:9000/runner-cache/project/9659330/molecule-no-container-engines-4-non_protected 
Created cache
Uploading artifacts for successful job 00:02
Cleaning up project directory and file based variables 00:00
Job succeeded

The cache retrieval and upload account for 2/3 of the job duration.

Which tests are failing ?

https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/-/jobs/9075178461

Since when has it been failing ?

not sure

Reason for failure

To investigate.

Anything else we need to know

We can't drop the cache completely, because otherwise we're facing 429 from vagrant boxes hosting.

I'm of the opinion that the least painful option would be to convert the molecule jobs provisioning to kubevirt but there might be better options.