Improvements to longruns pipeline

[Sbozzolo]

This PR:

• Removed the longrun depot. This avoid potential depot corruptions
• Switches from mpiexec to srun for a more reliable MPI experience.

[LenkaNovak]

LGTM, just checked this fix here: https://buildkite.com/clima/climacoupler-longruns/builds/448

[LenkaNovak]

LGTM, just checked this fix here: https://buildkite.com/clima/climacoupler-longruns/builds/448

Actually, even though the init is now passing, it looks like we need to constrain the Plots dependency again?

[Sbozzolo]

I cleaned up a couple of things in the manifest and it looks like it's working:

https://buildkite.com/clima/climacoupler-longruns/builds/449

[szy21]

There seem to be some artifact errors: https://buildkite.com/clima/climacoupler-longruns/builds/449#018dc853-7e02-4d95-a6d0-8e9acaeb0b47/219-8134

[szy21]

Also all the breaking simulations hang. Maybe we need to add SLURM_KILL_BAD_EXIT: 1 to the pipeline?

[Sbozzolo]

Also all the breaking simulations hang. Maybe we need to add SLURM_KILL_BAD_EXIT: 1 to the pipeline?

Added

There seem to be some artifact errors: https://buildkite.com/clima/climacoupler-longruns/builds/449#018dc853-7e02-4d95-a6d0-8e9acaeb0b47/219-8134

This seems like a random filesystem failure. I am trying again.

[Sbozzolo]

ArgumentError: '/central/scratch/esm/slurm-buildkite/climacoupler-longruns/450/depot/default/artifacts/845995bb777cf5a3920541585d62c087f62e5cb5

I think this is a race condition on downloading artifacts :(

https://buildkite.com/clima/climacoupler-longruns/builds/450#018dcc4a-c9ef-4d0d-84b0-2c7ec9ccddb5

[LenkaNovak]

ArgumentError: '/central/scratch/esm/slurm-buildkite/climacoupler-longruns/450/depot/default/artifacts/845995bb777cf5a3920541585d62c087f62e5cb5

I think this is a race condition on downloading artifacts :(

https://buildkite.com/clima/climacoupler-longruns/builds/450#018dcc4a-c9ef-4d0d-84b0-2c7ec9ccddb5

Hmm, I haven't seen this one before. Maybe we should deal with the topo files before running the job like the rest of the artifacts here, rather than on the fly when initiating atmos?

[Sbozzolo]

ArgumentError: '/central/scratch/esm/slurm-buildkite/climacoupler-longruns/450/depot/default/artifacts/845995bb777cf5a3920541585d62c087f62e5cb5
I think this is a race condition on downloading artifacts :(
https://buildkite.com/clima/climacoupler-longruns/builds/450#018dcc4a-c9ef-4d0d-84b0-2c7ec9ccddb5

Hmm, I haven't seen this one before. Maybe we should deal with the topo files before running the job like the rest of the artifacts here, rather than on the fly when initiating atmos?

Ideally we should fix that (xref: CliMA/ClimaLand.jl#467), but yes, best to ensure that all the artifacts are downloaded before starting the simulations for the time being.

[Sbozzolo]

Okay, this time everything is looking good:

https://buildkite.com/clima/climacoupler-longruns/builds/451