add GPU DYAMOND runs

[juliasloan25]

Purpose

closes #658

Only adding a longrun, no shortrun.
This run exceeds the memory available on P100s. Caltech's V100s have 16GB and 32 GB options, neither of which is large enough for this job, according to https://www.hpc.caltech.edu/resources. Instead of running on central like the rest of the longruns, this job will run on clima (which has A100s with 80GB of memory). I've opened an issue to address the allocations seen in this run: #683

view run on buildkite here: https://buildkite.com/clima/climacoupler-longruns/builds/480#_

Content

• add config file based on config/longrun_configs/dyamond_target.yml for longrun
  • set anim: false for gpu-compatibility
• add longrun using GPU

---

• I have read and checked the items on the review checklist.

[LenkaNovak]

Is there a way we could request an H100 for this job only? I don't think the allocation enhancements will be addressed anytime soon. If that's possible, I would suggest commenting out the regular CI job for now, but retaining the longrun one, which we only run once a week on Sundays.

[juliasloan25]

Is there a way we could request an H100 for this job only? I don't think the allocation enhancements will be addressed anytime soon. If that's possible, I would suggest commenting out the regular CI job for now, but retaining the longrun one, which we only run once a week on Sundays.

ClimaAtmos has a separate buildkite pipeline that runs target GPU simulations on clima (see the runs and the pipeline.yml itself). I can implement the same thing for us

[LenkaNovak]

Is there a way we could request an H100 for this job only? I don't think the allocation enhancements will be addressed anytime soon. If that's possible, I would suggest commenting out the regular CI job for now, but retaining the longrun one, which we only run once a week on Sundays.

ClimaAtmos has a separate buildkite pipeline that runs target GPU simulations on clima (see the runs and the pipeline.yml itself). I can implement the same thing for us

Does this allow us to specify the hardware for just one run though?

[juliasloan25]

Is there a way we could request an H100 for this job only? I don't think the allocation enhancements will be addressed anytime soon. If that's possible, I would suggest commenting out the regular CI job for now, but retaining the longrun one, which we only run once a week on Sundays.

ClimaAtmos has a separate buildkite pipeline that runs target GPU simulations on clima (see the runs and the pipeline.yml itself). I can implement the same thing for us

Does this allow us to specify the hardware for just one run though?

No, it would be a separate pipeline where this job would be run. I think this will be useful for GPU scaling runs too

[LenkaNovak]

LGTM, thank you, @juliasloan25. Just had a question about the sim length.

[LenkaNovak]

If it's a long run, could we run it for longer (e.g. 50 days) or does the simulation crash? 👀