O1.2.5 Atmos-land simulations on GPU at 1 SYPD on 4 A100s

[LenkaNovak]

The Climate Modeling Alliance

Software Design Issue 📜

Purpose

We need to promote the GPU capability into the coupler code, still sharing the same the distribution context between Atmos and the surfaces (which are on the same grid), as we do with MPI.

This will be done in two stages, first making the minimal changes needed to get simulations and tests to run, then secondly propagating these changes throughout the repo and ensuring high code quality throughout.

Cost/Benefits/Risks

b: faster AMIP + DYAMOND

People and Personnel

• Lead: @sriharshakandala
• Collaborators: @juliasloan25 @LenkaNovak
• Reviewers:

Inputs

• ClimaAtmos release (with a fixed insolation)
• ClimaLSM GPUfication (for AMIP and DYAMOND)

Results and Deliverables

• GPU Buildkite run with the target AMIP setup, which is within ~5% performance of the standalone Atmos run (as in the MPI case - though double check this).
• key cases in Buildkite on GPU: slabplanet, target AMIP, DYAMOND (excluding and including topography)

Task Breakdown And Schedule

initial runs [1 April 2024]

Beta Give feedback

1. setup a DYAMOND sim on CPU for future GPU profiling Revamp long run physics specification #457
2. update for the ClimaAtmos v0.16.3 for GPU compatibility use new package versions #502
3. run slabplanet sim on GPU #530
   🍃 leaf GPU +2
   
4. run GPU slabplanet with albedo from file #590
   🍃 leaf GPU +2
   
5. run simple AMIP sim on GPU #572
   🍃 leaf GPU +2
   
6. run target AMIP runs on GPU #630
   🍃 leaf GPU +2
   
7. fix GPU job agent requirements in pipeline #662
   🍃 leaf GPU bug +3
   
8. run GPU AMIP sims with bucket albedo from file #583
   🍃 leaf GPU +2
   
9. run DYAMOND on GPU #658
   🍃 leaf GPU +2
   
10. investigate GPU conservation #607
    🔥 Urgent 🍃 leaf GPU +3
    
11. Use 4xA100 to achieve > 1SYPD for target EDMF AMIP #740
    🔥 Urgent 🍃 leaf GPU +3
    

GPU performance [1 June 2024]

Beta Give feedback

1. add coupler summary table #705
   🍃 leaf GPU enhancement +3
   
2. Add 100km targets to longruns #752
   🍃 leaf +1
   
3. investigate allocations #683
   🍃 leaf GPU +2
   
4. Use 4xA100 to achieve > 1SYPD for target EDMF AMIP #740
   🔥 Urgent 🍃 leaf GPU +3
   

SDI Revision Log

CC

@tapios @cmbengue

[LenkaNovak]

@sriharshakandala , @juliasloan25 Thanks for the meeting. Above I've added some details on what we discussed - please feel free to modify it as you feel fit. After CliMA/ClimaAtmos.jl#2341 is merged, let's release ClimaAtmos, after which we can update the coupler and get started on the GPU slabplanet. Looking forward to some fast testing! 🚀