add coupler summary table

[juliasloan25]

Purpose

closes #705

status 4/23

view passing build in coupler benchmarks pipeline here
view output to slack here

To do

• get CPU runs working
• check slack output
• verify SYPD calculations (make sure all units are seconds, compare to estimate from buildkite timing)
• add individual rows for CPU allocations vs GPU allocations (shouldn't be compared directly)
• include buildkite id in output
• use 4 GPUs

To do in separate PR(s)

• save output as .pdf rather than .txt - not a priority
  • use latex backend for PrettyTables.jl?
• add atmos-only run without diagnostic edmf
  • follow ClimaAtmos gpu_aquaplanet_dyamond
• add cpu/gpu state comparison columns
• pass in config file to benchmark script
  • use this to get run names (instead of passing these in), resolution & dt (instead of hardcoding these in table)

note: I was using CUDA.memory_status to get the GPU memory usage, but this varied wildly between runs (3-30GB). I'm not sure why this is, but it doesn't seem to be a reliable metric so I won't be using it going forward. Here is the code block calling that function, in case we want to return to it in the future:

    if comms_ctx.device isa ClimaComms.CUDADevice
        # If `io` is `stdout`, print the memory status, otherwise store in buffer to return
        CUDA.memory_status(io)
        if io != stdout
            # Get the memory usage in GB by parsing the output of `CUDA.memory_status`
            str = String(take!(io))
            println(str)
            gpu_allocs_GB = "GPU: " * split(split(str, '(')[2], '/')[1]
        end
    end

[juliasloan25]

In some runs, atmos outputs an SYPD that's much larger than the SYPD calculated here by timing the coupling loop. For example, in the "GPU slabplanet: albedo from function" run here, atmos prints an SYPD around 17, and we calculate it to be 8.15. I'm not sure where this difference is coming from since I'm using CA.EfficiencyStats and CA.simulated_years_per_day to calculate SYPD in the coupler driver

[LenkaNovak]

Thanks so much for implementing this, @juliasloan25 ! A super useful PR! 🎉 I just had a few comments / suggestions.

[LenkaNovak]

I'm also a little surprised about the SYPD for all runs reported on Slack. For example, this build finished running 0.25 sim years in ~0.25 wallclock days (including initialization), so SYPD should be around 1. I think the config should be the same, but the slack results imply that the benchmark runs are 100x slower. 🤔

[juliasloan25]

In some runs, atmos outputs an SYPD that's much larger than the SYPD calculated here by timing the coupling loop. For example, in the "GPU slabplanet: albedo from function" run here, atmos prints an SYPD around 17, and we calculate it to be 8.15. I'm not sure where this difference is coming from since I'm using CA.EfficiencyStats and CA.simulated_years_per_day to calculate SYPD in the coupler driver

After speaking with Charlier, it sounds like this might be because we're using Base.@elapsed to time the coupling loop, and this doesn't work well with CUDA kernels. I've changed it to use ClimaComms.@elapsed, which will call CUDA.@elapsed when we're running on GPU and should give more accurate numbers

[LenkaNovak]

Looks good, thank you! 🚀 I have a few comments and the parsed args need a fix. Then, presuming all CI runs are passing without change from the main, I think we can merge this.

[LenkaNovak]

I presume Base.precompile(solve_coupler!, (typeof(cs),)) doesn't work here?

[juliasloan25]

I haven't tried it! I can give that a go

[juliasloan25]

It works :)

[LenkaNovak]

Although I've just noticed that the latest SYPD is still outputting the older estimate (including the precompilation), so we may need to revert previous version.

[juliasloan25]

I looked into this some more and chatted with Charlie (see conversation on slack here), and it sounds like precompile only works if it can infer all the types, and even then still may not actually compile all the code. I opened an issue for us to fix the type inference and change to using precompile (#775), but for now I'll just keep it as running for 2 timesteps before running the full simulation

[LenkaNovak]

I'm approving this since the table works, but please do check the discrepancy between atmos/coupler allocations and that we're not seeing any behavioral changes on Buildkite once CI is fixed. Thanks for the great work, @juliasloan25 ! 🚀

[juliasloan25]

I'm approving this since the table works, but please do check the discrepancy between atmos/coupler allocations and that we're not seeing any behavioral changes on Buildkite once CI is fixed. Thanks for the great work, @juliasloan25 ! 🚀

I fixed CI and all the plots look the same as main 😄

I also changed the functions we're using to calculate allocations. Now there's one metric that's the same between CPU and GPU runs, which calculates the maximum CPU memory used over the course of the simulation (max RSS). The numbers don't look exactly like what I would expect (i.e. there are more allocations for atmos-only than coupled), but I think this is something we can easily change later on if we decide this isn't a reliable metric, and I don't want it to hold up this whole PR. I'll see if I can figure out what's going on with the allocation measurement tomorrow, but if nothing comes up I'm inclined to merge this as-is and investigate later on as we trigger more runs

Update: I added a GC.gc() call right before the coupling loop. Atmos has a gc call right before their solve, so this should produce more comparable results between the coupler and atmos. Calling GC.gc() may decrease reported Sys.maxrss (see slack thread). It looks like this does produce allocation results more like what we would expect, but there's also variation in allocations between runs.