Memory allocations

[charleskawczynski]

Transferred from CA#686, ClimaTimeSteppers is the culprit of most allocations:

This build seems to have properly filtered out the load times:

[ Info: allocs_perf_target_unthreaded: 10 unique allocating sites, 410944 total bytes
┌─────────────────────────────────────────────────────────────────────┬─────────────┬───────────────┐
│ <file>:<line number>                                                │ Allocations │ Allocations % │
│                                                                     │   (bytes)   │    (xᵢ/∑x)    │
├─────────────────────────────────────────────────────────────────────┼─────────────┼───────────────┤
│ ClimaTimeSteppers/y3D2E/src/solvers/imex_ark.jl:398                 │   338464    │      82       │
│ ClimaAtmos.jl/examples/hybrid/staggered_nonhydrostatic_model.jl:330 │    43008    │      10       │
│ ClimaTimeSteppers/y3D2E/src/solvers/imex_ark.jl:257                 │    16128    │       4       │
│ ClimaTimeSteppers/y3D2E/src/solvers/newtons_method.jl:60            │    12096    │       3       │
│ ClimaTimeSteppers/y3D2E/src/integrators.jl:103                      │     464     │       0       │
│ ClimaTimeSteppers/y3D2E/src/solvers/imex_ark.jl:384                 │     448     │       0       │
│ ClimaTimeSteppers/y3D2E/src/integrators.jl:101                      │     144     │       0       │
│ ClimaTimeSteppers/y3D2E/src/integrators.jl:106                      │     96      │       0       │
│ ClimaTimeSteppers/y3D2E/src/integrators.jl:102                      │     64      │       0       │
│ ClimaTimeSteppers/y3D2E/src/integrators.jl:138                      │     32      │       0       │
└─────────────────────────────────────────────────────────────────────┴─────────────┴───────────────┘

This points to ClimaTimeSteppers 0.2.4.

[charleskawczynski]

The latest build's allocation table is:

[ Info: allocs_perf_target_unthreaded: 6 unique allocating sites, 134672 total bytes
┌─────────────────────────────────────────────────────────────────────────┬─────────────┬───────────────┐
│ <file>:<line number>                                                    │ Allocations │ Allocations % │
│                                                                         │   (bytes)   │    (xᵢ/∑x)    │
├─────────────────────────────────────────────────────────────────────────┼─────────────┼───────────────┤
│ ClimaAtmos.jl/ClimaTimeSteppers/KReuM/src/solvers/imex_ark.jl:410       │    73920    │      55       │
│ ClimaAtmos.jl/examples/hybrid/staggered_nonhydrostatic_model.jl:251     │    32256    │      24       │
│ ClimaAtmos.jl/ClimaTimeSteppers/KReuM/src/solvers/imex_ark.jl:257       │    16128    │      12       │
│ ClimaAtmos.jl/ClimaTimeSteppers/KReuM/src/solvers/newtons_method.jl:484 │    12096    │       9       │
│ ClimaAtmos.jl/ClimaTimeSteppers/KReuM/src/integrators.jl:245            │     208     │       0       │
│ ClimaAtmos.jl/ClimaTimeSteppers/KReuM/src/integrators.jl:241            │     64      │       0       │
└─────────────────────────────────────────────────────────────────────────┴─────────────┴───────────────┘

So, we're doing much better after adding type parameters to the imex ARK algo (originally it was left as ::Any).

[charleskawczynski]

The latest build's allocation table is:

[ Info: allocs_perf_target_unthreaded: 3 unique allocating sites, 416 total bytes
┌─────────────────────────────────────────────────────────────────────┬─────────────┬───────────────┐
│ <file>:<line number>                                                │ Allocations │ Allocations % │
│                                                                     │   (bytes)   │    (xᵢ/∑x)    │
├─────────────────────────────────────────────────────────────────────┼─────────────┼───────────────┤
│ ClimaAtmos.jl/ClimaTimeSteppers/ZtUqN/src/integrators.jl:220        │     208     │      50       │
│ ClimaAtmos.jl/examples/hybrid/staggered_nonhydrostatic_model.jl:251 │     144     │      35       │
│ ClimaAtmos.jl/ClimaTimeSteppers/ZtUqN/src/integrators.jl:216        │     64      │      15       │
└─────────────────────────────────────────────────────────────────────┴─────────────┴───────────────┘

It's a bit unclear which lines are responsible, it looks like Julia (or Coverage or something) is tracking lines incorrectly. These lines point to:

• https://github.com/CliMA/ClimaTimeSteppers.jl/blob/v0.6.0/src/integrators.jl#L216
• https://github.com/CliMA/ClimaTimeSteppers.jl/blob/v0.6.0/src/integrators.jl#L220

I think the next step would be to add an allocation script into CTS that uses all (but albeit trivial) tendencies, limiters etc. This should help with turn around time

[charleskawczynski]

The remaining step! allocation issue is primarily an upstream:

• Broadcast.broadcast_shape for BlockedUnitRanges fails inference JuliaArrays/BlockArrays.jl#310

[charleskawczynski]

This issue has been side-stepped in ClimaCore, and is not longer an issue (and we've upgraded in #253). We do still have a few allocations with push!/pop!, but they're no longer the leading allocator. In fact, they're not even showing up in the ClimaAtmos allocation flame graphs, so I think it's time to close this.