Comments/Discussion on Sparse

[Yurlungur]

This is a discussion of @jlippuner 's sparse walkthrough #586

Wishlist:

• Can we have packs that only pull in allocated variables on the host side?
• Currently comm buffers are all pre-allocated whether or not the variable is. This is something that we may want to change, depending on "runtime experience". There are trade-offs between computation and memory here.
• Some tensor support needs to be re-introduced. See, e.g., Properly implement rank >= 2 variables #600 and Output does not support variables with reduced rank #606
• Variable sparse allocations might fall out of sync. Can we modify the infrastructure to tie variables with a given sparse ID together?
• Can allocation/deallocation be optimized? Perhaps with a memory pool? Kokkos memory pool or umpire?
• Performance analysis needs doing

Miscelany

• Developers to be aware: shallow copies of views don't change shape when views change shape

[Yurlungur]

For Jonas's sparse advection example, numbers are:

• With sparse/fake sparse: ~38s
• Compile-time disabled sparse: ~34s

Implies about a 20% overhead for the sparse machinery. However this example is very artificial and essentially a worst-case scenario as far as sparse overhead imposed on the run time. Real examples probably have even less overhead

[chadmeyer]

Of the things above, my biggest concerns from a practical perspective are (in order) the pre-allocation of the communications buffers and the possibility that individual components could be falling out of sync.

The pre-allocation issue is important for components which are localized within a large simulation (especially in 3D). That can be a huge amount of memory, and it would not seem to be a huge issue to only allocate buffers where the neighbors have allocated data on the mesh proper.

The sync issue is perhaps more complicated. Certainly some quantities need to be in sync, even if some components are zero everywhere (near the boundary at least). Most quantities like that (tied to a material) need to be able to be allocated to be filled in as a source term if nothing else. There are likely other use cases that I'm not considering where different components assigned to the same sparse index need not all be allocated, but that seems like a later improvement to the infrastructure than the more relevant "allocate all of the quantities associated with index n now" path.