2022.11.17 Meeting Notes

Parthenon paper

PG will ping AG on GH, otherwise write bio and submit

Refine ahead (of information arrival) is important, but the "depth" (even 0) may be criterion dependent.
There may even be advantages to check neighboring blocks (or communicate the info) "spreading refinemement info" (Teyssier et al? [but in the context of cell-based amr])
short term sol: check all cells minus stencil width and add comment
intermediate/long term: add more flexible capability (keep track with issue, https://github.com/parthenon-hpc-lab/parthenon/issues/787)

Question about how data is allocated/stored in packs
- currently two ways: original "split" and new a single view of view (latter is preferred and should remain as only option)
In AthenaK data is currently a 5D array, which works fine for static meshes but causes issue with mesh refinement (requiring lots of deep copies)
It's not a priori clear which approach is faster, especially when load balancing may rebuild the entire tree
Will likely be able to compare soon

Parthenon currently crashes when more memory is requested than available.
Might be nice to have a user option to either keep crashing or throw a warning and get going (limiting refinement)
https://github.com/parthenon-hpc-lab/parthenon/issues/788

There are general issue with legacy and new intel compilers (PG encountered many ICE)
Compared to Athena++, the vectorization with Intel compiler is not that great in AthenaK
JD wrote an example on how to pull out pointer directly, which helped vectorization

They ran/profiled the sedov tests with AMR (phydro)
There's lot of room for improvement on the Riemann solve. Could optimize memory access patterns.
Another issue is on hierarchical parallelism. We may need to set the team size manually as we potentially have too large teams (when using small blocks)
There may also be benefits to use async D2D copies during stage init
Will discuss potential integration of AmgX next time

Userspace prolong, keep design as is and separate larger overhaul (e.g, to define variable dependencies) in separate PR
Extended ci is now triggered automatically when "auto-merge" is enabled. This should ensure that extended CI passes prior to go into develop.

1 Dec