Add Independent Sparse Thresholds and Sparse Allocation Control Fields #699

lroberts36 · 2022-07-27T02:50:51Z

PR Summary

This PR adds a few different sparse variable features and fixes some bugs:

Ability to specify the sparse allocation and deallocation thresholds for individual fields.
Functionality to allocate/deallocate sparse fields based on the allocation/deallocation status of another sparse field (i.e. controlled and controlling fields).
Allow for default values of unallocated or newly allocated fields to have values other than zero.
NaN payload tags for flagging unitialized, newly allocated sparse fields.
Sparse advection test that uses these new features.
There are also a couple fixes of small bugs in the sparse machinery.
Include better checks on whether or not packs containing sparse variables are stale. The old machinery could result in packs being used that were stale.

The first two features are being used downstream in Riot.

PR Checklist

Code passes cpplint
New features are documented.
Adds tests for new features.
Code is formatted
Changes are summarized in CHANGELOG.md
CI has been triggered on Darwin for performance regression tests.
(@lanl.gov employees) Update copyright on changed files

…rse-thresholds Conflicts: src/utils/concepts_lite.hpp

…t-sparse-thresholds Conflicts: src/interface/mesh_data.hpp src/kokkos_abstraction.hpp

…t-sparse-thresholds

lroberts36 · 2022-07-27T02:54:35Z

@par-hermes format

…t-sparse-thresholds

lroberts36 · 2022-11-01T02:49:34Z

@pgrete Can you give this a review when you get the chance? Although there might be some more change to the sparse framework as we work on things in Riot, I think at this point it would be good to get this into develop. It had develop had diverged from it significantly over the last month, so I think my original plan of waiting until things were completely stable in Riot may not be a good one (in particular I am thinking of changes coming as we include face, edge, and vertex variables).

pgrete

First bunch of comments, more tomorrow.

pgrete · 2022-11-23T16:27:11Z

docs/interface/sparse.md

+- *Allocation due to neighbors:* If a sparse field on a neighbor block passes ghost data that is 
+  anywhere above the `allocation_threshold`, the receiving block will allocate the sparse field if 
+  it is unallocated before loading the ghost data. `allocation_threshold` is set in the `Metadata` 
+  of a field.  A detailed description of sparse ghost zone communication is 
+  given [here](../sparse_boundary_communication.md). 
+- *Allocation due to other fields:* In some instances, we may want to allocate or deallocate a 


It may be good to document the logical relationship between those two mechanism, i.e., are they exclusive or, or, and and, if both are defined if there's an error or which one takes precedence.

I think that it should be or between allocation due to neighbors and allocation due to other fields. We should always default to allocating a field if anything requires allocation, since that won't degrade the accuracy of the calculation.

Right now, there is no mechanism for allocation due to other fields in the code.

Right now, there is no mechanism for allocation due to other fields in the code.

Now I'm confused. I thought this PR is about adding this functionality (as described also here in the doc). What am I missing?

Sorry, there are two ideas here. The first is that you would like to always allocate one field whenever some other field is allocated. This is what this PR introduced. The second, which is not implemented, is allocating a field based upon some more complicated criterion in the interior of a block (e.g. the temperature being above/below some threshold).

docs/interface/sparse.md

example/sparse_advection/sparse_advection_driver.cpp

src/utils/nan_payload_tag.hpp

tst/unit/test_nan_tags.cpp

pgrete

Alright. I'm done with the review. I left a couple of comments (nothing too big) that I'd like to see/hear an answer first before doing the final (swift) approve.

tst/regression/test_suites/advection_outflow/advection_outflow.py

tst/regression/test_suites/poisson/poisson.py

tst/regression/test_suites/output_hdf5/output_hdf5.py

src/bvals/cc/bnd_info.cpp

src/bvals/cc/boundary_communication.cpp

src/interface/update.cpp

src/interface/update.hpp

src/interface/variable.hpp

Co-authored-by: Philipp Grete <[email protected]>

lroberts36 · 2022-11-28T19:16:03Z

Thanks for the review @pgrete! I think I have responded to all your comments.

pgrete

Thanks for addressing my comments.
I'm approving now (but would still be happy for final replies to the unresolved conversations).

Also one final note on sparse in general. The complexity/logic of the implementation is now beyond what I was able to follow based on the docs.
I know there are more pressing concerns in terms of getting things work, but at some (hopefully not too distant future), I'd be very happy to see a more detailed (developer) doc for the internals of the parse machinery.
I'm probably also biased as I again spent hours this week trying to track down #659 with little progress,

lroberts36 added 11 commits July 25, 2022 17:07

Merge branch 'lroberts36/sparse-pack' into lroberts36/independent-spa…

f481c43

…rse-thresholds Conflicts: src/utils/concepts_lite.hpp

Merge branch 'lroberts36/new-sparse-bvals' into lroberts36/independen…

27aa06f

…t-sparse-thresholds Conflicts: src/interface/mesh_data.hpp src/kokkos_abstraction.hpp

Start on per variable sparse threshold

7bef4d7

Include sparse thresholds in metadata

e8b2497

Merge branch 'lroberts36/new-sparse-bvals' into lroberts36/independen…

a7b606f

…t-sparse-thresholds

comment

e366cdc

Allow for sparse pool sparse control variable base name

e8d93bb

format

89c2ff9

Add mechanism for resolving field control map

040c5c7

Add test of control resolution

c05f335

Fix output_hdf5 test

04970e1

lroberts36 added the enhancement New feature or request label Jul 27, 2022

Merge branch 'develop' into lroberts36/independent-sparse-thresholds

9841c69

par-hermes and others added 16 commits July 27, 2022 03:05

cpp-py-formatter

4c5c426

cleanup to meet codacy standards

87b3b73

minor code cleanup and switch to sparse communcation

9261928

Update doc

7271443

Initialize my_rank to stop codacy complaint

773086a

remove possible device access of host global variables

3b50c9e

Add intialization status for variables

8796cb5

Add hooks for intialization routines

8110a02

Call initilization routines in sparse_advection

62491cc

Add necessary logic for flagging intialization

b7c7729

Add nan tags

ed529e0

Actually add NaN test

bb197eb

Make NaN tags accessible on device

5be50ae

Add device check of NaN tags

1fc3938

Add by variable default sparse values

de42f9c

Merge branch 'lroberts36/new-sparse-bvals' into lroberts36/independen…

ae723cb

…t-sparse-thresholds

lroberts36 added 4 commits October 31, 2022 19:37

format and lint

20eccb8

Merge branch 'develop' into lroberts36/independent-sparse-thresholds

50c6040

Updates that didn't merge

bc0b86b

Updates and formats

d6f995b

pgrete mentioned this pull request Nov 9, 2022

Regression test fail on Spock/Crusher/Frontier/HIP #659

Closed

pgrete reviewed Nov 23, 2022

View reviewed changes

pgrete reviewed Nov 28, 2022

View reviewed changes

lroberts36 and others added 14 commits November 28, 2022 10:50

Update tst/unit/test_nan_tags.cpp

abd89f7

Co-authored-by: Philipp Grete <[email protected]>

Update tst/unit/test_nan_tags.cpp

40e0360

Co-authored-by: Philipp Grete <[email protected]>

Minor update to doc

9e341ba

Remove false statement

ac8f833

bug

1bd9c3a

Update comment

fd03236

possibly remove future threading issues

125a2c0

comment

e389b97

Fix bug

86fd8a1

Remove old packs

a937121

Remove debug statement

7b6313b

Remove debug

bf8e26e

format and lint

e8995db

Merge branch 'develop' into lroberts36/independent-sparse-thresholds

ed43367

lroberts36 enabled auto-merge (squash) November 28, 2022 19:16

This was referenced Nov 30, 2022

Document nan_payload_tags #792

Open

Check check_metadata for advection_outflow and poisson test #793

Open

pgrete approved these changes Nov 30, 2022

View reviewed changes

lroberts36 merged commit 249acbb into develop Nov 30, 2022

lroberts36 mentioned this pull request Nov 30, 2022

Update sparse documentation about initialization #794

Open

Yurlungur deleted the lroberts36/independent-sparse-thresholds branch December 3, 2022 15:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Independent Sparse Thresholds and Sparse Allocation Control Fields #699

Add Independent Sparse Thresholds and Sparse Allocation Control Fields #699

lroberts36 commented Jul 27, 2022 •

edited

Loading

lroberts36 commented Jul 27, 2022

lroberts36 commented Nov 1, 2022

pgrete left a comment

pgrete Nov 23, 2022

lroberts36 Nov 28, 2022

pgrete Nov 30, 2022

lroberts36 Nov 30, 2022

pgrete left a comment

lroberts36 commented Nov 28, 2022

pgrete left a comment

Add Independent Sparse Thresholds and Sparse Allocation Control Fields #699

Add Independent Sparse Thresholds and Sparse Allocation Control Fields #699

Conversation

lroberts36 commented Jul 27, 2022 • edited Loading

PR Summary

PR Checklist

lroberts36 commented Jul 27, 2022

lroberts36 commented Nov 1, 2022

pgrete left a comment

Choose a reason for hiding this comment

pgrete Nov 23, 2022

Choose a reason for hiding this comment

lroberts36 Nov 28, 2022

Choose a reason for hiding this comment

pgrete Nov 30, 2022

Choose a reason for hiding this comment

lroberts36 Nov 30, 2022

Choose a reason for hiding this comment

pgrete left a comment

Choose a reason for hiding this comment

lroberts36 commented Nov 28, 2022

pgrete left a comment

Choose a reason for hiding this comment

lroberts36 commented Jul 27, 2022 •

edited

Loading