Make AMR and sparse restarts bitwise exact #1073
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lroberts36
requested review from
Yurlungur,
pgrete,
bprather,
jdolence,
pdmullen and
brryan
lroberts36
changed the title
lroberts36
changed the title
Yurlungur
approved these changes
May 14, 2024
Co-authored-by: Patrick Mullen <[email protected]>
13 tasks
pgrete
approved these changes
May 15, 2024
12 tasks
pgrete
added a commit
that referenced
this pull request
May 29, 2024
…#1073) (#1089) * Fix deref couter reading from previous restarts * Fix logic. * make a separate dataset for derefinement count * don't set derefinement count to large number in restart test * Add warning * ignore derefinement_count in phdf_diff * only warn on rank 0 * remove hdf debug output --------- Co-authored-by: Luke Roberts <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Summary
This PR saves the derefinement count for each meshblock and the deallocation count for each sparse variable in the restart output (as well as in regular HDF5 output). This should ensure that restarted simulations give bitwise exact results to simulations that were run from start to finish.
Previously, AMR simulations that were restarted had the deallocation count of all meshblocks reset to zero. This meant that after restart, a block could be deallocated a few steps after it would have been deallocated if there was no restart. This resulted in differences in simulation results with and without restarts many orders of magnitude larger than machine precision, although still relatively small compared to characteristic scales of the problem. An example of this difference is in issue #1072 (the problem detailed there is fixed by this PR).
Similarly, the deallocation count of sparse variables was not stored and was reset to zero on restart. This was a known issue, and the restart regression test set the global deallocation count large enough that variables were never deallocated. This PR also stores the deallocation count of variables in the restart file and sets them to the correct values on restart. The restart test now works with a deallocation count of 5.
Also,
MeshRefinement::DereferenceCountis renamed toMeshRefinement::DeallocationCountsince dereference count doesn't really make sense.WARNING: This PR does not solve the edge case described in issue #1077
PR Checklist