Add ParaView Catalyst in-situ visualization capability #222
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file allows GRChombo examples to be built with Catalyst. Also add some more work on the CatalystAdaptor code.
This can be handled by the updated mkdep script in GRTLCollaboration/Chombo#32.
paraview-config is extremely slow and we only want to call it once and use the cached output otherwise.
Also add compile_commands.json database.
This currently passes ghosted boxes to Catalyst.
In theory only the variables requested in the script should be sent to Catalyst.
* Change constructor for each vtkAMRBox. * Generate ghost array for each vtkUniformGrid. * Audit grid at the end of construction.
This one includes resampling to an image and no raytracing.
Note that now the vtkAMRBox indices are (0,0,0) where the lower corner of the ghosted domain on the coarsest level is e.g. (-3,-3,-3) in Chombo indices. In theory this might make it annoying to calculate coordinates, but these should be calculated from the associated vtkUniformGrid objects anyway which have their origin at the correct place (i.e. Chombo's (0,0,0)). This is necessary as VTK expects the "origin" of the vtkAMRBoxes to be the lower bound.
Also change back to the old construction of vtkUniformGrids. I think the one that used the vtkAMRBoxes assumed node centering leading to the wrong dimensions.
These have been generated from more representative data of the current parameter file.
Add some recent new ones and remove some old ones.
This includes "Insitu" in the filename. Note that the default example executable will be built and linked with ParaView Catalyst if PARAVIEW_DIR is set correctly but will not be named accordingly.
This allows the user to restrict which variables are passed to Catalyst. In theory this could be determined from the Catalyst script but the functions to determine this don't seem to work well with newer Catalyst scripts (i.e. those generated by v5.9 or later) and always request all variables.
In order to do this, parameter names have been changed.
This reduces VTK log output to stderr and instead sends it to a catalyst_pout (this name can be changed with the catalyst_pout_prefix parameter) file (for each rank) under the usual pout_subpath like the normal Chombo output (in the case MPI=TRUE). There is also a catalyst_verbosity parameter which controls both the verbosity of the VTK log output and the messages from CatalystAdaptor in the normal pout file. The -rdynamic flag will also be passed to the linker for the insitu build target so that the VTK log stack trace (which is output in the case of some termination signal) has more information for the [GR]Chombo part of the stack.
This provides instructions on building and using ParaView Catalyst in-situ visualization with a GRChombo example.
Also refactor error/warning in CatalystAdaptor
These can be specified with the XTRA_VTK_COMPONENT_FLAGS and XTRA_PARAVIEW_COMPONENT_FLAGS variables in the makefile which will be passed to paraview-config.
This uses a script generated by ParaView 5.9.1 to extract an image of a slice and compares this image to a known valid one.
This would occur if Catalyst was used but the extraction parameters were not set (so minimum extraction_level parameter doesn't exist).
Cells that are marked as ghosts will no longer be visible in ParaView (e.g. those in the outer boundary).
This writes the created vtkOverlappingAMR object to a file. It's mainly for debugging but I guess in theory it allows a user to save the state of the grid at any timestep on any level rather than just a whole coarsest level timestep.
This shifts the set `origin` of the `vtkUniformGrid`s from the `global_origin` at (0,0,0) (in Chombo coordinates) to that of the low corner of the box. This seems to disagree with what is done in the ParaView Catalyst CxxOverlappingAMRExample but makes the outputted VTK XML files appear more sensible when loaded in ParaView.
Now, all of the Catalyst parameters used in `CatalystAdaptor` are in a `CatalystAdaptor::m_p`, a struct of type `CatalystAdaptor::params_t`. This will make it easier to add/modify parameters. Previously multiple functions across the `CatalystAdaptor` and `GRAMR` classes as well as a Setup function needed to be modified every time a Catalyst parameter was changed. Note that some parameters are used elsewhere (e.g. `catalyst_coprocess_level` in `GRAMRLevel` and are therefore not part of `CatalystAdaptor::params_t`. Some parameters have been renamed in order to make them more consistent (all Catalyst parameters are now prefixed with `catalyst_`). The Insitu README has been updated accordingly.
This allows the user to make Catalyst write VTK XML files for the 3D AMR data that is passed to Catalyst at each Catalyst coprocess and specify the filename prefix of these files.
The test should not pass with a PNG from a previous run if Catalyst fails to run for whatever reason.
This was probably obvious but the `tv_nsec` member of a `timespec` struct is only the number of nanoseconds since the last full second so it is necessary to compare the `tv_sec` member first.
Remove all the code that checks the modified timestamps and simply just delete the old file at the start. This way, if nothing is generated, the test will definitely fail.
The key step is to apply vtkParallelAMRUtilities::StripGhostLayers (which removes layers of fine ghost cells that only partially cover coarse cells) and vtkParallelAMRUtilties::BlankCells (which blanks out coarser cells that are covered by finer cells on a different rank -- I thought this was automatic previously) to the vtkOverlappingAMR object (this is done in the [VisIt Chombo Reader](https://gitlab.kitware.com/paraview/visitbridge/-/blob/df098f4148a96d62c388861c1d476039e02224ae/databases/AvtAlgorithms/vtkAvtSTMDFileFormatAlgorithm.cxx#L511-L514)). Since the former might require deep copying (in case stripping happens), a temporary vtkOverlappingAMR object is instantiated in CatalystAdaptor::coprocess which is used to create the grid and add the fields. This is then copied (either shallow or deep as necessary) to the m_vtk_grid_ptr member which is passed to Catalyst. Note that now, no cells are manually marked as ghosts in CatalystAdaptor::build_vtk_grid as I observed gaps when doing so. The only marking of ghosts now comes from BlankCells.
Previously the paraview-config command could fail and the build would continue going which made it difficult for the user to see where the problem is. Also commands that exit with non-zero exit codes are no longer cached by the cache.sh script so that the paraview-config error always appears.
These don't have the .SHELLSTATUS variable.
Also update CMake version to 3.22.6.
These are set as environment variables which can be recovered in the Catalyst Python script. Also update slice_chi.py.
The vtkPythonInterpreter::SetRedirectOutput() function was introduced with VTK version 9.1.0.
|
I was able to implement this successfully into one of my examples on Cambridge CSD3 |
amelialdrew
approved these changes
Feb 14, 2023
edjong302
approved these changes
Feb 16, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds ParaView Catalyst in-situ visualization capabilities to GRChombo.
User documentation
The user documentation for this feature can be found in the README so, for brevity, I will not repeat that here. Instead I will focus on implementation details below.
Once this PR is merged, I plan to add a version of this documentation page to the wiki.
Brief list and description of key changes
Add
CatalystAdaptorclassThis class translates the GRChombo grid into a
vtkOverlappingAMRdata structure that can be passed to ParaView. It also contains the functions to initialise ParaView/Catalyst and the pipelines, execute the user-provided pipelines at regular intervals and finalise ParaView/Catalyst at the end of the simulation.Add
Make.insitufile andcache.shscriptThe
Make.insitumakefile can be included in GRChombo exampleGNUMakefiles to add Catalyst capability to the example. It adds the necessary compiler and linker flags to do this using thecxxcppflagsandXTRALDFLAGSChombo makefile variables so these cannot be set in the Chombo Make.defs.local file or elsewhere. It gets these flags using theparaview-configPython script (which in turn callscmake) which is provided by suitable ParaView builds/installations. Since this script takes a long time to run and the Chombo build system calls it several times, thecache.shscript caches the output to temporary files in/tmp.Add changes to GRChombo in order to enable Catalyst support
Such changes include:
ChomboParameters.CatalystAdaptorobject which is stored asGRAMR::m_insitu.GRAMR::m_insituand other ParaView/Catalyst/VTK features inSetupFunctions.hpp.catalystCoProcess()andpreCatalystCoProcess()toGRAMRLevel. The former calls the mainCatalystAdaptor::coprocess()function which passes the GRChombo grid to Catalyst and then executes the ParaView pipelines. The latter is called bycatalystCoProcess()and is a virtual function which can be overloaded in example-specific derived classes. By default, it just exchanges ghost cells for requested variables.catalysCoProcess()is called inGRAMRLevel::postTimeStep()right afterspecificPostTimeStep().Note that all changes are conditional compiled on the
USE_CATALYSTmacro which is defined byMake.insituif thePARAVIEW_DIRenvironment variable is set so this shouldn't break building examples without Catalyst.Add a Catalyst Insitu test
This test just creates a simple 1-level GRChombo grid with data set by the
Polynomialclass of the AMR Interpolator test. It then runs a simple Catalyst pipeline which outputs a slice of the variableAas a PNG file. This file is then compared with an expected image to verify there isn't a regression in the in-situ code. Note that ifUSE_CATALYSTis undefined, the test is skipped (it does nothing and exits immediately).Add a GitHub action to run the Catalyst Insitu test
Since building with Catalyst is a little more complicated, this test is run separately from all the others. It is built using a version of ParaView built by a GitHub action in the GRChombo/paraview-build repository.
Add Catalyst support to the BinaryBH example
There are a small number of modifications to the BinaryBH example in order to add Catalyst support. The main one is the inclusion of
Make.insituto theGNUMakefile, but there are also some changes to ensureWeyl4is calculated inspecificPostTimeStep()if Catalyst is activated and to callcatalystCoProcess()at t=0.Add some example Catalyst Python scripts
See the comment lines at the top of these scripts (in the
Examples/BinaryBH/catalyst_scriptsdirectory) to see what they do. They all take slices through thecenter(which is passed from GRChombo using environment variables) of constant z.I appreciate this is a very large PR so please take your time and don't hesitate to ask me any questions you might have. I don't expect you to go through and understand all of the ParaView/Catalyst/VTK function calls/object instantiations but it would be good to check the GRChombo side of things and make sure things are consistent with the rest of the code.