Running KHARMA

To run a particular problem with KHARMA, you can usually simply invoke kharma.host for CPU compiles, and kharma.cuda for Nvidia GPUs. The problem initialization and parameters are specified in a .par file:

$ ./kharma.host -i pars/tests/orszag_tang.par

KHARMA benefits from certain runtime environment variables controlling CPU pinning and occupancy, which can be handled by the wrapper run.sh, described below.

Note that some MPI libraries may require that kharma.{host,cuda} be run with mpirun -n 1 ./kharma.host even when invoking a single process. NVHPC in particular simply hangs if invoked without an external MPI environment.

All the parameters specifying a simulation are provided by a single input "deck" in the .par file. Sample input files corresponding to standard tests and astrophysical systems are included in pars/. The same KHARMA binary can run any .par file, regardless of spacetime geometry, problem initialization, etc.

KHARMA will attempt to guess many parameters if they are not specified, e.g. boundary conditions and coordinate sizes for simulations in spherical polar coordinates, or interior boundary locations for black hole simulations based on keeping 5 zones inside the event horizon. Most of the core inferences are done in the function FixParameters in kharma.cpp, and most of the default values are specified in the Initialize() functions of their respective packages, e.g. in grmhd/grmhd.cpp.

Any parameter can be overridden from the command line, which is useful when scripting, or to tweak something when continuing a simulation. Simply pass the full parameter path and value, e.g., parthenon/output0/dt=1 to change the parameter dt in block <parthenon/output0> to 1.0. These overrides must be passed after the initial parameter file or restart file argument. For example, to turn on verbose console output:

$ ./kharma.host -i pars/orszag_tang.par debug/verbose=1

Parameters should not be preceded with the usual dashes or other flag characters.

Parthenon will warn (but not crash) if a new parameter is specified from the command line which is not present in the parameter file. Parameters which are unused or unknown to Parthenon or KHARMA will be silently ignored, which is a common cause of issues. Double-check your parameter spelling!

run.sh

The run.sh script exists mostly to source the machine file when running KHARMA, just as we source it when compiling. This is an easy way to keep a consistent compile-time and runtime environment, since the same module load statements, environment variables, etc are executed when compiling and running. run.sh does not require the extra arguments (e.g., clean or cuda) used in make.sh -- when make.sh is run, all arguments are recorded in the file make_args in KHARMA's source directory, and run.sh automatically reads this so that the $ARGS variable is set the same when running as it was when building.

Using run.sh is of course optional, it does nothing that cannot be done manually or in a batch script. The convenience is that any quirks to be aware of when running the code (extra MPI flags, environment variables such as MPICH_GPU_SUPPORT_ENABLED, etc, etc) can be centralized in the machine file. Then, whether invoked interactively or from a batch script, KHARMA has the same environment and MPI conventions. Or at least that's the hope; run.sh is an informal tool which can break easily, and not all machine files are written to support all of its options nicely.

In addition to parsing the machine file for defaults, run.sh allows a few command-line overrides. Note they must appear in the order listed here. Any run.sh options should appear before KHARMA options.

• trace, prof, nvprof, described below
• -n, -nt, number of MPI ranks and number of OpenMP threads per rank
• -b name of the KHARMA binary to use (e.g., usually kharma.host or kharma.cuda). Convenient if you have multiple executables you're testing against one another

Additionally, the MPI_EXE, MPI_NUM_PROCS, and MPI_EXTRA_ARGS environment variables are respected, if present.

The parsing in run.sh is extremely delicate; this is a known issue. An overhaul of make.sh and run.sh parameters is long overdue...

Tracing and Profiling

If the Kokkos tools are compiled alongside KHARMA (that is, not within KHARMA's source directory, but with the same parent folder) then one can invoke ./run.sh trace [other options] and see a running log of most KHARMA functions called, as well as every Kokkos kernel invoked over the run. Note that trace must be the first option listed! This is separate from KHARMA's own tracing; it is slightly more verbose, but the greatest benefit is that KHARMA does not need to be recompiled to use the Kokkos tools tracing.

Again assuming an installation of the Kokkos tools alongside KHARMA, profiling is available with ./run.sh prof [other options], which will print a summary of time spent in each Kokkos kernel at the end of execution, in JSON format. The NVProf connector can be invoked with ./run.sh nvprof [other options].

Outputs

Nearly all problems will write out the state of the simulation every once in a while, for plotting or analysis after a simulation is complete. The cadence and contents of these "dump" files are specified in the parameter file; all outputs are usually listed together at the end. The fluid state is generally output to HDF5 files named problem_name.out0.NNNNN.phdf, with the sequence number NNNNN counting up incrementally from 00000. Additionally, some problems compute reductions (total energy, accretion rate, etc.) and output these to a text file, problem_name.hst.

Output files are split first by mesh block ID, then vector index if present, then by cell in k, j, i order. Since data is split by block, reading the file for e.g. plotting, analysis, etc. means either treating every meshblock separately, or mapping their contents onto a single global mesh. Since these operations are tedious to reimplement, I recommend copying or using existing solutions:

• Parthenon provides a small python package, parthenon_tools, designed to read these output files and produce a few basic plots.
• The pyharm package provides reading, calculation, and plotting tools. These are well-developed for single-level simulations (anything without SMR/AMR) but treat any refined simulations at the base grid level only. This is likely enough for reductions (accretion rate, EH magnetization, radial profiles) but makes for underwhelming 2D and 3D movies.
• The development branch of pyharm can experimentally use yt for full AMR support. So far, this is only really useful for directly plotting primitive variables (e.g., $\rho$), because yt's native calculations are not aware of either the metric or KHARMA's code units. Support for a broader array of variables is planned, but this will require some work to define the metric and other GR quantities as yt variables.

Parthenon also has documentation on generating and handling output files. Note that some solutions there are not yet supported in KHARMA.

Restarting Simulations

In addition to the analysis outputs, most long-running problems will output "restart" or "checkpoint" files. These are named similarly to the science outputs, with the pattern problem_name.out1.NNNNN.rhdf (note the changed file extension as well as the output number).

A simulation can be resumed very simply from such a file. E.g. to restart a torus simulation from the 200th restart (by default corresponding to 20,000M of simulation time):

$ ./kharma.cuda -r torus.out1.00200.rhdf

All of the original problem parameters are saved to .rhdf files, and Parthenon will read them automatically when restarting a simulation, so you needn't specify a parameter file in addition to a restart file.

Parameters can be overridden when restarting, using exactly the same syntax as when running a new problem. Needless to say, be very careful with this, as there are no guardrails or parameter checks in place to prevent silly parameter changes. Many parameters are used only during problem initialization, so changes will produce no effect. Other changes will produce gibberish, and some, like the mesh size, will just crash the code.

If you know what you are doing and want to override many parameters at once, the command line is a tedious and ephemeral way to do so. Starting in KHARMA 2024.9 you can specify both -r <name>.rhdf and then -i <name>.par. All parameters present in the latter will override settings in the former, and any new settings will be added as well. This is most commonly used to restart a plain GRMHD simulation with additional physics enabled.