Failing to generate cardinal-opt

[zsh151]

I'm trying to install Cardinal without MOOSE's conda environment on Ubuntu.

Compiling was working fine until I ran make -j8. There were no obvious errors, but no cardinal-opt was generated after completion.

Set environment variables to

export ENABLE_DAGMC=yes

export NEKRS_HOME=$HOME/cardinal/install

export CC=mpicc
export CXX=mpicxx
export FC=mpif90

export OPENMC_CROSS_SECTIONS=${HOME}/cross_sections/endfb71_hdf5/cross_sections.xml

export NEKRS_OCCA_MODE_DEFAULT=CPU

Below is the whole output I get

The output after running make -j8
make.txt

The output after running ./contrib/moose/scripts/update_and_rebuild_petsc.sh
1.txt

The output after running ./contrib/moose/scripts/update_and_rebuild_libmesh.sh
2.txt

The output after running ./contrib/moose/scripts/update_and_rebuild_wasp.sh
3.txt

So, where did it go wrong?
Any help would be greatly appreciated.

[aprilnovak]

It looks like PETSc, libMesh, and wasp build fine. I don't see any errors in your make.txt, so I think the actual error went to the console instead. Can you try the following and then re-attach make.txt? The & will also put any errors into that file.

make -j8 >& make.txt

[zsh151]

After commenting it out, there will indeed be no errors, but after running make, there will still be no cardinal-opt generated.

Here are the contents of the cardinal/install/bin file

Here are the outputs after running make
make.txt

[aprilnovak]

cardinal-opt will actually appear in cardinal/cardinal-opt -- those executables in install/bin are extra binaries that are associated with NekRS, OpenMC, and DAGMC as standalone tools (i.e. these are bonuses so that you don't need to compile NekRS/OpenMC/DAGMC separately to get any of those utilities).

Is that text file you attached from doing make >& make.txt? I don't see any error messages in it, which makes me think they actually went to the console. Did you include the &?

[zsh151]

cardinal-opt can be generated after running rm -rf build/ install/ make clean and then make.

Here is the output with the &
make.txt

But it still seems something wrong with the cardinal-opt. When I run ./run_tests -j8, here is the result.

41 passed, 36 skipped, 0 pending, 50 FAILED
MAX FAILURES REACHED

And if I run tests again, an error appears at the end of tests.

runWorker Exception: Traceback (most recent call last):
  File "/home/en/cardinal/contrib/moose/python/TestHarness/schedulers/Scheduler.py", line 456, in runJob
    self.queueJobs(jobs, j_lock)
  File "/home/en/cardinal/contrib/moose/python/TestHarness/schedulers/Scheduler.py", line 262, in queueJobs
    self.run_pool.apply_async(self.runJob, (job, jobs, j_lock))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 455, in apply_async
    self._check_running()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 350, in _check_running
    raise ValueError("Pool not running")
ValueError: Pool not running

Here is the output in the second test.
test.txt

[aprilnovak]

That ValueError: Pool not running message just indicates that the test suite hit the maximum number of failures (50), in which case the tests stop running early.

Those NekRS-related failures look puzzling. Perhaps this is an indication that our commenting-out of that compiler error message (Unsupported compiler) in the NekRS CMake files is actually the culprit after all. The OpenMC-related failures are of less concern to me, because there is a known failure to reproduce the test suite of OpenMC on some OS due to small accumulations of roundoff/etc. errors (see: openmc-dev/openmc#801).

Here is what I would suggest for now, just to triple check our steps and be sure there's not something strange in your build files. I'd delete cardinal and start over from scratch, without touching that NekRS CMake file.

rm -rf cardinal
git clone https://github.com/neams-th-coe/cardinal.git
cd cardinal
./scripts/get-dependencies.sh
<build as usual>

If we hit errors related to that compiler message, I'll suggest you to post the Unsupported compiler error you're seeing on the NekRS dicussions page for us to get help: https://github.com/Nek5000/nekRS/discussions

[zsh151]

Thanks for your advice. I read the Getting Started content on https://github.com/Nek5000/nekRS.
Below are all the changes have been made

sudo apt install libopenmpi-dev   #provides openmpi
sudo apt install libx11-dev           #provides X11
sudo apt install libxt-dev             #provides X11

export FC=mpif77

And then build as usual from the beginning, without touching that NekRS CMake file. And the process went smoothly.

Here is the result of the rests.
Does it mean the installation was successful?

test:neutronics/triggers.k_std_dev .......................................................... FAILED (CSVDIFF)
test:neutronics/heat_source.overlap_all ..................................................... FAILED (EXODIFF)
test:neutronics/feedback/interchangeable.no_feedback_tally .................................. FAILED (CSVDIFF)
test:neutronics/relaxation/cell_tallies.global_with_alignment_unity_relax ................... FAILED (CSVDIFF)
test:neutronics/flux.flux ................................................................... FAILED (CSVDIFF)
test:neutronics/mesh_tally.no_coupling ...................................................... FAILED (CSVDIFF)
test:neutronics/feedback/lattice.pincell .................................................... FAILED (EXODIFF)
test:postprocessors/fission_tally_relative_error.fission_tally_relative_error ............... FAILED (CSVDIFF)
test:neutronics/dagmc/cell_tallies.null_skin ................................................ FAILED (EXODIFF)
test:neutronics/dagmc/mesh_tallies.null_skin ................................................ FAILED (EXODIFF)
test:neutronics/feedback/multi_component_density/multi.collate_density ...................... FAILED (CSVDIFF)
test:neutronics/relaxation/mesh_tallies.global_without_alignment_unity_relax ................ FAILED (EXODIFF)
test:neutronics/feedback/temperature/two_one.two_to_one ..................................... FAILED (CSVDIFF)
test:neutronics/openmc_xml.particles ........................................................ FAILED (CSVDIFF)
test:neutronics/fixed_source.overlap_all .................................................... FAILED (CSVDIFF)
test:neutronics/dagmc/density_skin.skin ..................................................... FAILED (EXODIFF)
tutorials/pebbles.solid ..................................................................... FAILED (CSVDIFF)
test:neutronics/feedback/triso/cache.identical .............................................. FAILED (EXODIFF)
test:neutronics/feedback/different_units.different_units .................................... FAILED (EXODIFF)
test:neutronics/feedback/single_level.pincell ............................................... FAILED (EXODIFF)
test:neutronics/feedback/temperature/one_one.one_to_one ..................................... FAILED (CSVDIFF)
test:neutronics/feedback/materials.pincell .................................................. FAILED (CSVDIFF)
test:neutronics/feedback/temperature/one_two.one_to_two ..................................... FAILED (CSVDIFF)
test:neutronics/heat_source/distrib_cell.solid .............................................. FAILED (EXODIFF)
test:postprocessors/eigenvalue.k_eigenvalue ................................................. FAILED (CSVDIFF)
test:neutronics/feedback/unmapped_moose.unmapped ............................................ FAILED (EXODIFF)
test:neutronics/triggers.k_std_dev_cutoff ................................................... FAILED (CSVDIFF)
test:neutronics/heat_source.from_postprocessor .............................................. FAILED (EXODIFF)
test:neutronics/feedback/interchangeable.T_feedback_tally ................................... FAILED (CSVDIFF)
test:neutronics/flux.flip_order ............................................................. FAILED (CSVDIFF)
test:neutronics/relaxation/cell_tallies.global_with_alignment_relax ......................... FAILED (CSVDIFF)
test:postprocessors/fission_tally_relative_error.multi_score ................................ FAILED (CSVDIFF)
test:neutronics/mesh_tally.one_mesh ......................................................... FAILED (EXODIFF)
test:neutronics/feedback/lattice.pincell_null_mat ........................................... FAILED (EXODIFF)
test:neutronics/feedback/multi_component_density/multi.multi_vars ........................... FAILED (CSVDIFF)
test:neutronics/dagmc/cell_tallies.skin ..................................................... FAILED (EXODIFF)
test:neutronics/feedback/temperature/two_one.two_to_one_multi ............................... FAILED (CSVDIFF)
test:neutronics/openmc_xml.inactive ......................................................... FAILED (CSVDIFF)
test:neutronics/dagmc/mesh_tallies.skin ..................................................... FAILED (EXODIFF)
test:neutronics/dagmc/density_skin.disjoint_bins ............................................ FAILED (EXODIFF)
test:neutronics/relaxation/mesh_tallies.global_without_alignment_relax ...................... FAILED (EXODIFF)
test:neutronics/feedback/different_units.different_units_null_fixed_mesh .................... FAILED (EXODIFF)
test:controls/openmc_nuclide_densities.mat .................................................. FAILED (CSVDIFF)
test:neutronics/feedback/single_level.pincell_null_fixed_mesh ............................... FAILED (EXODIFF)
test:neutronics/triggers.k_variance ......................................................... FAILED (CSVDIFF)
test:neutronics/heat_source.overlap_solid ................................................... FAILED (CSVDIFF)
utils/meshes/interassembly.specs ............................................................. FAILED (CODE 1)
utils/meshes/assembly.specs .................................................................. FAILED (CODE 1)
utils/meshes/interassembly_w_structures.specs ................................................ FAILED (CODE 1)
test:userobjects/hexagonal_gap_layered.gap_layered ............................................ FAILED (CRASH)
tutorials/standalone.standalone_nek ........................................................ FAILED (CODE -15)
test:nek_stochastic/quiet_init.driver_multi_1 ................................................. FAILED (CRASH)
--------------------------------------------------------------------------------------------------------------
Ran 288 tests in 160.2 seconds. Average test time 3.6 seconds, maximum test time 118.6 seconds.
236 passed, 35 skipped, 0 pending, 52 FAILED
MAX FAILURES REACHED

[aprilnovak]

That looks great! Glad this worked, it can get messy with stale build files if environment is changed part-way through.

Yes, since those are all OpenMC failures, and are only diff-type failures (no crashes), that most likely means that there's some small difference with your compiler that is giving differences in the 3rd, 4th, etc. decimal place for OpenMC output quantities. This is only noticeable when running with low numbers of particles (like in our test suite, so that it runs fast) -- you will see convergent results if you increase the number of particles, so this is not a bug/problem.

The failures for the utils/meshes tests are probably due to a missing python module on your machine, but that's easy to fix if you look further into the error messages for those tests (not a high priority, unless you want to use those meshing scripts).

[aprilnovak]

It could also be due to the cross section library you're using - our test suite is based on ENDF-b7.1 (https://cardinal.cels.anl.gov/cross_sections.html).