NekRS on Cluster

[philipJFpfahl]

Hello,

Recently we installed Cardinal with NekRS, openMC, and DAGMC on our cluster.
I ran into some trouble while doing the tutorial.
Every tutorial step seems to run (also the other coupled tutorials), except for the MSFR. All the files are added to the folder (I hope). If I run the openMC.i file it runs a bit and crashes when starting nekRS so I only run the nek.i file.
So I get the error below.

$ mpiexec -n 100 ../../cardinal_new/cardinal/cardinal-opt -i nek.i

  ____   ___   / /__ / __ \/ ___/
  / __ \ / _ \ / //_// /_/ /\__ \ 
 / / / //  __// ,<  / _, _/___/ / 
/_/ /_/ \___//_/|_|/_/ |_|/____/  v23.0.7 (907edeac)

COPYRIGHT (c) 2019-2023 UCHICAGO ARGONNE, LLC

MPI tasks: 100

Initializing device 
reading par file ...
general::filtering is deprecated and might be removed in the future!
general::filtermodes is deprecated and might be removed in the future!
general::filterweight is deprecated and might be removed in the future!
active occa mode: Serial

using NEKRS_HOME: /nfs/safe_shared/cardinal/install
using NEKRS_CACHE_DIR: /nfs/safe_shared/philip_share/msfr/.cache
using OCCA_CACHE_DIR: /nfs/safe_shared/philip_share/msfr/.cache/occa/

building nekInterface for lx1=8, lelt=6163 and lelg=616000 ... 
/nfs/safe_shared/cardinal/install/nek5000/core/map2.f:184:29:

  184 |       call read_con(wk4,size(wk),neli,nlv,ierr)
      |                             1
Error: Result of SIZE overflows its kind at (1)
/nfs/safe_shared/cardinal/install/nek5000/core/map2.f:189:30:

  189 |         call find_con(wk,size(wk),tol,ierr)
      |                              1
Error: Result of SIZE overflows its kind at (1)
/nfs/safe_shared/cardinal/install/nek5000/core/map2.f:192:32:

  192 |           call find_con(wk,size(wk),tol,ierr)
      |                                1
Error: Result of SIZE overflows its kind at (1)
make: *** [/nfs/safe_shared/philip_share/msfr/.cache/nek5000/makefile:167: /nfs/safe_shared/philip_share/msfr/.cache/nek5000/obj/map2.o] Error 1
make: *** Waiting for unfinished jobs....
Error in buildNekInterface: 
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 18 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

I hope you can help with this issue.

Thanks in advance and have a nice weekend

Philip

[aprilnovak]

Hi @philipJFpfahl I am sorry I missed your question and took so long to respond. For some reason I do not think Github sent me an email about it.

The MSFR case is our largest tutorial - those types of error messages mean you are running out of memory. Can you tell me a bit more about what HPC system you are running on?

[philipJFpfahl]

Hello,
No worries, it is also not a super high priority.

The cluster has 3 nodes with:

(base) [pfp01@node02 ~]$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              256
On-line CPU(s) list: 0-255
Thread(s) per core:  2
Core(s) per socket:  64
Socket(s):           2
NUMA node(s):        2
Vendor ID:           AuthenticAMD
CPU family:          25
Model:               1
Model name:          AMD EPYC 7763 64-Core Processor
Stepping:            1
CPU MHz:             2450.000
CPU max MHz:         3529,0520
CPU min MHz:         1500,0000
BogoMIPS:            4890.78
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            32768K
NUMA node0 CPU(s):   0-63,128-191
NUMA node1 CPU(s):   64-127,192-255

and

              total        used        free      shared  buff/cache   available
Mem:          1,0Ti       3,9Gi       977Gi        37Mi        25Gi       997Gi
Swap:         4,0Gi        63Mi       3,9Gi

So there is one Tb of RAM for each node with 256 cores.
I can run this slurm job:

1 #!/bin/bash
2 #SBATCH --job-name=nekRS            # create a short name for your job
3 #SBATCH -N 3                        # node count
4 #SBATCH -n 768                      # total number of tasks across all nodes
5 #SBATCH --mem=875g
6 
7 module purge
8 
9 module load cardinal
10 
11 mpiexec -n 768 ../../cardinal_new/cardinal/cardinal-opt -i nek.i  

and still get the error.

How much memory do you allocate when running the problem?

All the best,
Philip

[aprilnovak]

I ran this case on Summit, about 25 nodes. Each node has 2 Power9 processors, for ~500 GB RAM. The mesh for this case has 600,000 elements, which gives 1.2 billion grid points.

[aprilnovak]

You can try decreasing the polynomial order, and at some point you should be able to run on that machine, if you want to test things out.

[philipJFpfahl]

Hello,

I finally got to test it and indeed, with reduced polynomial order the simulation ran.

Thank you :)