Add support for traditional threading

[DusanJovic-NOAA]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• [] Commit 'test_changes.list' from previous step

---

Description:

This PR adds support for traditional (non esmf managed) threading. Currently, all '2threads' tests have been converted to traditional threading. ESMF managed threading is still the default.

To use tradition threading, set ESMF_THREADING=false in test's configuration file and set THRD to number of threads, for example:

$ cat tests/cpld_2threads_p8
. . . 
ESMF_THREADING=false
THRD=$THRD_cpl_thrd

Many tests had two versions of ufs.configure templates (_esmf.IN and the version without _esmf) with the only difference being the value of globalResourceControl:, true for esmf and false for non-esmf threading. Those are now unified in a single ufs.configure template.

Commit Message:

* UFSWM - This PR adds support for traditional (non esmf managed) threading. ESMF managed threading is still default.

Priority:

• Normal

Git Tracking

UFSWM:

• Closes Set traditional threading as default #2246

Sub component Pull Requests:

• None

UFSWM Blocking Dependencies:

• None

---

Changes

Regression Test Changes (Please commit test_changes.list):

• No Baseline Changes.

Input data Changes:

• None.

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[NickSzapiro-NOAA]

Does traditional threading require one value of #threads per node?

If yes, I'm not sure these lines in rt.sh and run_test.sh work when components under-fill nodes

TPN=$(( TPN / THRD ))
NODES=$(( TASKS / TPN ))
if (( NODES * TPN < TASKS )); then
  NODES=$(( NODES + 1 ))
fi
PPN=$(( TASKS / NODES ))
if (( TASKS - ( PPN * NODES ) > 0 )); then
  PPN=$((PPN + 1))
fi

and maybe compute_petbounds_and_task functions could export NODES too

[DusanJovic-NOAA]

Does traditional threading require one value of #threads per node?

If yes, I'm not sure these lines in rt.sh and run_test.sh work when components under-fill nodes

TPN=$(( TPN / THRD ))
NODES=$(( TASKS / TPN ))
if (( NODES * TPN < TASKS )); then
  NODES=$(( NODES + 1 ))
fi
PPN=$(( TASKS / NODES ))
if (( TASKS - ( PPN * NODES ) > 0 )); then
  PPN=$((PPN + 1))
fi

and maybe compute_petbounds_and_task functions could export NODES too

I am not sure what you mean by '#threads per node'. Are you referring to TPN variable? TPN is a number of (MPI) tasks per node, which is initially set to be equal to the number of physical cores on a given platform, 80 on Hercules, 40 on Hera, 128 on WCOSS2 etc. Then it is divided by number of threads (THRD) to avoid having multiple threads running on the same physical core. Once we have tasks per node (TPN) and total number of tasks (TASKS), we compute the number of nodes (NODES). That is done in this block:

TPN=$(( TPN / THRD ))
NODES=$(( TASKS / TPN ))
if (( NODES * TPN < TASKS )); then
  NODES=$(( NODES + 1 ))
fi

the following block:

PPN=$(( TASKS / NODES ))
if (( TASKS - ( PPN * NODES ) > 0 )); then
  PPN=$((PPN + 1))
fi

has been added for 'derecho support', although I never fully understood why that was needed and what is special about Derecho to require it.

[jkbk2004]

merged with #2538