Add -DAPP=<app_name> cmake option. Add timing info to rt logs.

[DusanJovic-NOAA]

Description

Add timing info for all compile jobs to the main log file.
Fix Rocoto option.
Save timestamps for each rt job

As the number of applications within the UFS grows, this PR collects the applications and re-defines it in terms of components that need to be enabled/disabled for the application. It also allows the user the ability to define their own application based on a unique App Name.

The currently supported valid applications are:

App Name	App Description	Components this App will turn ON
ATM	Atmosphere Only	FMS, FV3
ATMW	Atmosphere with Waves	FMS, FV3, WW3
S2S	Coupled model	FMS, FV3, MOM6, CICE6, CMEPS
S2SW	Coupled model with Waves	FMS, FV3, MOM6, CICE6, WW3, CMEPS
DATM	Coupled Model with (CDEPS) Data atmosphere	FMS, MOM6, CICE6, CMEPS, CDEPS
DATM_NEMS	Coupled Model with (NEMS) Data atmosphere	FMS, MOM6, CICE6, CMEPS, NEMSdatm

CMake expects an option -DAPP=<APP_NAME>, where APP_NAME is the item in the 1st column from the above table.
Without a valid application, the configuration will exit with a message.

To add new applications, the user will define a unique application name in the list defined as VALID_APPS in CMakeLists.txt and proceed to enable the components for the new application in cmake/configure_apps.cmake.

Testing

How were these changes tested? tested on hera.intel
What compilers / HPCs was it tested with? intel
Are the changes covered by regression tests? (If not, why? Do new tests need to be added?)
Have regression tests and unit tests (utests) been run? yes
On which platforms and with which compilers? hera, intel

• hera.intel
• hera.gnu
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss_cray
• wcoss_dell_p3

Two tests failed on wcoss (fv3_gfs_v15p2 and fv3_gfs_v15p2_RRTMGP).

Dependencies

No.

Do PRs in upstream repositories need to be merged first?
No.

[BrianCurtis-NOAA]

Machine: gaea
Compiler: intel
Repo location: /lustre/f2/pdata/ncep/Brian.Curtis/test/594080688/20210322111506/ufs-weather-model
Please manually delete: /lustre/f2/scratch/Brian.Curtis/FV3_RT/rt_31109
Test datm_debug_cfsr 102 failed in check_result failed
Test datm_debug_cfsr 102 failed in run_test failed
Please make changes and add the following label back:
gaea-intel-RT

[DusanJovic-NOAA]

Machine: gaea
Compiler: intel
Repo location: /lustre/f2/pdata/ncep/Brian.Curtis/test/594080688/20210322111506/ufs-weather-model
Please manually delete: /lustre/f2/scratch/Brian.Curtis/FV3_RT/rt_31109
Test datm_debug_cfsr 102 failed in check_result failed
Test datm_debug_cfsr 102 failed in run_test failed
Please make changes and add the following label back:
gaea-intel-RT

$ cat log_gaea.intel/rt_102_datm_debug_cfsr.log

baseline dir = /lustre/f2/pdata/esrl/gsd/ufs/ufs-weather-model/RT/NEMSfv3gfs/develop-20210318/INTEL/datm_debug_cfsr
working dir  = /lustre/f2/scratch/Brian.Curtis/FV3_RT/rt_31109/datm_debug_cfsr
Checking test 102 datm_debug_cfsr results ....
 Comparing RESTART/MOM.res.nc ............ALT CHECK......NOT OK
 Comparing RESTART/iced.2011-10-01-21600.nc ............ALT CHECK......NOT OK
 Comparing RESTART/DATM_CFSR.cpl.r.2011-10-01-21600.nc ............ALT CHECK......NOT OK

The total amount of wall time                        = 43.506641

Test 102 datm_debug_cfsr FAIL

[climbfuji]

Is there a default for APP or should there be one? (APP=ATM).

This PR does a ton more than what is in the title or the description. It would be great to add a bit of information so that we can find those PRs easier if needed in the future.

[MinsukJi-NOAA]

CI errors have to do with EC2 instance, and not ufs-weather-model. I am looking into this.

[MinsukJi-NOAA]

CI errors have to do with EC2 instance, and not ufs-weather-model. I am looking into this.

Should be good to go now.

[aerorahul]

All changes look good.
One change in README.md is requested.

[aerorahul]

Looks good.
Thanks for incorporating changes from PR #473.
PR #473 address and closes #416