Use jinja templates instead of @VARNAME@ in config files

[aerorahul]

Description

This PR:

• replaces the template pattern used in configs e.g. @MACHINE@ in config.base with jinja template pattern {{ MACHINE }}
• setup_expt.py script is updated to remove custom code that was written to search and replace @VARNAME@ with values from the experiment yamls. Jinja provides a much cleaner way to interface those two.
• updated configs that are jinja templates now have an extension of .j2 in parm/config/<system> directory. e.g. config.base is now named config.base.j2 in parm/config/gfs. It gets rendered as config.base in the experiment directory after setup_expt.py completes the execution.
• makes use of --account argument in setup_expt.py as opposed to replacing ACCOUNT from hosts/host.yaml.
• organizes hosts/host.yaml into sections for Paths, BQS properties, HPSS properties and Features for that host.

In addition, this PR also:

• improves logging of create_experiment.py, setup_expt.py and setup_xml.py.
• allows host.py to honor MACHINE_ID set by sourcing detect_machine.sh.

Resolves #3439

Type of change

• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO

How has this been tested?

Experiment directories were created before and after this change and compared. Changes were in the expected places where template patterns were updated.

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[github-advanced-security]

ShellCheck found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[aerorahul]

I am going to close this PR for now as I work through the unresolved issues.

[emcbot]

Checkout Failed on Hercules in Build# 8: Remote call on Hercules-EMC failed

[emcbot]

Build FAILED on Hercules in Build# 9 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3411/global-workflow/sorc/logs/gsi_enkf.log

Follow link here to view the contents of the above file(s): (link)

[aerorahul]

_CD/HERCULES/CI/3411/global-workflow/sorc/logs/gsi_enkf.log

@TerrenceMcGuinness-NOAA
Looks like the build job timed out on this one on Hercules. Here is the last snippet.

-- Installing: /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3411/global-workflow/sorc/gsi_enkf.fd/install/include/gsi/stpvismod.mod
-- Installing: /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3411/global-workflow/sorc/gsi_enkf.fd/install/include/gsi/into3lmod.mod
-- Installing: /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3411/global-workflow/sorc/gsi_enkf.fd/install/include/gsi/gsi_gustoper.mod
slurmstepd: error: *** JOB 4881370 ON hercules-03-31 CANCELLED AT 2025-03-26T18:19:32 DUE TO TIME LIMIT ***

I am going to reset the label and hope that it kicks this back up

[aerorahul]

Looking at the CI run on WCOSS2, it appears all tests have passed successfully under /lfs/h2/emc/ptmp/emc.global/PR/PR_3411
I performed a rocotostat | grep -v "SUCCEEDED" under each experiment directory and got no hits on failed jobs.
Marking the label CI_WCOSS_Passed

[emcbot]

CI Passed on Hercules in Build# 10
Built and ran in directory /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3411

Experiment C48_ATM_fdf80467 Completed 1 Cycles: *SUCCESS* at Thu Mar 27 10:00:24 CDT 2025
Experiment C48mx500_hybAOWCDA_fdf80467 Completed 2 Cycles: *SUCCESS* at Thu Mar 27 10:06:19 CDT 2025
Experiment C96mx100_S2S_fdf80467 Completed 1 Cycles: *SUCCESS* at Thu Mar 27 10:12:29 CDT 2025
Experiment C96C48_hybatmDA_fdf80467 Completed 3 Cycles: *SUCCESS* at Thu Mar 27 11:00:40 CDT 2025
Experiment C96_atm3DVar_fdf80467 Completed 3 Cycles: *SUCCESS* at Thu Mar 27 11:18:11 CDT 2025
Experiment C48_S2SW_fdf80467 Completed 1 Cycles: *SUCCESS* at Thu Mar 27 11:56:06 CDT 2025
Experiment C48_S2SWA_gefs_fdf80467 Completed 1 Cycles: *SUCCESS* at Thu Mar 27 12:15:21 CDT 2025
Experiment C48mx500_3DVarAOWCDA_fdf80467 Completed 2 Cycles: *SUCCESS* at Thu Mar 27 14:21:34 CDT 2025