Skip to content

Adding Jenkins CI to GaeaC6 using role account#3389

Merged
DavidHuber-NOAA merged 31 commits into
NOAA-EMC:developfrom
TerrenceMcGuinness-NOAA:ci_jenkins_role_gaea6
Mar 3, 2025
Merged

Adding Jenkins CI to GaeaC6 using role account#3389
DavidHuber-NOAA merged 31 commits into
NOAA-EMC:developfrom
TerrenceMcGuinness-NOAA:ci_jenkins_role_gaea6

Conversation

@TerrenceMcGuinness-NOAA
Copy link
Copy Markdown
Collaborator

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA commented Feb 25, 2025 •
edited
Loading

Description

This PR has the necessary updates to the CI scripts in Jenkins for running on Gaea C6.
The GaeaC6-EMC Node is all ready configured and connected to the Gaea C6 Head Node gaea64

PyGithub is not in Spack Stack yet and is needed in support of the GitHub logging and gist creation python codes. This python module will be installed in $HOME/.local in the role accounts until the module is added to Spack Stack.

Same with GitHub CLI (which will be supported in Spack Stack 1.9) it will be installed in ${ROLE_HOME}/utils/bin which should be in role account's $PATH where ROLE_HOME is at the same level as fix and data.

This is setup to run in the role account with the following added directories and utils:

/gpfs/f6/drsa-precip3/proj-shared/global/CI

Location on disk where the CI test cases actually build and run specified in Jenkinsfile

└── GFS_CI_CD
    ├── global-workflow
    │   └── workflow
    ├── Bash
    ├── GitLab
    └── Jenkins
        ├── agent
        └── workspace

Top level CI directories (this will file structure be duplicated for each role account)

GFS_CI_CD

CI system directory for Jenkins launching scripts and in support of BASH CI

GFS_CI_CD/global-workflow

Directory where the JAVA agent is launched from

Jenkins/agent

Jenkins work space (system level workspace)

Jenkins/workspace

GitHub CLI (gh utility):

Location where gh was installed (pending when supporting Spack Stack 1.9)

gpfs/f6/drsa-precip3/world-shared/role.glopara/utils/bin/gh

soft link to gh utility for Jenkins to find

/ncrc/home1/role.glopara/bin

Resolves #3375

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

This PR can be tested in place after Dave's PR is merged for cycling on GaeaC6

github-advanced-security[bot]
github-advanced-security AI found potential problems Feb 25, 2025
View reviewed changes
Comment thread ci/scripts/utils/launch_java_agent.sh Fixed
aerorahul
aerorahul reviewed Feb 25, 2025
View reviewed changes
Comment thread ci/scripts/utils/launch_java_agent.sh Outdated
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA changed the title Ci jenkins role gaea6 Adding Jenkins CI to GaeaC6 using role account Feb 25, 2025
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added the CI/CD Issue related to CI/CD label Feb 25, 2025
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA marked this pull request as draft February 25, 2025 19:51
@TerrenceMcGuinness-NOAA
added new GaeaC6-EMC node to the NodeParameterDefinition list for res…
4f34f5b 
…tarting with parameters in the Jenkins controler
aerorahul
aerorahul reviewed Feb 25, 2025
View reviewed changes
Comment thread ci/scripts/utils/launch_java_agent.sh Outdated
github-advanced-security[bot]
github-advanced-security AI found potential problems Feb 25, 2025
View reviewed changes
Comment thread ci/scripts/utils/launch_java_agent.sh Fixed
Comment thread ci/scripts/utils/launch_java_agent.sh Fixed
github-advanced-security[bot]
github-advanced-security AI found potential problems Feb 27, 2025
View reviewed changes
Comment thread ci/scripts/utils/launch_java_agent.sh Fixed
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Mar 3, 2025
DavidHuber-NOAA
DavidHuber-NOAA reviewed Mar 3, 2025
View reviewed changes
Comment thread ci/scripts/utils/githubpr.py
from wxflow import which, Logger

# Initialize logger with environment variable for logging level
logger = Logger(level=os.environ.get("LOGGING_LEVEL", "DEBUG"), colored_log=False)
Copy link
Copy Markdown
Contributor

@DavidHuber-NOAA DavidHuber-NOAA Mar 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking, should this remain at LOGGING_LEVEL=DEBUG? Or was that just for testing?

Also, I have been looking for a way to disable colors when logging. Thanks for showing off this colored_log=False!

Copy link
Copy Markdown
Collaborator Author

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Mar 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

David, All of the python scripts that use the wxflow Logger are set at the DEBUG level, so I thought it best to follow form here. Notice that we can alter the Level at runtime with the environment variable LOGGING_LEVEL, so it appears the intent was to have DEBUG as the default.

DavidHuber-NOAA
DavidHuber-NOAA reviewed Mar 3, 2025
View reviewed changes
Comment thread ci/scripts/utils/launch_java_agent.sh Outdated
@emcbot emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Mar 3, 2025
@TerrenceMcGuinness-NOAA @DavidHuber-NOAA
Update ci/scripts/utils/launch_java_agent.sh
6f48a11 
Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>
TerrenceMcGuinness-NOAA
TerrenceMcGuinness-NOAA commented Mar 3, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator Author

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Committing reviews

Comment thread ci/Jenkinsfile
Comment thread ci/platforms/config.gaeac6
Comment thread ci/scripts/utils/launch_java_agent.sh Outdated
Comment thread ci/scripts/utils/githubpr.py
from wxflow import which, Logger

# Initialize logger with environment variable for logging level
logger = Logger(level=os.environ.get("LOGGING_LEVEL", "DEBUG"), colored_log=False)
Copy link
Copy Markdown
Collaborator Author

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Mar 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

David, All of the python scripts that use the wxflow Logger are set at the DEBUG level, so I thought it best to follow form here. Notice that we can alter the Level at runtime with the environment variable LOGGING_LEVEL, so it appears the intent was to have DEBUG as the default.

@emcbot
Copy link
Copy Markdown

emcbot commented Mar 3, 2025

Experiment C96C48_hybatmDA FAILED on Hera in Build# 6 with error logs:

/scratch1/NCEPDEV/global/CI/3389/RUNTESTS/COMROOT/C96C48_hybatmDA_7bcda9d6/logs/2021122100/enkfgdas_earc_tars_01.log

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Mar 3, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented Mar 3, 2025

Experiment C96C48_hybatmDA FAILED on Hera in Build# 6 in
/scratch1/NCEPDEV/global/CI/3389/RUNTESTS/EXPDIR/C96C48_hybatmDA_7bcda9d6

@emcbot
Copy link
Copy Markdown

emcbot commented Mar 3, 2025

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 6 with error logs:

/scratch1/NCEPDEV/global/CI/3389/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_7bcda9d6/logs/2021122018/enkfgdas_earc_tars_01.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link
Copy Markdown

emcbot commented Mar 3, 2025

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 6 in
/scratch1/NCEPDEV/global/CI/3389/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_7bcda9d6

@emcbot
Copy link
Copy Markdown

emcbot commented Mar 3, 2025

Experiment C48mx500_hybAOWCDA FAILED on Hera in Build# 6 with error logs:

/scratch1/NCEPDEV/global/CI/3389/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_7bcda9d6/logs/2021032500/enkfgdas_earc_tars_01.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link
Copy Markdown

emcbot commented Mar 3, 2025

Experiment C48mx500_hybAOWCDA FAILED on Hera in Build# 6 in
/scratch1/NCEPDEV/global/CI/3389/RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_7bcda9d6

@DavidHuber-NOAA
Copy link
Copy Markdown
Contributor

DavidHuber-NOAA commented Mar 3, 2025

These earc failures are expected and will be resolved by #3339. CI is running as intended. Merging.

@DavidHuber-NOAA DavidHuber-NOAA merged commit f5ddb9f into NOAA-EMC:develop Mar 3, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented Mar 3, 2025

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 6 with error logs:

/scratch1/NCEPDEV/global/CI/3389/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_7bcda9d6/logs/2024022400/enkfgdas_earc_tars_01.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link
Copy Markdown

emcbot commented Mar 3, 2025

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 6 in
/scratch1/NCEPDEV/global/CI/3389/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_7bcda9d6

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Mar 3, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented Mar 3, 2025

CI Failed on Hera in Build# 6
Built and ran in directory /scratch1/NCEPDEV/global/CI/3389


Experiment C96mx100_S2S_7bcda9d6 Completed 1 Cycles: *SUCCESS* at Mon Mar  3 20:02:54 UTC 2025
Experiment C48_ATM_7bcda9d6 Completed 1 Cycles: *SUCCESS* at Mon Mar  3 20:33:15 UTC 2025
Experiment C96C48_hybatmDA_7bcda9d6 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Mon Mar  3 20:45:32 UTC 2025
Experiment C96C48_hybatmDA_7bcda9d6 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3389/RUNTESTS/COMROOT/C96C48_hybatmDA_7bcda9d6/logs/2021122100/enkfgdas_earc_tars_01.log
Experiment C96C48_hybatmaerosnowDA_7bcda9d6 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Mon Mar  3 20:47:10 UTC 2025
Experiment C96C48_hybatmaerosnowDA_7bcda9d6 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3389/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_7bcda9d6/logs/2021122018/enkfgdas_earc_tars_01.log
Experiment C48mx500_hybAOWCDA_7bcda9d6 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Mon Mar  3 20:51:36 UTC 2025
Experiment C48mx500_hybAOWCDA_7bcda9d6 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3389/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_7bcda9d6/logs/2021032500/enkfgdas_earc_tars_01.log
Experiment C96C48_ufs_hybatmDA_7bcda9d6 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Mon Mar  3 21:34:26 UTC 2025
Experiment C96C48_ufs_hybatmDA_7bcda9d6 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3389/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_7bcda9d6/logs/2024022400/enkfgdas_earc_tars_01.log
Experiment C96_atm3DVar_7bcda9d6 Completed 3 Cycles: *SUCCESS* at Mon Mar  3 21:58:51 UTC 2025
Experiment C48_S2SW_7bcda9d6 Completed 1 Cycles: *SUCCESS* at Mon Mar  3 22:29:21 UTC 2025
Experiment C48_S2SWA_gefs_7bcda9d6 Completed 1 Cycles: *SUCCESS* at Mon Mar  3 22:43:51 UTC 2025
Experiment C48mx500_3DVarAOWCDA_7bcda9d6 Completed 2 Cycles: *SUCCESS* at Mon Mar  3 23:26:18 UTC 2025

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Mar 10, 2025
@KateFriedman-NOAA
Merge remote-tracking branch 'origin/develop' into feature/bugzilla_1373
6de1f91 
* origin/develop:
  Enable HPSS via Globus on Hercules and Orion
  Remove job name from product files & update GEMPAK module. (NOAA-EMC#3415)
  `link` instead of `copy` in staging jobs (NOAA-EMC#3410)
  Migrate CI Jenkins to role account on Hera (NOAA-EMC#3414)
  Add rocotorc documentation when using scrontab (NOAA-EMC#3417)
  Update jgdas atmos verfozn and verfrad with COMIN/COMOUT prefix instead of COM (NOAA-EMC#3342)
  Add configuration for empirically-corrected ozone parameters (NOAA-EMC#3386)
  Enable global-workflow to run C768C384 GSI on Gaea-C6 (NOAA-EMC#3412)
  Move logical checks into if blocks (NOAA-EMC#3339)
  Adding Jenkins CI to GaeaC6 using role account (NOAA-EMC#3389)
  Enable GDASApp g-w CI cases to run on wcoss2 (NOAA-EMC#3399)
  CI/CD Test on Gaea C5- And update config.gaea under ci/platform (NOAA-EMC#3280)
  Enable cycling support for Gaea C6 (NOAA-EMC#3323)
  Update enkf archive jobs to use COMIN/COMOUT (NOAA-EMC#3393)
  Copy marine ensemble output observation diags and spread (NOAA-EMC#3407)
  Ci testing on aws 2 (NOAA-EMC#3408)
  Disable METplus jobs on Hera (NOAA-EMC#3403)
  Add the mean EnKF soil increment to the deterministic member (NOAA-EMC#3295)
  Add mpich/8.1.19 to the WCOSS2 LD_LIBRARY_PATH for GDASApp jobs (NOAA-EMC#3396)
  Change order of RUNs (NOAA-EMC#3335)
  CI testing on aws (NOAA-EMC#3391)
danholdaway added a commit to danholdaway/global-workflow that referenced this pull request Mar 26, 2025
@danholdaway
Merge branch 'develop' into feature/remove_var_name_aliasing
9a20019 
* develop:
  Delete DATAROOT when running generate_workflows.sh (NOAA-EMC#3504)
  Fix 3244 garbled change (NOAA-EMC#3492)
  Enable ensemble archiving via Globus (NOAA-EMC#3479)
  Update MSU FIX_DIR paths (NOAA-EMC#3488)
  Updates for AOWCDA and hybatmaerosnowDA cases on Gaea C6 (NOAA-EMC#3487)
  Update GOCART path for GDAS/GFS/GCAFS implementations  (NOAA-EMC#3455)
  Make RUN Variables Explicit in `config.resources` (NOAA-EMC#3478)
  Remove unused key from enkfgdas_earc_vrfy (NOAA-EMC#3473)
  Bug fix to the failing early cycle marine DA ensemble re-centering (NOAA-EMC#3454)
  Make marine LETKF optional (NOAA-EMC#3462)
  When sourcing for RUN=enkf*, use CASE_ENS (NOAA-EMC#3475)
  Updates for Gaea: verif-global tag, tracker tag, Fit2Obs tag, and C768 analysis resources (NOAA-EMC#3463)
  Update gefswave glo_025 mesh file with new mask (NOAA-EMC#3457)
  Update MSU glopara paths to new role-global space (NOAA-EMC#3443)
  Enable CI testing on AWS (NOAA-EMC#3459)
  Enable Gaea C5 Jenkins CI (NOAA-EMC#3447)
  Job reference removal from WMO product names (NOAA-EMC#3460)
  Turn off aerosol prognostic radiative feedback for GDAS NOAA-EMC#2926 (NOAA-EMC#3445)
  Add DO_GEMPAK check to postsnd subtask (NOAA-EMC#3451)
  Add a force option to setup_xml to ignore unwritable directories (NOAA-EMC#3448)
  Remove the eomg job (NOAA-EMC#3331)
  Migration to role account for Jenkins on Orion (NOAA-EMC#3440)
  Eliminate `_gfs`, `_gdas`, etc, variables and add necessary if blocks (NOAA-EMC#3420)
  Update workflow staging for sfcanl tiles and waveinit (NOAA-EMC#3429)
  Improve messaging to display clear warning when missing snogrb file (NOAA-EMC#3317)
  JEDI-based ensemble recentering and analysis calculation (NOAA-EMC#3312)
  Enable HPSS archiving on C5/6 (NOAA-EMC#3437)
  Check if HOMEDIR STMP and PTMP are writable (NOAA-EMC#3430)
  Update UFS_Utils and GFS-utils hashes to update Gaea support and ocean/ice post products (NOAA-EMC#3433)
  Enable C1152 forecasts on gaea C6 (NOAA-EMC#3438)
  Migration to role account for Jenkins on Hercules (NOAA-EMC#3423)
  Remove Direct Linking to COM from DATA for `extractvars` Job (NOAA-EMC#3379)
  Enable HPSS via Globus on Hercules and Orion
  Remove job name from product files & update GEMPAK module. (NOAA-EMC#3415)
  `link` instead of `copy` in staging jobs (NOAA-EMC#3410)
  Migrate CI Jenkins to role account on Hera (NOAA-EMC#3414)
  Add rocotorc documentation when using scrontab (NOAA-EMC#3417)
  Update jgdas atmos verfozn and verfrad with COMIN/COMOUT prefix instead of COM (NOAA-EMC#3342)
  Add configuration for empirically-corrected ozone parameters (NOAA-EMC#3386)
  Enable global-workflow to run C768C384 GSI on Gaea-C6 (NOAA-EMC#3412)
  Move logical checks into if blocks (NOAA-EMC#3339)
  Adding Jenkins CI to GaeaC6 using role account (NOAA-EMC#3389)
  Enable GDASApp g-w CI cases to run on wcoss2 (NOAA-EMC#3399)
  CI/CD Test on Gaea C5- And update config.gaea under ci/platform (NOAA-EMC#3280)
  Enable cycling support for Gaea C6 (NOAA-EMC#3323)
  Update enkf archive jobs to use COMIN/COMOUT (NOAA-EMC#3393)
  Copy marine ensemble output observation diags and spread (NOAA-EMC#3407)
  Ci testing on aws 2 (NOAA-EMC#3408)
  Disable METplus jobs on Hera (NOAA-EMC#3403)
  Add the mean EnKF soil increment to the deterministic member (NOAA-EMC#3295)
  Add mpich/8.1.19 to the WCOSS2 LD_LIBRARY_PATH for GDASApp jobs (NOAA-EMC#3396)
  Change order of RUNs (NOAA-EMC#3335)
  CI testing on aws (NOAA-EMC#3391)
  Rename Gulf of Mexico in bufr station list in GFSv17 (NOAA-EMC#3384)
  Enabling AWS CI/testing (NOAA-EMC#3383)
  Update issue templates to use new issue type field (NOAA-EMC#3369)
  Replace WAVECUR_DID variable with "rtofs" (NOAA-EMC#3337)
  Allow for C1152 ATM-Aero cycled DA to run on WCOSS2 (NOAA-EMC#3309)
  Remove Direct Linking to COM from DATA for `wavepostsbs` Job (NOAA-EMC#3303)
  Update jgdas enkf update job with COMIN or COMOUT prefix instead of COM (NOAA-EMC#3333)
  Add capability to run diff resolutions for marine anl and background (NOAA-EMC#3238)
  Update high resolution tests and fix minor wave issues  (NOAA-EMC#3289)
  Add sfs as valid system (NOAA-EMC#3243)
  Add missing arch_tars dependencies (NOAA-EMC#3319)
  Fix the empty aerosol DA aerostat tar file issue (NOAA-EMC#3332)
  Add missing file safeguard for IMS prep in snow analysis tasks (NOAA-EMC#3329)
tsga added a commit to tsga/global-workflow that referenced this pull request May 1, 2025
@tsga
Merge branch 'develop' into feature/land_iau
ad208da 
* develop:
  Update GSI hash and GSI fix version to resolve bugs (NOAA-EMC#3626)
  Add missing marine DA files to archiving  (NOAA-EMC#3596)
  Add a low resolution test to mimic GFSv17 cycling as much as possible (NOAA-EMC#3617)
  Add the setting to use the reject list for station t/q observations in GSI based soil DA (NOAA-EMC#3599)
  GitLab CI Framework for schedule PR cases and ctests on multi hosts (NOAA-EMC#3603)
  Avoid parallel restart I/O on WCOSS2 (NOAA-EMC#3615)
  Enables user toggling of GDASApp g-w ctests (NOAA-EMC#3587)
  COM variable updates for prep and some external downstream jobs (NOAA-EMC#3608)
  Remove MOS from system (NOAA-EMC#3612)
  Updates to enable soil DA  (NOAA-EMC#3452)
  Unexport SHELLOPTS when running htar (NOAA-EMC#3601)
  Fix check for netcdf wave restart (NOAA-EMC#3594)
  Call err_chk/err_exit for fatal errors in post JJobs/ex-scripts (NOAA-EMC#3571)
  Remove support for Jet and S4 (NOAA-EMC#3572)
  Hotfix in GitLab pipline for Nightly (env MACHINE breaks build on head node) (NOAA-EMC#3578)
  [hotfix] Missed a path during merging develop (NOAA-EMC#3577)
  Prepare for ops readiness - part 1 (NOAA-EMC#3557)
  Update UFS weather-model to 20250328 hash (NOAA-EMC#3528)
  Fix SFS fcst config (NOAA-EMC#3574)
  Use err_chk in GDAS j-jobs (NOAA-EMC#3570)
  Perform compute builds on Gaea head nodes (NOAA-EMC#3560)
  Add initial capability to produce JEDI-based observation space summary stat files (NOAA-EMC#3471)
  Spread epos over more nodes on Hera to increase allocated memory (NOAA-EMC#3567)
  Create separate gists when multiple files are published on GitHub (NOAA-EMC#3551)
  Use err_chk in GSI J-Jobs and scripts (NOAA-EMC#3549)
  Add unified jinja obs list to marine DA (NOAA-EMC#3530)
  Save snow and aerosol analysis increments (and logs and YAMLs) every cycle (NOAA-EMC#3537)
  Add Dependencies to SFS Cleanup Job (NOAA-EMC#3559)
  Updates archiving to reflect current naming of marine anl output (NOAA-EMC#3541)
  Temporarily disable compute builds on C6 (NOAA-EMC#3558)
  Update gdas.cd hash to resolve msu prod_util failure (NOAA-EMC#3556)
  COMIN/COMOUT updates for enkf chgres and downstream product jobs (NOAA-EMC#3518)
  Call err_chk in forecast scripts for fatal errors (NOAA-EMC#3515)
  Add Rocoto Jobs for the Missing Products of GEFS (NOAA-EMC#3466)
  Download subset fix data with python script (NOAA-EMC#3400)
  Check that partition should be set (NOAA-EMC#3543)
  Rename wave output and refactor some wave scripts to use MPMD, and fix some bugzillas along the way (NOAA-EMC#3517)
  Add support for dual batch partitions on AWS NOAA-EMC#3483
  Update CI build and run directories for GitLab Nightlies on C6 and added GitLab support on Hera (NOAA-EMC#3536)
  Hotfix path for CI in Jenkins on Gaea C6 to it's world-share path (NOAA-EMC#3532)
  Create single ocean grib2 product file (NOAA-EMC#3529)
  Scheduled Nightly CI/CD Pipeline Script in GitLab on Gaea C6 (NOAA-EMC#3493)
  make sure cold starts are handled correctly when DOIAU=YES (issue NOAA-EMC#3516) (NOAA-EMC#3520)
  Add check for DO_AERO_FCST before copying fv_tracer files (NOAA-EMC#3485)
  Use jinja templates instead of `@VARNAME@` in config files (NOAA-EMC#3411)
  Replace "status" (or comparable) with "err" in preparation for moving to err_chk/err_exit (NOAA-EMC#3507)
  Error in Java launch script for CI (NOAA-EMC#3465)
  Delete DATAROOT when running generate_workflows.sh (NOAA-EMC#3504)
  Fix 3244 garbled change (NOAA-EMC#3492)
  Enable ensemble archiving via Globus (NOAA-EMC#3479)
  Update MSU FIX_DIR paths (NOAA-EMC#3488)
  Updates for AOWCDA and hybatmaerosnowDA cases on Gaea C6 (NOAA-EMC#3487)
  Update GOCART path for GDAS/GFS/GCAFS implementations  (NOAA-EMC#3455)
  Make RUN Variables Explicit in `config.resources` (NOAA-EMC#3478)
  Remove unused key from enkfgdas_earc_vrfy (NOAA-EMC#3473)
  Bug fix to the failing early cycle marine DA ensemble re-centering (NOAA-EMC#3454)
  Make marine LETKF optional (NOAA-EMC#3462)
  When sourcing for RUN=enkf*, use CASE_ENS (NOAA-EMC#3475)
  Updates for Gaea: verif-global tag, tracker tag, Fit2Obs tag, and C768 analysis resources (NOAA-EMC#3463)
  Update gefswave glo_025 mesh file with new mask (NOAA-EMC#3457)
  Update MSU glopara paths to new role-global space (NOAA-EMC#3443)
  Enable CI testing on AWS (NOAA-EMC#3459)
  Enable Gaea C5 Jenkins CI (NOAA-EMC#3447)
  Job reference removal from WMO product names (NOAA-EMC#3460)
  Turn off aerosol prognostic radiative feedback for GDAS NOAA-EMC#2926 (NOAA-EMC#3445)
  Add DO_GEMPAK check to postsnd subtask (NOAA-EMC#3451)
  Add a force option to setup_xml to ignore unwritable directories (NOAA-EMC#3448)
  Remove the eomg job (NOAA-EMC#3331)
  Migration to role account for Jenkins on Orion (NOAA-EMC#3440)
  Eliminate `_gfs`, `_gdas`, etc, variables and add necessary if blocks (NOAA-EMC#3420)
  Update workflow staging for sfcanl tiles and waveinit (NOAA-EMC#3429)
  Improve messaging to display clear warning when missing snogrb file (NOAA-EMC#3317)
  JEDI-based ensemble recentering and analysis calculation (NOAA-EMC#3312)
  Enable HPSS archiving on C5/6 (NOAA-EMC#3437)
  Check if HOMEDIR STMP and PTMP are writable (NOAA-EMC#3430)
  Update UFS_Utils and GFS-utils hashes to update Gaea support and ocean/ice post products (NOAA-EMC#3433)
  Enable C1152 forecasts on gaea C6 (NOAA-EMC#3438)
  Migration to role account for Jenkins on Hercules (NOAA-EMC#3423)
  Remove Direct Linking to COM from DATA for `extractvars` Job (NOAA-EMC#3379)
  Enable HPSS via Globus on Hercules and Orion
  Remove job name from product files & update GEMPAK module. (NOAA-EMC#3415)
  `link` instead of `copy` in staging jobs (NOAA-EMC#3410)
  Migrate CI Jenkins to role account on Hera (NOAA-EMC#3414)
  Add rocotorc documentation when using scrontab (NOAA-EMC#3417)
  Update jgdas atmos verfozn and verfrad with COMIN/COMOUT prefix instead of COM (NOAA-EMC#3342)
  Add configuration for empirically-corrected ozone parameters (NOAA-EMC#3386)
  Enable global-workflow to run C768C384 GSI on Gaea-C6 (NOAA-EMC#3412)
  Move logical checks into if blocks (NOAA-EMC#3339)
  Adding Jenkins CI to GaeaC6 using role account (NOAA-EMC#3389)
  Enable GDASApp g-w CI cases to run on wcoss2 (NOAA-EMC#3399)
  CI/CD Test on Gaea C5- And update config.gaea under ci/platform (NOAA-EMC#3280)
  Enable cycling support for Gaea C6 (NOAA-EMC#3323)
  Update enkf archive jobs to use COMIN/COMOUT (NOAA-EMC#3393)
  Copy marine ensemble output observation diags and spread (NOAA-EMC#3407)
  Ci testing on aws 2 (NOAA-EMC#3408)
  Disable METplus jobs on Hera (NOAA-EMC#3403)
  Add the mean EnKF soil increment to the deterministic member (NOAA-EMC#3295)
  Add mpich/8.1.19 to the WCOSS2 LD_LIBRARY_PATH for GDASApp jobs (NOAA-EMC#3396)
  Change order of RUNs (NOAA-EMC#3335)
  CI testing on aws (NOAA-EMC#3391)
  Rename Gulf of Mexico in bufr station list in GFSv17 (NOAA-EMC#3384)
  Enabling AWS CI/testing (NOAA-EMC#3383)
  Update issue templates to use new issue type field (NOAA-EMC#3369)
  Replace WAVECUR_DID variable with "rtofs" (NOAA-EMC#3337)
  Allow for C1152 ATM-Aero cycled DA to run on WCOSS2 (NOAA-EMC#3309)
  Remove Direct Linking to COM from DATA for `wavepostsbs` Job (NOAA-EMC#3303)
  Update jgdas enkf update job with COMIN or COMOUT prefix instead of COM (NOAA-EMC#3333)
  Add capability to run diff resolutions for marine anl and background (NOAA-EMC#3238)
  Update high resolution tests and fix minor wave issues  (NOAA-EMC#3289)
  Add sfs as valid system (NOAA-EMC#3243)
  Add missing arch_tars dependencies (NOAA-EMC#3319)
  Fix the empty aerosol DA aerostat tar file issue (NOAA-EMC#3332)
  Add missing file safeguard for IMS prep in snow analysis tasks (NOAA-EMC#3329)
  Fix memory unsetting on Gaea (NOAA-EMC#3325)
  Fix error log parsing in compute build CI (NOAA-EMC#3301)
  Remove marineanlvrfy task from global-workflow (NOAA-EMC#3314)
  Add `gfs_wavepostpnt` dependencies to gfs_cleanup (NOAA-EMC#3313)
  Increase the GDASApp build wallclock (NOAA-EMC#3298)
  Capture build fail in Jenkins pipeline when no error logs are produced (NOAA-EMC#3297)
  Add/update config files for Gaea and check existence before sourcing config files in generate_workflows.sh (NOAA-EMC#3286)
  Fix ocean restarts when cold starting with DOIAU=YES (NOAA-EMC#3278)
  Splitting up the archive task (NOAA-EMC#3242)
  CTests extended validation for C48_ATM and staged C48_S2SW for gfs_fcst and gfs_atmos (NOAA-EMC#3256)
  Add esnowanl to enkfgfs cycle (NOAA-EMC#3283)
  Add gfs cycles to C48mx500_3DVarAOWCDA (NOAA-EMC#3249)
  Add fetch job and update stage_ic to work with fetched ICs (NOAA-EMC#3141)
  Remove WAFS files and references from `develop` (NOAA-EMC#3263)
  fix intel stack version number on c5 (NOAA-EMC#3258)
  Update gsi_monitor and ufs_utils hashes to recent hashes for C5/C6 build and run (NOAA-EMC#3252)
  Enable DA cycling on gaea C5/C6 (NOAA-EMC#3255)
  Copy post-processed sea ice increment for diagnostics (NOAA-EMC#3235)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DavidHuber-NOAA DavidHuber-NOAA DavidHuber-NOAA approved these changes

@WalterKolczynski-NOAA WalterKolczynski-NOAA Awaiting requested review from WalterKolczynski-NOAA

@KateFriedman-NOAA KateFriedman-NOAA Awaiting requested review from KateFriedman-NOAA

@aerorahul aerorahul Awaiting requested review from aerorahul

Assignees

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA

Labels

CI/CD Issue related to CI/CD CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Migrate Jenkins CI automation to Role Accounts on Gaea6

5 participants

@TerrenceMcGuinness-NOAA @emcbot @DavidHuber-NOAA @aerorahul @github-advanced-security