Platform-594 AutoRT Jenkins

[kbooker79]

Description

This PR adds the codebase a Jenkins variant of existing crontab initiated AutoRT. These are the Python and shell script code changes necessary to run AutoRT in the context of the Jenkins CI/CD system on all of the tier-1 platforms.

Top of commit queue on: TBD

Input data additions/changes

• No changes are expected to input data.
• There will be new input data.
• Input data will be updated.

Anticipated changes to regression tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

Subcomponents involved:

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Combined with PR's (If Applicable):

Commit Queue Checklist:

• Link PR's from all sub-components involved
• Confirm reviews completed in sub-component PR's
• Add all appropriate labels to this PR.
• Run full RT suite on either Hera/Cheyenne with both Intel/GNU compilers
• Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

Testing Day Checklist:

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
• Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

Testing Log (for CM's):

• RDHPCS
  • Intel
    • Hera
    • Orion
    • Jet
    • Gaea
    • Cheyenne
  • GNU
    • Hera
    • Cheyenne
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[jkbk2004]

#1531 was closed and this pr is just created. There are still separate steps we need to deactivate crontab to start test one machine at a time though. I like to start getting review comments. @BrianCurtis-NOAA I am separately following up with baseline storage location issues. Jenkins hookup is done by epic role account but our baselines are under emc.nemspara: write permission issue across two role accounts. We can talk more about that issue at tagup.

[github-actions]

@kbooker79 please bring these up to date with respective authoritative repositories

• ufs-weather-model NOT up to date
• fv3 NOT up to date

[github-actions]

@kbooker79 please bring these up to date with respective authoritative repositories

• ufs-weather-model NOT up to date
• fv3 NOT up to date

[github-actions]

@kbooker79 please bring these up to date with respective authoritative repositories

• ufs-weather-model NOT up to date
• fv3 NOT up to date

[jkbk2004]

@kbooker79 can you sync up branch?

[BrianCurtis-NOAA]

Automated RT Failure Notification
Machine: gaea
Compiler: intel
Job: RT
[RT] Repo location: /lustre/f2/pdata/ncep/emc.nemspara/autort/pr/1215793622/20230125200007/ufs-weather-model
[RT] Error: Test hrrr_control_decomp_dyn32_phy32 106 failed in run_test failed
Please make changes and add the following label back: gaea-intel-RT

[jkbk2004]

@kbooker79 is there any reason you are not working on your forked feature branch?

[jkbk2004]

@kbooker79 I see /glade/scratch/dtcufsrt on cheyenne test. it should be epicufsrt.

[kbooker79]

@kbooker79 I see /glade/scratch/dtcufsrt on cheyenne test. it should be epicufsrt.

@kbooker79 I see /glade/scratch/dtcufsrt on cheyenne test. it should be epicufsrt.

@jkbk2004, I noticed that as well...I just made the change.

[jkbk2004]

The cheyenne jenkins pipeline runs ok without any confliction. This PR is ready.

[sadeghitabas]

I did an RT for hera.intel and hera.gnu as well, you can find it below:
/scratch1/NCEPDEV/nems/Sadegh.Tabas/git/PR1580/

[BrianCurtis-NOAA]

One thing I notice is that the removal of the labels is through my account still (using Cheyenne as an example). The goal is to remove my account from all aspects of the autort process.

[jkbk2004]

One thing I notice is that the removal of the labels is through my account still (using Cheyenne as an example). The goal is to remove my account from all aspects of the autort process.

@kbooker79 can you make sure again?

[DeniseWorthen]

@kbooker79 Please add a description for what this PR does, as requested in the PR template:

Provide a detailed description of what this PR does. What bug does it fix, or what feature does it add? Is a change of answers expected from this PR? Are any library updates included in this PR (modulefiles etc.)?

[kbooker79]

@kbooker79 Please add a description for what this PR does, as requested in the PR template:

Provide a detailed description of what this PR does. What bug does it fix, or what feature does it add? Is a change of answers expected from this PR? Are any library updates included in this PR (modulefiles etc.)?

This has been updated as requested.

[kbooker79]

One thing I notice is that the removal of the labels is through my account still (using Cheyenne as an example). The goal is to remove my account from all aspects of the autort process.

@kbooker79 can you make sure again?

@jkbk2004, it's not using your account specifically. This is being taken care of via the automation and is no way tied to you.

[jkbk2004]

One thing I notice is that the removal of the labels is through my account still (using Cheyenne as an example). The goal is to remove my account from all aspects of the autort process.

@kbooker79 can you make sure again?

@jkbk2004, it's not using your account specifically. This is being taken care of via the automation and is no way tied to you.

@kbooker79 the point is why it still uses Brian's account when it removes label. can you check again?

[jkbk2004]

looks good! all sets.