Skip to content

Stage atmospheric backgrounds and UFS cubed-sphere history files#2792

Merged
WalterKolczynski-NOAA merged 122 commits into
NOAA-EMC:developfrom
DavidNew-NOAA:feature/cubed_sphere_bkg
Aug 8, 2024
Merged

Stage atmospheric backgrounds and UFS cubed-sphere history files#2792
WalterKolczynski-NOAA merged 122 commits into
NOAA-EMC:developfrom
DavidNew-NOAA:feature/cubed_sphere_bkg

Conversation

@DavidNew-NOAA
Copy link
Copy Markdown
Contributor

@DavidNew-NOAA DavidNew-NOAA commented Jul 25, 2024 •
edited
Loading

Description

This PR changes the G-W so that backgrounds for the atmospheric analyses are staged as UFS cubed-sphere history files rather than FMS restarts. The motivation being that FMS restarts include many data we don't need for the analyses and thus take up too much disk space/bandwidth.

This is a companion PR to GDASApp #1236 and JCB-GDAS #19.

Type of change

  • New feature (adds functionality)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? YES
  • Does this change require a documentation update? NO

How has this been tested?

  • Build on Hera
  • Cycling experiment on Hera
  • GDASApp jjob tests

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • I have made corresponding changes to the documentation if necessary

danholdaway and others added 30 commits May 29, 2024 14:35
@danholdaway
Use rename_atm of jcb parm erpos
b5eb345
@DavidNew-NOAA
Initial commit
3a8f527
@DavidNew-NOAA
Update atm_berror_bump_fix.yaml.j2
6904982
@DavidNew-NOAA
Update atm_berror_gsibec_fix.yaml.j2
0b37fe5
@DavidNew-NOAA
Update atm_berror_identity_fix.yaml.j2
d770b63
@DavidNew-NOAA
Update atm_bkg_fix.yaml.j2
0187bba
@DavidNew-NOAA
Rename some things
4dcd47a
@DavidNew-NOAA
Merge branch 'feature/stage_from_yaml' of https://github.com/DavidNew…
e61963d 
…-NOAA/global-workflow into feature/stage_from_yaml
@DavidNew-NOAA
Rename some things
f91297d
@DavidNew-NOAA
Add changes to analysis python scripts
983d720
@danholdaway
update hashes and config
9436f4b
@danholdaway
Merge remote-tracking branch 'upsteam/develop' into feature/rename_atm
97a63a7 
* upsteam/develop:
  Archiving cleanup (NOAA-EMC#2621)
  Switch to Rocky 9 built external packages on Hercules (NOAA-EMC#2608)
  Add the capability to use slurm reservation nodes (NOAA-EMC#2627)
  Update forecast job to use COMIN/COMOUT (NOAA-EMC#2622)
  Update to add 1-deg global wave grid (NOAA-EMC#2619)
  Add C384mx025_3DVarAOWCDA yamls (NOAA-EMC#2625)
  Script to keep Jenkins Agent persistent from cron (NOAA-EMC#2634)
@DavidNew-NOAA
Merge branch 'develop' into feature/stage_from_yaml
6e44a5f
@DavidNew-NOAA
Merge branch 'develop' into feature/stage_from_yaml
2490049
@DavidNew-NOAA
Minor fix
3022393
@DavidNew-NOAA
Minor update
cc1a3b6
@DavidNew-NOAA
Minor update
2097116
@DavidNew-NOAA
Coding norms
5327711
@danholdaway
Remove jcb from GW submodules
2702ef0
@danholdaway
Remove JCB
b6456be
@danholdaway
adjust link to point to new jcb location in gdasapp
2a5f301
@danholdaway
Update GDASapp Hash
9f425a6
@DavidNew-NOAA
Move some stuff around
4d36caf
@danholdaway
Merge remote-tracking branch 'upsteam/develop' into feature/rename_atm
55358e5 
* upsteam/develop:
  Update ufs-weather-model  (NOAA-EMC#2646)
  Update wmo parm files to fix WMO header (NOAA-EMC#2652)
  Add IAU to snow DA (and its test) (NOAA-EMC#2610)
@danholdaway
gdas hash
9e357f9
@DavidNew-NOAA
Add time loop for some staging
ba2d12d
@danholdaway
bump gdas
d0d4429
@danholdaway
bump gdas
ee1aa30
@DavidNew-NOAA
Merge branch 'NOAA-EMC:develop' into feature/stage_from_yaml
2138df1
@danholdaway
hashes
09ec021
@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Aug 7, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Aug 7, 2024
@emcbot
Copy link
Copy Markdown

emcbot commented Aug 7, 2024

Build FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2792/gfs/sorc/logs/build_ufs.log

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Aug 7, 2024
@WalterKolczynski-NOAA
Copy link
Copy Markdown
Contributor

WalterKolczynski-NOAA commented Aug 7, 2024

Now it's failing due to lack of space, I think.

@WalterKolczynski-NOAA WalterKolczynski-NOAA removed the CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed label Aug 7, 2024
@WalterKolczynski-NOAA
Copy link
Copy Markdown
Contributor

WalterKolczynski-NOAA commented Aug 7, 2024

Now it's failing due to lack of space, I think.

Actually, a sysadmin killed all the CI running on Hera because we had three PRs building at once and it hammered the node. Going to relaunch a bit at a time.

@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Aug 7, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Aug 7, 2024
@emcbot
Copy link
Copy Markdown

emcbot commented Aug 8, 2024

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2792


Experiment C48_ATM_4784d241 Completed 1 Cycles: *SUCCESS* at Wed Aug  7 23:17:06 UTC 2024
Experiment C48mx500_3DVarAOWCDA_4784d241 Completed 2 Cycles: *SUCCESS* at Wed Aug  7 23:42:56 UTC 2024
Experiment C96_atm3DVar_4784d241 Completed 3 Cycles: *SUCCESS* at Thu Aug  8 00:48:40 UTC 2024
Experiment C48_S2SWA_gefs_4784d241 Completed 1 Cycles: *SUCCESS* at Thu Aug  8 00:52:02 UTC 2024
Experiment C96C48_hybatmDA_4784d241 Completed 3 Cycles: *SUCCESS* at Thu Aug  8 01:06:46 UTC 2024
Experiment C48_S2SW_4784d241 Completed 1 Cycles: *SUCCESS* at Thu Aug  8 01:11:56 UTC 2024
Experiment C96_atmaerosnowDA_4784d241 Completed 3 Cycles: *SUCCESS* at Thu Aug  8 02:05:42 UTC 2024

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit e2c0f06 into NOAA-EMC:develop Aug 8, 2024
@DavidNew-NOAA
Copy link
Copy Markdown
Contributor Author

DavidNew-NOAA commented Aug 8, 2024

Thank you @WalterKolczynski-NOAA

@EricSinsky-NOAA EricSinsky-NOAA mentioned this pull request Aug 8, 2024
Merged
7 tasks
DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this pull request Aug 9, 2024
@DavidHuber-NOAA
Merge remote-tracking branch 'origin/develop' into fix/c768_hera
638a588 
* origin/develop:
  Stage atmospheric backgrounds and UFS cubed-sphere history files (NOAA-EMC#2792)
  Check that a PR driver is still running before trying to kill it (NOAA-EMC#2799)
  Feature/get arch adds an empty archive job to GEFS system (NOAA-EMC#2772)
  Marine DA updates (NOAA-EMC#2802)
  Revert MSU FIX_DIRs back to glopara (NOAA-EMC#2811)
  Bugfix for updating label states in Jenkins (NOAA-EMC#2808)
  Clean-up temporary rundirs - take 2. (NOAA-EMC#2753)
  Change land surface for HR4 (NOAA-EMC#2787)
  Run METplus serially and correct the name of prod tasks (NOAA-EMC#2804)
  Update Java Agent launching script for Jenkins connections (NOAA-EMC#2762)
  Fix erroneous cdump addition (NOAA-EMC#2803)
  Update ocean post-processing triggers (NOAA-EMC#2784)
  Update the gfs_utils repository hash (NOAA-EMC#2801)
  Add fixes for metplus jobs when gfs_cyc=2 or 4 (NOAA-EMC#2791)
  Simplify resource-related variables, remove CDUMP where unneeded (NOAA-EMC#2727)
  Remove f000 from atmos rocoto tasks for replay cases (NOAA-EMC#2778)
DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this pull request Aug 9, 2024
@DavidHuber-NOAA
Merge remote-tracking branch 'origin/develop' into fix/c768_hera
17efba2 
* origin/develop:
  Stage atmospheric backgrounds and UFS cubed-sphere history files (NOAA-EMC#2792)
  Check that a PR driver is still running before trying to kill it (NOAA-EMC#2799)
  Feature/get arch adds an empty archive job to GEFS system (NOAA-EMC#2772)
  Marine DA updates (NOAA-EMC#2802)
  Revert MSU FIX_DIRs back to glopara (NOAA-EMC#2811)
  Bugfix for updating label states in Jenkins (NOAA-EMC#2808)
  Clean-up temporary rundirs - take 2. (NOAA-EMC#2753)
  Change land surface for HR4 (NOAA-EMC#2787)
  Run METplus serially and correct the name of prod tasks (NOAA-EMC#2804)
  Update Java Agent launching script for Jenkins connections (NOAA-EMC#2762)
  Fix erroneous cdump addition (NOAA-EMC#2803)
  Update ocean post-processing triggers (NOAA-EMC#2784)
  Update the gfs_utils repository hash (NOAA-EMC#2801)
  Add fixes for metplus jobs when gfs_cyc=2 or 4 (NOAA-EMC#2791)
  Simplify resource-related variables, remove CDUMP where unneeded (NOAA-EMC#2727)
  Remove f000 from atmos rocoto tasks for replay cases (NOAA-EMC#2778)
@RussTreadon-NOAA RussTreadon-NOAA mentioned this pull request Sep 11, 2024
Closed
@DavidNew-NOAA DavidNew-NOAA deleted the feature/cubed_sphere_bkg branch November 15, 2024 01:28
@DavidNew-NOAA DavidNew-NOAA mentioned this pull request Jun 11, 2025
Closed
3 tasks
DavidNew-NOAA added a commit to NOAA-EMC/GDASApp that referenced this pull request Jan 6, 2026
@DavidNew-NOAA
Stage atmospheric backgrounds and UFS cubed-sphere history files (#19)
7b3b8c6 
This PR changes the G-W so that backgrounds for the atmospheric analyses
are staged as UFS cubed-sphere history files rather than FMS restarts.
The motivation being that FMS restarts include many data we don't need
for the analyses and thus take up too much disk space/bandwidth.

This is a companion PR to Global Workflow
[#2791](NOAA-EMC/global-workflow#2792) and
GDASApp [#1236](#1236).
DavidNew-NOAA added a commit to NOAA-EMC/GDASApp that referenced this pull request Jan 16, 2026
@DavidNew-NOAA
Stage atmospheric backgrounds and UFS cubed-sphere history files (#1236)
96a97dc 
This PR changes the GDASApp so that backgrounds for the atmospheric
analyses are staged as UFS cubed-sphere history files rather than FMS
restarts. jjob tests and test data are also updated to accommodate these
changes. The motivation being that FMS restarts include many data we
don't need for the analyses and thus take up too much disk
space/bandwidth.

This is a companion PR to Global Workflow
[#2792](NOAA-EMC/global-workflow#2792) and
JCB-GDAS [#19](NOAA-EMC/jcb-gdas#19).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@RussTreadon-NOAA RussTreadon-NOAA RussTreadon-NOAA left review comments

Assignees

No one assigned

Labels

CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@DavidNew-NOAA @RussTreadon-NOAA @emcbot @WalterKolczynski-NOAA @danholdaway @aerorahul @DavidHuber-NOAA