Skip to content

Add functionality to assimilate the new snow observations#4132

Merged
DavidHuber-NOAA merged 16 commits into
NOAA-EMC:developfrom
jiaruidong2017:feature/snocvr_snomad
Oct 15, 2025
Merged

Add functionality to assimilate the new snow observations#4132
DavidHuber-NOAA merged 16 commits into
NOAA-EMC:developfrom
jiaruidong2017:feature/snocvr_snomad

Conversation

@jiaruidong2017
Copy link
Copy Markdown
Contributor

@jiaruidong2017 jiaruidong2017 commented Oct 6, 2025

Description

This PR adds an entry to enable assimilation of the newly introduced snow observations. This PR also includes functionality to prepare the combined SNOCVR and SNOMAD observations for global snow analysis.

Resolves #1888

Type of change

  • New feature (adds functionality)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO

  • Does this change require a documentation update? NO

  • Does this change require an update to any of the following submodules? YES

How has this been tested?

I conducted cycled experiments with DO_JEDISNOWDA=YES. These runs also require snow observations from GDA after September 2025, when the new SNOMAD products began being produced operationally.

  • Cycled test on Ursa

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@jiaruidong2017 jiaruidong2017 mentioned this pull request Oct 6, 2025
Merged
7 tasks
CoryMartin-NOAA
CoryMartin-NOAA reviewed Oct 6, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@CoryMartin-NOAA CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar changes need added for esnowanl

Comment thread dev/parm/config/gfs/config.snowanl.j2 Outdated

export PREP_SNOCVR_SNOMAD_YAML="${PARMgfs}/gdas/snow/prep/prep_snocvr_snomad.yaml.j2"
export OBSBUILDER="${USHgfs}/bufr_snocvr_snomad.py"
export PYTHONPATH=${HOMEgfs}/lib/python3.11/site-packages:${PYTHONPATH}
Copy link
Copy Markdown
Contributor

@CoryMartin-NOAA CoryMartin-NOAA Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't hard code python path for python version, nor should PYTHONPATH be defined here (it should be in the dev/ush/ module script

Copy link
Copy Markdown
Contributor Author

@jiaruidong2017 jiaruidong2017 Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be included in the dev/ush/load_ufsda_modules.sh, correct? Thanks.

Copy link
Copy Markdown
Contributor

@CoryMartin-NOAA CoryMartin-NOAA Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

Copy link
Copy Markdown
Contributor Author

@jiaruidong2017 jiaruidong2017 Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made the changes and added the changes for esnowanl.

CoryMartin-NOAA pushed a commit to NOAA-EMC/GDASApp that referenced this pull request Oct 6, 2025
@jiaruidong2017
Support assimilation of new snow observations (#1930)
cdb9122 
# Description

This PR adds a BUFR-query–based obsBuilder script that concatenates the
new SNOMAD observation into SNOCVR, and processes the result into
IODA-format files for snow DA in the global-workflow.

# Companion PRs

This PR depends NOAA-EMC/jcb-gdas#188
This PR contributes
NOAA-EMC/global-workflow#4132

# Issues

Resolves #1888 

# Automated CI tests to run in Global Workflow
<!-- Which Global Workflow CI tests are required to adequately test this
PR? -->
- [ ] atm_jjob <!-- JEDI atm single cycle DA !-->
- [ ] C96C48_ufs_hybatmDA <!-- JEDI atm cycled DA !-->
- [ ] C96C48_hybatmsnowDA <!-- JEDI snow cycled DA !-->
- [ ] C96_gcafs_cycled <!-- JEDI aerosol cycled DA !-->
- [ ] C48mx500_3DVarAOWCDA <!-- JEDI low-res marine 3DVar cycled DA !-->
- [ ] C48mx500_hybAOWCDA <!-- JEDI marine hybrid envar cycled DA !-->
- [ ] C96C48_hybatmDA <!-- GSI atm cycled DA !-->
@jiaruidong2017 jiaruidong2017 added the JEDI Feature development to support JEDI-based DA label Oct 6, 2025
CoryMartin-NOAA
CoryMartin-NOAA reviewed Oct 7, 2025
View reviewed changes
Comment thread ush/python/pygfs/task/snow_analysis.py Outdated
Comment on lines +157 to +164
# Ensure the IODA snow depth GHCN file is produced by the IODA converter
# If so, copy to DATA/obs/
if not os.path.isfile(f"{os.path.join(localconf.DATA, output_file)}"):
logger.exception(f"{self.task_config.OBSBUILDER} failed to produce {output_file}")
raise FileNotFoundError(f"{os.path.join(localconf.DATA, output_file)}")
else:
logger.info(f"Copy {output_file} successfully generated")
FileHandler(prep_snocvr_snomad_config.netcdf).sync()
Copy link
Copy Markdown
Contributor

@CoryMartin-NOAA CoryMartin-NOAA Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we should allow the analysis to still run even if this file fails to be produced? Thoughts?

Copy link
Copy Markdown
Contributor Author

@jiaruidong2017 jiaruidong2017 Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will revise the warning and ensure the process continues when the file is not produced, once Ursa is back online. Thank you for your suggestions.

Copy link
Copy Markdown
Contributor Author

@jiaruidong2017 jiaruidong2017 Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made the changes.

@jiaruidong2017
Update snow_analysis.py
54e24df 
Update warning and ensure the process continues
@jiaruidong2017
Update snowens_analysis.py
63a8620 
Change to warning and ensure the process continues
CoryMartin-NOAA
CoryMartin-NOAA previously approved these changes Oct 7, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@CoryMartin-NOAA CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't test but looks good, thanks @jiaruidong2017

@CoryMartin-NOAA
Copy link
Copy Markdown
Contributor

CoryMartin-NOAA commented Oct 10, 2025

@jiaruidong2017 yes please ensure the obs list is up to date so that we can assimilate all necessary obs

@CoryMartin-NOAA
Copy link
Copy Markdown
Contributor

CoryMartin-NOAA commented Oct 10, 2025

@jiaruidong2017 I think this is the file that is needed to be updated per @DavidNew-NOAA https://github.com/NOAA-EMC/GDASApp/blob/develop/parm/snow/snow_obs_list.yaml.j2

@jiaruidong2017 jiaruidong2017 mentioned this pull request Oct 10, 2025
Merged
7 tasks
DavidNew-NOAA pushed a commit to NOAA-EMC/GDASApp that referenced this pull request Oct 10, 2025
@jiaruidong2017
Update the observation list for snow DA (#1935)
f25f340 
# Description

Update the observation list to include the new snow observations

# Companion PRs

Contribute to NOAA-EMC/global-workflow#4132

# Issues

<!-- Enter any issues referenced or resolved by this PR here. Use
keywords "Resolves" or "Refs".
Resolves #1234
Refs #4321
Refs NOAA-EMC/repo#5678
-->

# Automated CI tests to run in Global Workflow
<!-- Which Global Workflow CI tests are required to adequately test this
PR? -->
- [ ] atm_jjob <!-- JEDI atm single cycle DA !-->
- [ ] C96C48_ufs_hybatmDA <!-- JEDI atm cycled DA !-->
- [ ] C96C48_hybatmsnowDA <!-- JEDI snow cycled DA !-->
- [ ] C96_gcafs_cycled <!-- JEDI aerosol cycled DA !-->
- [ ] C48mx500_3DVarAOWCDA <!-- JEDI low-res marine 3DVar cycled DA !-->
- [ ] C48mx500_hybAOWCDA <!-- JEDI marine hybrid envar cycled DA !-->
- [ ] C96C48_hybatmDA <!-- GSI atm cycled DA !-->
DavidNew-NOAA
DavidNew-NOAA previously approved these changes Oct 10, 2025
View reviewed changes
@aerorahul aerorahul mentioned this pull request Oct 12, 2025
Closed
2 tasks
@CatherineThomas-NOAA CatherineThomas-NOAA moved this to In Progress in GFS v17 Oct 14, 2025
DavidHuber-NOAA
DavidHuber-NOAA reviewed Oct 14, 2025
View reviewed changes
Comment thread ush/python/pygfs/task/snow_analysis.py Outdated
DavidHuber-NOAA
DavidHuber-NOAA previously approved these changes Oct 14, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@DavidHuber-NOAA
Copy link
Copy Markdown
Contributor

DavidHuber-NOAA commented Oct 14, 2025

Launching CI on C6.

@emcbot emcbot added CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 CI-Gaeac6-Running CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully and removed CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 CI-Gaeac6-Running labels Oct 14, 2025
@DavidHuber-NOAA
Copy link
Copy Markdown
Contributor

DavidHuber-NOAA commented Oct 15, 2025

All tests passed. Merging.

@DavidHuber-NOAA DavidHuber-NOAA merged commit 43a29b3 into NOAA-EMC:develop Oct 15, 2025
5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in GFS v17 Oct 15, 2025
@CatherineThomas-NOAA CatherineThomas-NOAA mentioned this pull request Nov 4, 2025
Closed
weihuang-jedi added a commit to NOAA-EPIC/global-workflow-cloud that referenced this pull request Nov 5, 2025
@weihuang-jedi
Merge branch 'feature/use_container_spack-stack-1.9.2' of github.com:…
bd195a9 
…NOAA-EPIC/global-workflow-cloud into feature/use_container_spack-stack-1.9.2

* 'feature/use_container_spack-stack-1.9.2' of github.com:NOAA-EPIC/global-workflow-cloud:
  reverse few changes
  re-sync with EMC repo
  Add master grib files to GFS HPSS archive for AIGFS (NOAA-EMC#4203)
  Update Snow filenames to comply with EE2 (NOAA-EMC#4195)
  Rename files for JEDI atm EE2 (NOAA-EMC#4193)
  Generate `pres_b` files for `RUN=gdas` and update `APCP` to `598`. (NOAA-EMC#4196)
  Update checks for MOM6 restarts when performing a re-run on failure (NOAA-EMC#4179)
  Decrease HPSS storage for GFS retros and address hpss bugs (NOAA-EMC#4184)
  Add noaacloud to ufsda case in dev/ush/load_modules (NOAA-EMC#4198)
  Remove replay from global workflow  (NOAA-EMC#4182)
  Add IODA stats text file to COM (NOAA-EMC#4176)
  Update UFS_UTILS submodule (NOAA-EMC#4178)
  Atm COM File Rename to Standardized Form  (NOAA-EMC#4117)
  Replace cp with cpfs/cpreq for atomic copies to COM directories (NOAA-EMC#4130)
  Create a UPP module for the global workflow (NOAA-EMC#4174)
  Refactor marine DA tasks (NOAA-EMC#4160)
  Delay ocean post-processing trigger to next-next forecast (NOAA-EMC#4167)
  Make options hashes
  Remove multiple option from static data template
  Fix static_data yaml (descriptions and labels)
  Fix static_data yaml (remove colon)
  Add Ursa to and remove C5 from list of HPCs in the bug report template (NOAA-EMC#4164)
  Rename marine (ocean/ice) files following EE2 conventions (NOAA-EMC#4162)
  Add attributes to Gaussian grid sfcanl file (NOAA-EMC#4149)
  Remove the snow analysis from archive (NOAA-EMC#4157)
  Update verif-global to fix pcp failures on special cases (NOAA-EMC#4154)
  Add CRTM fix directory paths to global-workflow (NOAA-EMC#4143)
  Update UFS Model  (NOAA-EMC#4138)
  Add functionality to assimilate the new snow observations (NOAA-EMC#4132)
DavidNew-NOAA pushed a commit to NOAA-EMC/GDASApp that referenced this pull request Jan 16, 2026
@jiaruidong2017
Support assimilation of new snow observations (#1930)
11b1244 
# Description

This PR adds a BUFR-query–based obsBuilder script that concatenates the
new SNOMAD observation into SNOCVR, and processes the result into
IODA-format files for snow DA in the global-workflow.

# Companion PRs

This PR depends NOAA-EMC/jcb-gdas#188
This PR contributes
NOAA-EMC/global-workflow#4132

# Issues

Resolves #1888 

# Automated CI tests to run in Global Workflow
<!-- Which Global Workflow CI tests are required to adequately test this
PR? -->
- [ ] atm_jjob <!-- JEDI atm single cycle DA !-->
- [ ] C96C48_ufs_hybatmDA <!-- JEDI atm cycled DA !-->
- [ ] C96C48_hybatmsnowDA <!-- JEDI snow cycled DA !-->
- [ ] C96_gcafs_cycled <!-- JEDI aerosol cycled DA !-->
- [ ] C48mx500_3DVarAOWCDA <!-- JEDI low-res marine 3DVar cycled DA !-->
- [ ] C48mx500_hybAOWCDA <!-- JEDI marine hybrid envar cycled DA !-->
- [ ] C96C48_hybatmDA <!-- GSI atm cycled DA !-->
DavidNew-NOAA pushed a commit to NOAA-EMC/GDASApp that referenced this pull request Jan 16, 2026
@jiaruidong2017
Update the observation list for snow DA (#1935)
640b547 
# Description

Update the observation list to include the new snow observations

# Companion PRs

Contribute to NOAA-EMC/global-workflow#4132

# Issues

<!-- Enter any issues referenced or resolved by this PR here. Use
keywords "Resolves" or "Refs".
Resolves #1234
Refs #4321
Refs NOAA-EMC/repo#5678
-->

# Automated CI tests to run in Global Workflow
<!-- Which Global Workflow CI tests are required to adequately test this
PR? -->
- [ ] atm_jjob <!-- JEDI atm single cycle DA !-->
- [ ] C96C48_ufs_hybatmDA <!-- JEDI atm cycled DA !-->
- [ ] C96C48_hybatmsnowDA <!-- JEDI snow cycled DA !-->
- [ ] C96_gcafs_cycled <!-- JEDI aerosol cycled DA !-->
- [ ] C48mx500_3DVarAOWCDA <!-- JEDI low-res marine 3DVar cycled DA !-->
- [ ] C48mx500_hybAOWCDA <!-- JEDI marine hybrid envar cycled DA !-->
- [ ] C96C48_hybatmDA <!-- GSI atm cycled DA !-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DavidHuber-NOAA DavidHuber-NOAA DavidHuber-NOAA approved these changes

@DavidNew-NOAA DavidNew-NOAA DavidNew-NOAA left review comments

@aerorahul aerorahul Awaiting requested review from aerorahul

@RussTreadon-NOAA RussTreadon-NOAA Awaiting requested review from RussTreadon-NOAA RussTreadon-NOAA is a code owner

@JessicaMeixner-NOAA JessicaMeixner-NOAA Awaiting requested review from JessicaMeixner-NOAA JessicaMeixner-NOAA is a code owner

@CoryMartin-NOAA CoryMartin-NOAA Awaiting requested review from CoryMartin-NOAA

Assignees

No one assigned

Labels

CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully JEDI Feature development to support JEDI-based DA

Projects

GFS v17
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add SNOMAD observations for snow DA

6 participants

@jiaruidong2017 @CoryMartin-NOAA @DavidNew-NOAA @DavidHuber-NOAA @CatherineThomas-NOAA @emcbot