Skip to content

Fixing window_begin and end in yaml for redating ADT#1565

Merged
guillaumevernieres merged 3 commits into
developfrom
bugfix/prepoceanobs_window
Mar 24, 2025
Merged

Fixing window_begin and end in yaml for redating ADT#1565
guillaumevernieres merged 3 commits into
developfrom
bugfix/prepoceanobs_window

Conversation

@apchoiCMD
Copy link
Copy Markdown
Collaborator

@apchoiCMD apchoiCMD commented Mar 20, 2025

Description

This PR fixes the generation of marine DA yamls by setting appropriate window_begin and window_end values to enable the use of the redate function in the ADT and INSITU concatenator.

Companion PRs

Issues

Automated CI tests to run in Global Workflow

  • atm_jjob
  • C96C48_ufs_hybatmDA
  • C96C48_hybatmaerosnowDA
  • C48mx500_3DVarAOWCDA
  • C48mx500_hybAOWCDA
  • C96C48_hybatmDA

Copy link
Copy Markdown
Contributor

@guillaumevernieres guillaumevernieres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @apchoiCMD

@guillaumevernieres guillaumevernieres added the hera-GW-RT Queue for automated testing with global-workflow on Hera label Mar 20, 2025
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Mar 20, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented Mar 20, 2025

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Thu Mar 20 21:24:12 UTC 2025 on hfe02
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Mar 20 22:08:54 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C48mx500_hybAOWCDA|C96C48_hybatmDA
Tests:                                  *Failed*
Tests: Failed at Thu Mar 20 23:33:15 UTC 2025
Tests: 98% tests passed, 1 tests failed out of 43
	2074 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
Tests: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1565/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Mar 20, 2025
@apchoiCMD
Copy link
Copy Markdown
Collaborator Author

apchoiCMD commented Mar 21, 2025

From the log at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1565/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci/C48mx500_3DVarAOWCDA/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032500/gdas_marineanlfinal.log

Any thought? There is no marinevariational folder in /scratch1/NCEPDEV/stmp2/role.jedipara/RUNDIRS/C48mx500_3DVarAOWCDA/gdas.2021032500/gdasmarineanalysis.2021032500

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1565/global-workflow/scripts/exglobal_marine_analysis_finalize.py", line 24, in <module>
    MarineAnl.finalize()
  File "/scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1565/global-workflow/ush/python/wxflow/logger.py", line 266, in wrapper
    retval = func(*args, **kwargs)
  File "/scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1565/global-workflow/ush/python/pygfs/task/marine_analysis.py", line 419, in finalize
    FileHandler({'copy': post_file_list}).sync()
  File "/scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1565/global-workflow/ush/python/wxflow/file_utils.py", line 55, in sync
    sync_factory[action](files)
  File "/scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1565/global-workflow/ush/python/wxflow/file_utils.py", line 59, in copy_req
    FileHandler._copy_files(filelist, required=True)
  File "/scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1565/global-workflow/ush/python/wxflow/file_utils.py", line 95, in _copy_files
    raise FileNotFoundError(f"Source file '{src}' does not exist")
FileNotFoundError: Source file '/scratch1/NCEPDEV/stmp2/role.jedipara/RUNDIRS/C48mx500_3DVarAOWCDA/gdas.2021032500/gdasmarineanalysis.2021032500/marinevariational/ocn.inc.nc' does not exist
+ JGLOBAL_MARINE_ANALYSIS_FINALIZE[1]: postamble JGLOBAL_MARINE_ANALYSIS_FINALIZE 1742513565 1
+ preamble.sh[70]: set +x
End JGLOBAL_MARINE_ANALYSIS_FINALIZE at 23:32:59 with error code 1 (time elapsed: 00:00:14)
+ marineanlfinal.sh[1]: postamble marineanlfinal.sh 1742513535 1
+ preamble.sh[70]: set +x
End marineanlfinal.sh at 23:32:59 with error code 1 (time elapsed: 00:00:44)

@apchoiCMD
Copy link
Copy Markdown
Collaborator Author

It looks like successfully passed the ctest on Hera on both low and high res test-

image

@guillaumevernieres
Copy link
Copy Markdown
Contributor

Probably a glitch in the matrix @apchoiCMD !

@guillaumevernieres guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera and removed hera-GW-RT-Failed Automated testing with global-workflow failed on Hera labels Mar 21, 2025
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Mar 21, 2025
Copy link
Copy Markdown
Collaborator

@AndrewEichmann-NOAA AndrewEichmann-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little perplexed how this worked in the first place

@guillaumevernieres
Copy link
Copy Markdown
Contributor

I'm a little perplexed how this worked in the first place

@AndrewEichmann-NOAA : It looks like the time stamp was always set to the start of the window, so it never did what we intended. Not a huge deal for ADT obs.

@guillaumevernieres
Copy link
Copy Markdown
Contributor

I'm not sure why it's failing @apchoiCMD , the latest ci failed in the init phase of the marine DA ...

@apchoiCMD
Copy link
Copy Markdown
Collaborator Author

Let us see what comes out at this time! Thanks @guillaumevernieres to look into it-

@guillaumevernieres
Copy link
Copy Markdown
Contributor

Let us see what comes out at this time! Thanks @guillaumevernieres to look into it-

It isn't reported yet but it failed @apchoiCMD . A problem for Monday.

@emcbot
Copy link
Copy Markdown

emcbot commented Mar 22, 2025

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Fri Mar 21 12:45:40 UTC 2025 on hfe02
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Mar 21 13:29:08 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C48mx500_hybAOWCDA|C96C48_hybatmDA
Tests:                                  *Failed*
Tests: Failed at Sat Mar 22 02:00:35 UTC 2025
Tests: 88% tests passed, 5 tests failed out of 43
	2071 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000 (Failed)
	2073 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	2070 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000 (Timeout)
	2072 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 (Timeout)
	2074 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 (Timeout)
Tests: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1565/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Mar 22, 2025
@guillaumevernieres guillaumevernieres removed the hera-GW-RT-Failed Automated testing with global-workflow failed on Hera label Mar 24, 2025
@guillaumevernieres guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera hercules-GW-RT Queue for automated testing with global-workflow on Hercules labels Mar 24, 2025
@emcbot emcbot added hercules-GW-RT-Running Automated testing with global-workflow running on Hercules hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hercules-GW-RT Queue for automated testing with global-workflow on Hercules hera-GW-RT Queue for automated testing with global-workflow on Hera labels Mar 24, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented Mar 24, 2025

Automated GW-GDASApp Testing Results:
Machine: hercules

Start: Mon Mar 24 09:57:44 CDT 2025 on hercules-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                  *FAILED*
Build: Failed at Mon Mar 24 10:03:13 CDT 2025
Build: see output at /work2/noaa/da/role-da/CI/hercules/GDASApp/workflow/PR/1565/global-workflow/sorc/log.build

@emcbot emcbot added hercules-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hercules-GW-RT-Running Automated testing with global-workflow running on Hercules labels Mar 24, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented Mar 24, 2025

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Mon Mar 24 15:13:23 UTC 2025 on hfe02
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Mar 24 16:00:34 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C48mx500_3DVarAOWCDA|C96C48_hybatmDA
Tests:                                 *SUCCESS*
Tests: Completed at Mon Mar 24 16:20:54 UTC 2025
Tests: 100% tests passed, 0 tests failed out of 46

@emcbot emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Mar 24, 2025
@guillaumevernieres guillaumevernieres merged commit 0888683 into develop Mar 24, 2025
@guillaumevernieres guillaumevernieres deleted the bugfix/prepoceanobs_window branch March 24, 2025 16:31
@emcbot
Copy link
Copy Markdown

emcbot commented Mar 24, 2025

Automated GW-GDASApp Testing Results:
Machine: hercules

Start: Mon Mar 24 10:17:25 CDT 2025 on hercules-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Mar 24 12:44:48 CDT 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C48mx500_3DVarAOWCDA|C96C48_hybatmDA
Tests:                                 *SUCCESS*
Tests: Completed at Mon Mar 24 13:08:25 CDT 2025
Tests: 100% tests passed, 0 tests failed out of 46

@emcbot emcbot added the hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules label Mar 24, 2025
jspsl pushed a commit that referenced this pull request Mar 26, 2025
# Description

This PR fixes the generation of marine DA `yamls` by setting appropriate
`window_begin` and `window_end` values to enable the use of the redate
function in the ADT and INSITU concatenator.

# Companion PRs

<!-- Enter links to any companion PRs here. -->

# Issues
- Resolves #1535 

![adt_redate](https://github.com/user-attachments/assets/b7a662cc-eab3-4b2a-8ce9-4421fbe951e2)
<!-- Enter any issues referenced or resolved by this PR here. Use
keywords "Resolves" or "Refs".

-->

# Automated CI tests to run in Global Workflow
<!-- Which Global Workflow CI tests are required to adequately test this
PR? -->
- [ ] atm_jjob <!-- JEDI atm single cycle DA !-->
- [ ] C96C48_ufs_hybatmDA <!-- JEDI atm cycled DA !-->
- [ ] C96C48_hybatmaerosnowDA  <!-- JEDI aero/snow cycled DA !-->
- [ ] C48mx500_3DVarAOWCDA <!-- JEDI low-res marine 3DVar cycled DA !-->
- [x] C48mx500_hybAOWCDA <!-- JEDI marine hybrid envar cycled DA !-->
- [ ] C96C48_hybatmDA <!-- GSI atm cycled DA !-->
danholdaway added a commit that referenced this pull request Mar 26, 2025
* develop:
  added original date (#1570)
  Update OCEAN_BASIN_FILE_ORION path for marine ctest (#1569)
  Fixing window_begin and end in yaml for redating ADT (#1565)
  Save LETKF mean analysis and mean increment for verification (#1566)
  Update JEDI hashes (20250324) (#1567)
  Removes obsolete json files from marine bufr2ioda (#1564)
  Add hybrid 4denvar as a phase3 validation configuration (#1552)
  Bugfix/ocn ecen (#1550)
  Bugfix SSS obs for LETKF (#1560)
  Bugfix ocean ensemble recentering (#1558)
  apply ci path patch only when running g-w ctests (#1554)
  Update JEDI hashes (20250317)  (#1531)
  add option to create_experiment.py in create_exp.sh  (#1546)
  Update jcb hashes and support initial IODA stats capability (#1547)
  Fix GW-CI CTest dependencies (#1534)
  Converts prepoceanobs to use JCB-style obs list and config files (#1527)
  JEDI-based ensemble recentering and analysis calculation (#1488)
  tools for phase3 validation (#1526)
  Update JEDI hashes (20250306) (#1520)
  New jcb-gdas # and insitu obs prep bug fix  (#1523)
  Marine in situ obs update (SSS, moorings, drifters) (#1511)
  Save marine background and analysis spread (#1517)
  Minor build and install updates and ctest cleanup (#1512)
  correct logic for upper loop limit in chem_diagb (#1516)
  Incorporate multi-cycle concatination and separation of variables in post-BUFR observations (#1504)
  Update JEDI hashes (20250225) (#1475)
  Add a depth dependent e-fold scale (#1494)
  Update jcb and da-utils hashes to support land DA. (#1509)
  Add options to inflate the ensemble while recentering (#1501)
  Add components to use jcb yamls for marine letkf job (#1486)
  Add GNSSRO bufr2ioda and json files (#1448)
  Removal of the marine verification tools (#1499)
  update logic to identify MSU machine (#1498)
  IODA Converter with In-Situ Observations Concatenation and ObsError Inflation (#1472)
  Add sea ice recentering (#1478)
  marine vrfy bugfix (#1483)
  add LD_LIBRARY_PATH patch for wcoss2 build (#1484)
  Bugfix for ensemble recenter + add a test (#1477)
  Add code changes for the implementation of the  low-resolution B (#1441)
  update spack-stack path in Hera modulefiles (#1471)
  fixed glider output filename (#1469)
  add gdas_waveinit to C48mx500_3DVarAOWCDA ctest suite (#1470)
  update jcb-gdas hash 20250128 (#1466)
  Time series of omb stats and html doc (#1463)
  Update marine vrfy HTML codes (#1464)
  Automate the verify tools to submit from start pdy to end pdy (#1458)
  Update an offline marine vrfy tool (#1447)
  Update JEDI hashes (20250127) (#1460)
DavidNew-NOAA pushed a commit that referenced this pull request Jan 16, 2026
# Description

This PR fixes the generation of marine DA `yamls` by setting appropriate
`window_begin` and `window_end` values to enable the use of the redate
function in the ADT and INSITU concatenator.

# Companion PRs

<!-- Enter links to any companion PRs here. -->

# Issues
- Resolves #1535 

![adt_redate](https://github.com/user-attachments/assets/b7a662cc-eab3-4b2a-8ce9-4421fbe951e2)
<!-- Enter any issues referenced or resolved by this PR here. Use
keywords "Resolves" or "Refs".

-->

# Automated CI tests to run in Global Workflow
<!-- Which Global Workflow CI tests are required to adequately test this
PR? -->
- [ ] atm_jjob <!-- JEDI atm single cycle DA !-->
- [ ] C96C48_ufs_hybatmDA <!-- JEDI atm cycled DA !-->
- [ ] C96C48_hybatmaerosnowDA  <!-- JEDI aero/snow cycled DA !-->
- [ ] C48mx500_3DVarAOWCDA <!-- JEDI low-res marine 3DVar cycled DA !-->
- [x] C48mx500_hybAOWCDA <!-- JEDI marine hybrid envar cycled DA !-->
- [ ] C96C48_hybatmDA <!-- GSI atm cycled DA !-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hera-GW-RT-Passed Automated testing with global-workflow successful on Hera hercules-GW-RT-Failed Automated testing with global-workflow failed on Hera hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Possibly wrong time when using overlapping window for ADT

5 participants