add option to ignore qm=9 for T2m and q2m in prepbufr files. by ClaraDraper-NOAA · Pull Request #866 · NOAA-EMC/GSI

ClaraDraper-NOAA · 2025-04-16T18:46:13Z

Description

Updates read_prepbufr with an explicit flag (ignore_2mQM) to over-ride quality mark of 9 (no obs error) in prepbufr files for q2m and T2m. Previously, these quality marks were ignored for all cases when the 2m obs are assimilated. With the coming update to the obs table in GSI-fix PR 33 to include the obs errors in the prepbufr, we now only need to ignore the quality marks when using archived prepbufr files (i.e., for the reanalysis).

Also fixes bug that was preventing q2m from being assimilated, by not properly resettings the quality mark.

Resolves #861

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
[ x] New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

Compared:
-ignore_2mQM = .true. and original obs table (with no obs errors for q2m and T2m)
-ignore_2mQM = .False. and obs table from GSI-fix PR 33 .

The two are not the same, but differences can be explained. Specifically, the latter excludes a handful more obs that the former, due to these obs have quality mark = 9 due to missing pressure obs errors.

Checklist

[x ] My code follows the style guidelines of this project
[ x] I have performed a self-review of my own code
[ x] I have commented my code, particularly in hard-to-understand areas
New and existing tests pass with my changes
Any dependent changes have been merged and published

Is dependent on GSI-fix PR 33

ClaraDraper-NOAA · 2025-04-16T18:48:17Z

Creating as a draft for now, until GSI-fix PR 33 is merged.

@jswhit2

RussTreadon-NOAA · 2025-04-17T11:22:54Z

We should run GSI ctests with the fix files changes from GSI-fix PR #33.

RussTreadon-NOAA

Consider echoing the values of hofx_2m_sfcfile and ignore_2mQM to stdout when gsi.x runs.

ClaraDraper-NOAA · 2025-04-17T21:27:00Z

We should run GSI ctests with the fix files changes from GSI-fix PR #33.

Running them now. I'll confirm once they've passed.

ClaraDraper-NOAA · 2025-04-17T23:01:06Z

We should run GSI ctests with the fix files changes from GSI-fix PR #33.

Running them now. I'll confirm once they've passed.

The global tests have passed on hera, using this PR and GSI-fix PR 33.

RussTreadon-NOAA · 2025-04-18T10:25:31Z

@ClaraDraper-NOAA , GSI-fix PR #33 has been merged into GSI-fix develop. Done at 8ea29cd.

The fix submodule hash in this PR can be updated.

RussTreadon-NOAA

Approve pending updating of fix submodule to 8ea29cd

RussTreadon-NOAA · 2025-04-18T10:28:26Z

@ClaraDraper-NOAA , we need two peer reviews and approvals of this PR before we can merge into develop. Who would you like me to assign as peer reviewers?

ClaraDraper-NOAA · 2025-04-18T16:17:52Z

Approve pending updating of fix submodule to 8ea29cd

I've update the hash for the fix submodule.

ClaraDraper-NOAA · 2025-04-18T16:19:22Z

@ClaraDraper-NOAA , we need two peer reviews and approvals of this PR before we can merge into develop. Who would you like me to assign as peer reviewers?

@jiaruidong2017 , @ilianagenkova , and/or @tsga would all be good options.

jiaruidong2017 · 2025-04-18T18:34:52Z

                     tqm(k)=2
                     if (kx==187) obserr(3,k)=2.0_r_double
                     if (kx==181) obserr(3,k)=2.0_r_double
                     if (kx==183) obserr(3,k)=2.0_r_double


Suggested change

if (kx==183) obserr(3,k)=2.0_r_double

Do you think it is necessary to discard the obs type 183 and keep consistent with new obs error table? If yes, please make change for the qob case below.

Good point. I removed 183 from the global_2m_land option higher up.

jiaruidong2017

Looks good, and great efforts.

ilianagenkova · 2025-04-18T21:29:41Z

The error table changes look good.

The global_convinfo.txt file changes achieve the following:

allow GOES-19 winds to be assimilated (types 245,246,247) , monitored (240) , skipped altogether (251) which is the intend here, I believe

allow saildrone (type 180,280 , subtype 02) to be monitoried

adds gps types 803 and 804
If that was the goal, then the changes are good.

RussTreadon-NOAA · 2025-04-21T12:17:59Z

Thank you @ilianagenkova for noting the impact of the updated global_convinfo.txt. Tagging @ADCollard and @XuanliLi-NOAA for their input regarding what your comment / question.

Andrew and Xuanli: Do we want the noted changes to global_convinfo.txt to be the defaults for GSI develop?

XuanliLi-NOAA · 2025-04-21T13:04:45Z

@RussTreadon-NOAA: Yes, please add gps 803 and 804 in global_convinfo as the default settings for GSI.

ClaraDraper-NOAA · 2025-04-21T16:30:55Z

@RussTreadon-NOAA Would you mind if I updated the GSI-fix hash once more in this PR to include the new reject list as well (merged in PR ? It's a minor udpdate, adding a single SYNOP station to the reject list for T and q.

RussTreadon-NOAA · 2025-04-21T16:34:54Z

@ClaraDraper-NOAA : updating the GSI-fix hash is fine. Once the updated hash is in ClaraDraper-NOAA:feature/ignore_2mQC we need to rerun ctests and and get a new round of peer reviews & approvals.

ClaraDraper-NOAA · 2025-04-21T16:38:40Z

Thanks @RussTreadon-NOAA . I'll request approvals once the tests have passed.

ClaraDraper-NOAA · 2025-04-21T21:55:45Z

The global ctests have passed on hera with the new hash for GSI-fix, which includes an initial station to be rejected for T and q obs.

/scratch2/BMC/gsienkf/Clara.Draper/gerrit-hera/PRtesting/gsi/GSI-EMC/build $ ctest -R global
Test project /scratch2/BMC/gsienkf/Clara.Draper/gerrit-hera/PRtesting/gsi/GSI-EMC/build
Start 1: global_4denvar
1/2 Test #1: global_4denvar ................... Passed 2205.11 sec
Start 6: global_enkf
2/2 Test #6: global_enkf ...................... Passed 1428.31 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) = 3633.48 sec

ilianagenkova · 2025-04-21T23:39:20Z

@ClaraDraper-NOAA and @RussTreadon-NOAA, I missed the moment when adding fix/rejectlist_global.txt was discussed. I am falling behind on all the GSI changes, yet based on past experience I'd say that introducing fix/rejectlist_global.txt is probably a good idea. Is it just the Land DA that will read/use that file?

Another way to "turn off" a single station (not an entire kx data type!) would have been through the sdmedit list ( /lfs/h1/ops/prod/dcom/sdmedit ) - including a station in that list will assign the observations a QM value such that all models will filter out the data.

@ADCollard, tagging you for awareness.

RussTreadon-NOAA · 2025-04-22T12:19:07Z

WCOSS2 ctests

Install ClaraDraper-NOAA:feature/ignore_2mQC at 05b07b8 and develop at bed9513 on Cactus. Run ctests with following results

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/pr866/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  731.18 sec
2/6 Test #6: global_enkf ......................   Passed  922.15 sec
3/6 Test #2: rtma .............................   Passed  1035.32 sec
4/6 Test #5: hafs_3denvar_hybens ..............   Passed  1341.54 sec
5/6 Test #4: hafs_4denvar_glbens ..............***Failed  1401.03 sec
6/6 Test #1: global_4denvar ...................   Passed  1984.07 sec

83% tests passed, 1 tests failed out of 6

Total Test time (real) = 1984.09 sec

The hafs_4denvar_glbens failure is due to

The runtime for hafs_4denvar_glbens_hiproc_updat is 269.198716 seconds.  This has exceeded maximum allowable threshold time of 264.816746 seconds, resulting in Failure of timethresh2 the regression test.

This is not a fatal failure.

Note: The updated global_convinfo.txt in this PR turns on assimilation of gpsro data from satellite ids 803 and 804. Thus, merger of this PR into GSI develop alters global gsi.x analysis results. @xuanli confirms that we want this change in GSI develop. This note is added as a reminder to all so no one is caught off guard by altered global analysis results after this PR is merged into develop.

RussTreadon-NOAA · 2025-04-22T12:22:37Z

@ClaraDraper-NOAA: This PR can be scheduled for merger into GSI develop pending one or more peer reviews and approvals. Please reach out to peer reviewers.

RussTreadon-NOAA

Approve pending sufficient peer reviews and approvals.

RussTreadon-NOAA · 2025-04-22T14:49:35Z

@ClaraDraper-NOAA and @jiaruidong2017 : Have you opened a g-w issue to update the staged GSI fix files. g-w does NOT get GSI fix files from $HOMEgfs/sorc/gsi_enk.fd/fix. g-w gets GSI fix files from $HOMEgfs/fix/gsi

jiaruidong2017 · 2025-04-22T15:01:46Z

Thanks @RussTreadon-NOAA for your information. We have opened an issue in g-w NOAA-EMC/global-workflow#3546

…C#866)

add option to ignore qm=9 for T2m and q2m in prepbufr files.

eb59d57

RussTreadon-NOAA assigned ClaraDraper-NOAA Apr 17, 2025

RussTreadon-NOAA reviewed Apr 17, 2025

View reviewed changes

Comment thread src/gsi/gsimod.F90

Output ignore_2mQM and hofx_2m_sfcfile to stdout.

14cf117

RussTreadon-NOAA mentioned this pull request Apr 18, 2025

Update prepobs_errtable.global for GSI based soil DA. NOAA-EMC/GSI-fix#33

Merged

RussTreadon-NOAA previously approved these changes Apr 18, 2025

View reviewed changes

RussTreadon-NOAA marked this pull request as ready for review April 18, 2025 10:28

Update fix hash.

19211dd

ClaraDraper-NOAA dismissed RussTreadon-NOAA’s stale review via 19211dd April 18, 2025 16:12

RussTreadon-NOAA requested review from ilianagenkova, jiaruidong2017 and tsga April 18, 2025 16:23

tsga approved these changes Apr 18, 2025

View reviewed changes

jiaruidong2017 reviewed Apr 18, 2025

View reviewed changes

Removed ovs type 183 from global_2m_land

6d24286

jiaruidong2017 approved these changes Apr 18, 2025

View reviewed changes

ilianagenkova approved these changes Apr 18, 2025

View reviewed changes

ClaraDraper-NOAA mentioned this pull request Apr 21, 2025

Update error table for surface T,Q observations NOAA-EMC/global-workflow#3546

Closed

4 tasks

CoryMartin-NOAA previously approved these changes Apr 21, 2025

View reviewed changes

jiaruidong2017 mentioned this pull request Apr 21, 2025

Add the setting to use the reject list for station t/q observations in GSI based soil DA NOAA-EMC/global-workflow#3599

Merged

22 tasks

Updated fix hash to include reject list.

05b07b8

ClaraDraper-NOAA dismissed CoryMartin-NOAA’s stale review via 05b07b8 April 21, 2025 16:40

ClaraDraper-NOAA requested review from CoryMartin-NOAA, RussTreadon-NOAA, ilianagenkova, jiaruidong2017 and tsga April 21, 2025 21:55

ilianagenkova approved these changes Apr 21, 2025

View reviewed changes

RussTreadon-NOAA requested a review from ADCollard April 22, 2025 10:16

CoryMartin-NOAA approved these changes Apr 22, 2025

View reviewed changes

RussTreadon-NOAA approved these changes Apr 22, 2025

View reviewed changes

jiaruidong2017 approved these changes Apr 22, 2025

View reviewed changes

RussTreadon-NOAA merged commit 0912493 into NOAA-EMC:develop Apr 22, 2025

xyzemc pushed a commit to xyzemc/GSI_develop-v16-tms that referenced this pull request Feb 27, 2026

add option to ignore qm=9 for T2m and q2m in prepbufr files. (NOAA-EM…

cce50a0

…C#866)

Conversation

ClaraDraper-NOAA commented Apr 16, 2025 • edited by RussTreadon-NOAA Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ClaraDraper-NOAA commented Apr 16, 2025

Uh oh!

RussTreadon-NOAA commented Apr 17, 2025

Uh oh!

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ClaraDraper-NOAA commented Apr 17, 2025

Uh oh!

ClaraDraper-NOAA commented Apr 17, 2025

Uh oh!

RussTreadon-NOAA commented Apr 18, 2025

Uh oh!

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

RussTreadon-NOAA commented Apr 18, 2025

Uh oh!

ClaraDraper-NOAA commented Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ClaraDraper-NOAA commented Apr 18, 2025

Uh oh!

jiaruidong2017 Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

ClaraDraper-NOAA Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

jiaruidong2017 left a comment

Choose a reason for hiding this comment

Uh oh!

ilianagenkova Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

RussTreadon-NOAA commented Apr 21, 2025

Uh oh!

XuanliLi-NOAA commented Apr 21, 2025

Uh oh!

ClaraDraper-NOAA commented Apr 21, 2025

Uh oh!

RussTreadon-NOAA commented Apr 21, 2025

Uh oh!

ClaraDraper-NOAA commented Apr 21, 2025

Uh oh!

ClaraDraper-NOAA commented Apr 21, 2025

Uh oh!

ilianagenkova commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RussTreadon-NOAA commented Apr 22, 2025

Uh oh!

RussTreadon-NOAA commented Apr 22, 2025

Uh oh!

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

RussTreadon-NOAA commented Apr 22, 2025

Uh oh!

jiaruidong2017 commented Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ClaraDraper-NOAA commented Apr 16, 2025 •

edited by RussTreadon-NOAA

Loading

ClaraDraper-NOAA commented Apr 18, 2025 •

edited

Loading

ilianagenkova commented Apr 21, 2025 •

edited

Loading

jiaruidong2017 commented Apr 22, 2025 •

edited

Loading