Follow-up modifications for GFS post process by WenMeng-NOAA · Pull Request #2467 · NOAA-EMC/global-workflow

WenMeng-NOAA · 2024-04-08T13:31:51Z

Description

This PR is the follow-up modifications for gfs post process component:

Use new naming convention for inline post outputs
Add missing parm files for gfs pgrb2/pgrb2 generation
Use new naming conventions of master/flux files in gfs pgrb2/pgrb2 generation
Resolves GFS job JGLOBAL_ATMOS_PRODUCTS failed at finding master and flux files #2191

Type of change

Bug fix (fixes something broken)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

Yes, ran the 'JGLOBAL_ATMOS_PRODUCTS' job on WCOSS2

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

aerorahul

The changes here looks good.
The archive script should also be updated to ensure that these updated files are in the list of files that should be archived.
For reference, the archive script used is:
https://github.com/NOAA-EMC/global-workflow/blob/develop/ush/hpssarch_gen.sh

WenMeng-NOAA · 2024-04-08T13:56:58Z

The changes here looks good. The archive script should also be updated to ensure that these updated files are in the list of files that should be archived. For reference, the archive script used is: https://github.com/NOAA-EMC/global-workflow/blob/develop/ush/hpssarch_gen.sh

@aerorahul Good catching. Updated hpssarch_gen.sh.

aerorahul · 2024-04-08T14:19:13Z

The changes here looks good. The archive script should also be updated to ensure that these updated files are in the list of files that should be archived. For reference, the archive script used is: https://github.com/NOAA-EMC/global-workflow/blob/develop/ush/hpssarch_gen.sh

@aerorahul Good catching. Updated hpssarch_gen.sh.

A few more instances where sfluxgrb appears in hpssarch_gen.sh. Lines 398, 399, 411.

WenMeng-NOAA · 2024-04-08T14:39:37Z

@aerorahul

A few more instances where sfluxgrb appears in hpssarch_gen.sh. Lines 398, 399, 411.
Updated.

KateFriedman-NOAA

Changes look good.

aerorahul

Need some more updates for jobs to be triggered off the new names.
Also, analysis UPP needs to create the updated filename.

aerorahul · 2024-04-09T17:08:28Z


 # Files needed by ${INTERP_ATMOS_MASTERSH}
-MASTER_FILE="${COM_ATMOS_MASTER}/${PREFIX}master.grb2${fhr3}"
+MASTER_FILE="${COM_ATMOS_MASTER}/${PREFIX}master${fhr3}.grb2"


@WenMeng-NOAA
The workflow looks for the master file to kick off the products generation job.
Please update this line to:

- 'history_file_tmpl': f'{self.cdump}.t@Hz.master.grb2#dep#'}, + 'history_file_tmpl': f'{self.cdump}.t@Hz.master#dep#.grb2'},

The analysis products job trigger also needs to be updated. Please update this line to:

- data = f'{atm_master_path}/{self.cdump}.t@Hz.master.grb2anl' + data = f'{atm_master_path}/{self.cdump}.t@Hz.masteranl.grb2'

Finally, the offline UPP for analysis needs to write out the updated filename to COM. Please update these lines with:

- - ["{{ DATA }}/GFSPRS.GrbF00", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.master.grb2anl"] - - ["{{ DATA }}/GFSPRS.GrbF00.idx", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.master.grb2ianl"] + - ["{{ DATA }}/GFSPRS.GrbF00", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.masteranl.grb2"] + - ["{{ DATA }}/GFSPRS.GrbF00.idx", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.masteranl.grb2.idx"]

Note more updates will be needed in the hpssarch_gen.sh for these changes in analysis filenames.

I will work on that.

@WenMeng-NOAA
I just looked at the EE2 implementation standards and the correct extension for grib2 files is grib2. We should replace grb2 with grib2.

@aerorahul Just confirm with you that updates will be in all master/flux/goes/pgrb2/pgrb2b.

@aerorahul Updated.

aerorahul · 2024-04-09T17:21:01Z

@WenMeng-NOAA
Similar workflow updates will be needed in gefs_tasks.py

aerorahul

some in hpssarch_gen.sh were left to be updated.

aerorahul · 2024-04-09T19:31:35Z

+        ${NLN} "${COM_ATMOS_MASTER}/${RUN}.t${cyc}z.masterf${FH3}.grb2" "GFSPRS.GrbF${FH2}"
+        ${NLN} "${COM_ATMOS_MASTER}/${RUN}.t${cyc}z.sfluxf${FH3}.grb2" "GFSFLX.GrbF${FH2}"


Suggested change

${NLN} "${COM_ATMOS_MASTER}/${RUN}.t${cyc}z.masterf${FH3}.grb2" "GFSPRS.GrbF${FH2}"

${NLN} "${COM_ATMOS_MASTER}/${RUN}.t${cyc}z.sfluxf${FH3}.grb2" "GFSFLX.GrbF${FH2}"

${NLN} "${COM_ATMOS_MASTER}/${RUN}.t${cyc}z.masterf${FH3}.grib2" "GFSPRS.GrbF${FH2}"

${NLN} "${COM_ATMOS_MASTER}/${RUN}.t${cyc}z.sfluxf${FH3}.grib2" "GFSFLX.GrbF${FH2}"

aerorahul · 2024-04-09T19:32:09Z

+        echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2"
+        echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2.idx"


Suggested change

echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2"

echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2.idx"

echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grib2"

echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grib2.idx"

aerorahul · 2024-04-09T19:32:30Z

+      echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2"
+      echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2.idx"


Suggested change

echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2"

echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2.idx"

echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grib2"

echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grib2.idx"

aerorahul · 2024-04-09T19:32:51Z

    flist="001 002 004 005 007 008"
    for fhr in ${flist}; do
-      file="${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxgrbf${fhr}.grib2"
+      file="${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2"


Suggested change

file="${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2"

file="${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grib2"

@aerorahul Updated.

WalterKolczynski-NOAA · 2024-04-10T18:13:27Z

    copy:
-      - ["{{ DATA }}/GFSPRS.GrbF00", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.master.grb2anl"]
-      - ["{{ DATA }}/GFSPRS.GrbF00.idx", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.master.grb2ianl"]
+      - ["{{ DATA }}/GFSPRS.GrbF00", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.masteranl.grib2"]


There should be a period separating the master/sflux from the anl/${fhr3} in all these just like the lat-lon files have.

@WalterKolczynski-NOAA Updated.

WenMeng-NOAA · 2024-04-11T00:57:32Z

@aerorahul @WalterKolczynski-NOAA Should all pgrb2/pgrb2b be changed as "gfs.t{CC}z.pgrb2a.?p??.f{FFF}.grib2" and gfs.t{CC}z.pgrb2b.?p??.f{FFF}.grib2"?

WalterKolczynski-NOAA · 2024-04-11T02:44:36Z

@aerorahul @WalterKolczynski-NOAA Should all pgrb2/pgrb2b be changed as "gfs.t{CC}z.pgrb2a.?p??.f{FFF}.grib2" and gfs.t{CC}z.pgrb2b.?p??.f{FFF}.grib2"?

The target filenames for these are RUN.atmos.tHHz.pres_a.LpLL.fFFF.grib2 and RUN.atmos.tHHz.pres_b.LpLL.fFFF.grib2.idx. Changing these will require a lot of downstream changes in the secondary products (e.g. gempak).

WalterKolczynski-NOAA · 2024-04-11T02:45:27Z

We have a spreadsheet documenting the planned filename changes (still in-progress and subject to change): https://docs.google.com/spreadsheets/d/1SzXQnfoUMF0W82Q906A7h9ZLA-vq59kjDH6EGRJuXDY

WenMeng-NOAA · 2024-04-11T03:06:50Z

@aerorahul @WalterKolczynski-NOAA Should all pgrb2/pgrb2b be changed as "gfs.t{CC}z.pgrb2a.?p??.f{FFF}.grib2" and gfs.t{CC}z.pgrb2b.?p??.f{FFF}.grib2"?

The target filenames for these are RUN.atmos.tHHz.pres_a.LpLL.fFFF.grib2 and RUN.atmos.tHHz.pres_b.LpLL.fFFF.grib2.idx. Changing these will require a lot of downstream changes in the secondary products (e.g. gempak).

@WalterKolczynski-NOAA I can only update file naming convention only in pgrb generation, which is included in my GFS post-processing standalone test. Please let me know your preferred approach.

WalterKolczynski-NOAA · 2024-04-11T17:59:31Z

I've scheduled a brief meeting for 15:00 today to discuss. If that doesn't work, we can pick another time.

aerorahul · 2024-04-15T18:10:51Z

awaiting finalizing filenames confirmation.

…EMC#2475) * UFSWM - Updates cpld_control_sfs to use the most up-to-date configuration specs * UFSWM - Update units attributes in netCDF output to be machine-readable * FV3 - Update units attributes in netCDF output to be machine-readable * atmos_cubed_sphere - Update units attributes in netCDF output to be machine-readable

WenMeng-NOAA added 2 commits April 8, 2024 13:13

Follow-up fixes for gfs post process.

4e4c108

Remove commented lines

49760d2

WenMeng-NOAA requested review from WalterKolczynski-NOAA and aerorahul April 8, 2024 13:32

aerorahul reviewed Apr 8, 2024

View reviewed changes

Update flux file naming convention

9d23951

More updates in flux file name in hpssarch_gen.sh.

4c7f570

Merge remote-tracking branch 'upstream/develop' into post_gfs

8f0367e

KateFriedman-NOAA approved these changes Apr 8, 2024

View reviewed changes

aerorahul requested changes Apr 9, 2024

View reviewed changes

aerorahul mentioned this pull request Apr 9, 2024

Improve error messaging to resolve bugzilla #2468

Merged

7 tasks

Update with Rahul's comments

afd97d1

aerorahul requested changes Apr 9, 2024

View reviewed changes

further modifications

e92e404

WalterKolczynski-NOAA requested changes Apr 10, 2024

View reviewed changes

Update naming conventions per Walter's comments

9d6b6b8

Merge branch 'develop' into post_gfs

c0c9541

aerorahul closed this Apr 15, 2024

		${NLN} "${COM_ATMOS_MASTER}/${RUN}.t${cyc}z.masterf${FH3}.grb2" "GFSPRS.GrbF${FH2}"
		${NLN} "${COM_ATMOS_MASTER}/${RUN}.t${cyc}z.sfluxf${FH3}.grb2" "GFSFLX.GrbF${FH2}"

		echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2"
		echo "${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2.idx"

	file="${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grb2"
	file="${COM_ATMOS_MASTER/${ROTDIR}\//}/${head}sfluxf${fhr}.grib2"

Conversation

WenMeng-NOAA commented Apr 8, 2024

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

WenMeng-NOAA commented Apr 8, 2024

Uh oh!

aerorahul commented Apr 8, 2024

Uh oh!

WenMeng-NOAA commented Apr 8, 2024

Uh oh!

KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aerorahul commented Apr 9, 2024

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WenMeng-NOAA commented Apr 11, 2024

Uh oh!

WalterKolczynski-NOAA commented Apr 11, 2024

Uh oh!

WalterKolczynski-NOAA commented Apr 11, 2024

Uh oh!

WenMeng-NOAA commented Apr 11, 2024

Uh oh!

WalterKolczynski-NOAA commented Apr 11, 2024

Uh oh!

aerorahul commented Apr 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants