Skip to content

Update global atmos upp job to use COMIN/COMOUT #2867

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aerorahul merged 22 commits into from
Sep 16, 2024
Merged

Update global atmos upp job to use COMIN/COMOUT #2867

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
91969c3
changing COM_ into COMIN
mingshichen-noaa Aug 12, 2024
f93185c
changing COM_ into COMIN_
mingshichen-noaa Aug 12, 2024
e4446c6
change COMIN_ATMOS_MASTER into COMOUT_ATMOS_MASTER
mingshichen-noaa Aug 15, 2024
d842dd8
change COMIN_ATMOS_MASTER into COMOUT_ATMOS_MASTER
mingshichen-noaa Aug 15, 2024
f296840
changing COM/ into COMIN/ or COMOUT/
mingshichen-noaa Aug 15, 2024
ba61fad
update COM_ into COMIN_ or COMOUT_
mingshichen-noaa Aug 15, 2024
0be05d7
Merge branch 'NOAA-EMC:develop' into feature/atmos_upp_com_in_out
mingshichen-noaa Aug 19, 2024
435bdd7
Merge branch 'NOAA-EMC:develop' into feature/atmos_upp_com_in_out
mingshichen-noaa Aug 27, 2024
668cfbd
change COM to COMIN or COMOUT with explicitly specifying the template
mingshichen-noaa Aug 28, 2024
4b08226
gdas.cd and ufs_model.fd have been updated
mingshichen-noaa Aug 28, 2024
06ed674
remove COMIN, only keep COMOUT
mingshichen-noaa Aug 28, 2024
70d6e15
modified indent space
mingshichen-noaa Aug 29, 2024
09871b2
update gdas.cd wxflow
mingshichen-noaa Aug 29, 2024
5c74d5c
recover the add gdas.cd ufs_model.fd
mingshichen-noaa Sep 10, 2024
0aeef60
resolved file conflict of upp.yaml
mingshichen-noaa Sep 10, 2024
99536e1
recover wxflow
mingshichen-noaa Sep 10, 2024
f2c099f
removed -m 755
mingshichen-noaa Sep 11, 2024
c459290
resolve conflicts of upp.yaml
mingshichen-noaa Sep 11, 2024
de934ee
adding _TMPL on suffix of COM_ATMOS_HISTORY and COM_ATMOS_MASTER
mingshichen-noaa Sep 13, 2024
90f67ab
Update parm/post/upp.yaml
aerorahul Sep 14, 2024
dddcaa8
Merge branch 'develop' into feature/atmos_upp_com_in_out
aerorahul Sep 14, 2024
970998c
Merge branch 'develop' into feature/atmos_upp_com_in_out
aerorahul Sep 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions jobs/JGLOBAL_ATMOS_UPP
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ source "${HOMEgfs}/ush/jjob_header.sh" -e "upp" -c "base upp"
##############################################

# Construct COM variables from templates
YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_ATMOS_ANALYSIS COM_ATMOS_HISTORY COM_ATMOS_MASTER
if [[ ! -d ${COM_ATMOS_MASTER} ]]; then mkdir -m 775 -p "${COM_ATMOS_MASTER}"; fi
YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COMIN_ATMOS_ANALYSIS COMIN_ATMOS_HISTORY COMOUT_ATMOS_MASTER
Comment thread
mingshichen-noaa marked this conversation as resolved.
Outdated
if [[ ! -d ${COMOUT_ATMOS_MASTER} ]]; then mkdir -m 775 -p "${COMOUT_ATMOS_MASTER}"; fi

@aerorahul aerorahul Sep 3, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if [[ ! -d ${COMOUT_ATMOS_MASTER} ]]; then mkdir -m 775 -p "${COMOUT_ATMOS_MASTER}"; fi
if [[ ! -d ${COMOUT_ATMOS_MASTER} ]]; then mkdir -p "${COMOUT_ATMOS_MASTER}"; fi

Please remove -m 775

@mingshichen-noaa mingshichen-noaa Sep 3, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have removed "-m 775" on line 19.

@aerorahul aerorahul Sep 11, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mingshichen-noaa
I still see -m 775 in this PR. Did you push your change?

@mingshichen-noaa mingshichen-noaa Sep 11, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aerorahul
After I removed "-m 775" and then git add file, git commit -m ""-m 775" ,
finally git push origin HEAD:feature/atmos_upp_com_in_out.
Just now I tried to do the above mentioned final step again.
But after I checked jobs/JGLOBAL_ATMOS_UPP in the PR, "-m 775" is still over there. So far I have not identified the cause.

@mingshichen-noaa mingshichen-noaa Sep 12, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aerorahul
I identified the reason why you see still "-m 775 in this PR". This is because I created two branches: origin/atmos_upp_com_in_out and origin/feature/atmos_upp_com_in_out. I did wrongly git push the modified files into origin/atmos_upp_com_in_out. in order to avoid the mistakes again, I have deleted the origin/atmos_upp_com_in_out.
Please review the PR again. Thanks.



###############################################################
Expand Down
28 changes: 14 additions & 14 deletions parm/post/upp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@ analysis:
data_in:
copy:
- ["{{ PARMgfs }}/post/postxconfig-NT-GFS-ANL.txt", "{{ DATA }}/postxconfig-NT.txt"]
- ["{{ COM_ATMOS_ANALYSIS }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.atmanl.nc", "{{ DATA }}/{{ atmos_filename }}"]
- ["{{ COM_ATMOS_ANALYSIS }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.sfcanl.nc", "{{ DATA }}/{{ flux_filename }}"]
- ["{{ COMIN_ATMOS_ANALYSIS }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.atmanl.nc", "{{ DATA }}/{{ atmos_filename }}"]
- ["{{ COMIN_ATMOS_ANALYSIS }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.sfcanl.nc", "{{ DATA }}/{{ flux_filename }}"]
data_out:
copy:
- ["{{ DATA }}/GFSPRS.GrbF00", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.master.grb2anl"]
- ["{{ DATA }}/GFSPRS.GrbF00.idx", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.master.grb2ianl"]
- ["{{ DATA }}/GFSPRS.GrbF00", "{{ COMOUT_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.master.grb2anl"]
- ["{{ DATA }}/GFSPRS.GrbF00.idx", "{{ COMOUT_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.master.grb2ianl"]

forecast:
config:
Expand All @@ -36,14 +36,14 @@ forecast:
{% else %}
- ["{{ PARMgfs }}/post/postxconfig-NT-GFS-TWO.txt", "{{ DATA }}/postxconfig-NT.txt"]
{% endif %}
- ["{{ COM_ATMOS_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.atmf{{ '%03d' % forecast_hour }}.nc", "{{ DATA }}/{{ atmos_filename }}"]
- ["{{ COM_ATMOS_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.sfcf{{ '%03d' % forecast_hour }}.nc", "{{ DATA }}/{{ flux_filename }}"]
- ["{{ COMIN_ATMOS_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.atmf{{ '%03d' % forecast_hour }}.nc", "{{ DATA }}/{{ atmos_filename }}"]
- ["{{ COMIN_ATMOS_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.sfcf{{ '%03d' % forecast_hour }}.nc", "{{ DATA }}/{{ flux_filename }}"]
data_out:
copy:
- ["{{ DATA }}/GFSPRS.GrbF{{ '%02d' % forecast_hour }}", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.master.grb2f{{ '%03d' % forecast_hour }}"]
- ["{{ DATA }}/GFSFLX.GrbF{{ '%02d' % forecast_hour }}", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.sfluxgrbf{{ '%03d' % forecast_hour }}.grib2"]
- ["{{ DATA }}/GFSPRS.GrbF{{ '%02d' % forecast_hour }}.idx", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.master.grb2if{{ '%03d' % forecast_hour }}"]
- ["{{ DATA }}/GFSFLX.GrbF{{ '%02d' % forecast_hour }}.idx", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.sfluxgrbf{{ '%03d' % forecast_hour }}.grib2.idx"]
- ["{{ DATA }}/GFSPRS.GrbF{{ '%02d' % forecast_hour }}", "{{ COMOUT_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.master.grb2f{{ '%03d' % forecast_hour }}"]
- ["{{ DATA }}/GFSFLX.GrbF{{ '%02d' % forecast_hour }}", "{{ COMOUT_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.sfluxgrbf{{ '%03d' % forecast_hour }}.grib2"]
- ["{{ DATA }}/GFSPRS.GrbF{{ '%02d' % forecast_hour }}.idx", "{{ COMOUT_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.master.grb2if{{ '%03d' % forecast_hour }}"]
- ["{{ DATA }}/GFSFLX.GrbF{{ '%02d' % forecast_hour }}.idx", "{{ COMOUT_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.sfluxgrbf{{ '%03d' % forecast_hour }}.grib2.idx"]

goes:
config:
Expand Down Expand Up @@ -82,9 +82,9 @@ goes:
- ["{{ 'CRTM_FIX' | getenv }}/AerosolCoeff.bin", "{{ DATA }}/"]
- ["{{ 'CRTM_FIX' | getenv }}/CloudCoeff.bin", "{{ DATA }}/"]
- ["{{ PARMgfs }}/post/postxconfig-NT-GFS-GOES.txt", "{{ DATA }}/postxconfig-NT.txt"]
- ["{{ COM_ATMOS_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.atmf{{ '%03d' % forecast_hour }}.nc", "{{ DATA }}/{{ atmos_filename }}"]
- ["{{ COM_ATMOS_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.sfcf{{ '%03d' % forecast_hour }}.nc", "{{ DATA }}/{{ flux_filename }}"]
- ["{{ COMIN_ATMOS_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.atmf{{ '%03d' % forecast_hour }}.nc", "{{ DATA }}/{{ atmos_filename }}"]
- ["{{ COMIN_ATMOS_HISTORY }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.sfcf{{ '%03d' % forecast_hour }}.nc", "{{ DATA }}/{{ flux_filename }}"]
data_out:
copy:
- ["{{ DATA }}/GFSPRS.GrbF{{ '%02d' % forecast_hour }}", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.special.grb2f{{ '%03d' % forecast_hour }}"]
- ["{{ DATA }}/GFSPRS.GrbF{{ '%02d' % forecast_hour }}.idx", "{{ COM_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.special.grb2if{{ '%03d' % forecast_hour }}"]
- ["{{ DATA }}/GFSPRS.GrbF{{ '%02d' % forecast_hour }}", "{{ COMOUT_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.special.grb2f{{ '%03d' % forecast_hour }}"]

@aerorahul aerorahul Sep 14, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GFSPRS change needs to be reverted. That's the failure right now.

- ["{{ DATA }}/GFSPRS.GrbF{{ '%02d' % forecast_hour }}.idx", "{{ COMOUT_ATMOS_MASTER }}/{{ RUN }}.t{{ current_cycle | strftime('%H') }}z.special.grb2if{{ '%03d' % forecast_hour }}"]
Comment thread
aerorahul marked this conversation as resolved.
Outdated
2 changes: 1 addition & 1 deletion scripts/exglobal_atmos_upp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ def main():

# Pull out all the configuration keys needed to run the rest of UPP steps
keys = ['HOMEgfs', 'DATA', 'current_cycle', 'RUN', 'NET',
'COM_ATMOS_ANALYSIS', 'COM_ATMOS_HISTORY', 'COM_ATMOS_MASTER',
'COMIN_ATMOS_ANALYSIS', 'COMIN_ATMOS_HISTORY', 'COMOUT_ATMOS_MASTER',
'upp_run',
'APRUN_UPP',
'forecast_hour', 'valid_datetime',
Expand Down
6 changes: 3 additions & 3 deletions ush/python/pygfs/task/upp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@ def _call_executable(exec_cmd: Executable) -> None:
@logit(logger)
def finalize(upp_run: Dict, upp_yaml: Dict) -> None:
"""Perform closing actions of the task.
Copy data back from the DATA/ directory to COM/
Copy data back from the DATA/ directory to COMIN/ or COMOUT/
Comment thread
mingshichen-noaa marked this conversation as resolved.
Outdated

Parameters
----------
Expand All @@ -259,6 +259,6 @@ def finalize(upp_run: Dict, upp_yaml: Dict) -> None:
Fully resolved upp.yaml dictionary
"""

# Copy "upp_run" specific generated data to COM/ directory
logger.info(f"Copy '{upp_run}' processed data to COM/ directory")
# Copy "upp_run" specific generated data to COMIN/ or COMOUT/ directory
logger.info(f"Copy '{upp_run}' processed data to COMIN/ or COMOUT/ directory")
Comment thread
mingshichen-noaa marked this conversation as resolved.
Outdated
FileHandler(upp_yaml[upp_run].data_out).sync()