Update CHGRP_RSTPROD check for MAKE_NSSTBUFR#100
Conversation
|
FWIW, NCO/Operations doesn't like files to be overwritten in $COMROOT, so any nsstbufr processing should be done & dusted before copying the file to $COMROOT. |
Thanks for the feedback @ilianagenkova ! @TaylorRoper-NOAA please make sure the permissions are set on the nsstbufr file before it gets moved to COM, thanks! |
|
@KateFriedman-NOAA Please look at the changes I made. I restructured to apply permissions to nsstbufr before copying to COM using |
… up extra spacing
KateFriedman-NOAA
left a comment
KateFriedman-NOAA
left a comment
There was a problem hiding this comment.
Changes look good, thanks @TaylorRoper-NOAA !
ilianagenkova
left a comment
ilianagenkova
left a comment
There was a problem hiding this comment.
Looks good, Taylor.
|
Thanks for your review and feedback @ilianagenkova ! I'll now merge this PR into the |
This PR enables running prep jobs on AWS. This required supporting `obsproc` and `prepobs` on the cloud (NOAA-EMC/obsproc#99 and NOAA-EMC/prepobs#43) and deciding on placement of `BASE_GIT` directory which houses `obsproc` and `prepobs`. `BASE_GIT` is proposed to be `/contrib/git` on AWS (and other NOAA CSPs), and `obsproc` and `prepobs` have been cloned and installed in `/contrib/git` on the EPIC's AWS `/contrib` filesystem for future CI testing. The obs data used was made available on AWS via #3494, and the `DMPDIR` is defaulted to `/bucket/dump` or user-defined via `DATA_MOUNT` for AWS. Since restricted data is not yet supported on NOAA CSPs, AWS defaults are updated to include `CHGRP_RSTPROD='NO'` and `MAKE_NSSTBUFR='YES'`. Updating `CHGRP_RSTPROD` is necessary since NSST buffer files are assumed in `obsproc` to consist of restricted data and are overwritten as empty when `chgrp rstprod` fails. Allowing making the NSST buffer file on the fly with `CHGRP_RSTPROD='NO'` was done via NOAA-EMC/obsproc#100. Resolves #3497
3 participants
Reorganized
MAKE_NSSTBUFRblock to checkCHGRP_RSTPROD. Since the restricted data is not used on the cloud at this time,CHGRP_RSTPRODis defaulted to NO for cloud environments. Previously, the NSST buffer file was only created ifchgrp rstprodcommand is successful causing the NSST buffer file to be empty when made on the cloud using unrestricted data.Tested this change within the prep steps of the global-workflow C96_atm3DVar test on Hera (with
CHGRP_RSTPROD='YES') and NOAA cloud AWS (withCHGRP_RSTPROD='YES'to check null file is created in place of NSST buffer file whenchgrp rstprodcommand fails and withCHGRP_RSTPROD='NO'). Associated log files and resulting NSST buffer files for both environments can be found on Hera in/scratch1/BMC/qosap/Taylor.Roper/shared(labeled*.heraand*.cloudrespectively).Resolves #101