Feature/script update

[jkuang-exploro]

Modularized forecast script reproducing the develop branch.

All vital components of the modularized forecast scripts are in place for a
successful ATM-only experiment replicating the develop branch. Please
comment on how could this PR be tested, as well as the test plan of the
"final" merge toward the develop.

Issue tracker: #365 and #366

Comment: I think maybe a specific hashtag should be used instead of the
vague "develop branch"

Jian Kuang's test contents for the modularized script:
ROTDIR: /work/noaa/marine/Jian.Kuang/comroot/modular-192
EXPDIR: /work/noaa/marine/Jian.Kuang/expdir/modular-192
RUNDIR: /work/noaa/stmp/jkuang/RUNDIRS/modular-192/2020090118/gfs/fcst.143393

07/31 test contents, this run is NOT completed successfully, due to the following error which also appear in feature/coupled-sprint branch as of that date:

FATAL from PE 0: mpp_domains_define.inc: At least one pe in pelist is not used by any tile in the mosaic

Repo: /work/noaa/marine/Jian.Kuang/coupled-sprint
EXPDIR: /work/noaa/marine/Jian.Kuang/expdir/modular_0731
ROTDIR: /work/noaa/marine/Jian.Kuang/comroot/modular_0731
RUNDIR: /work/noaa/stmp/jkuang/RUNDIRS/modular_0731/2020090118/gfs/fcst.355316

[jkuang-exploro]

@aerorahul This is the 1st version that I put together. All vital components are in place for a successful ATM-only experiment replicating the develop branch. I made it a draft because it wasn't tested yet.

[WalterKolczynski-NOAA]

Please start following global-workflow commit message guidelines (see https://github.com/NOAA-EMC/global-workflow/wiki/Development-of-Global-Workflow#commit-message-standards)

[jkuang-exploro]

@WalterKolczynski-NOAA Appreciate bringing this up. I updated the description.

[JessicaMeixner-NOAA]

@JianKuang-UMD this PR is already out of date with feature/coupled-crow after a PR from @WalterKolczynski-NOAA was merged yesterday. There is one (at most 2) more PRs from @WalterKolczynski-NOAA which has changes needed for p7 but actually will likely help by adding features in the forecast script that are in the develop branch but not in feature/coupled-crow yet.

[jkuang-exploro]

@JessicaMeixner-NOAA So should I update this PR now or after the other upcoming PR goes into coupled-crow?

[jkuang-exploro]

@WalterKolczynski-NOAA Should I update this PR now or after the other upcoming PR goes into coupled-crow?

[WalterKolczynski-NOAA]

I'd wait until after coupled-crow settles. The next one has the bigger forecast script changes.

[aerorahul]

@WalterKolczynski-NOAA
We agreed no more big commits to the coupled crow branch.

[WalterKolczynski-NOAA]

This is still more work for the next prototype.

[aerorahul]

How are they ready to run prototypes in 2 weeks if more work is required?
That does not make sense or there is a miscommunication.
We will discuss tomorrow at the review.

[WalterKolczynski-NOAA]

We've said the whole time we're trying to have things ready by the end of this week, though now it may slip to Monday.

[KateFriedman-NOAA]

I ran the develop branch on Orion and compared my output to Jian's run with the modular forecast script. Same ICs used for both runs. There is a reproducibility issue with this hash of ufs-weather-model but the forecast output up through f15 matched. See comparison below.

Orion-login-4[27] /work/noaa/stmp/kfriedma/comrot/fcsdev/gfs.20200901/18/atmos$ compare_ncfile.py /work/noaa/marine/Jian.Kuang/comroot/modular-192/gfs.20200901/18/atmos/gfs.t18z.atmf015.nc gfs.t18z.atmf015.nc
grid_xt min/max 1=0.0,359.53125 min/max 2=0.0,359.53125 max abs diff=0.0000000000
lon min/max 1=0.0,359.53125 min/max 2=0.0,359.53125 max abs diff=0.0000000000
grid_yt min/max 1=-89.64164807259336,89.64164807259336 min/max 2=-89.64164807259336,89.64164807259336 max abs diff=0.0000000000
lat min/max 1=-89.64164807259336,89.64164807259336 min/max 2=-89.64164807259336,89.64164807259336 max abs diff=0.0000000000
pfull min/max 1=0.012781459,998.7807 min/max 2=0.012781459,998.7807 max abs diff=0.0000000000
phalf min/max 1=0.00999,1000.0 min/max 2=0.00999,1000.0 max abs diff=0.0000000000
time min/max 1=15.0,15.0 min/max 2=15.0,15.0 max abs diff=0.0000000000
cld_amt min/max 1=0.0,1.0000001 min/max 2=0.0,1.0000001 max abs diff=0.0000000000
clwmr min/max 1=-6.5368904e-20,0.001467026 min/max 2=-6.5368904e-20,0.001467026 max abs diff=0.0000000000
delz min/max 1=-3640.2764,-15.929722 min/max 2=-3640.2764,-15.929722 max abs diff=0.0000000000
dpres min/max 1=0.6059999,1763.2124 min/max 2=0.6059999,1763.2124 max abs diff=0.0000000000
dzdt min/max 1=-2.1237776,1.8675699 min/max 2=-2.1237776,1.8675699 max abs diff=0.0000000000
grle min/max 1=-1.850584e-20,0.0018258409 min/max 2=-1.850584e-20,0.0018258409 max abs diff=0.0000000000
hgtsfc min/max 1=-55.19578,5677.363 min/max 2=-55.19578,5677.363 max abs diff=0.0000000000
icmr min/max 1=-8.362246e-21,0.00076990423 min/max 2=-8.362246e-21,0.00076990423 max abs diff=0.0000000000
o3mr min/max 1=1.9768066e-08,1.7678107e-05 min/max 2=1.9768066e-08,1.7678107e-05 max abs diff=0.0000000000
pressfc min/max 1=51402.188,103400.016 min/max 2=51402.188,103400.016 max abs diff=0.0000000000
rwmr min/max 1=-6.4957633e-21,0.0026575255 min/max 2=-6.4957633e-21,0.0026575255 max abs diff=0.0000000000
snmr min/max 1=-1.2146279e-19,0.004904586 min/max 2=-1.2146279e-19,0.004904586 max abs diff=0.0000000000
spfh min/max 1=6.052613e-08,0.022263074 min/max 2=6.052613e-08,0.022263074 max abs diff=0.0000000000
tmp min/max 1=180.51543,317.53638 min/max 2=180.51543,317.53638 max abs diff=0.0000000000
ugrd min/max 1=-72.20959,159.68596 min/max 2=-72.20959,159.68596 max abs diff=0.0000000000
vgrd min/max 1=-84.675446,117.3277 min/max 2=-84.675446,117.3277 max abs diff=0.0000000000

Orion-login-4[28] /work/noaa/stmp/kfriedma/comrot/fcsdev/gfs.20200901/18/atmos$ compare_ncfile.py /work/noaa/marine/Jian.Kuang/comroot/modular-192/gfs.20200901/18/atmos/gfs.t18z.sfcf015.nc gfs.t18z.sfcf015.nc
grid_xt min/max 1=0.0,359.53125 min/max 2=0.0,359.53125 max abs diff=0.0000000000
lon min/max 1=0.0,359.53125 min/max 2=0.0,359.53125 max abs diff=0.0000000000
grid_yt min/max 1=-89.64164807259336,89.64164807259336 min/max 2=-89.64164807259336,89.64164807259336 max abs diff=0.0000000000
lat min/max 1=-89.64164807259336,89.64164807259336 min/max 2=-89.64164807259336,89.64164807259336 max abs diff=0.0000000000
time min/max 1=15.0,15.0 min/max 2=15.0,15.0 max abs diff=0.0000000000
acond min/max 1=7.6655255e-05,0.37537307 min/max 2=7.6655255e-05,0.37537307 max abs diff=0.0000000000
albdo_ave min/max 1=0.0,87.2877 min/max 2=0.0,87.2877 max abs diff=0.0000000000
alnsf min/max 1=0.01905039,0.6540536 min/max 2=0.01905039,0.6540536 max abs diff=0.0000000000
alnwf min/max 1=0.018865336,0.65163565 min/max 2=0.018865336,0.65163565 max abs diff=0.0000000000
alvsf min/max 1=0.014070779,0.6519257 min/max 2=0.014070779,0.6519257 max abs diff=0.0000000000
alvwf min/max 1=0.013472418,0.6519257 min/max 2=0.013472418,0.6519257 max abs diff=0.0000000000
aod550 min/max 1=0.009115649,1.481971 min/max 2=0.009115649,1.481971 max abs diff=0.0000000000
bc_aod550 min/max 1=0.0008537131,0.104453824 min/max 2=0.0008537131,0.104453824 max abs diff=0.0000000000
c0 min/max 1=0.0,0.077707395 min/max 2=0.0,0.077707395 max abs diff=0.0000000000
cd min/max 1=-580.73987,0.0 min/max 2=-580.73987,0.0 max abs diff=0.0000000000
cduvb_ave min/max 1=0.0,22.131285 min/max 2=0.0,22.131285 max abs diff=0.0000000000
cnvprcp min/max 1=0.0,0.0010432269 min/max 2=0.0,0.0010432269 max abs diff=0.0000000000
cnwat min/max 1=0.0,0.5 min/max 2=0.0,0.5 max abs diff=0.0000000000
cpofp min/max 1=0.0,1.0000001 min/max 2=0.0,1.0000001 max abs diff=0.0000000000
cprat_ave min/max 1=0.0,0.0007404335 min/max 2=0.0,0.0007404335 max abs diff=0.0000000000
cpratb_ave min/max 1=0.0,0.0013801329 min/max 2=0.0,0.0013801329 max abs diff=0.0000000000
crain min/max 1=0.0,1.0 min/max 2=0.0,1.0 max abs diff=0.0000000000
csdlf min/max 1=69.70384,474.11243 min/max 2=69.70384,474.11243 max abs diff=0.0000000000
csdsf min/max 1=0.0,1010.89355 min/max 2=0.0,1010.89355 max abs diff=0.0000000000
csulf min/max 1=115.540825,676.82556 min/max 2=115.540825,676.82556 max abs diff=0.0000000000
csulftoa min/max 1=118.66877,367.13226 min/max 2=118.66877,367.13226 max abs diff=0.0000000000
csusf min/max 1=0.0,770.80884 min/max 2=0.0,770.80884 max abs diff=0.0000000000
csusftoa min/max 1=0.0,713.62964 min/max 2=0.0,713.62964 max abs diff=0.0000000000
cwork_aveclm min/max 1=0.0,1561.4344 min/max 2=0.0,1561.4344 max abs diff=0.0000000000
dconv min/max 1=0.0,6.244938 min/max 2=0.0,6.244938 max abs diff=0.0000000000
dlwrf min/max 1=70.71169,501.53412 min/max 2=70.71169,501.53412 max abs diff=0.0000000000
dlwrf_ave min/max 1=69.91925,491.6324 min/max 2=69.91925,491.6324 max abs diff=0.0000000000
dswrf min/max 1=0.0,1060.8348 min/max 2=0.0,1060.8348 max abs diff=0.0000000000
dswrf_ave min/max 1=0.0,1010.86163 min/max 2=0.0,1010.86163 max abs diff=0.0000000000
dswrf_avetoa min/max 1=0.0,1303.6658 min/max 2=0.0,1303.6658 max abs diff=0.0000000000
dtcool min/max 1=0.0,1.4649872 min/max 2=0.0,1.4649872 max abs diff=0.0000000000
du_aod550 min/max 1=0.00072498986,0.547553 min/max 2=0.00072498986,0.547553 max abs diff=0.0000000000
duvb_ave min/max 1=0.0,21.922491 min/max 2=0.0,21.922491 max abs diff=0.0000000000
evbs_ave min/max 1=0.0,400.80872 min/max 2=0.0,400.80872 max abs diff=0.0000000000
evcw_ave min/max 1=0.0,546.1592 min/max 2=0.0,546.1592 max abs diff=0.0000000000
f10m min/max 1=0.9316632,1.312024 min/max 2=0.9316632,1.312024 max abs diff=0.0000000000
facsf min/max 1=0.0,1.0 min/max 2=0.0,1.0 max abs diff=0.0000000000
facwf min/max 1=-4.2632563e-16,1.0 min/max 2=-4.2632563e-16,1.0 max abs diff=0.0000000000
ffhh min/max 1=1.181344,52.21402 min/max 2=1.181344,52.21402 max abs diff=0.0000000000
ffmm min/max 1=1.3604072,54.366844 min/max 2=1.3604072,54.366844 max abs diff=0.0000000000
fldcp min/max 1=0.0,0.419691 min/max 2=0.0,0.419691 max abs diff=0.0000000000
fricv min/max 1=0.0073574255,2.191457 min/max 2=0.0073574255,2.191457 max abs diff=0.0000000000
gflux min/max 1=-496.3348,159.95694 min/max 2=-496.3348,159.95694 max abs diff=0.0000000000
gflux_ave min/max 1=-445.56363,161.64084 min/max 2=-445.56363,161.64084 max abs diff=0.0000000000
hgt_hyblev1 min/max 1=7.9631543,11.536963 min/max 2=7.9631543,11.536963 max abs diff=0.0000000000
hpbl min/max 1=8.056323,5687.9355 min/max 2=8.056323,5687.9355 max abs diff=0.0000000000
icec min/max 1=0.0,1.0 min/max 2=0.0,1.0 max abs diff=0.0000000000
icetk min/max 1=0.0,5.0065074 min/max 2=0.0,5.0065074 max abs diff=0.0000000000
land min/max 1=0.0,2.0 min/max 2=0.0,2.0 max abs diff=0.0000000000
lhtfl min/max 1=-65.13178,911.34393 min/max 2=-65.13178,911.34393 max abs diff=0.0000000000
lhtfl_ave min/max 1=-64.09785,840.77875 min/max 2=-64.09785,840.77875 max abs diff=0.0000000000
nbdsf_ave min/max 1=0.0,510.1645 min/max 2=0.0,510.1645 max abs diff=0.0000000000
nddsf_ave min/max 1=0.0,369.62607 min/max 2=0.0,369.62607 max abs diff=0.0000000000
oc_aod550 min/max 1=0.0019335842,1.1782336 min/max 2=0.0019335842,1.1782336 max abs diff=0.0000000000
orog min/max 1=-62.2226,5702.785 min/max 2=-62.2226,5702.785 max abs diff=0.0000000000
pevpr min/max 1=-30.324945,2574.2175 min/max 2=-30.324945,2574.2175 max abs diff=0.0000000000
pevpr_ave min/max 1=-26.324642,2615.0718 min/max 2=-26.324642,2615.0718 max abs diff=0.0000000000
prate_ave min/max 1=0.0,0.0037249955 min/max 2=0.0,0.0037249955 max abs diff=0.0000000000
prateb_ave min/max 1=0.0,0.0080602635 min/max 2=0.0,0.0080602635 max abs diff=0.0000000000
pres_avehcb min/max 1=5468.017,50491.145 min/max 2=5468.017,50491.145 max abs diff=0.0000000000
pres_avehct min/max 1=4784.438,47155.79 min/max 2=4784.438,47155.79 max abs diff=0.0000000000
pres_avelcb min/max 1=65145.93,103085.266 min/max 2=65145.93,103085.266 max abs diff=0.0000000000
pres_avelct min/max 1=64158.492,102376.65 min/max 2=64158.492,102376.65 max abs diff=0.0000000000
pres_avemcb min/max 1=40300.88,74896.36 min/max 2=40300.88,74896.36 max abs diff=0.0000000000
pres_avemct min/max 1=39157.105,73459.56 min/max 2=39157.105,73459.56 max abs diff=0.0000000000
prescnvclb min/max 1=38310.215,102175.81 min/max 2=38310.215,102175.81 max abs diff=0.0000000000
prescnvclt min/max 1=7564.2275,97245.31 min/max 2=7564.2275,97245.31 max abs diff=0.0000000000
pressfc min/max 1=51402.08,103399.96 min/max 2=51402.08,103399.96 max abs diff=0.0000000000
pwatclm min/max 1=0.1179889,92.85657 min/max 2=0.1179889,92.85657 max abs diff=0.0000000000
qrain min/max 1=-5.1567416,53.36178 min/max 2=-5.1567416,53.36178 max abs diff=0.0000000000
sbsno_ave min/max 1=0.0,146.56505 min/max 2=0.0,146.56505 max abs diff=0.0000000000
sfcr min/max 1=1e-07,2.653 min/max 2=1e-07,2.653 max abs diff=0.0000000000
sfexc min/max 1=8.3723935e-05,0.3580269 min/max 2=8.3723935e-05,0.3580269 max abs diff=0.0000000000
shdmax min/max 1=0.0,0.99 min/max 2=0.0,0.99 max abs diff=0.0000000000
shdmin min/max 1=0.0,0.8874654 min/max 2=0.0,0.8874654 max abs diff=0.0000000000
shtfl min/max 1=-303.62604,689.8421 min/max 2=-303.62604,689.8421 max abs diff=0.0000000000
shtfl_ave min/max 1=-231.70915,528.4035 min/max 2=-231.70915,528.4035 max abs diff=0.0000000000
sltyp min/max 1=0.0,9.0 min/max 2=0.0,9.0 max abs diff=0.0000000000
snoalb min/max 1=0.0,0.84000003 min/max 2=0.0,0.84000003 max abs diff=0.0000000000
snod min/max 1=0.0,2.7449734 min/max 2=0.0,2.7449734 max abs diff=0.0000000000
snohf min/max 1=-0.33085096,258.92303 min/max 2=-0.33085096,258.92303 max abs diff=0.0000000000
snowc_ave min/max 1=0.0,100.0 min/max 2=0.0,100.0 max abs diff=0.0000000000
soill1 min/max 1=0.02,1.0 min/max 2=0.02,1.0 max abs diff=0.0000000000
soill2 min/max 1=0.02,1.0 min/max 2=0.02,1.0 max abs diff=0.0000000000
soill3 min/max 1=0.02,1.0 min/max 2=0.02,1.0 max abs diff=0.0000000000
soill4 min/max 1=0.02,1.0 min/max 2=0.02,1.0 max abs diff=0.0000000000
soilm min/max 1=40.0,2000.0 min/max 2=40.0,2000.0 max abs diff=0.0000000000
soilt1 min/max 1=213.63599,322.24103 min/max 2=213.63599,322.24103 max abs diff=0.0000000000
soilt2 min/max 1=215.41876,316.31528 min/max 2=215.41876,316.31528 max abs diff=0.0000000000
soilt3 min/max 1=218.00589,315.588 min/max 2=218.00589,315.588 max abs diff=0.0000000000
soilt4 min/max 1=220.98828,313.90536 min/max 2=220.98828,313.90536 max abs diff=0.0000000000
soilw1 min/max 1=0.02,1.0 min/max 2=0.02,1.0 max abs diff=0.0000000000
soilw2 min/max 1=0.02,1.0 min/max 2=0.02,1.0 max abs diff=0.0000000000
soilw3 min/max 1=0.02,1.0 min/max 2=0.02,1.0 max abs diff=0.0000000000
soilw4 min/max 1=0.02,1.0 min/max 2=0.02,1.0 max abs diff=0.0000000000
sotyp min/max 1=0.0,16.0 min/max 2=0.0,16.0 max abs diff=0.0000000000
spfh2m min/max 1=2.1114145e-05,0.022690546 min/max 2=2.1114145e-05,0.022690546 max abs diff=0.0000000000
spfh_hyblev1 min/max 1=2.2855056e-05,0.02227621 min/max 2=2.2855056e-05,0.02227621 max abs diff=0.0000000000
spfhmax_max2m min/max 1=2.2431675e-05,0.022846607 min/max 2=2.2431675e-05,0.022846607 max abs diff=0.0000000000
spfhmin_min2m min/max 1=1.8100864e-05,0.0224638 min/max 2=1.8100864e-05,0.0224638 max abs diff=0.0000000000
ss_aod550 min/max 1=0.00016142667,0.16363867 min/max 2=0.00016142667,0.16363867 max abs diff=0.0000000000
ssrun_acc min/max 1=0.0,72.80799 min/max 2=0.0,72.80799 max abs diff=0.0000000000
su_aod550 min/max 1=0.0035001924,0.626611 min/max 2=0.0035001924,0.626611 max abs diff=0.0000000000
sunsd_acc min/max 1=0.0,10800.001 min/max 2=0.0,10800.001 max abs diff=0.0000000000
tcdc_avebndcl min/max 1=0.0,100.0 min/max 2=0.0,100.0 max abs diff=0.0000000000
tcdc_aveclm min/max 1=0.0,100.0 min/max 2=0.0,100.0 max abs diff=0.0000000000
tcdc_avehcl min/max 1=0.0,100.0 min/max 2=0.0,100.0 max abs diff=0.0000000000
tcdc_avelcl min/max 1=0.0,100.0 min/max 2=0.0,100.0 max abs diff=0.0000000000
tcdc_avemcl min/max 1=0.0,100.0 min/max 2=0.0,100.0 max abs diff=0.0000000000
tcdccnvcl min/max 1=0.0,80.0 min/max 2=0.0,80.0 max abs diff=0.0000000000
tg3 min/max 1=215.48914,304.02972 min/max 2=215.48914,304.02972 max abs diff=0.0000000000
tisfc min/max 1=214.31944,345.20267 min/max 2=214.31944,345.20267 max abs diff=0.0000000000
tmax_max2m min/max 1=218.98985,318.39227 min/max 2=218.98985,318.39227 max abs diff=0.0000000000
tmin_min2m min/max 1=217.22723,313.29218 min/max 2=217.22723,313.29218 max abs diff=0.0000000000
tmp2m min/max 1=218.98985,318.39227 min/max 2=218.98985,318.39227 max abs diff=0.0000000000
tmp_avehct min/max 1=181.49324,261.49017 min/max 2=181.49324,261.49017 max abs diff=0.0000000000
tmp_avelct min/max 1=225.09671,299.7898 min/max 2=225.09671,299.7898 max abs diff=0.0000000000
tmp_avemct min/max 1=220.84262,282.62845 min/max 2=220.84262,282.62845 max abs diff=0.0000000000
tmp_hyblev1 min/max 1=219.86157,317.33945 min/max 2=219.86157,317.33945 max abs diff=0.0000000000
tmpsfc min/max 1=214.31944,345.20267 min/max 2=214.31944,345.20267 max abs diff=0.0000000000
tprcp min/max 1=0.0,0.003266652 min/max 2=0.0,0.003266652 max abs diff=0.0000000000
trans_ave min/max 1=0.0,540.5403 min/max 2=0.0,540.5403 max abs diff=0.0000000000
tref min/max 1=212.89056,336.36594 min/max 2=212.89056,336.36594 max abs diff=0.0000000000
u-gwd_ave min/max 1=-4.880547,10.564625 min/max 2=-4.880547,10.564625 max abs diff=0.0000000000
uflx_ave min/max 1=-3.3018656,3.1014175 min/max 2=-3.3018656,3.1014175 max abs diff=0.0000000000
ugrd10m min/max 1=-36.08228,38.67949 min/max 2=-36.08228,38.67949 max abs diff=0.0000000000
ugrd_hyblev1 min/max 1=-36.50257,39.120895 min/max 2=-36.50257,39.120895 max abs diff=0.0000000000
ulwrf min/max 1=120.312096,758.6273 min/max 2=120.312096,758.6273 max abs diff=0.0000000000
ulwrf_ave min/max 1=116.28082,707.26935 min/max 2=116.28082,707.26935 max abs diff=0.0000000000
ulwrf_avetoa min/max 1=84.10931,367.13226 min/max 2=84.10931,367.13226 max abs diff=0.0000000000
uswrf min/max 1=0.0,645.65546 min/max 2=0.0,645.65546 max abs diff=0.0000000000
uswrf_ave min/max 1=0.0,759.22595 min/max 2=0.0,759.22595 max abs diff=0.0000000000
uswrf_avetoa min/max 1=0.0,909.6714 min/max 2=0.0,909.6714 max abs diff=0.0000000000
v-gwd_ave min/max 1=-6.181528,8.3638525 min/max 2=-6.181528,8.3638525 max abs diff=0.0000000000
vbdsf_ave min/max 1=0.0,430.7707 min/max 2=0.0,430.7707 max abs diff=0.0000000000
vddsf_ave min/max 1=0.0,405.05283 min/max 2=0.0,405.05283 max abs diff=0.0000000000
veg min/max 1=0.0,99.0 min/max 2=0.0,99.0 max abs diff=0.0000000000
vflx_ave min/max 1=-3.4080212,2.8411288 min/max 2=-3.4080212,2.8411288 max abs diff=0.0000000000
vgrd10m min/max 1=-31.631641,38.28689 min/max 2=-31.631641,38.28689 max abs diff=0.0000000000
vgrd_hyblev1 min/max 1=-31.981686,38.738304 min/max 2=-31.981686,38.738304 max abs diff=0.0000000000
vtype min/max 1=0.0,20.0 min/max 2=0.0,20.0 max abs diff=0.0000000000
w0 min/max 1=-314.09402,0.6492158 min/max 2=-314.09402,0.6492158 max abs diff=0.0000000000
watr_acc min/max 1=0.0,73.03194 min/max 2=0.0,73.03194 max abs diff=0.0000000000
wd min/max 1=-0.98544055,336.92212 min/max 2=-0.98544055,336.92212 max abs diff=0.0000000000
weasd min/max 1=0.0,415.8015 min/max 2=0.0,415.8015 max abs diff=0.0000000000
wilt min/max 1=0.0,0.39735234 min/max 2=0.0,0.39735234 max abs diff=0.0000000000
xs min/max 1=-1.6352863,0.09205757 min/max 2=-1.6352863,0.09205757 max abs diff=0.0000000000
xt min/max 1=0.0,4.074173 min/max 2=0.0,4.074173 max abs diff=0.0000000000
xtts min/max 1=-146.40254,30.948713 min/max 2=-146.40254,30.948713 max abs diff=0.0000000000
xu min/max 1=-1.6308749,1.726182 min/max 2=-1.6308749,1.726182 max abs diff=0.0000000000
xv min/max 1=-1.5081172,1.475466 min/max 2=-1.5081172,1.475466 max abs diff=0.0000000000
xz min/max 1=0.32580042,30.0 min/max 2=0.32580042,30.0 max abs diff=0.0000000000
xzts min/max 1=-182.07448,1232531.5 min/max 2=-182.07448,1232531.5 max abs diff=0.0000000000
zc min/max 1=0.0,0.005362158 min/max 2=0.0,0.005362158 max abs diff=0.0000000000
Orion-login-4[29] /work/noaa/stmp/kfriedma/comrot/fcsdev/gfs.20200901/18/atmos$

[aerorahul]

The indents in the scripts are completely whacky.
Please use 2 space indents in the scripts. 4 spaces are acceptable too, but need to be consistent.
exglobal_forecast.sh is removed, why? And if it is removed, how are you running it.

Please don't add new exglobal scripts with _bk. What is the intention of having those?
Version control takes care of this stuff.

[aerorahul]

@KateFriedman-NOAA @JianKuang-UMD
Can you post paths to your EXPDIR, ROTDIR and the RUNDIR for the forecast.

[KateFriedman-NOAA]

For my tests:
EXPDIR: /work/noaa/global/kfriedma/expdir/fcsdev
ROTDIR: /work/noaa/stmp/kfriedma/comrot/fcsdev
Forgot to set KEEPDATA=YES so rundir was wiped at end of job....however, I killed the initial run of the gfsfcst job when I realized I set the FHMAX to be too large and that rundir from that initial attempt was not wiped. I saved the output from the initial attempt and it was matching (/work/noaa/stmp/kfriedma/comrot/fcsdev/gfs.20200901/18/atmos/SAVE_1).
RUNDIR: /work/noaa/stmp/kfriedma/RUNDIRS/fcsdev/2020090118/gfs/fcst.325028

[jkuang-exploro]

I updated the contents of feature/coupled-sprint into this PR, which will let us keep up with the develop branch.

I created a tag named "coupled_sprint_start" pointing to the hash of coupled-crow when I initially started the merge. I will update this PR with coupled-crow branch later today.

[aerorahul]

@JianKuang-UMD
Is there a reason these templates need to have exec permissions?

[aerorahul]

@JianKuang-UMD
There are still trailing whitespaces in the parsing files.

[aerorahul]

Trailing spaces in the forecast_*.sh files too.

[jkuang-exploro]

@aerorahul Unfortunately I wasn't able to see those trailing whitespaces. And for some reason I am unable to simply push a change for file permissions.

[WalterKolczynski-NOAA]

A couple changes that were supposedly made haven't actually been made. Maybe forgot to push some commits? Also one syntax issue.

[WalterKolczynski-NOAA]

This still needs to be changed back to .false.

[aerorahul]

Please see comments.

[WalterKolczynski-NOAA]

@WalterKolczynski-NOAA
Please explain why one-liners need to be replaced with explicit if-then-else blocks?

Does it improve readability?
Is there a flaw in the execution?

By extension, will the use of ternary operators be disallowed?

Improved readability. We can discuss ternary operators as a group, though I don't have a big issue with them since there are at least clearly assignments and they are designed to do that. But I also know I had trouble with them once upon a time, so I wouldn't say no to prohibiting them to keep code more readable. Bash doesn't have them anyway.