Merge feature/hafs_couplehycom_cdeps back to support/HAFS

[BinLiu-NOAA]

Merge the feature/hafs_couplehycom_cdeps branch back to support/HAFS branch.

[lliu1220]

@uturuncoglu @BinLiu-NOAA
Testing hafs_couplehycom_cdeps branch for HAT10,
recompiled fv3 and linked into run directory, ran into the following issue:
This has been tested to work back in April, now seeking help/suggestions, thanks appreciate.
There is no 'issue' where I could post my error msg and so I 'm posting here mostly relevant topic:
Thanks again... Ling

20210725 204543.549 ERROR PET632 ESMF_Comp.F90:759 ESMF_CompConstruct Value unrecognized or out of range - Conflict between petlist and global pet count
20210725 204543.555 ERROR PET632 ESMF_GridComp.F90:568 ESMF_GridCompCreate Value unrecognized or out of range - Internal subroutine call returned Error
20210725 204543.707 ERROR PET632 module_NEMS_GRID_COMP.F90:311 Value unrecognized or out of range - Breaking out of subroutine
20210725 204543.707 ERROR PET632 MAIN_NEMS.F90:476 Value unrecognized or out of range - Aborting NEMS

[uturuncoglu]

@lliu1220 The error "Conflict between petlist and global pet count” generally indicates the configuration issue. The number of PEs that is defined in model_configure, nems.configure, input.nml (decomposition paramaters) and job submission script need to be consistent.

[lliu1220]

@lliu1220 The error "Conflict between petlist and global pet count” generally indicates the configuration issue. The number of PEs that is defined in model_configure, nems.configure, input.nml (decomposition paramaters) and job submission script need to be consistent.

@uturuncoglu

Thanks for pinpointing the problem. It looks like nems.configure was taylored for 1340 PE as for the following:
MED_model: cmeps
MED_petlist_bounds: 1280 1339

The problem is when i try to submit job with :
srun --label -n 1340 ./fv3_32bit.exe

It said "number of PE not available", I suspect it was linked to my account marine-cpu, for instance -n 633 would work.

So my next question is, if I want to use srun --lable -n 633, what should I do with nems.config at the MED_petlist bounds category?

Thank you.
Ling

[uturuncoglu]

@lliu1220 In input.nml, the layout set as 40x30, it means you will assign 1280 core for FV3. In nems.configure, you have

ATM_petlist_bounds: 0 1279
OCN_petlist_bounds: 1280 1339
MED_petlist_bounds: 1280 1339

and ATM PEs seems consistent with the input.nml and you also set 80 PEs for write component. That is fine. The total number of processor used by this job is 1340 and in your job card you need to provide same number of core. So, update 1135 to 1340 in the sun command. That will probably solve the issue.

[uturuncoglu]

The correct number is 1340 and you might try to run the case with another account such as nems rather than marine-cpu.

[uturuncoglu]

I am not sure. Once your configuration works, I think you could easily compare the namelist files to see what is changed. Let me know if you have any issue.

…

On Jul 26, 2021, at 6:56 PM, Ling Liu ***@***.***> wrote: The correct number is 1340 and you might try to run the case with another account such as nems rather than marine-cpu. @uturuncoglu <https://github.com/uturuncoglu> thanks for letting me know. I wonder why it was fine to change the number back in April, but not right now, what was the change? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJMBR7HF53H2S7DKVUOQJDTZX75TANCNFSM5ARBY3CQ>.

[lliu1220]

I am not sure. Once your configuration works, I think you could easily compare the namelist files to see what is changed. Let me know if you have any issue.
…
On Jul 26, 2021, at 6:56 PM, Ling Liu @.***> wrote: The correct number is 1340 and you might try to run the case with another account such as nems rather than marine-cpu. @uturuncoglu https://github.com/uturuncoglu thanks for letting me know. I wonder why it was fine to change the number back in April, but not right now, what was the change? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJMBR7HF53H2S7DKVUOQJDTZX75TANCNFSM5ARBY3CQ.

@uturuncoglu

Thank you. Submitted Orion ticket, waiting for a response for now...

[lliu1220]

@uturuncoglu
It worked ! thanks a lot!
One question, how to bring PETESMFLogFile back?
I see you disabled them.. I don't know how to have them back.. to help debug just in case

[uturuncoglu]

Hi, That is great! You could set following print_esmf: .true. in model_configure. —ufuk

…

On Jul 28, 2021, at 7:30 AM, Ling Liu ***@***.***> wrote: @uturuncoglu <https://github.com/uturuncoglu> It worked ! thanks a lot! One question, how to bring PETESMFLogFile back? I see you disabled them.. I don't know how to have them back.. to help debug just in case — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJMBR44MOYZKYTPVR4U72LT2AA6PANCNFSM5ARBY3CQ>.

[lliu1220]

@uturuncoglu
quick question,
If I want to cycle , is it just in model_config
RUN_CONTINUE: .true.

the default was .false.
Where do I specify END_time?

I just want to confirm on this, currently only runs one day , and the queue time was quite long for me

[uturuncoglu]

I think if you want to restart the model, you need to modify also input.nml (i.e. nstf option and others). I think there is a documentation to restart the FV3 but I don't know the exact location. BTW, we never tired to restart the configurations that includes data ocean before. This requires additional work such as creating new ESMF run sequence for warm start and setting up namelist files. So, it might require additional work.

[lliu1220]

@uturuncoglu

Our project requires us to run through CDEPS coupling for a Hurricane period of time, such as Isaias, 2020, July 20-August 6 of 2020.

I'm not saying to restart the experiment, Just to confirm, you meant that CDEPS does not cycle experiment for longer than a day with data ocean?

If this is the case, can you give us instruction on how to enable cycling experiments?

Thank you

[uturuncoglu]

@lliu1220 In the docn.streams, there is an option called taxmode01. This is currently set to limit. So, if your run extend to the the data available in the DOCN, then the model stops and put warning to you. there is no cycle in this case. If you want to cycle same data through the run, then you could set this option to cycle.

[lliu1220]

@uturuncoglu
I might be confusing you. For instance, from the current out log file, I have the following:

0: StartTime= 2019 8 10 0 0 0
0: CurrTime = 2019 8 10 0 0 0
0: StopTime = 2019 8 11 0 0 0

My actual question is how to lift StopTime = 2019 8 11 0 0 0
to say StopTime = 2019 9 11 0 0 0

So the model doesn t just run one day? Say I have plenty of data under DOCN_INPUT until 2022...

[uturuncoglu]

I think if you change the forecast time nhours_fcst from 24 to 48 in the model_configure that will be fine. This is mainly used by the NEMS driver to setup ESMF clock. I think that RUN_CONTINUE needs to be same and it controls something else.

[lliu1220]

@uturuncoglu
I saw under your current INPUT , the gfs_bndy.tile7.024.nc only have 24 hourrs.
I tried to copy over from my April experiments gfs_bndy.tile7.027.nc-gfs_bndy.tile7.120.nc
But realized they were on different resolution:
lon = 2888 ;
lat = 1920 ;
whereas 24 hours' resolution is:
lon = 728 ;
lat = 600 ;
I wonder if you could provide longer than 24 hours of INPUT files with the same resolution?

Another question is, in this cpl history file
ufs.hafs.cpl.hi.2019-08-11-00000.nc
I seem only have 1 time of record, how do I bring all time record, say if I ran:
08/10-08/15,
am I going to archive all the following?:
ufs.hafs.cpl.hi.2019-08-11-00000.nc
ufs.hafs.cpl.hi.2019-08-12-00000.nc
ufs.hafs.cpl.hi.2019-08-13-00000.nc
ufs.hafs.cpl.hi.2019-08-14-00000.nc
ufs.hafs.cpl.hi.2019-08-15-00000.nc

Thank you.

[lliu1220]

@uturuncoglu
I followed the instruction and was able to run through HAFS, the goal is to generate the following boundary grid files for longer than 24 hours.
netcdf gfs_bndy.tile7.024 {
dimensions:
lon = 728 ;
lat = 600 ;
lonp = 729 ;
latm = 599 ;
halo = 14 ;
halop = 15 ;
lev = 92 ;
levp = 93 ;

But I was only able to generate them in high resolution lon = 2888 ;
lat = 1920 ;

Do you have user guide on how to change the resolution in the HAFS workflow? Thank you.