Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

History variable FATES_TRANSITION_MATRIX_LULU contains garbage for FatesColdDryDepSatPhen and FatesColdMeganSatPhen #2656

Closed
slevis-lmwg opened this issue Jul 22, 2024 · 9 comments
Closed

History variable FATES_TRANSITION_MATRIX_LULU contains garbage for FatesColdDryDepSatPhen and FatesColdMeganSatPhen #2656

slevis-lmwg opened this issue Jul 22, 2024 · 9 comments
Assignees
@glemieux
Labels
bug something is working incorrectly external issue needs to be addressed elsewhere (submodule); issue here for the sake of project tracking FATES API update Changes to the FATES version that also REQUIRE an API change in CTSM

Comments

@slevis-lmwg
Copy link
Contributor

slevis-lmwg commented Jul 22, 2024 •
edited by glemieux
Loading

Yep, those cases are definitely garbage, but they've been that way for these two tests since that variable was introduced in ctsm5.2.013. Those values should be zero'd out. I'm not sure why the FatesColdSatPhen correctly zeros FATES_TRANSITION_MATRIX_LULU, but FatesColdDryDepSatPhen and FatesColdMeganSatPhen does not.

Originally posted by @glemieux in #2605 (comment)
@slevis-lmwg slevis-lmwg added the bug something is working incorrectly label Jul 22, 2024
@slevis-lmwg
Copy link
Contributor Author

I originally spotted the problem in #2605 as a strange difference from baseline in these two tests:

SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdDryDepSatPhen.GC.0719-175606de_gnu/SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdDryDepSatPhen.GC.0719-175606de_gnu.clm2.h0.2000-01-01-00000.nc.cprnc.out: RMS FATES_TRANSITION_MATRIX_LULU       Infinity            NORMALIZED    Infinity
SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdMeganSatPhen.GC.0719-175606de_gnu/SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdMeganSatPhen.GC.0719-175606de_gnu.clm2.h0.2000-01-01-00000.nc.cprnc.out: RMS FATES_TRANSITION_MATRIX_LULU       Infinity            NORMALIZED    Infinity

@glemieux glemieux changed the title History variable FATES_TRANSITION_MATRIX_LULU contains garbage History variable FATES_TRANSITION_MATRIX_LULU contains garbage for FatesColdDryDepSatPhen and FatesColdMeganSatPhen Jul 22, 2024
@glemieux glemieux changed the title History variable FATES_TRANSITION_MATRIX_LULU contains garbage for FatesColdDryDepSatPhen and FatesColdMeganSatPhen History variable FATES_TRANSITION_MATRIX_LULU contains garbage for FatesColdDryDepSatPhen and FatesColdMeganSatPhen Jul 22, 2024
@glemieux
Copy link
Collaborator

glemieux commented Jul 22, 2024 •
edited
Loading

Checking the difference between FatesColdSatPhen testmod, which appropriately zeros FATES_TRANSITION_MATRIX_LULU, and these failing testmods, the only difference are the expected settings in shell_commands (i.e. CLM_BLDNML_OPTS set to '--drydep' or '--megan').

@glemieux
Copy link
Collaborator

glemieux commented Jul 22, 2024 •
edited
Loading

This might simply be an initialization error on the fates side that is only showing up with gnu compiler tests. I noticed that the instances of FatesColdSatPhen that I've been checking are either intel or nvhpc. Testing.

@glemieux
Copy link
Collaborator

This might simply be an initialization error on the fates side that is only showing up with gnu compiler tests. I noticed that the instances of FatesColdSatPhen that I've been checking are either intel or nvhpc. Testing.

It looks like this is actually the case. Running gnu compiled version of the FatesColdSatPhen test results in the transition matrix reporting junk. I'll open an issue on the fates-side and cross reference here.

@glemieux
Copy link
Collaborator

This probably is being missed in more tests that are based on the Fates testmod as history output is being cleared and this output is not added.

This was referenced Jul 23, 2024
Closed
Closed
@glemieux
Copy link
Collaborator

fates issue: NGEET/fates#1230

@glemieux glemieux mentioned this issue Jul 23, 2024
Merged
4 tasks
glemieux added a commit to rgknox/ctsm that referenced this issue Jul 24, 2024
@glemieux
Merge tag 'ctsm5.2.015' into fates-hist-flush
a000f9c 
Update submodule tags to pass runoff from cism to rof

- Update MOSART, CMEPS, and CISM so CISM runoff goes to ROF rather than CTSM
- Update RTM with fix needed for Paleo LGM work

Contributors:
@mvertens, @jedwards4b, @billsacks, @Katetc, @ekluzek, @slevis-lmwg

 Fixes ESCOMP#2590 Update CMEPS/MOSART/CISM/RTM tags
 Fixes ESCOMP/RTM#50 Likely wrong RTM river flux to MOM6 within cesm2_3_beta17

 Differences in namelist 'mosart_inparm':
  missing variable: 'do_rtmflood'
  missing variable: 'finidat_rtm'
  missing variable: 'frivinp_rtm'
  missing variable: 'rtmhist_fexcl1'
  missing variable: 'rtmhist_fexcl2'
  missing variable: 'rtmhist_fexcl3'
  missing variable: 'rtmhist_fincl1'
  missing variable: 'rtmhist_fincl2'
  missing variable: 'rtmhist_fincl3'
  missing variable: 'rtmhist_mfilt'
  missing variable: 'rtmhist_ndens'
  missing variable: 'rtmhist_nhtfrq'
  found extra variable: 'budget_frq'
  found extra variable: 'fexcl1'
  found extra variable: 'fexcl2'
  found extra variable: 'fexcl3'
  found extra variable: 'fincl1'
  found extra variable: 'fincl2'
  found extra variable: 'fincl3'
  found extra variable: 'finidat'
  found extra variable: 'frivinp'
  found extra variable: 'mfilt'
  found extra variable: 'mosart_euler_calc'
  found extra variable: 'mosart_tracers'
  found extra variable: 'ndens'
  found extra variable: 'nhtfrq'
  found extra variable: 'use_halo_option'

Changes answers
- what code configurations: mosart and rtm
- what platforms/compilers: all
- nature of change: mosart roundoff; rtm larger than roundoff due to bug fix; the latter also affects bgc variables

 We are ignoring strange diffs from baseline in two tests in variable
 FATES_TRANSITION_MATRIX_LULU as explained in issue ESCOMP#2656.
glemieux added a commit to rgknox/ctsm that referenced this issue Jul 24, 2024
@glemieux
update fates satellite phenology test coverage
4728918 
This moves one of the FatesColdSatPhen tests to a gnu compiler to
provide compiler coverage for issues like ESCOMP#2656.  This also adds an
nvhpc compiler test to the fates test suite.
@glemieux
Copy link
Collaborator

Note that this will be resolved via NGEET/fates#1231, which will likely be fates tag sci.1.77.2_api.36.0.0. This tag could come in with #2594.

@glemieux glemieux mentioned this issue Jul 24, 2024
Merged
ekluzek added a commit to ekluzek/CTSM that referenced this issue Jul 24, 2024
@ekluzek
Merge tag 'ctsm5.2.015' into dustcontrolincmeps
28fbfa7 
Update submodule tags to pass runoff from cism to rof

- Update MOSART, CMEPS, and CISM so CISM runoff goes to ROF rather than CTSM
- Update RTM with fix needed for Paleo LGM work

Contributors:
@mvertens, @jedwards4b, @billsacks, @Katetc, @ekluzek, @slevis-lmwg

 Fixes ESCOMP#2590 Update CMEPS/MOSART/CISM/RTM tags
 Fixes ESCOMP/RTM#50 Likely wrong RTM river flux to MOM6 within cesm2_3_beta17

 Differences in namelist 'mosart_inparm':
  missing variable: 'do_rtmflood'
  missing variable: 'finidat_rtm'
  missing variable: 'frivinp_rtm'
  missing variable: 'rtmhist_fexcl1'
  missing variable: 'rtmhist_fexcl2'
  missing variable: 'rtmhist_fexcl3'
  missing variable: 'rtmhist_fincl1'
  missing variable: 'rtmhist_fincl2'
  missing variable: 'rtmhist_fincl3'
  missing variable: 'rtmhist_mfilt'
  missing variable: 'rtmhist_ndens'
  missing variable: 'rtmhist_nhtfrq'
  found extra variable: 'budget_frq'
  found extra variable: 'fexcl1'
  found extra variable: 'fexcl2'
  found extra variable: 'fexcl3'
  found extra variable: 'fincl1'
  found extra variable: 'fincl2'
  found extra variable: 'fincl3'
  found extra variable: 'finidat'
  found extra variable: 'frivinp'
  found extra variable: 'mfilt'
  found extra variable: 'mosart_euler_calc'
  found extra variable: 'mosart_tracers'
  found extra variable: 'ndens'
  found extra variable: 'nhtfrq'
  found extra variable: 'use_halo_option'

Changes answers
- what code configurations: mosart and rtm
- what platforms/compilers: all
- nature of change: mosart roundoff; rtm larger than roundoff due to bug fix; the latter also affects bgc variables

 We are ignoring strange diffs from baseline in two tests in variable
 FATES_TRANSITION_MATRIX_LULU as explained in issue ESCOMP#2656.

 Conflicts:
	bld/unit_testers/build-namelist_test.pl
slevis-lmwg added a commit to slevis-lmwg/ctsm that referenced this issue Jul 24, 2024
@slevis-lmwg
Merge tag 'ctsm5.2.015' into new_rawpftlai
1a0dc60 
Update submodule tags to pass runoff from cism to rof

- Update MOSART, CMEPS, and CISM so CISM runoff goes to ROF rather than CTSM
- Update RTM with fix needed for Paleo LGM work

Contributors:
@mvertens, @jedwards4b, @billsacks, @Katetc, @ekluzek, @slevis-lmwg

 Fixes ESCOMP#2590 Update CMEPS/MOSART/CISM/RTM tags
 Fixes ESCOMP/RTM#50 Likely wrong RTM river flux to MOM6 within cesm2_3_beta17

 Differences in namelist 'mosart_inparm':
  missing variable: 'do_rtmflood'
  missing variable: 'finidat_rtm'
  missing variable: 'frivinp_rtm'
  missing variable: 'rtmhist_fexcl1'
  missing variable: 'rtmhist_fexcl2'
  missing variable: 'rtmhist_fexcl3'
  missing variable: 'rtmhist_fincl1'
  missing variable: 'rtmhist_fincl2'
  missing variable: 'rtmhist_fincl3'
  missing variable: 'rtmhist_mfilt'
  missing variable: 'rtmhist_ndens'
  missing variable: 'rtmhist_nhtfrq'
  found extra variable: 'budget_frq'
  found extra variable: 'fexcl1'
  found extra variable: 'fexcl2'
  found extra variable: 'fexcl3'
  found extra variable: 'fincl1'
  found extra variable: 'fincl2'
  found extra variable: 'fincl3'
  found extra variable: 'finidat'
  found extra variable: 'frivinp'
  found extra variable: 'mfilt'
  found extra variable: 'mosart_euler_calc'
  found extra variable: 'mosart_tracers'
  found extra variable: 'ndens'
  found extra variable: 'nhtfrq'
  found extra variable: 'use_halo_option'

Changes answers
- what code configurations: mosart and rtm
- what platforms/compilers: all
- nature of change: mosart roundoff; rtm larger than roundoff due to bug fix; the latter also affects bgc variables

 We are ignoring strange diffs from baseline in two tests in variable
 FATES_TRANSITION_MATRIX_LULU as explained in issue ESCOMP#2656.
@ekluzek ekluzek mentioned this issue Jul 25, 2024
Merged
10 tasks
@ekluzek ekluzek added the next this should get some attention in the next week or two. Normally each Thursday SE meeting. label Jul 25, 2024
samsrabin added a commit to samsrabin/CTSM that referenced this issue Jul 25, 2024
@samsrabin
Merge tag 'ctsm5.2.015' into scale-mat-reqs-pr3-mergemaster
c5cc03c 
Update submodule tags to pass runoff from cism to rof

- Update MOSART, CMEPS, and CISM so CISM runoff goes to ROF rather than CTSM
- Update RTM with fix needed for Paleo LGM work

Contributors:
@mvertens, @jedwards4b, @billsacks, @Katetc, @ekluzek, @slevis-lmwg

 Fixes ESCOMP#2590 Update CMEPS/MOSART/CISM/RTM tags
 Fixes ESCOMP/RTM#50 Likely wrong RTM river flux to MOM6 within cesm2_3_beta17

 Differences in namelist 'mosart_inparm':
  missing variable: 'do_rtmflood'
  missing variable: 'finidat_rtm'
  missing variable: 'frivinp_rtm'
  missing variable: 'rtmhist_fexcl1'
  missing variable: 'rtmhist_fexcl2'
  missing variable: 'rtmhist_fexcl3'
  missing variable: 'rtmhist_fincl1'
  missing variable: 'rtmhist_fincl2'
  missing variable: 'rtmhist_fincl3'
  missing variable: 'rtmhist_mfilt'
  missing variable: 'rtmhist_ndens'
  missing variable: 'rtmhist_nhtfrq'
  found extra variable: 'budget_frq'
  found extra variable: 'fexcl1'
  found extra variable: 'fexcl2'
  found extra variable: 'fexcl3'
  found extra variable: 'fincl1'
  found extra variable: 'fincl2'
  found extra variable: 'fincl3'
  found extra variable: 'finidat'
  found extra variable: 'frivinp'
  found extra variable: 'mfilt'
  found extra variable: 'mosart_euler_calc'
  found extra variable: 'mosart_tracers'
  found extra variable: 'ndens'
  found extra variable: 'nhtfrq'
  found extra variable: 'use_halo_option'

Changes answers
- what code configurations: mosart and rtm
- what platforms/compilers: all
- nature of change: mosart roundoff; rtm larger than roundoff due to bug fix; the latter also affects bgc variables

 We are ignoring strange diffs from baseline in two tests in variable
 FATES_TRANSITION_MATRIX_LULU as explained in issue ESCOMP#2656.
adrifoster pushed a commit to adrifoster/CTSM that referenced this issue Jul 25, 2024
Merge tag 'ctsm5.2.015' into merge-master-20240725
0d1ea16 
Update submodule tags to pass runoff from cism to rof

- Update MOSART, CMEPS, and CISM so CISM runoff goes to ROF rather than CTSM
- Update RTM with fix needed for Paleo LGM work

Contributors:
@mvertens, @jedwards4b, @billsacks, @Katetc, @ekluzek, @slevis-lmwg

 Fixes ESCOMP#2590 Update CMEPS/MOSART/CISM/RTM tags
 Fixes ESCOMP/RTM#50 Likely wrong RTM river flux to MOM6 within cesm2_3_beta17

 Differences in namelist 'mosart_inparm':
  missing variable: 'do_rtmflood'
  missing variable: 'finidat_rtm'
  missing variable: 'frivinp_rtm'
  missing variable: 'rtmhist_fexcl1'
  missing variable: 'rtmhist_fexcl2'
  missing variable: 'rtmhist_fexcl3'
  missing variable: 'rtmhist_fincl1'
  missing variable: 'rtmhist_fincl2'
  missing variable: 'rtmhist_fincl3'
  missing variable: 'rtmhist_mfilt'
  missing variable: 'rtmhist_ndens'
  missing variable: 'rtmhist_nhtfrq'
  found extra variable: 'budget_frq'
  found extra variable: 'fexcl1'
  found extra variable: 'fexcl2'
  found extra variable: 'fexcl3'
  found extra variable: 'fincl1'
  found extra variable: 'fincl2'
  found extra variable: 'fincl3'
  found extra variable: 'finidat'
  found extra variable: 'frivinp'
  found extra variable: 'mfilt'
  found extra variable: 'mosart_euler_calc'
  found extra variable: 'mosart_tracers'
  found extra variable: 'ndens'
  found extra variable: 'nhtfrq'
  found extra variable: 'use_halo_option'

Changes answers
- what code configurations: mosart and rtm
- what platforms/compilers: all
- nature of change: mosart roundoff; rtm larger than roundoff due to bug fix; the latter also affects bgc variables

 We are ignoring strange diffs from baseline in two tests in variable
 FATES_TRANSITION_MATRIX_LULU as explained in issue ESCOMP#2656.
adrifoster pushed a commit to adrifoster/CTSM that referenced this issue Jul 25, 2024
Merge tag 'ctsm5.2.015' into merge-master-2024-0725
3839e4d 
Update submodule tags to pass runoff from cism to rof

- Update MOSART, CMEPS, and CISM so CISM runoff goes to ROF rather than CTSM
- Update RTM with fix needed for Paleo LGM work

Contributors:
@mvertens, @jedwards4b, @billsacks, @Katetc, @ekluzek, @slevis-lmwg

 Fixes ESCOMP#2590 Update CMEPS/MOSART/CISM/RTM tags
 Fixes ESCOMP/RTM#50 Likely wrong RTM river flux to MOM6 within cesm2_3_beta17

 Differences in namelist 'mosart_inparm':
  missing variable: 'do_rtmflood'
  missing variable: 'finidat_rtm'
  missing variable: 'frivinp_rtm'
  missing variable: 'rtmhist_fexcl1'
  missing variable: 'rtmhist_fexcl2'
  missing variable: 'rtmhist_fexcl3'
  missing variable: 'rtmhist_fincl1'
  missing variable: 'rtmhist_fincl2'
  missing variable: 'rtmhist_fincl3'
  missing variable: 'rtmhist_mfilt'
  missing variable: 'rtmhist_ndens'
  missing variable: 'rtmhist_nhtfrq'
  found extra variable: 'budget_frq'
  found extra variable: 'fexcl1'
  found extra variable: 'fexcl2'
  found extra variable: 'fexcl3'
  found extra variable: 'fincl1'
  found extra variable: 'fincl2'
  found extra variable: 'fincl3'
  found extra variable: 'finidat'
  found extra variable: 'frivinp'
  found extra variable: 'mfilt'
  found extra variable: 'mosart_euler_calc'
  found extra variable: 'mosart_tracers'
  found extra variable: 'ndens'
  found extra variable: 'nhtfrq'
  found extra variable: 'use_halo_option'

Changes answers
- what code configurations: mosart and rtm
- what platforms/compilers: all
- nature of change: mosart roundoff; rtm larger than roundoff due to bug fix; the latter also affects bgc variables

 We are ignoring strange diffs from baseline in two tests in variable
 FATES_TRANSITION_MATRIX_LULU as explained in issue ESCOMP#2656.
glemieux added a commit to rgknox/ctsm that referenced this issue Jul 26, 2024
@glemieux
update fates tag
d393cbf 
This addresses ESCOMP#2656
@glemieux
Copy link
Collaborator

This is fixed on fates main via sci.1.77.2_api.36.0.0.

@slevis-lmwg slevis-lmwg mentioned this issue Aug 2, 2024
Merged
@samsrabin samsrabin removed the next this should get some attention in the next week or two. Normally each Thursday SE meeting. label Aug 8, 2024
@samsrabin samsrabin added FATES API update Changes to the FATES version that also REQUIRE an API change in CTSM external issue needs to be addressed elsewhere (submodule); issue here for the sake of project tracking labels Aug 8, 2024
@samsrabin samsrabin mentioned this issue Aug 13, 2024
Merged
@glemieux
Copy link
Collaborator

glemieux commented Sep 5, 2024

This should be fixed by #2624

@glemieux glemieux closed this as completed Sep 5, 2024
samsrabin pushed a commit to samsrabin/CTSM that referenced this issue Sep 17, 2024
@glemieux @samsrabin
update fates satellite phenology test coverage
3bde763 
This moves one of the FatesColdSatPhen tests to a gnu compiler to
provide compiler coverage for issues like ESCOMP#2656.  This also adds an
nvhpc compiler test to the fates test suite.
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@glemieux glemieux

Labels
bug something is working incorrectly external issue needs to be addressed elsewhere (submodule); issue here for the sake of project tracking FATES API update Changes to the FATES version that also REQUIRE an API change in CTSM
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@glemieux @samsrabin @ekluzek @slevis-lmwg