Izumi updates

[dabail10]

For detailed information about submitting Pull Requests (PRs) to the CICE-Consortium,
please refer to: https://github.com/CICE-Consortium/About-Us/wiki/Resource-Index#information-for-developers

PR checklist

• Short (1 sentence) summary of your PR:
  Compiler updates on izumi.
• Developer(s):
  dabail10 (D. Bailey)
• Suggest PR reviewers from list in the column to the right.
• Please copy the PR test results link or provide a summary of testing completed below.
  https://github.com/CICE-Consortium/Test-Results/wiki/icepack_by_hash_forks#b299062d2e46acd561ca329a9434aff03fc338c5
• How much do the PR code changes differ from the unmodified code?
  • bit for bit
  • different at roundoff level
  • more substantial
• Does this PR create or have dependencies on CICE or any other models?
  • Yes
  • No
• Does this PR add any new test cases?
  • Yes
  • No
• Is the documentation being updated? ("Documentation" includes information on the wiki or in the .rst files from doc/source/, which are used to create the online technical docs at https://readthedocs.org/projects/cice-consortium-cice/.)
  • Yes
  • No, does the documentation need to be updated at a later time?
    • Yes
    • No
• Please provide any additional information or relevant details below:
  I only had baselines for intel and nag before the compiler upgrades. There also seem to be some issues with gnu debug to sort out.

[apcraig]

Thanks Dave. I think at the very least, we'll want to run full test suites with pgi and gnu even if we don't have a baseline to compare to. If we are doing it, we might as well create a baseline. Or you can use /scratch/cluster/tcraig/ICEPACK_BASELINES_MASTER/icepack.09a5e19f00.200530-100005 which are the full test suites from last weekend's testing. Just do an

ln -s $MY_ICEPACK_BASELINES_DIR /scratch/cluster/tcraig/ICEPACK_BASELINES_MASTER/icepack.09a5e19f00.200530-100005

and then --bcmp icepack.09a5e19f00.200530-100005.

Is it likely the same module updates are going to work in CICE? I see you noted problems with gnu debug, is that in Icepack or CICE or both? Do we want to work on that before merging or after?

[dabail10]

Excellent. I didn't know where the baselines were sitting on izumi. I will re-run with gnu and pgi. There might be a compiler issue with gnu and I haven't tried with CICE yet.

[dabail10]

Mark is rebooting izumi tomorrow, so I will wait until after that.

[dabail10]

Just an update. I ran the comparison against your baselines and everything was bfb that was there. There was a lot of missing baselines and the 30 GNU tests are still failing today. I'm adding some print statements.

538 measured results of 538 total results
389 of 538 tests PASSED
0 of 538 tests PENDING
119 of 538 tests MISSING data
30 of 538 tests FAILED

[dabail10]

My sandbox is /home/dbailey/CICE_master. The testsuite is izumi2. I ran base_suite.

[dabail10]

Sorry, my brain is not working today.

[dabail10]

My testing is done for Icepack. A couple notes.

1. All tests are bfb for intel, gnu, and nag.

2. There is a bug fix here that was caught by gnu debug.

3. There was one missing test in your baselines:
   MISS izumi_intel_restart_col_1x1_bgcISPOL compare icepack.09a5e19f00.200530-100005 missing-data

4. All of the PGI tests are not bfb. This indicates changes to the compiler.

FAIL izumi_pgi_smoke_col_1x1_diag1_run1year compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_smoke_col_1x1_debug_run1year compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_smoke_col_1x1_bgcISPOL_debug compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_smoke_col_1x1_bgcNICE_debug compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_smoke_col_1x1_bgcsklNICE_debug compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_smoke_col_1x1_debug_run1year_thermo1 compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_smoke_col_1x1_debug_run1year_swccsm3 compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_smoke_col_1x1_alt03_debug_run1year compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_smoke_col_1x1_alt04_debug_run1year compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_smoke_col_1x1_debug_dt30min_leap_run1year compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_smoke_col_1x1_debug_dyn_run1year compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_smoke_col_1x1_debug_fsd12_run1year compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_smoke_col_1x1_debug_fsd1_run1year compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_debug compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_diag1 compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_pondcesm compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_pondlvl compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_pondtopo compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_bgcISPOL compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_thermo1 compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_swccsm3 compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_isotope compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_alt01 compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_alt02 compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_alt03 compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_alt04 compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_dyn compare icepack.09a5e19f00.200530-100005 different-data
FAIL izumi_pgi_restart_col_1x1_fsd12 compare icepack.09a5e19f00.200530-100005 different-data

[apcraig]

Excellent, thanks @dabail10! For reference, the full test suite results can be seen here,

https://github.com/CICE-Consortium/Test-Results/wiki/icepack_by_hash_forks#b299062d2e46acd561ca329a9434aff03fc338c5

I will merge this unless you feel there are any other outstanding issues.