Add MpasClimatologyTask and RemapMpasClimatologySubtask

[xylar]

MpasClimatologyTask computes climatologies for all requested
variables and for all requested seasons. All analysis tasks
that were previously computing climatologies using ncclimo now
simply add the required variables and seasons to this task and
all climatologies are computed simultaneously.

RemapMpasClimatologySubtask masks and then remaps MPAS
climatologies produced by MpasClimatologyTask to the desired
comparison grid. This task operates as a subtask whichever
analysis task will plot the remapped climatology.

subtasks

Support has been added for each analysis task to have one or more
subtasks. Each subtask has the same task name as the "parent" task
but also has a subtask name. The full name of the subtask when it
is printed is taskName: subtaskName, while the name in log files, etc.
is taskName_subtaskName.

Currently, the only subtask is RemapMpasClimatologySubtask, but
future subtasks will allow several seasons, depths, etc. to be plotted
simultaneously (in parallel) within a given master task.

prerequisite tasks

Support has been added for prerequisite tasks that must run before
a given task. Currently, the only prerequisite tasks are
MpasClimatologyTask and RemapMpasClimatologySubtask.
MpasClimatologyTask must run before any task that uses a
climatology computed with ncclimo, as this task computes all
climatologies simultaneously. All RemapMpasClimatologySubtasks
wait until the corresponding MpasClimatologyTask has finished
before proceeding. All tasks with a RemapMpasClimatologySubtask
wait until that subtask has finished before making their plots.

[xylar]

@milenaveneziani and @pwolfram, stand by on this PR for now. I think I will break it into 2, since it's quite large and includes a lot of changes.

Also, I seem to need to fix CI...

[xylar]

This PR should not be merged until #259 has been merged.

[pwolfram]

Ok @xylar, I will wait until you split this into two PRs

[xylar]

@pwolfram, I've done so already. #259 is the other PR. I can split it into a 3rd one for subtasks but there's no way for anyone to test subtasks until I implement a use for them, so I think it wouldn't be very useful to split any further. Let me know if you disagree.

In any case, #259 does need to be reviewed before this PR.

[milenaveneziani]

@xylar: so that I understand the order of some of these PR's: does this one build on the design document PR #209?

[xylar]

Yes, the subtasks and prerequisites are from #209. Then I made a prerequisite and a subtask to actually use the functionality here.

[xylar]

Testing

Runs successfully with:

• the usual G-QU240 test case on my laptop.
• the default configuration on Anvil (in batch mode)
• the default configuration (except shorter climatology and time-series time span) on Theta (on login node)
• the MatchBoth_orig configuration on grizzly in batch mode (with indexNino34 disabled, see error in indexNino34 on LANL IC #269 )
• the 20170313.beta1.A_WCYCL1850S.ne30_oECv3_ICG.edison configuration on edison in batch mode (with fixed MOC region file as in Switch example config files to use mapping directory #266).

I'm having trouble with the MatchBoth_orig configuration on wolf in batch mode (with indexNino34 disabled, see #269 ). I am getting errors trying to run ESMF_RegridWeightGen. With the changes from #266 (pointing to existing mapping files), I was able to run successfully.

[xylar]

@milenaveneziani, this is ready for further testing if you have a chance. I'll try to test on theta and anvil tomorrow.

[xylar]

@milenaveneziani, do you have time to test these today or tomorrow? Maybe on one or more of IC, Edison or OLCF, which I have not yet tested?

[milenaveneziani]

I've been wanting to get to this in the past 2 days.. I can test on edison for sure between later today and tomorrow. I could also test at Oak Ridge, but perhaps I shouldn't promise it.

Do you test both in batch and on the login node?

[xylar]

I tested in batch mode on Anvil and on the login node on Theta (my batch job didn't finish in the allotted hour, seemingly because theta is slow for this stuff...)

[milenaveneziani]

First test on edison, on the login node, passed. Results here:
http://portal.nersc.gov/project/acme/milena/20170915.beta2.A_WCYCL1850S.ne30_oECv3_ICG.edisonClimoTest/
I am now going to try a batch job.

[milenaveneziani]

Second test on edison in batch mode also passed.

I am now running tests on titan, where I have beta2 data that I have transferred recently to test a-prime, and I am having a weird problem there. These are the messages I get when running on the login node:

...
Traceback (most recent call last):
  File "./run_mpas_analysis", line 194, in add_task_and_subtasks
    analysisTask.setup_and_check()
  File "/autofs/nccs-svm1_home1/milena/MPAS-git-repositories/MPAS-Analysis/mpas_analysis/shared/climatology/remap_mpas_climatology_subtask.py", line 152, in setup_and_check
    raise IOError('No MPAS restart file found: need at least one '
IOError: No MPAS restart file found: need at least one restart file to perform remapping of climatologies.
ERROR: analysis task climatologyMapSeaIceConcNH: remapMpasClimatology failed during check and will not be run
ERROR: a subtask of analysis task climatologyMapSeaIceConcNH failed during check,
       so this task will not be run
Traceback (most recent call last):
  File "./run_mpas_analysis", line 194, in add_task_and_subtasks
    analysisTask.setup_and_check()
  File "/autofs/nccs-svm1_home1/milena/MPAS-git-repositories/MPAS-Analysis/mpas_analysis/shared/climatology/remap_mpas_climatology_subtask.py", line 152, in setup_and_check
    raise IOError('No MPAS restart file found: need at least one '
IOError: No MPAS restart file found: need at least one restart file to perform remapping of climatologies.
ERROR: analysis task climatologyMapSeaIceThickNH: remapMpasClimatology failed during check and will not be run
ERROR: a subtask of analysis task climatologyMapSeaIceThickNH failed during check,
       so this task will not be run
Traceback (most recent call last):
  File "./run_mpas_analysis", line 194, in add_task_and_subtasks
    analysisTask.setup_and_check()
  File "/autofs/nccs-svm1_home1/milena/MPAS-git-repositories/MPAS-Analysis/mpas_analysis/shared/climatology/remap_mpas_climatology_subtask.py", line 152, in setup_and_check
    raise IOError('No MPAS restart file found: need at least one '
IOError: No MPAS restart file found: need at least one restart file to perform remapping of climatologies.
ERROR: analysis task climatologyMapSeaIceConcSH: remapMpasClimatology failed during check and will not be run
ERROR: a subtask of analysis task climatologyMapSeaIceConcSH failed during check,
       so this task will not be run
Traceback (most recent call last):
  File "./run_mpas_analysis", line 194, in add_task_and_subtasks
    analysisTask.setup_and_check()
  File "/autofs/nccs-svm1_home1/milena/MPAS-git-repositories/MPAS-Analysis/mpas_analysis/shared/climatology/remap_mpas_climatology_subtask.py", line 152, in setup_and_check
    raise IOError('No MPAS restart file found: need at least one '
IOError: No MPAS restart file found: need at least one restart file to perform remapping of climatologies.
ERROR: analysis task climatologyMapSeaIceThickSH: remapMpasClimatology failed during check and will not be run
ERROR: a subtask of analysis task climatologyMapSeaIceThickSH failed during check,
       so this task will not be run
Running mpasClimatologyOcean
Running timeSeriesOHC
Running timeSeriesSST
Running indexNino34
Running mpasClimatologySeaIce
Running timeSeriesSeaIceAreaVol
   Task mpasClimatologySeaIce has finished successfully.
   Task timeSeriesSST has finished successfully.
   Task indexNino34 has finished successfully.
   Task timeSeriesSeaIceAreaVol has finished successfully.
   Task mpasClimatologyOcean has finished successfully.
Running climatologyMapMLD: remapMpasClimatology
Running climatologyMapSST: remapMpasClimatology
Running climatologyMapSSS: remapMpasClimatology
Running meridionalHeatTransport
Running streamfunctionMOC
   Task meridionalHeatTransport has finished successfully.
   Task climatologyMapMLD: remapMpasClimatology has finished successfully.
Running climatologyMapMLD
   Task climatologyMapSSS: remapMpasClimatology has finished successfully.
Running climatologyMapSSS
   Task climatologyMapSST: remapMpasClimatology has finished successfully.
Running climatologyMapSST
   Task climatologyMapSSS has finished successfully.
   Task climatologyMapMLD has finished successfully.
   Task climatologyMapSST has finished successfully.
   Task timeSeriesOHC has finished successfully.
   Task streamfunctionMOC has finished successfully.
Generating webpage for viewing results...

So, for some reason, the sea-ice climatology tasks are not recognizing that there is an mpas restart file available, but the ocean climatology tasks do.
After checking the html link, I can confirm that the only diagnostics missing are the sea-ice climos.

Do you understand this @xylar?

[xylar]

@milenaveneziani, I think one of the changes I made may have removed the ability to use an ocean restart file in lieu of a sea-ice restart file. Is that an important capability? I had the impression it was kind of a sloppy way of handling cases where we hadn't transferred a sea-ice restart file, as opposed to something we wanted to be using in that way. I'll look into it as I have time tomorrow.

[milenaveneziani]

Hi @xylar, I personally don't see it that way, but rather as our solution to the fact that mesh information is not part of a static run-specific file that contains mesh and vertical coordinate details, among other things. Unless we were envisioning a near future where mpas-seaice had different mesh characteristics than mpas-o, I don't see why we should increase the data requirement to add a mpas-seaice restart file at the moment. That's just my opinion; we can discuss further if you feel strongly the other way.

[xylar]

@milenaveneziani, okay, so looking into it further, the issue isn't that I eliminated code on the sea-ice side. Rather, when I added the remapping task, I had it assume there was a restart file available for each component to avoid special logic for the sea-ice component. I can pretty easily add in a special case in that task if the component is sea ice.

I'll let you know later today as soon as I've done that.

[milenaveneziani]

hmm, I see. Don't know, I don't like special cases either.. Maybe we should just leave it as is.
Let me try again tomorrow after I have transferred a sea-ice restart file.

[milenaveneziani]

@xylar: all went well on titan when I added the mpas-cice restart file. Seems to me this is enough testing, what do you think?

I realized we don't mention what files are needed by MPAS-Analysis to run in the documentation (we do that in a-prime). So, what about adding this piece to the README file:

# List of MPAS output files that are needed by MPAS-Analysis:
       - mpas-o files:
               mpaso.hist.am.timeSeriesStatsMonthly.*.nc (Note: since OHC
                   anomalies are computed wrt the first year of the simulation,
                   if OHC diagnostics is activated, the analysis will need the
                   first full year of mpaso.hist.am.timeSeriesStatsMonthly.*.nc
                   files, no matter what begin_yr_ts, end_yr_ts are. This is
                   especially important to know if short term archiving is
                   used in the run to analyze: in that case, set short_term_archive
                   to 1 and choose begin_yr_ts, end_yr_ts to include only data
                   that have been short-term-archived).
               mpaso.hist.am.meridionalHeatTransport.0001-03-01.nc (or any hist.am.meridionalHeatTransport file)
               mpaso.rst.0002-01-01_00000.nc (or any other mpas-o restart file)
               streams.ocean
               mpas-o_in
       - mpas-cice files:
               mpascice.hist.am.timeSeriesStatsMonthly.*.nc
               mpascice.rst.0002-01-01_00000.nc (or any other mpas-cice restart file)
               streams.cice
               mpas-cice_in

[xylar]

I'll add that right away. Thanks for testing!

[milenaveneziani]

I think this is good to go, @xylar. Thanks for your patience in the last couple of weeks.