Support comparison with a reference run

[xylar]

This merge adds the capability to compare the output of a simulation against a reference simulation.

In the case of climatologies, the comparison with the reference run is performed instead of the comparison with observations (or state estimates, reanalyses, etc.).

In the case of time series, an additional plot is added for the reference run.

Notes:

• Legend information has been moved from the title to the legend in all plots, since this is much easier to handle in the case of plots with anywhere from 1 to 4 time series
• Comparison with a reference run is not supported for the MOC (though this could conceivably be added in a future PR)
• It isn't obvious how to do a meaningful comparison of Hovmoller (trend vs. depth) plots with a reference run, so this isn't currently supported

The reference run is specified through the [runs]/referenceRunConfigFile option, which points to a config file describing analysis that must have already been performed on the reference run. This saves computing the reference climatologies and time series again and again if the same reference run is used several times. (It also seems to fix some NetCDF/HDF5 errors I encountered when trying to compare several runs at once to the same reference run.) Additionally, the reference config file can specify a range of years for the climatology and/or time series that differs from the main run.

The startYear and endYear for climatologies, time series and indices in the reference config file must be the actual range that was used to compute them. Otherwise, the results may not be found correctly or the results may not plot correctly. One way to make sure this is the case is to use the config file saved to the logs directory in when the reference analysis is performed.

[xylar]

This PR branches off of #288 so will only be merged after that PR has been fulfilled

[xylar]

This PR is intended to address #212

[xylar]

@vanroekel, I would like to discuss with you how best to add a new plot to the Nino34 Index plots. It becomes cumbersome to have 4 plots in a row. Do we want to do 2x2 plots when we include a reference run? Or do we want to remove one or the other of the observations plots? How would you lay things out?

[xylar]

Testing

I tested this with the QU240 run on my laptop and with several EC60to30v3wLI runs on Edison. An example of the latter can be found here:
http://portal.nersc.gov/project/m2833/mpas_analysis_output/param_study_vs_ctrl/20171223.T62_oEC60to30v3wLI_ICG.cori-knl.land_ice_flux_attenuation_coefficient_5/

Note: there is a problem with the reference melt rate field -- it appears to be all zeros.

I experienced NetCDF errors in climatologyMapSeaIceThickSH: remapRefMpasClimatology in 3 of the 4 runs but this did not recur when I reran the analysis. I am making a note to see if this was a file system glitch or if this turns out to be a recurring problem.

Update: the melt problem has been fixed. The HDF5 errors seem to persist. When several tasks are analyzing the same files, the first task seems to be successful but remaining tasks show hdf5 errors like:

HDF5-DIAG: Error detected in HDF5 (1.8.18) thread 46912496680640:
  #000: H5Adeprec.c line 299 in H5Aopen_idx(): unable to open attribute
    major: Attribute
    minor: Can't open object
...

This could be an issue with e3sm-unified/1.1.2 but I tried with the usual python/anaconda-2.7-acme and that had the same errors.

Update 2: I switched the approach used to specify the reference run. Now, a config file is specified and the reference analysis must have already been performed in advance. This saves computing the reference climatologies and time series over and over, and also seems to have fixed the issue with file conflicts. This allows a different year range to be specified for the reference run and for the main run (for climatologies, time series and climate indices). This has been tested on the QU240 run.

[vanroekel]

@xylar what would the 4th NINO plot be? Are you thinking 2 OBS, 1 MainSimulation, and 1 REF RUN? Or something else? If you are thinking just Obs + 2 simulations, I would personally lean to removing 1 of the observation plots and retain three plots. Perhaps we could plot the same length of observational record as the simulation (or as close as possible) to eliminate one plot. I don't think we should do a difference plot for NINO34 simulation output against OBS or another run. I don't think we would gain any useful insight and are much more likely to get distracted and/or misled by the direct comparison.

[xylar]

@vanroekel, thanks for the reply

what would the 4th NINO plot be? Are you thinking 2 OBS, 1 MainSimulation, and 1 REF RUN? Or something else?

Yes, that's exactly what I had in mind

If you are thinking just Obs + 2 simulations, I would personally lean to removing 1 of the observation plots and retain three plots. Perhaps we could plot the same length of observational record as the simulation (or as close as possible) to eliminate one plot.

I think that's a very good idea and shouldn't be too hard. Would you want to do that instead of the 30-year plot in all cases? Or would you want to keep the 3 plots the same as what we currently have if we're not comparing with a reference run?

I don't think we should do a difference plot for NINO34 simulation output against OBS or another run. I don't think we would gain any useful insight and are much more likely to get distracted and/or misled by the direct comparison.

Oh, I completely agree. Whereas difference plots make sense for climatologies, I wasn't planning on doing them for any time-series fields.

[xylar]

@vanroekel, I added new panel for the reference run as follows (this is obviously too short and low res a run to get a proper feel for these plots). First panel is obs, second is the main run, 3rd is the reference run. I'm going to add text "Reference run: " before the last plot, just to make things clearer:

I'll post an example with a longer time series once I have one.

[xylar]

This branch now also depends on #293 and will only be merged after that PR.

[milenaveneziani]

@xylar: what does this PR need, besides some testing? can it still be considered WIP?

[xylar]

@milenaveneziani, yes, feel free to test it out. Since it's based off of #294, you'll need a python environment with xarray>=0.10.0.

@stephenprice, as you have time, could you see if this will work for you as well? You'll need to use the acme-unified/1.1.2 environment on Edison. Run the analysis on some reference run. Then, point the analysis for the config file for the reference run via the referenceRunConfigFile option in the [input] section.

[vanroekel]

@xylar finally testing this now. I've successfully done a comparison between different years of the 18to6. I want to test a comparison with low res and high res and will approve if that goes well (should be done this morning). One minor suggestion / request. When this feature is used to compare different years in the same run it is hard to know which time period is which on the time series plots e.g.

Is there an easy way to add a label to one or the other to delineate? I was thinking something as simple as main and ref tagged to each label. Not critical, but I think it would be helpful.

[xylar]

@vanroekel, thanks for your testing. My suggested solution to the case you show above would be to change mainRunName to be different in your 2 config files (e.g. appending _main and _ref or the year range). This is what I have done in my testing. In fact, you could just name the runs "main" and "ref" if you wanted.

[xylar]

By the way, that OHC plot looks really strange. I'm worried something has gone wrong with the computation.

[vanroekel]

@xylar thanks for the great suggestion, makes perfect sense. As for the OHC plot, you are right. It is odd the zero line doesn't move across the whole plot and it also looks like the black lines may be on a different vertical scale when compared to previous OHC plots of this run. I'll take a look.

[xylar]

I believe the problems with the OHC plot are unrelated to this PR but they should be posted as an issue and get fixed. If you can just post the plot and give some basic info (e.g. which years you used for each time series), that should help me debug.

[vanroekel]

all tests went well. I can post an issue on the odd OHC plot above. Thanks @xylar!

[xylar]

Thanks for testing, @vanroekel! Much appreciated.