Support searching ESGF for a specific version of a file #1822

bouweandela · 2022-11-23T11:03:13Z

Description

This adds support to esmvalcore.esgf.find_files for searching for a specific version of a file on ESGF. This a prerequisite to support specific versions of datasets in the recipe, which is part of #1609. It also adds an esmvalcore.esgf.ESGFFile.facets attribute, which can be used to inspect the facets of the ESGFFile.

Example usage:

import esmvalcore.esgf
facets = {
    'dataset': 'inmcm4',
    'ensemble': 'r1i1p1',
    'exp': ['historical', 'rcp85'],
    'mip': 'Amon',
    'project': 'CMIP5',
    'short_name': 'tas',
    'version': 'v20130207',
}
files = esmvalcore.esgf.find_files(**facets)

result

[
    ESGFFile:cmip5/output1/INM/inmcm4/historical/mon/atmos/Amon/r1i1p1/v20130207/tas_Amon_inmcm4_historical_r1i1p1_185001-200512.nc on hosts ['esgf-data1.ceda.ac.uk', 'esgf2.dkrz.de'],
    ESGFFile:cmip5/output1/INM/inmcm4/rcp85/mon/atmos/Amon/r1i1p1/v20130207/tas_Amon_inmcm4_rcp85_r1i1p1_200601-210012.nc on hosts ['esgf-data1.ceda.ac.uk', 'esgf2.dkrz.de'],
]

and

import pprint
print(pprint.pformat(files[0].facets))

result:

{   'dataset': 'inmcm4',
    'ensemble': 'r1i1p1',
    'exp': 'historical',
    'frequency': 'mon',
    'institute': 'INM',
    'mip': 'Amon',
    'product': 'output1',
    'project': 'CMIP5',
    'realm': 'atmos',
    'short_name': 'tas',
    'version': 'v20130207'}

Link to documentation: https://esmvaltool--1822.org.readthedocs.build/projects/ESMValCore/en/1822/api/esmvalcore.esgf.html

Before you get started

☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🧪 The new functionality is relevant and scientifically sound
🛠 This pull request has a descriptive title and labels
🛠 Code is written according to the code quality guidelines
🧪 and 🛠 Documentation is available
🛠 Unit tests have been added
🛠 Changes are backward compatible
🛠 The list of authors is up to date
🛠 All checks below this pull request were successful

To help with the number pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

codecov · 2022-11-23T11:05:04Z

Codecov Report

Merging #1822 (05d859e) into main (b0120d8) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1822      +/-   ##
==========================================
+ Coverage   91.54%   91.57%   +0.02%     
==========================================
  Files         203      203              
  Lines       10945    10979      +34     
==========================================
+ Hits        10020    10054      +34     
  Misses        925      925

Impacted Files	Coverage Δ
esmvalcore/esgf/facets.py	`100.00% <ø> (ø)`
esmvalcore/esgf/_download.py	`100.00% <100.00%> (ø)`
esmvalcore/esgf/_search.py	`100.00% <100.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

valeriupredoi

(possible) changes requested, depending on what @bouweandela thinks of my comment; otherwise the code and functionality, inc tests look great. Just remember to add to the documentation of course, but am guessing that'll happen in #1609

esmvalcore/esgf/_search.py

bouweandela · 2022-11-25T15:52:08Z

Just remember to add to the documentation of course, but am guessing that'll happen in #1609

Documentation for the new features introduced here should be included in this pull request. I did add some docs, is there anything you're missing?

valeriupredoi · 2022-11-25T16:18:40Z

Just remember to add to the documentation of course, but am guessing that'll happen in #1609

Documentation for the new features introduced here should be included in this pull request. I did add some docs, is there anything you're missing?

yes, an example where a specific version of the file is requested - I didn't see it in the docs (not the API documentation, but rather, the actual doc where esgf stuff is documented)

valeriupredoi · 2022-11-28T15:48:54Z

@bouweandela you may also want to close this #1523 too after this gets merged

bouweandela · 2022-11-28T17:59:30Z

That's a different problem, not solved in this pull request. It might be best to solve that after we have a similar interface for searching for files locally as proposed in #1825 because if the files on ESGF have different timespans than the ones available locally, it will still cause trouble even if fixed for one of the two search methods.

…ffile-facets

bouweandela · 2022-12-01T12:04:41Z

@valeriupredoi I added some more examples to explain the functionality better. Could you have another look, please?

valeriupredoi

purrfect, cheers @bouweandela 🍺

valeriupredoi · 2022-12-05T14:43:05Z

@ESMValGroup/technical-lead-development-team this is good for merge, any of you good folk want to have a last look and merge it, please? 🍺

schlunma

Looks good to me!

Just two small questions on the code and one general one: has this been tested with real recipes? I think it would be good to test some recipes (with offline=False) with that to make sure nothing broke (at least one that uses timerange, one that uses start_date/end_date, and one that uses wildcards in timerange). I guess this does not allow to use version in the recipe yet?

Cheers! 🚀

tests/integration/esgf/test_search_download.py

esmvalcore/esgf/_search.py

…ffile-facets

bouweandela · 2022-12-06T09:29:12Z

I guess this does not allow to use version in the recipe yet?

That will probably only work when getting all data from ESGF. Support for doing that with local data is implemented in #1835.

bouweandela · 2022-12-06T09:30:28Z

has this been tested with real recipes? I think it would be good to test some recipes (with offline=False) with that to make sure nothing broke (at least one that uses timerange, one that uses start_date/end_date, and one that uses wildcards in timerange).

@valeriupredoi Did you already have a go at this? I just did it myself and everything seems fine, but it's probably best done by someone else than the author of the changes.

valeriupredoi · 2022-12-06T11:49:57Z

has this been tested with real recipes? I think it would be good to test some recipes (with offline=False) with that to make sure nothing broke (at least one that uses timerange, one that uses start_date/end_date, and one that uses wildcards in timerange).

@valeriupredoi Did you already have a go at this? I just did it myself and everything seems fine, but it's probably best done by someone else than the author of the changes.

no I didn't - but the tests are nice and all-inclusive, I don't see any problem with running it. Would you still want me to run tests? I'd rather do it when we already have an example in ESMValTool 😁

bouweandela · 2022-12-06T11:59:56Z

Would you still want me to run tests?

Yes, I think that's what @schlunma just asked for.

Here is a (slightly modified in the time selection part) version of the example Python recipe that I just used for testing:

# ESMValTool
# recipe_python.yml
---
documentation:
  description: |
    Example recipe that plots a map and timeseries of temperature.

  title: Recipe that runs an example diagnostic written in Python.

  authors:
    - andela_bouwe
    - righi_mattia

  maintainer:
    - schlund_manuel

  references:
    - acknow_project

  projects:
    - esmval
    - c3s-magic

datasets:
  - {dataset: BCC-ESM1, project: CMIP6, exp: historical, ensemble: r1i1p1f1, grid: gn}
  - {dataset: CanESM2, project: CMIP5, exp: historical, ensemble: r1i1p1}

preprocessors:

  select_january:
    extract_month:
      month: 1
    convert_units:
      units: degree_Celsius

  annual_mean_amsterdam:
    extract_point:
      latitude: 52.379189
      longitude: 4.899431
      scheme: linear
    annual_statistics:
      operator: mean
    multi_model_statistics:
      statistics:
        - mean
      span: overlap

  annual_mean_global:
    area_statistics:
      operator: mean
      fx_variables:
        areacella:
    annual_statistics:
      operator: mean

diagnostics:

  map:
    description: Global map of temperature in January 2000.
    themes:
      - phys
    realms:
      - atmos
    variables:
      tas:
        mip: Amon
        preprocessor: select_january
        start_year: 2000
        end_year: 2000
    scripts:
      script1:
        script: examples/diagnostic.py
        quickplot:
          plot_type: pcolormesh
          cmap: Reds

  timeseries:
    description: Annual mean temperature in Amsterdam and global mean since 1850.
    themes:
      - phys
    realms:
      - atmos
    variables:
      tas_amsterdam:
        short_name: tas
        mip: Amon
        preprocessor: annual_mean_amsterdam
        timerange: '*'
      tas_global:
        short_name: tas
        mip: Amon
        preprocessor: annual_mean_global
        timerange: '1850/2000'
    scripts:
      script1:
        script: examples/diagnostic.py
        quickplot:
          plot_type: plot

valeriupredoi · 2022-12-06T12:10:53Z

# ESMValTool
# recipe_python.yml
---
documentation:
  description: |
    Example recipe that plots a map and timeseries of temperature.

  title: Recipe that runs an example diagnostic written in Python.

  authors:
    - andela_bouwe
    - righi_mattia

  maintainer:
    - schlund_manuel

  references:

...

Cheers, bud! Works very well 🍻

schlunma · 2022-12-06T12:14:54Z

Awesome, thanks very much both! Will merge now 🚀

valeriupredoi · 2022-12-06T12:18:19Z

great work, chaps!

Support searching ESGF for a specific version of a file

d0c0553

bouweandela added enhancement New feature or request api Notebook API labels Nov 23, 2022

Add some tests

aa97817

bouweandela marked this pull request as ready for review November 24, 2022 20:22

bouweandela added this to the v2.8.0 milestone Nov 24, 2022

valeriupredoi requested changes Nov 25, 2022

View reviewed changes

esmvalcore/esgf/_search.py Show resolved Hide resolved

bouweandela added 5 commits November 29, 2022 15:26

Merge branch 'main' of github.com:ESMValGroup/ESMValCore into add-esg…

44b7f2e

…ffile-facets

Add some more documentation

a09bf12

Fix test

9d7db26

Fix indentation

3c251af

Add example of searching for a specific version of a file

246d2c5

valeriupredoi approved these changes Dec 5, 2022

View reviewed changes

schlunma reviewed Dec 5, 2022

View reviewed changes

tests/integration/esgf/test_search_download.py Show resolved Hide resolved

esmvalcore/esgf/_search.py Show resolved Hide resolved

bouweandela added 2 commits December 6, 2022 10:23

Restore support for start_year/end_year

48c35f2

Merge branch 'main' of github.com:ESMValGroup/ESMValCore into add-esg…

05d859e

…ffile-facets

schlunma merged commit a56e611 into main Dec 6, 2022

schlunma deleted the add-esgffile-facets branch December 6, 2022 12:15

Support searching ESGF for a specific version of a file #1822

Support searching ESGF for a specific version of a file #1822

Uh oh!

Conversation

bouweandela commented Nov 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Before you get started

Checklist

Uh oh!

codecov bot commented Nov 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

valeriupredoi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bouweandela commented Nov 25, 2022

Uh oh!

valeriupredoi commented Nov 25, 2022

Uh oh!

valeriupredoi commented Nov 28, 2022

Uh oh!

bouweandela commented Nov 28, 2022

Uh oh!

bouweandela commented Dec 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

valeriupredoi left a comment

Choose a reason for hiding this comment

Uh oh!

valeriupredoi commented Dec 5, 2022

Uh oh!

schlunma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bouweandela commented Dec 6, 2022

Uh oh!

bouweandela commented Dec 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

valeriupredoi commented Dec 6, 2022

Uh oh!

bouweandela commented Dec 6, 2022

Uh oh!

valeriupredoi commented Dec 6, 2022

Uh oh!

schlunma commented Dec 6, 2022

Uh oh!

valeriupredoi commented Dec 6, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bouweandela commented Nov 23, 2022 •

edited

Loading

codecov bot commented Nov 23, 2022 •

edited

Loading

bouweandela commented Dec 1, 2022 •

edited

Loading

bouweandela commented Dec 6, 2022 •

edited

Loading