v3.0 (#46)

* Remove `variants` key and use `features` instead (#30) * Remove variants from models.yaml * Remove spinup variant and use the spinup variable instead * GitHub Action: Apply Pep8-formatting * Remove variant check * Install specific icontools version --------- Co-authored-by: github-actions <[email protected]> * Convert config files to yaml format (#36) * Remove variants from models.yaml * Remove spinup variant and use the spinup variable instead * GitHub Action: Apply Pep8-formatting * Remove variant check * Install specific icontools version * Starting to work on cosmo case * GitHub Action: Apply Pep8-formatting * GitHub Action: Apply Pep8-formatting * Add cases and paths * Create cfg object from Config class * GitHub Action: Apply Pep8-formatting * Format output of configuration * Add email and simplify print function * GitHub Action: Apply Pep8-formatting * Improve output * GitHub Action: Apply Pep8-formatting * Make cams a dict and adapt prepare_data * GitHub Action: Apply Pep8-formatting * Adapt emissions * GitHub Action: Apply Pep8-formatting * Adapt biofluxes * Change user variable naming * GitHub Action: Apply Pep8-formatting * Fix path * Adapt oem * GitHub Action: Apply Pep8-formatting * Modify whole config.yaml * Adapt online_vprm * Work on int2lm, some restructuring * GitHub Action: Apply Pep8-formatting * Move times to config file * GitHub Action: Apply Pep8-formatting * Further work on int2lm * GitHub Action: Apply Pep8-formatting * Finish int2lm * GitHub Action: Apply Pep8-formatting * Fix post_int2lm * GitHub Action: Apply Pep8-formatting * Use correct config variable in post_int2lm * Work on cosmo job/config * GitHub Action: Apply Pep8-formatting * Make cosmo job run * GitHub Action: Apply Pep8-formatting * Fix hstart and hstop for spinup * GitHub Action: Apply Pep8-formatting * Fix post_cosmo and add functions to set config variables * GitHub Action: Apply Pep8-formatting * Rename cosmo-ghg test case and reduce simulation time * Minor fixes for other jobs * Fix cosmo job * Remove config.py * Change time interval * Remove cosmo_spec * Fix inidates for spinup runs * GitHub Action: Apply Pep8-formatting * Set correct inidate for int2lm job * Fix and unify post_int2lm * GitHub Action: Apply Pep8-formatting * Fix hstart and hstop for spinup runs * Remove unnecessary inidate variables * GitHub Action: Apply Pep8-formatting * Add cosmo-ghg restart test case * Create separate inidate variable for int2lm * GitHub Action: Apply Pep8-formatting * Work on icon case * Further work on ICON and make job_id and chain_root config variables * GitHub Action: Apply Pep8-formatting * Input files for ICON * GitHub Action: Apply Pep8-formatting * Add class function for easy-access variables * Further work on ICON * GitHub Action: Apply Pep8-formatting * Print config variables before duplicating dict variables * ICON: prepare_data * Remove icon_spec * Add class function to convert relative to absolute paths * Further work on prepare_data * GitHub Action: Apply Pep8-formatting * Add prefix to meteo nameformat * GitHub Action: Apply Pep8-formatting * Fix time variable for xarray * GitHub Action: Apply Pep8-formatting * Remove print * Further work on icon job * GitHub Action: Apply Pep8-formatting * Better naming for variables and restart setting * GitHub Action: Apply Pep8-formatting * Add GEOSP file for restart simulations * GitHub Action: Apply Pep8-formatting * Set procs for restart etc. in config * Set correct endtime in prepare_data * Set correct nodes for icon (cpu + gpu) * Fix error * Further work on icon restart * GitHub Action: Apply Pep8-formatting * Fix restart and finish icon-test case * Remove print * Update jenkins tests * Fix cosmo-ghg test * fix int2lm job script * Fix for GEOSP data copy * Fix jobid date * GitHub Action: Apply Pep8-formatting * Add ISO 8601 duration converter * Fix icon test jenkins files * GitHub Action: Apply Pep8-formatting * Modify comment in config file to match case --------- Co-authored-by: github-actions <[email protected]> * Use `cfg.enddate` instead of `hstart` and `hstop` (#39) * Remove variants from models.yaml * Remove spinup variant and use the spinup variable instead * GitHub Action: Apply Pep8-formatting * Remove variant check * Install specific icontools version * Starting to work on cosmo case * GitHub Action: Apply Pep8-formatting * GitHub Action: Apply Pep8-formatting * Add cases and paths * Create cfg object from Config class * GitHub Action: Apply Pep8-formatting * Format output of configuration * Add email and simplify print function * GitHub Action: Apply Pep8-formatting * Improve output * GitHub Action: Apply Pep8-formatting * Make cams a dict and adapt prepare_data * GitHub Action: Apply Pep8-formatting * Adapt emissions * GitHub Action: Apply Pep8-formatting * Adapt biofluxes * Change user variable naming * GitHub Action: Apply Pep8-formatting * Fix path * Adapt oem * GitHub Action: Apply Pep8-formatting * Modify whole config.yaml * Adapt online_vprm * Work on int2lm, some restructuring * GitHub Action: Apply Pep8-formatting * Move times to config file * GitHub Action: Apply Pep8-formatting * Further work on int2lm * GitHub Action: Apply Pep8-formatting * Finish int2lm * GitHub Action: Apply Pep8-formatting * Fix post_int2lm * GitHub Action: Apply Pep8-formatting * Use correct config variable in post_int2lm * Work on cosmo job/config * GitHub Action: Apply Pep8-formatting * Make cosmo job run * GitHub Action: Apply Pep8-formatting * Fix hstart and hstop for spinup * GitHub Action: Apply Pep8-formatting * Fix post_cosmo and add functions to set config variables * GitHub Action: Apply Pep8-formatting * Rename cosmo-ghg test case and reduce simulation time * Minor fixes for other jobs * Fix cosmo job * Remove config.py * Change time interval * Remove cosmo_spec * Fix inidates for spinup runs * GitHub Action: Apply Pep8-formatting * Set correct inidate for int2lm job * Fix and unify post_int2lm * GitHub Action: Apply Pep8-formatting * Fix hstart and hstop for spinup runs * Remove unnecessary inidate variables * GitHub Action: Apply Pep8-formatting * Add cosmo-ghg restart test case * Create separate inidate variable for int2lm * GitHub Action: Apply Pep8-formatting * Work on icon case * Further work on ICON and make job_id and chain_root config variables * GitHub Action: Apply Pep8-formatting * Input files for ICON * GitHub Action: Apply Pep8-formatting * Add class function for easy-access variables * Further work on ICON * GitHub Action: Apply Pep8-formatting * Print config variables before duplicating dict variables * ICON: prepare_data * Remove icon_spec * Add class function to convert relative to absolute paths * Further work on prepare_data * GitHub Action: Apply Pep8-formatting * Add prefix to meteo nameformat * GitHub Action: Apply Pep8-formatting * Fix time variable for xarray * GitHub Action: Apply Pep8-formatting * Remove print * Further work on icon job * GitHub Action: Apply Pep8-formatting * Better naming for variables and restart setting * GitHub Action: Apply Pep8-formatting * Add GEOSP file for restart simulations * GitHub Action: Apply Pep8-formatting * Set procs for restart etc. in config * Set correct endtime in prepare_data * Set correct nodes for icon (cpu + gpu) * Fix error * Further work on icon restart * GitHub Action: Apply Pep8-formatting * Fix restart and finish icon-test case * Remove print * Update jenkins tests * Fix cosmo-ghg test * fix int2lm job script * Fix for GEOSP data copy * Fix jobid date * GitHub Action: Apply Pep8-formatting * Add ISO 8601 duration converter * Fix icon test jenkins files * GitHub Action: Apply Pep8-formatting * Use startdate and enddate in iter_hours * Merge init * Work on startdate, enddate, etc. * GitHub Action: Apply Pep8-formatting * Some cleanup for startdates * GitHub Action: Apply Pep8-formatting * Set jobid based on enddate * Provide restart step in iso format (icon-test) * GitHub Action: Apply Pep8-formatting * Adapt run_chain for enddate * GitHub Action: Apply Pep8-formatting * Remove obsolete time occurences * GitHub Action: Apply Pep8-formatting * Fix last commit * GitHub Action: Apply Pep8-formatting * Set missing variables * Remove hstop in prepare_data * GitHub Action: Apply Pep8-formatting * Adapt namelist templates * fix for namelist templates * Remove config.py * Further fixes * Adapt main function parameters for all jobs * Adapt function parameters for other jobs and cfg_variables * Correct function arguments when called * Fix icon job * GitHub Action: Apply Pep8-formatting * Don't pass startdate and enddate anymore * GitHub Action: Apply Pep8-formatting * Further removal of startdate and enddate * GitHub Action: Apply Pep8-formatting * Remove check at the end of int2lm runscript * Sync int2lm scripts * Remove unused imports * Fix cosmo-ghg jobs * GitHub Action: Apply Pep8-formatting * Modify docstring for iter_hours function * Fix chain for spinup * Remove print from post_int2lm * Fix spinup * Fix for second simulation in spinup * GitHub Action: Apply Pep8-formatting * Some docstring cleanup --------- Co-authored-by: github-actions <[email protected]> * Add icon-art-oem-test (#40) * Add config.yaml * Replace inidate with startdate_sim * Fix icontools scripts * Fix chem file names * Set correct input file names * Small fix for latbc grid file name * Add prefix to chem file * GitHub Action: Apply Pep8-formatting * Fix for chem file * GitHub Action: Apply Pep8-formatting * More corrections for chem file * GitHub Action: Apply Pep8-formatting * Fix art namelist * Fix art input folder * Update on how to build icon(-art) * Update .gitignore * Correcting case name in comment * Use config variables for num_procs in namelist * Fix ICON namelists * Remove config.py * Only process initial file if present * GitHub Action: Apply Pep8-formatting * Add start of icontools runscript to logger --------- Co-authored-by: github-actions <[email protected]> * Add icon-art-global-test (#41) * Add config.yaml and modify icon_runjob * Change variables to scratch input files * Further modify variable names * GitHub Action: Apply Pep8-formatting * Fix restart file and disable create_oh_for_restart() * GitHub Action: Apply Pep8-formatting * Full log from runscript --------- Co-authored-by: github-actions <[email protected]> * GitHub Pages (#44) * Initial setup * Deploy only rc branch * Modify docs.yml * New preview action * Cleanup * GitHub Action: Apply Pep8-formatting * Remove PSphinxTheme * Deploy preview for PR 43 🛫 * Fix docs.yml * Remove pr-preview * Remove autodoc * GitHub Action: Apply Pep8-formatting * Select correct branch * Remvoe autofunctions * Formatting --------- Co-authored-by: github-actions <[email protected]> Co-authored-by: mjaehn <[email protected]> * Update docs.yml * Add resume option and update docstrings in run_chain.py (#42) * Add resume option * GitHub Action: Apply Pep8-formatting * Add and modify docstrings in run_chain.py * Change some wording --------- Co-authored-by: github-actions <[email protected]> * Update README (#45) * Remove readthedocs yaml file * Update README * Fix link * Deploy docs to latest and tags (#47) * Deploy docs to latest and tags * Fix syntax error --------- Co-authored-by: github-actions <[email protected]> Co-authored-by: mjaehn <[email protected]>
C2SM · Nov 11, 2023 · 717f2c9 · 717f2c9
1 parent bfaaee4
commit 717f2c9
Show file tree

Hide file tree

Showing 127 changed files with 2,772 additions and 2,146 deletions.
diff --git a/.github/workflows/docs-latest.yml b/.github/workflows/docs-latest.yml
@@ -0,0 +1,27 @@
+name: Build and Deploy Documentation to Latest
+env:
+  DOCS_VERSION: latest
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  deploy-docs:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+      - name: Build docs
+        uses: C2SM/sphinx-action@sphinx-latest
+        with:
+          pre-build-command: "pip install sphinx_rtd_theme && pip install sphinx-copybutton"
+          build-command: "sphinx-build -b html . _build"
+          docs-folder: "docs/"
+      - uses: peaceiris/actions-gh-pages@v3
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ./docs/_build
+          destination_dir: ${{ env.DOCS_VERSION }}
+          allow_empty_commit: true
diff --git a/.github/workflows/docs-pr-preview.yml b/.github/workflows/docs-pr-preview.yml
@@ -0,0 +1,32 @@
+name: Deploy PR previews
+
+on:
+  pull_request:
+    types:
+      - opened
+      - reopened
+      - synchronize
+      - closed
+
+concurrency: preview-${{ github.ref }}
+
+jobs:
+  deploy-preview:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+      - name: Build docs
+        uses: C2SM/sphinx-action@sphinx-latest
+        with:
+          pre-build-command: "pip install sphinx_rtd_theme && pip install sphinx-copybutton"
+          build-command: "sphinx-build -b html . _build"
+          docs-folder: "docs/"
+
+      - name: Deploy preview
+        uses: rossjrw/pr-preview-action@v1
+        with:
+          source-dir: docs/_build
+          preview-branch: gh-pages
+          umbrella-dir: pr-preview
+          action: auto
diff --git a/.github/workflows/docs-tag.yml b/.github/workflows/docs-tag.yml
@@ -0,0 +1,30 @@
+name: Build and Deploy Documentation to Tags
+
+on:
+  push:
+    tags:
+      - '*'
+
+jobs:
+  deploy-docs:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+      - name: Get release
+        id: get_release
+        uses: bruceadams/[email protected]
+        env: 
+          GITHUB_TOKEN: ${{ github.token }}
+      - name: Build docs
+        uses: C2SM/sphinx-action@sphinx-latest
+        with:
+          pre-build-command: "pip install sphinx_rtd_theme && pip install sphinx-copybutton"
+          build-command: "sphinx-build -b html . _build"
+          docs-folder: "docs/"
+      - uses: peaceiris/actions-gh-pages@v3
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ./docs/_build
+          destination_dir: ${{ steps.get_release.outputs.tag_name }}
+          allow_empty_commit: true
diff --git a/.gitignore b/.gitignore
@@ -7,3 +7,4 @@ input/
 output/
 work/
 src/*/
+*.code-workspace
diff --git a/.readthedocs.yaml b/.readthedocs.yaml
diff --git a/README.md b/README.md
@@ -2,7 +2,7 @@
 
 The Processing Chain is a python script that prepares necessary input
 data, submits compute-jobs to the queue on Piz Daint and does
-post-processing steps. It supports different types of simulations,
+post-processing steps. It supports different types of models and simulation types,
 including **COSMO**, **COSMO-GHG**, **COSMO-ART**, **ICON** and
 **ICON-ART**. The chain can flexibly be adapted according to your needs,
 e.g., by creating your own case or adding new jobs.
@@ -12,7 +12,8 @@ e.g., by creating your own case or adding new jobs.
 The following steps allow you to create and use your own virtual
 environment to run the Processing Chain. We recommend to use a conda
 environment for the usage of the provided scripts. Please follow the
-instruction for the installation.
+instruction for the installation. The following steps only need to be 
+performed once.
 
 ### 1\. Install Miniconda
 
@@ -57,6 +58,10 @@ these files within your home directory:
     echo <your_account_id> > ~/.acct
     echo <your_email_address> > ~/.forward
 
+These settings are optional. The Processing Chain will first check the content
+of those files. If desired, the corresponding variables can be overridden by setting
+the `compute_account` and `user_mail` variables in the `config.yaml` file.
+
 ## Run the Chain
 
 Once everything has been set up correctly according to the above steps,
@@ -66,20 +71,21 @@ environment (if not done already):
     conda activate proc-chain
 
 To test if your environment has been successfully set, use the command
-line help to see the available arguments for the main script:
+line help to display the available arguments for the main script:
 
     python run_chain.py -h
 
 To run the test cases with their standard jobs, please ensure
 that you clone the Processing Chain to `$SCRATCH`, as input and
-output data are stored in subdirectories of the chain itself.
+output data are stored in subdirectories of the Processing Chain repository
+itself.
 
 > **Note**: For your own setups, you can use the Processing Chain
 > on a backed-up file system like `/project` or `/store`. In that case,
-> adapt the configuration file `config.py` in your case folder in
-> a way that output files are written to a specified folder on `$SCRATCH`.
+> adapt the configuration file `config.yaml` in your case folder so that
+> the output files are written to a specified folder on `$SCRATCH`.
 
-For these test cases, you can use the Jenkins script
+For these pre-defined test cases, you can use the Jenkins script
 
     ./jenkins/scripts/jenkins.sh
 
@@ -89,26 +95,25 @@ They will
 - setup spack-c2sm
 - download input data to `input/`
 - build `int2lm`, `cosmo-ghg`, `icon` and `icon-art`
-- test `cosmo-ghg`, `icon`, `icon-art-oem`, `icon-art-global`
+- test the following cases:
+    - `cosmo-ghg-spinup-test`
+    - `cosmo-ghg-test`
+    - `icon-test`
+    - `icon-art-oem-test`
+    - `icon-art-global-test`
 
 To run the test cases manually, type:
 
 ```bash
-    # COSMO-GHG
-    python run_chain.py cosmo-ghg-11km-test 2015-01-01 0 24
-    # ICON
-    python run_chain.py icon-test 2018-01-01 0 24 -j prepare_data icon
-    # ICON-ART (OEM)
-    python run_chain.py icon-art-oem-test 2018-01-01 0 24 -j prepare_data oae icon
-    # ICON-ART (global)
-    python run_chain.py icon-art-global-test 2018-01-01 0 24 -j prepare_data oae icon
+    # replace <casename> with one of the above tests
+    python run_chain.py <casename>
 ```
 
 ## Documentation
 
-To set up your own simulation case or to check all available
-configuration options, please read the official
-[documentation](https://processing-chain.readthedocs.io).
+For more information about the file structure, configuration options,
+namelist templates etc., please read the official
+[documentation](https://c2sm.github.io/processing-chain/).
 
 ## Contributing
 
@@ -117,7 +122,7 @@ others, we encourage you to create a pull request for this repository.
 
 ## Credits
 
-The Processing Chain was originally developed at
+The Processing Chain was originally developed in 2018 at
 [Empa](https://www.empa.ch) by the [Atmospheric Modeling and Remote
 Sensing](https://www.empa.ch/web/s503/modelling-remote-sensing) group.
 The following persons contributed significantly to the initial
@@ -131,4 +136,7 @@ development (in alphabetic order):
   - David Ochsner
   - Michael Steiner
 
+Since 2021, the code is public and hosted by C2SM. More information can
+be found at the
+[C2SM User Landing Page](https://c2sm.github.io/tools/processing_chain.html).
 The current code owner is Michael Jähn (<[email protected]>).