Processing Chain v3.1 #62
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Remove `variants` key and use `features` instead (#30) * Remove variants from models.yaml * Remove spinup variant and use the spinup variable instead * GitHub Action: Apply Pep8-formatting * Remove variant check * Install specific icontools version --------- Co-authored-by: github-actions <[email protected]> * Convert config files to yaml format (#36) * Remove variants from models.yaml * Remove spinup variant and use the spinup variable instead * GitHub Action: Apply Pep8-formatting * Remove variant check * Install specific icontools version * Starting to work on cosmo case * GitHub Action: Apply Pep8-formatting * GitHub Action: Apply Pep8-formatting * Add cases and paths * Create cfg object from Config class * GitHub Action: Apply Pep8-formatting * Format output of configuration * Add email and simplify print function * GitHub Action: Apply Pep8-formatting * Improve output * GitHub Action: Apply Pep8-formatting * Make cams a dict and adapt prepare_data * GitHub Action: Apply Pep8-formatting * Adapt emissions * GitHub Action: Apply Pep8-formatting * Adapt biofluxes * Change user variable naming * GitHub Action: Apply Pep8-formatting * Fix path * Adapt oem * GitHub Action: Apply Pep8-formatting * Modify whole config.yaml * Adapt online_vprm * Work on int2lm, some restructuring * GitHub Action: Apply Pep8-formatting * Move times to config file * GitHub Action: Apply Pep8-formatting * Further work on int2lm * GitHub Action: Apply Pep8-formatting * Finish int2lm * GitHub Action: Apply Pep8-formatting * Fix post_int2lm * GitHub Action: Apply Pep8-formatting * Use correct config variable in post_int2lm * Work on cosmo job/config * GitHub Action: Apply Pep8-formatting * Make cosmo job run * GitHub Action: Apply Pep8-formatting * Fix hstart and hstop for spinup * GitHub Action: Apply Pep8-formatting * Fix post_cosmo and add functions to set config variables * GitHub Action: Apply Pep8-formatting * Rename cosmo-ghg test case and reduce simulation time * Minor fixes for other jobs * Fix cosmo job * Remove config.py * Change time interval * Remove cosmo_spec * Fix inidates for spinup runs * GitHub Action: Apply Pep8-formatting * Set correct inidate for int2lm job * Fix and unify post_int2lm * GitHub Action: Apply Pep8-formatting * Fix hstart and hstop for spinup runs * Remove unnecessary inidate variables * GitHub Action: Apply Pep8-formatting * Add cosmo-ghg restart test case * Create separate inidate variable for int2lm * GitHub Action: Apply Pep8-formatting * Work on icon case * Further work on ICON and make job_id and chain_root config variables * GitHub Action: Apply Pep8-formatting * Input files for ICON * GitHub Action: Apply Pep8-formatting * Add class function for easy-access variables * Further work on ICON * GitHub Action: Apply Pep8-formatting * Print config variables before duplicating dict variables * ICON: prepare_data * Remove icon_spec * Add class function to convert relative to absolute paths * Further work on prepare_data * GitHub Action: Apply Pep8-formatting * Add prefix to meteo nameformat * GitHub Action: Apply Pep8-formatting * Fix time variable for xarray * GitHub Action: Apply Pep8-formatting * Remove print * Further work on icon job * GitHub Action: Apply Pep8-formatting * Better naming for variables and restart setting * GitHub Action: Apply Pep8-formatting * Add GEOSP file for restart simulations * GitHub Action: Apply Pep8-formatting * Set procs for restart etc. in config * Set correct endtime in prepare_data * Set correct nodes for icon (cpu + gpu) * Fix error * Further work on icon restart * GitHub Action: Apply Pep8-formatting * Fix restart and finish icon-test case * Remove print * Update jenkins tests * Fix cosmo-ghg test * fix int2lm job script * Fix for GEOSP data copy * Fix jobid date * GitHub Action: Apply Pep8-formatting * Add ISO 8601 duration converter * Fix icon test jenkins files * GitHub Action: Apply Pep8-formatting * Modify comment in config file to match case --------- Co-authored-by: github-actions <[email protected]> * Use `cfg.enddate` instead of `hstart` and `hstop` (#39) * Remove variants from models.yaml * Remove spinup variant and use the spinup variable instead * GitHub Action: Apply Pep8-formatting * Remove variant check * Install specific icontools version * Starting to work on cosmo case * GitHub Action: Apply Pep8-formatting * GitHub Action: Apply Pep8-formatting * Add cases and paths * Create cfg object from Config class * GitHub Action: Apply Pep8-formatting * Format output of configuration * Add email and simplify print function * GitHub Action: Apply Pep8-formatting * Improve output * GitHub Action: Apply Pep8-formatting * Make cams a dict and adapt prepare_data * GitHub Action: Apply Pep8-formatting * Adapt emissions * GitHub Action: Apply Pep8-formatting * Adapt biofluxes * Change user variable naming * GitHub Action: Apply Pep8-formatting * Fix path * Adapt oem * GitHub Action: Apply Pep8-formatting * Modify whole config.yaml * Adapt online_vprm * Work on int2lm, some restructuring * GitHub Action: Apply Pep8-formatting * Move times to config file * GitHub Action: Apply Pep8-formatting * Further work on int2lm * GitHub Action: Apply Pep8-formatting * Finish int2lm * GitHub Action: Apply Pep8-formatting * Fix post_int2lm * GitHub Action: Apply Pep8-formatting * Use correct config variable in post_int2lm * Work on cosmo job/config * GitHub Action: Apply Pep8-formatting * Make cosmo job run * GitHub Action: Apply Pep8-formatting * Fix hstart and hstop for spinup * GitHub Action: Apply Pep8-formatting * Fix post_cosmo and add functions to set config variables * GitHub Action: Apply Pep8-formatting * Rename cosmo-ghg test case and reduce simulation time * Minor fixes for other jobs * Fix cosmo job * Remove config.py * Change time interval * Remove cosmo_spec * Fix inidates for spinup runs * GitHub Action: Apply Pep8-formatting * Set correct inidate for int2lm job * Fix and unify post_int2lm * GitHub Action: Apply Pep8-formatting * Fix hstart and hstop for spinup runs * Remove unnecessary inidate variables * GitHub Action: Apply Pep8-formatting * Add cosmo-ghg restart test case * Create separate inidate variable for int2lm * GitHub Action: Apply Pep8-formatting * Work on icon case * Further work on ICON and make job_id and chain_root config variables * GitHub Action: Apply Pep8-formatting * Input files for ICON * GitHub Action: Apply Pep8-formatting * Add class function for easy-access variables * Further work on ICON * GitHub Action: Apply Pep8-formatting * Print config variables before duplicating dict variables * ICON: prepare_data * Remove icon_spec * Add class function to convert relative to absolute paths * Further work on prepare_data * GitHub Action: Apply Pep8-formatting * Add prefix to meteo nameformat * GitHub Action: Apply Pep8-formatting * Fix time variable for xarray * GitHub Action: Apply Pep8-formatting * Remove print * Further work on icon job * GitHub Action: Apply Pep8-formatting * Better naming for variables and restart setting * GitHub Action: Apply Pep8-formatting * Add GEOSP file for restart simulations * GitHub Action: Apply Pep8-formatting * Set procs for restart etc. in config * Set correct endtime in prepare_data * Set correct nodes for icon (cpu + gpu) * Fix error * Further work on icon restart * GitHub Action: Apply Pep8-formatting * Fix restart and finish icon-test case * Remove print * Update jenkins tests * Fix cosmo-ghg test * fix int2lm job script * Fix for GEOSP data copy * Fix jobid date * GitHub Action: Apply Pep8-formatting * Add ISO 8601 duration converter * Fix icon test jenkins files * GitHub Action: Apply Pep8-formatting * Use startdate and enddate in iter_hours * Merge init * Work on startdate, enddate, etc. * GitHub Action: Apply Pep8-formatting * Some cleanup for startdates * GitHub Action: Apply Pep8-formatting * Set jobid based on enddate * Provide restart step in iso format (icon-test) * GitHub Action: Apply Pep8-formatting * Adapt run_chain for enddate * GitHub Action: Apply Pep8-formatting * Remove obsolete time occurences * GitHub Action: Apply Pep8-formatting * Fix last commit * GitHub Action: Apply Pep8-formatting * Set missing variables * Remove hstop in prepare_data * GitHub Action: Apply Pep8-formatting * Adapt namelist templates * fix for namelist templates * Remove config.py * Further fixes * Adapt main function parameters for all jobs * Adapt function parameters for other jobs and cfg_variables * Correct function arguments when called * Fix icon job * GitHub Action: Apply Pep8-formatting * Don't pass startdate and enddate anymore * GitHub Action: Apply Pep8-formatting * Further removal of startdate and enddate * GitHub Action: Apply Pep8-formatting * Remove check at the end of int2lm runscript * Sync int2lm scripts * Remove unused imports * Fix cosmo-ghg jobs * GitHub Action: Apply Pep8-formatting * Modify docstring for iter_hours function * Fix chain for spinup * Remove print from post_int2lm * Fix spinup * Fix for second simulation in spinup * GitHub Action: Apply Pep8-formatting * Some docstring cleanup --------- Co-authored-by: github-actions <[email protected]> * Add icon-art-oem-test (#40) * Add config.yaml * Replace inidate with startdate_sim * Fix icontools scripts * Fix chem file names * Set correct input file names * Small fix for latbc grid file name * Add prefix to chem file * GitHub Action: Apply Pep8-formatting * Fix for chem file * GitHub Action: Apply Pep8-formatting * More corrections for chem file * GitHub Action: Apply Pep8-formatting * Fix art namelist * Fix art input folder * Update on how to build icon(-art) * Update .gitignore * Correcting case name in comment * Use config variables for num_procs in namelist * Fix ICON namelists * Remove config.py * Only process initial file if present * GitHub Action: Apply Pep8-formatting * Add start of icontools runscript to logger --------- Co-authored-by: github-actions <[email protected]> * Add icon-art-global-test (#41) * Add config.yaml and modify icon_runjob * Change variables to scratch input files * Further modify variable names * GitHub Action: Apply Pep8-formatting * Fix restart file and disable create_oh_for_restart() * GitHub Action: Apply Pep8-formatting * Full log from runscript --------- Co-authored-by: github-actions <[email protected]> * GitHub Pages (#44) * Initial setup * Deploy only rc branch * Modify docs.yml * New preview action * Cleanup * GitHub Action: Apply Pep8-formatting * Remove PSphinxTheme * Deploy preview for PR 43 🛫 * Fix docs.yml * Remove pr-preview * Remove autodoc * GitHub Action: Apply Pep8-formatting * Select correct branch * Remvoe autofunctions * Formatting --------- Co-authored-by: github-actions <[email protected]> Co-authored-by: mjaehn <[email protected]> * Update docs.yml * Add resume option and update docstrings in run_chain.py (#42) * Add resume option * GitHub Action: Apply Pep8-formatting * Add and modify docstrings in run_chain.py * Change some wording --------- Co-authored-by: github-actions <[email protected]> * Update README (#45) * Remove readthedocs yaml file * Update README * Fix link * Deploy docs to latest and tags (#47) * Deploy docs to latest and tags * Fix syntax error * Edit intro text * Reduce toc depth * Edit file structure * Futher modify file structure * Add environment section and move file structure * Add environment info * Format note * Add features * Add logo to README * Shorten title * Add logo to docs * Fix logo * Add favicon * Use small logo for docs * Add features * Update README.md * Update README * Add 'v' to version * Update README * Work on how-to-run * Add autodoc * Add syspath * GitHub Action: Apply Pep8-formatting * Add more paths * Test * Small fixes * Enable local html build * Add html baseurl * Remove baseurl * Install conda env before building docs * Fix for pr-preview * Update docs-pr-preview.yml * Update docs-pr-preview.yml * Update docs-pr-preview.yml * Update docs-pr-preview.yml * Adapt requirements files * Fix requirement file * Update gitignore * Remove python version * Remove shell env * Unify docs actions * Fix listing and remove autofunction * Add autofunctinos * Show command-line arguments * Add autofunctions * Update file structure * Fix indentation * Another small fix * Add lines * Rename src to externals * Add test cases section * Fix tree formatting * Remove obsolte file * Revise code structure * Update index for code structure * Add general table * Add more examples * Correct heading size * Fix typo * Add info for variables in namelist templates * List dict variables and job occurences * GitHub Action: Apply Pep8-formatting * Fix table * Edit namelist section * Fix colons * Fix autofunctions * Fix table * Settings for autofunctions * GitHub Action: Apply Pep8-formatting * Settings for autofunctions * Settings for toctree * Revert last commit * Add variable information * Fix table * Fix typo * Minor corrections * Minor corrections * Update jobs * Sort imports alphabetically * Update docstring for biofluxes * Rename externals -> ext * Update docstring for biofluxes * Update docstring for check_output * Update docstring for cosmo * Update docstring for emissions * Remove headers * Update docstring for icon * Update docstring for int2lm * Update docstring for obs_nudging and octe * Remove headers * Update docstring for oem * Update docstring for online_vprm * Update docstring for photo_rate * Update docstring for post_cosmo * Update docstring for post_int2lm * Update docstring for prepare_date * Update docstring for reduce_output * Update docstring for verify_chain * Change src to ext directory in scripts and config files * Add tools * Add autofunctions * Fix formatting * Fix formatting * Remove non-existing tool functions * Fix tool function * Change structure * Fix functions * restructuring * Unify docstrings * Fix directory structure formatting * Update README.md * Remove variant from config file and docs --------- Co-authored-by: github-actions <[email protected]> Co-authored-by: mjaehn <[email protected]>
* Added ICON interpolator (to point locations) * GitHub Action: Apply Pep8-formatting * Changes after review * GitHub Action: Apply Pep8-formatting --------- Co-authored-by: github-actions <[email protected]>
* GitHub Action: Apply Pep8-formatting * Small fix * GitHub Action: Apply Pep8-formatting * Format duration * Remove --wait from submit function * Let geosp be a separate job * GitHub Action: Apply Pep8-formatting * Add run_chain argument to process specified chunk * GitHub Action: Apply Pep8-formatting * Fix datetime comparison in int2lm * Submit all jobs as sbatch * GitHub Action: Apply Pep8-formatting * Use existing submit() function and only return job_id * GitHub Action: Apply Pep8-formatting * Fix prepare_data * Disable logging for individual jobs * GitHub Action: Apply Pep8-formatting * Write casename into logfile * Fix timers for jobs * GitHub Action: Apply Pep8-formatting * Use UTC times * GitHub Action: Apply Pep8-formatting * More async jobs and workflows * GitHub Action: Apply Pep8-formatting * Add logging to new jobs * GitHub Action: Apply Pep8-formatting * Remove unused imports * Remove global icon-art from prepare_data * GitHub Action: Apply Pep8-formatting * Replace os.path with Pathlib * GitHub Action: Apply Pep8-formatting * Fix for PosixPaths * ADD: convenience function for slurm job info * REF: move get_job_info method to the end of the class definition * GitHub Action: Apply Pep8-formatting * Add some docstrings * Fix quotes for f-strings * Add imports for art-global job * Add dependencies for all icon workflows * Fix pathname * GitHub Action: Apply Pep8-formatting * Fix chunk calculation * Add icontools job * GitHub Action: Apply Pep8-formatting * Add icontools to dependencies * Modify docstrings * Some code cleanup * fix(icontools): remove unsused packages + fix dependency logic - The `icontools` job already depends on `prepapre_data`, no need to add a `copy_id` in the dependencies (which was anyways undefinied in this scope). - The name of the first argument of `cfg.submit()` must be 'icontools' so that all the sub job ids are associated with the icontools job and are taken into account for other jobs depending on it (icon). Otherwise, icon will find no dependency in `cfg.job_ids['current']['icontools']`. - `cfg.get_job_info()` should be used later in `run_chain.py` to monitor time for all async jobs. Currently the reported time is only the submission time. * Make prepare_art_global dependent on previous icon * Fix import * Explicitly set async variable * Add logfile for icontools * Set variables for icontools * Revert "Explicitly set async variable" This reverts commit 99b96cf. * Define sequential workflow in icon-seq-test * Make cosmo-ghg workflow async * Add info about (a)sync mode * fix: also launch waiting job when some workflow jobs failed * add: other keys to default job info dict * fix: empty current job ids before each chunk * GitHub Action: Apply Pep8-formatting * Split prepare_data into cosmo and icon * Remove unused imports * GitHub Action: Apply Pep8-formatting * Rename prepare_data job in workflows * Rename prepare_data in icon-seq case * Set additional cfg variables in prepare_icon * GitHub Action: Apply Pep8-formatting * Time logging for all jobs * Merge geosp into prepare_art * GitHub Action: Apply Pep8-formatting * Include oem in icon-art-oem workflow * Fix icon job * Move geosp to icontools job * GitHub Action: Apply Pep8-formatting * add: placeholder for monitoring async jobs * Run geosp after icontools * Remove oem job from icon-art-oem case * Change to current logfile within jobs * GitHub Action: Apply Pep8-formatting * Add missing log inits * Configure root logger * set logger * Fix logger * Format logging output * Introduce BASIC_PYTHON_JOB option to call jobs directly in async mode * GitHub Action: Apply Pep8-formatting * add: only submit basic python jobs through a nested run_chain * GitHub Action: Apply Pep8-formatting * fix: loop over jobs in run_chunk * ref: job_id becomes chunk_id reflects reality and avoids confusion with actual jobs id * GitHub Action: Apply Pep8-formatting * fix: leftover `job_id` -> `chunk_id` * GitHub Action: Apply Pep8-formatting * add(untested): Slurm monitoring * GitHub Action: Apply Pep8-formatting * Add BASIC_PYHTON_JOB to missing jobs * Fix function arguments * Small fix and hint to KeyError * GitHub Action: Apply Pep8-formatting * ref(slurm summary): streamline code a bit * GitHub Action: Apply Pep8-formatting * Some settings for cosmo-ghg case * afterany -> afterok for wait job * Comment some function calls * add:ref: remove unused `info_requests` + print failing jobs * fix: job summary for previous chunk, not current * fix: reactivate empty current job ids at beginning of chunk * fix: only wait and monitor if not very first chunk * GitHub Action: Apply Pep8-formatting * ref: move icon-art error handling in slurm job itself * fix: remove BASIC_PYTHON_JOB workaround for icon * fix(icon.py): escape curly brackets for string formatting * fix(icon): escape curly brackets in python way * Add walltime and remove conda activation * GitHub Action: Apply Pep8-formatting * Set walltime correctly * Fix call to handle_error function * Set walltimes for icon workflows * Debug * Remove job_ids override * ref(config): clean up * Add smaller walltime to wait job * Remove prints * Some fixes for cosmo case * Remove model check * Don't pass logfile anymore * GitHub Action: Apply Pep8-formatting * Submit int2lm and cosmo jobs correctly * GitHub Action: Apply Pep8-formatting * Small fixes for int2lm and cosmo * Store job scripts in separate directory * Add BASIC_PYTHON_JOB to int2lm * GitHub Action: Apply Pep8-formatting * Add post_cosmo dependency * Bugfixes for cosmo jobs * Further fixes * Make post_cosmo a submit job * Fix post_cosmo * Fix post_cosmo * ref: refactor cycling and monitoring - Regroup waiting, monitoring nd cycling in a single `Config.cycle()` method. - write chunk monitoring info into chain log file * GitHub Action: Apply Pep8-formatting * Fix config variables * Fix log file output * Formatting * Remove time logging from jobs * GitHub Action: Apply Pep8-formatting * Fix for icon job * GitHub Action: Apply Pep8-formatting * ref: exception handling in Config.submit() * Change table cell widths * Remove old way of logging * Cleanup * NNodes -> N * GitHub Action: Apply Pep8-formatting * Fix for N/NNodes * GitHub Action: Apply Pep8-formatting * Yet another fix for N/NNodes * Simplify job names * Change missing job names * Fix syntax error from commit 8810dcf * GitHub Action: Apply Pep8-formatting * Just jobname for jobs * Cleaner console output * GitHub Action: Apply Pep8-formatting * Unify restart and spinup runs * GitHub Action: Apply Pep8-formatting * Complete config file for spinup test * Fix for spinup * GitHub Action: Apply Pep8-formatting * Fix 2 for spinup * Fancy formatting * Define separate spinup workflow * Fix formatting (hopefully) * Aligning case + workflow * Custom workflow_name and improve check * Remove workflow_name check in jobs * GitHub Action: Apply Pep8-formatting * Bugfix in cosmo job * Remove restart info * Compute chunks separately * GitHub Action: Apply Pep8-formatting * Add function to get previous chunk ID * GitHub Action: Apply Pep8-formatting * Directly get previous chunk id * Further cleanup and refactoring * GitHub Action: Apply Pep8-formatting * Fix for chunk_id_prev * Fix * Save total chunk list * Fix restart variables * Don't print chunk list twice * Fix for cosmo_restart_out * Remove sequential part * GitHub Action: Apply Pep8-formatting * Remove sequential case * Add dependencies to all workflows * Remove is_async config variable * Incorporate review * GitHub Action: Apply Pep8-formatting * Fix for basename * Add --wait again for seq. jobs in nested run_chain * Fixes for spinup * GitHub Action: Apply Pep8-formatting * ref: remove unnecessary if levels * del: remove unused imports * ref(basic python jobs): merge generation of script and submission The create_sbatch_script method is used nowhere else => No need to seperate generation of job script from its submission --------- Co-authored-by: github-actions <[email protected]> Co-authored-by: Matthieu <[email protected]>
* Use config instead of local variables * Add logging for whole chain * GitHub Action: Apply Pep8-formatting * Add logging for jobs * GitHub Action: Apply Pep8-formatting * Small fix * GitHub Action: Apply Pep8-formatting * Format duration * Remove --wait from submit function * Let geosp be a separate job * GitHub Action: Apply Pep8-formatting * Add run_chain argument to process specified chunk * GitHub Action: Apply Pep8-formatting * Fix datetime comparison in int2lm * Submit all jobs as sbatch * GitHub Action: Apply Pep8-formatting * Use existing submit() function and only return job_id * GitHub Action: Apply Pep8-formatting * Fix prepare_data * Disable logging for individual jobs * GitHub Action: Apply Pep8-formatting * Write casename into logfile * Fix timers for jobs * GitHub Action: Apply Pep8-formatting * Use UTC times * GitHub Action: Apply Pep8-formatting * More async jobs and workflows * GitHub Action: Apply Pep8-formatting * Add logging to new jobs * GitHub Action: Apply Pep8-formatting * Remove unused imports * Remove global icon-art from prepare_data * GitHub Action: Apply Pep8-formatting * Replace os.path with Pathlib * GitHub Action: Apply Pep8-formatting * Fix for PosixPaths * ADD: convenience function for slurm job info * REF: move get_job_info method to the end of the class definition * GitHub Action: Apply Pep8-formatting * Add some docstrings * Fix quotes for f-strings * Add imports for art-global job * Add dependencies for all icon workflows * Fix pathname * GitHub Action: Apply Pep8-formatting * Fix chunk calculation * Add icontools job * GitHub Action: Apply Pep8-formatting * Add icontools to dependencies * Modify docstrings * Some code cleanup * fix(icontools): remove unsused packages + fix dependency logic - The `icontools` job already depends on `prepapre_data`, no need to add a `copy_id` in the dependencies (which was anyways undefinied in this scope). - The name of the first argument of `cfg.submit()` must be 'icontools' so that all the sub job ids are associated with the icontools job and are taken into account for other jobs depending on it (icon). Otherwise, icon will find no dependency in `cfg.job_ids['current']['icontools']`. - `cfg.get_job_info()` should be used later in `run_chain.py` to monitor time for all async jobs. Currently the reported time is only the submission time. * Make prepare_art_global dependent on previous icon * Fix import * Explicitly set async variable * Add logfile for icontools * Set variables for icontools * Revert "Explicitly set async variable" This reverts commit 99b96cf. * Define sequential workflow in icon-seq-test * Make cosmo-ghg workflow async * Add info about (a)sync mode * fix: also launch waiting job when some workflow jobs failed * add: other keys to default job info dict * fix: empty current job ids before each chunk * GitHub Action: Apply Pep8-formatting * Split prepare_data into cosmo and icon * Remove unused imports * GitHub Action: Apply Pep8-formatting * Rename prepare_data job in workflows * Rename prepare_data in icon-seq case * Set additional cfg variables in prepare_icon * GitHub Action: Apply Pep8-formatting * Time logging for all jobs * Merge geosp into prepare_art * GitHub Action: Apply Pep8-formatting * Include oem in icon-art-oem workflow * Fix icon job * Move geosp to icontools job * GitHub Action: Apply Pep8-formatting * add: placeholder for monitoring async jobs * Run geosp after icontools * Remove oem job from icon-art-oem case * Change to current logfile within jobs * GitHub Action: Apply Pep8-formatting * Add missing log inits * Configure root logger * set logger * Fix logger * Format logging output * Introduce BASIC_PYTHON_JOB option to call jobs directly in async mode * GitHub Action: Apply Pep8-formatting * add: only submit basic python jobs through a nested run_chain * GitHub Action: Apply Pep8-formatting * fix: loop over jobs in run_chunk * ref: job_id becomes chunk_id reflects reality and avoids confusion with actual jobs id * GitHub Action: Apply Pep8-formatting * fix: leftover `job_id` -> `chunk_id` * GitHub Action: Apply Pep8-formatting * add(untested): Slurm monitoring * GitHub Action: Apply Pep8-formatting * Add BASIC_PYHTON_JOB to missing jobs * Fix function arguments * Small fix and hint to KeyError * GitHub Action: Apply Pep8-formatting * ref(slurm summary): streamline code a bit * GitHub Action: Apply Pep8-formatting * Some settings for cosmo-ghg case * afterany -> afterok for wait job * Comment some function calls * add:ref: remove unused `info_requests` + print failing jobs * fix: job summary for previous chunk, not current * fix: reactivate empty current job ids at beginning of chunk * fix: only wait and monitor if not very first chunk * GitHub Action: Apply Pep8-formatting * ref: move icon-art error handling in slurm job itself * fix: remove BASIC_PYTHON_JOB workaround for icon * fix(icon.py): escape curly brackets for string formatting * fix(icon): escape curly brackets in python way * Add walltime and remove conda activation * GitHub Action: Apply Pep8-formatting * Set walltime correctly * Fix call to handle_error function * Set walltimes for icon workflows * Debug * Remove job_ids override * ref(config): clean up * Add smaller walltime to wait job * Remove prints * Some fixes for cosmo case * Remove model check * Don't pass logfile anymore * GitHub Action: Apply Pep8-formatting * Submit int2lm and cosmo jobs correctly * GitHub Action: Apply Pep8-formatting * Small fixes for int2lm and cosmo * Store job scripts in separate directory * Add BASIC_PYTHON_JOB to int2lm * GitHub Action: Apply Pep8-formatting * Add post_cosmo dependency * Bugfixes for cosmo jobs * Further fixes * Make post_cosmo a submit job * Fix post_cosmo * Fix post_cosmo * ref: refactor cycling and monitoring - Regroup waiting, monitoring nd cycling in a single `Config.cycle()` method. - write chunk monitoring info into chain log file * GitHub Action: Apply Pep8-formatting * Fix config variables * Fix log file output * Formatting * Remove time logging from jobs * GitHub Action: Apply Pep8-formatting * Fix for icon job * GitHub Action: Apply Pep8-formatting * ref: exception handling in Config.submit() * Change table cell widths * Remove old way of logging * Cleanup * NNodes -> N * GitHub Action: Apply Pep8-formatting * Fix for N/NNodes * GitHub Action: Apply Pep8-formatting * Yet another fix for N/NNodes * Simplify job names * Change missing job names * Fix syntax error from commit 8810dcf * GitHub Action: Apply Pep8-formatting * Just jobname for jobs * Cleaner console output * GitHub Action: Apply Pep8-formatting * Unify restart and spinup runs * GitHub Action: Apply Pep8-formatting * Complete config file for spinup test * Fix for spinup * GitHub Action: Apply Pep8-formatting * Fix 2 for spinup * Fancy formatting * Define separate spinup workflow * Fix formatting (hopefully) * Aligning case + workflow * Custom workflow_name and improve check * Remove workflow_name check in jobs * GitHub Action: Apply Pep8-formatting * Bugfix in cosmo job * Remove restart info * Compute chunks separately * GitHub Action: Apply Pep8-formatting * Add function to get previous chunk ID * GitHub Action: Apply Pep8-formatting * Directly get previous chunk id * Further cleanup and refactoring * GitHub Action: Apply Pep8-formatting * Fix for chunk_id_prev * Fix * Save total chunk list * Fix restart variables * Don't print chunk list twice * Fix for cosmo_restart_out * Remove sequential part * GitHub Action: Apply Pep8-formatting * Remove sequential case * Add dependencies to all workflows * Remove is_async config variable * Include renaming scheme for chemicals for OEM * GitHub Action: Apply Pep8-formatting * re-order in and output tracers * Shortened variable name * GitHub Action: Apply Pep8-formatting --------- Co-authored-by: Michael Jähn <[email protected]> Co-authored-by: github-actions <[email protected]> Co-authored-by: Matthieu <[email protected]>
* Edit features * CHange intro text * Add hint for running multiple cases * Update 'how to run' * Update code structure * Adapt config.rst * Further changes in config.rst * Add missing info * Small adaption in namelists * Adjust jobs to new async mode * Some minor corrections and additions * Revise intro text * model -> workflow * Format variable names in config * Fix table * Fix code block * Change some variable descriptions * Test code-block * Codeblock fix * Codeblock fix * Always use code-blocks * Add flowchart * Fix figure
|
launch jenkins |
|
|
launch jenkins |
1 similar comment
|
launch jenkins |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.