diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 01cf03781a..efaddc7a24 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -1,36 +1 @@ -How to contribute -================= - -We want Iris to be driven by the community - your contributions are -invaluable to us! This page lists the guidelines for contributors which -will help ease the process of getting your hard work accepted into Iris, -and shared back to the world. - -Getting started ---------------- - -1. If you've not already got one, sign up for a - [GitHub account](https://github.com/signup/free). -1. Fork the Iris repository, create your new fix/feature branch, and - start commiting code. - - Our - [development guide](https://scitools.org.uk/iris/docs/latest/developers_guide/index.html) - has more detail. -1. Make sure you've added appropriate tests, and that *all* the tests - pass. - - -Submitting changes ------------------- - -1. Read and sign the Contributor Licence Agreement (CLA). - - See our [governance page](http://scitools.org.uk/governance.html) - for the CLA and what to do with it. -1. Push your branch to your fork of Iris. -1. Submit your pull request. -1. Chillax. - - -If in doubt, please -[contact us](https://groups.google.com/forum/?fromgroups=#!forum/scitools-iris) -on our Google Group, and we'll be happy to help you. +See the [Developers Guide](https://scitools-iris.readthedocs.io/en/latest/index.html#development-index) \ No newline at end of file diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index c3e3c386a5..597c413457 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -6,4 +6,4 @@ --- -[Consult Iris pull request check list]( https://scitools-iris.readthedocs.io/en/latest/developers_guide/pulls.html#the-iris-check-list) +[Consult Iris pull request check list]( https://scitools-iris.readthedocs.io/en/latest/developers_guide/contributing_pull_request_checklist.html) diff --git a/docs/iris/src/common_links.inc b/docs/iris/src/common_links.inc new file mode 100644 index 0000000000..420186cf50 --- /dev/null +++ b/docs/iris/src/common_links.inc @@ -0,0 +1,24 @@ +.. _SciTools: https://github.com/SciTools +.. _Iris: https://github.com/SciTools/iris +.. _Iris GitHub: https://github.com/SciTools/iris +.. _iris mailing list: https://groups.google.com/forum/#!forum/scitools-iris +.. _issue: https://github.com/SciTools/iris/issues +.. _issues: https://github.com/SciTools/iris/issues +.. _pull request: https://github.com/SciTools/iris/pulls +.. _pull requests: https://github.com/SciTools/iris/pulls +.. _contributor: https://github.com/SciTools/scitools.org.uk/blob/master/contributors.json +.. _core developers: https://github.com/SciTools/scitools.org.uk/blob/master/contributors.json +.. _iris-test-data: https://github.com/SciTools/iris-test-data +.. _iris-sample-data: https://github.com/SciTools/iris-sample-data +.. _test-iris-imagehash: https://github.com/SciTools/test-iris-imagehash +.. _readthedocs.yml: https://github.com/SciTools/iris/blob/master/requirements/ci/readthedocs.yml +.. _travis-ci: https://travis-ci.org/github/SciTools/iris +.. _.travis.yml: https://github.com/SciTools/iris/blob/master/.travis.yml +.. _.stickler.yml: https://github.com/SciTools/iris/blob/master/.stickler.yml +.. _.flake8.yml: https://github.com/SciTools/iris/blob/master/.flake8 +.. _GitHub Help Documentation: https://docs.github.com/en/github +.. _using git: https://docs.github.com/en/github/using-git +.. _generating sss keys for GitHub: https://docs.github.com/en/github/authenticating-to-github/adding-a-new-ssh-key-to-your-github-account +.. _New Issue: https://github.com/scitools/iris/issues/new/choose +.. _matplotlib: https://matplotlib.org/ +.. _conda: https://docs.conda.io/en/latest/ diff --git a/docs/iris/src/conf.py b/docs/iris/src/conf.py index 9b061f5ec6..0af6d8fcce 100644 --- a/docs/iris/src/conf.py +++ b/docs/iris/src/conf.py @@ -243,6 +243,7 @@ def autolog(message): "https://github.com/SciTools/iris/commit/69597eb3d8501ff16ee3d56aef1f7b8f1c2bb316#diff-1680206bdc5cfaa83e14428f5ba0f848", "http://www.wmo.int/pages/prog/www/DPFS/documents/485_Vol_I_en_colour.pdf", "http://code.google.com/p/msysgit/downloads/list", + "http://schacon.github.com/git", ] # list of sources to exclude from the build. diff --git a/docs/iris/src/developers_guide/ci_checks.png b/docs/iris/src/developers_guide/ci_checks.png new file mode 100644 index 0000000000..cf93239dea Binary files /dev/null and b/docs/iris/src/developers_guide/ci_checks.png differ diff --git a/docs/iris/src/developers_guide/code_format.rst b/docs/iris/src/developers_guide/code_format.rst deleted file mode 100644 index c889146269..0000000000 --- a/docs/iris/src/developers_guide/code_format.rst +++ /dev/null @@ -1,28 +0,0 @@ -.. _iris_code_format: - -Code formatting -*************** - -To enforce a consistent code format throughout Iris, we recommend using `pre-commit `_ to run -`black `_ formatting and `flake8 `_ -linting automatically prior to each ``git commit``. - -Please install ``pre-commit`` in your development environment using ``pip``:: - - $ pip install pre-commit - -or alternatively using ``conda``:: - - $ conda install -c conda-forge pre-commit - -Then from the root directory of Iris run:: - - $ pre-commit install - -to install the ``pre-commit`` git hooks defined in our ``.pre-commit-config.yaml`` file. - -Upon performing a ``git commit``, your code will now be automatically formatted to the ``black`` configuration defined -in our ``pyproject.toml`` file, and linted according to our ``.flake8`` configuration file. Note that, ``pre-commit`` -will automatically download and install the necessary packages for each ``.pre-commit-config.yaml`` git hook. - -Additionally, you may wish to enable ``black`` for your preferred `editor/IDE `_. diff --git a/docs/iris/src/developers_guide/contributing_changes.rst b/docs/iris/src/developers_guide/contributing_changes.rst new file mode 100644 index 0000000000..a752986ec4 --- /dev/null +++ b/docs/iris/src/developers_guide/contributing_changes.rst @@ -0,0 +1,11 @@ + +.. _contributing.changes: + +Contributing your changes +========================= + +.. toctree:: + :maxdepth: 3 + + documenting/whats_new_contributions + contributing_pull_request_checklist diff --git a/docs/iris/src/developers_guide/contributing_ci_tests.rst b/docs/iris/src/developers_guide/contributing_ci_tests.rst new file mode 100644 index 0000000000..d2d41a3e87 --- /dev/null +++ b/docs/iris/src/developers_guide/contributing_ci_tests.rst @@ -0,0 +1,65 @@ +.. _developer_testing_ci: + +.. include:: ../common_links.inc + +Continuous Integration (CI) Testing +=================================== + +The `Iris`_ GitHub repository is configured to run checks on the code +automatically when a pull request is created, updated or merged against +Iris **master**. The checks performed are: + +* :ref:`testing_cla` +* :ref:`testing_travis` +* :ref:`testing_stickler` + + +.. _testing_cla: + +SciTools CLA Checker +******************** + +A bot that checks the user who created the pull request has signed the +**Contributor's License Agreement (CLA)**. For more information on this this +please see https://scitools.org.uk/organisation.html#governance + + +.. _testing_stickler: + +Stickler CI +*********** + +Automatically enforces coding standards. The configuration file named +`.stickler.yml`_ is in the Iris_ root directory. For more information see +https://stickler-ci.com/. + + +.. _testing_travis: + +Travis-CI +********* + +The unit and integration tests in Iris are an essential mechanism to ensure +that the Iris code base is working as expected. :ref:`developer_running_tests` +may be run manually but to ensure the checks are performed a +continuous integration testing tool named `travis-ci`_ is used. + +A `travis-ci`_ configuration file named `.travis.yml`_ +is in the Iris repository which tells travis-ci what commands to run. The +commands include retrieving the Iris code base and associated test files using +conda and then running the tests. `travis-ci`_ allows for a matrix of tests to +be performed to ensure that all expected variations test successfully. + +The `travis-ci`_ tests are run automatically against the `Iris`_ master +repository when a pull request is submitted, updated or merged. + +GitHub Checklist +**************** + +An example snapshot from a successful GitHub pull request shows all test passing. + +.. image:: ci_checks.png + +If any CI checks fail, then the pull request is unlikely to be merged to the Iris +target branch by a core developer. + diff --git a/docs/iris/src/developers_guide/contributing_code_formatting.rst b/docs/iris/src/developers_guide/contributing_code_formatting.rst new file mode 100644 index 0000000000..b3f23f655a --- /dev/null +++ b/docs/iris/src/developers_guide/contributing_code_formatting.rst @@ -0,0 +1,63 @@ +.. include:: ../common_links.inc + +.. _code_formatting: + +Code formatting +=============== + +To ensure a consistent code format throughout Iris, we recommend using +tools to check the source directly. + +* `black`_ for an opinionated coding auto-formatter +* `flake8`_ linting checks + +The preferred way to run these tools automatically is to setup and configure +`pre-commit`_. + +You can install ``pre-commit`` in your development environment using ``pip``:: + + $ pip install pre-commit + +or alternatively using ``conda``:: + + $ conda install -c conda-forge pre-commit + +.. note:: If you have setup your Python environment using the guide + :ref:`installing_from_source` then ``pre-commit`` should already + be present. + +In order to install the ``pre-commit`` git hooks defined in our +``.pre-commit-config.yaml`` file, you must now run the following command from +the root directory of Iris:: + + $ pre-commit install + +Upon performing a ``git commit``, your code will now be automatically formatted +to the ``black`` configuration defined in our ``pyproject.toml`` file, and +linted according to our ``.flake8`` configuration file. Note that, +``pre-commit`` will automatically download and install the necessary packages +for each ``.pre-commit-config.yaml`` git hook. + +Additionally, you may wish to enable ``black`` for your preferred +`editor/IDE `_. + +With the ``pre-commit`` configured, the output of performing a ``git commit`` +will look similar to:: + + Check for added large files..............................................Passed + Check for merge conflicts................................................Passed + Debug Statements (Python)............................(no files to check)Skipped + Don't commit to branch...................................................Passed + black................................................(no files to check)Skipped + flake8...............................................(no files to check)Skipped + [contribution_overhaul c8513187] this is my commit message + 2 files changed, 10 insertions(+), 9 deletions(-) + + +.. note:: You can also run `black`_ and `flake8`_ manually. Please see the + their officially documentation for more information. + + +.. _black: https://black.readthedocs.io/en/stable/ +.. _flake8: https://flake8.pycqa.org/en/stable/ +.. _pre-commit: https://pre-commit.com/ diff --git a/docs/iris/src/developers_guide/contributing_codebase_index.rst b/docs/iris/src/developers_guide/contributing_codebase_index.rst new file mode 100644 index 0000000000..8d7eed8c84 --- /dev/null +++ b/docs/iris/src/developers_guide/contributing_codebase_index.rst @@ -0,0 +1,13 @@ +.. _contributing.documentation.codebase: + +Contributing to the code base +============================= + +.. toctree:: + :maxdepth: 3 + + contributing_code_formatting + documenting/docstrings + documenting/rest_guide + contributing_deprecations + contributing_testing_index diff --git a/docs/iris/src/developers_guide/deprecations.rst b/docs/iris/src/developers_guide/contributing_deprecations.rst similarity index 100% rename from docs/iris/src/developers_guide/deprecations.rst rename to docs/iris/src/developers_guide/contributing_deprecations.rst diff --git a/docs/iris/src/developers_guide/contributing_documentation.rst b/docs/iris/src/developers_guide/contributing_documentation.rst index 618e5fbd08..959169b2fb 100644 --- a/docs/iris/src/developers_guide/contributing_documentation.rst +++ b/docs/iris/src/developers_guide/contributing_documentation.rst @@ -1,49 +1,43 @@ -.. toctree:: - :maxdepth: 2 - .. _contributing.documentation: Contributing to the documentation -================================== +--------------------------------- Documentation is important and we encourage any improvements that can be made. -If you believe the documentation is not clear please contribute a change to +If you believe the documentation is not clear please contribute a change to improve the documentation for all users. -Any change to the Iris project whether it is a bugfix, new feature or -documentation update must use the :ref:`development-workflow`. +Any change to the Iris project whether it is a bugfix, new feature or +documentation update must use the :ref:`development-workflow`. .. contents:: Contents: :local: Requirements ------------- +~~~~~~~~~~~~ -The documentation uses specific packages that need to be present. Please see +The documentation uses specific packages that need to be present. Please see :ref:`installing_iris` for instructions. .. _contributing.documentation.building: Building --------- - -The build is run from the documentation directory ``iris/docs/iris/src``. In -this directory run:: +~~~~~~~~ - make html +The build can be run from the documentation directory ``iris/docs/iris/src``. The build output for the html is found in the ``_build/html`` sub directory. -When updating the documentation ensure the html build has *no errors* or +When updating the documentation ensure the html build has *no errors* or *warnings* otherwise it may fail the automated `travis-ci`_ build. Once the build is complete, if it is rerun it will only rebuild the impacted build artefacts so should take less time. -There is also an option to perform a build but skip the -:ref:`contributing.documentation.gallery` creation completely. This can be +There is also an option to perform a build but skip the +:ref:`contributing.documentation.gallery` creation completely. This can be achieved via:: make html-noplot @@ -55,10 +49,10 @@ If you wish to run a clean build you can run:: This is useful for a final test before committing your changes. -.. note:: In addition to the automated `travis-ci`_ build of the documentation, - the https://readthedocs.org/ service is also used. The configuration - of this held in a file in the root of the - `github Iris project `_ named +.. note:: In addition to the automated `travis-ci`_ build of the documentation, + the https://readthedocs.org/ service is also used. The configuration + of this held in a file in the root of the + `github Iris project `_ named ``.readthedocs.yml``. .. _travis-ci: https://travis-ci.org/github/SciTools/iris @@ -66,30 +60,36 @@ This is useful for a final test before committing your changes. .. _contributing.documentation.testing: Testing -------- +~~~~~~~ There are a ways to test various aspects of the documentation. The ``make`` commands shown below can be run in the ``iris/docs/iris`` or ``iris/docs/iris/src`` directory. -Each :ref:`contributing.documentation.gallery` entry has a corresponding test. +Each :ref:`contributing.documentation.gallery` entry has a corresponding test. To run the tests:: make gallerytest Many documentation pages includes python code itself that can be run to ensure -it is still valid:: +it is still valid or to demonstrate examples. To ensure these tests pass +run:: make doctest +See :data:`iris.cube.Cube.data` for an example of using the `doctest`_ +approach. + +.. _doctest: http://www.sphinx-doc.org/en/stable/ext/doctest.html + The hyperlinks in the documentation can be checked automatically. If there is a link that is known to work it can be excluded from the checks by adding it to the ``linkcheck_ignore`` array that is defined in the `conf.py`_. The hyperlink check can be run via:: - make linkcheck + make linkcheck -If this fails check the output for the text **broken** and then correct +If this fails check the output for the text **broken** and then correct or ignore the url. .. comment @@ -101,10 +101,10 @@ or ignore the url. The spelling check may pull up many technical abbreviations and acronyms. This can be managed by using an **allow** list in the form of a file. This file, or list of files is set in the `conf.py`_ using the string list - ``spelling_word_list_filename``. + ``spelling_word_list_filename``. -.. note:: All of the above tests are automatically run as part of the +.. note:: All of the above tests are automatically run as part of the `travis-ci`_ automated build. .. _conf.py: https://github.com/SciTools/iris/blob/master/docs/iris/src/conf.py @@ -113,36 +113,36 @@ or ignore the url. .. _contributing.documentation.api: Generating API documentation ----------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In order to auto generate the API documentation based upon the docstrings a -custom set of python scripts are used, these are located in the directory -``iris/docs/iris/src/sphinxext``. Once the ``make html`` command has been run, +custom set of python scripts are used, these are located in the directory +``iris/docs/iris/src/sphinxext``. Once the ``make html`` command has been run, the output of these scripts can be found in -``iris/docs/iris/src/_build/generated/api``. +``iris/docs/iris/src/generated/api``. If there is a particularly troublesome module that breaks the ``make html`` you can exclude the module from the API documentation. Add the entry to the -``exclude_modules`` tuple list in the +``exclude_modules`` tuple list in the ``iris/docs/iris/src/sphinxext/generate_package_rst.py`` file. .. _contributing.documentation.gallery: Gallery -------- +~~~~~~~ -The Iris :ref:`sphx_glr_generated_gallery` uses a sphinx extension named +The Iris :ref:`sphx_glr_generated_gallery` uses a sphinx extension named `sphinx-gallery `_ -that auto generates reStructuredText (rst) files based upon a gallery source +that auto generates reStructuredText (rst) files based upon a gallery source directory that abides directory and filename convention. The code for the gallery entries are in ``iris/docs/iris/gallery_code``. -Each sub directory in this directory is a sub section of the gallery. The +Each sub directory in this directory is a sub section of the gallery. The respective ``README.rst`` in each folder is included in the gallery output. -For each gallery entry there must be a corresponding test script located in -``iris/docs/iris/gallery_tests``. +For each gallery entry there must be a corresponding test script located in +``iris/docs/iris/gallery_tests``. To add an entry to the gallery simple place your python code into the appropriate sub directory and name it with a prefix of ``plot_``. If your @@ -150,12 +150,8 @@ gallery entry does not fit into any existing sub directories then create a new directory and place it in there. The reStructuredText (rst) output of the gallery is located in -``iris/docs/iris/src/_build/generated/gallery``. +``iris/docs/iris/src/generated/gallery``. For more information on the directory structure and options please see the `sphinx-gallery getting started `_ documentation. - - - - diff --git a/docs/iris/src/developers_guide/contributing_getting_involved.rst b/docs/iris/src/developers_guide/contributing_getting_involved.rst new file mode 100644 index 0000000000..a336bde38b --- /dev/null +++ b/docs/iris/src/developers_guide/contributing_getting_involved.rst @@ -0,0 +1,37 @@ +.. _development_where_to_start: + +.. include:: ../common_links.inc + +Getting Involved +---------------- + +Iris_ is an Open Source project hosted on Github and as such anyone with a +GitHub account may create an `issue`_ on our `Iris GitHub`_ project page for +raising: + +* bug reports +* feature requests +* documentation improvements + +The `Iris GitHub`_ project has been configured to use templates for each of +the above issue types when creating a `new issue`_ to ensure the appropriate +information is provided. + +A `pull request`_ may also be created by anyone who has become a +**contributor** to Iris_. Permissions to merge pull requests to the +main code base (master) are only given to **core developers** of Iris_, this is +to ensure a measure of control. + +To get started we suggest reading recent `issues`_ and `pull requests`_ +for Iris. + +If you are new to using GitHub we recommend reading the +`GitHub getting started`_ + +.. note:: For more information on becoming a contributor_ including a link to + the Contributors Licence Agreement (CLA) see the + `Governance `_ + section of the `SciTools`_ ogranization web site. + + +.. _GitHub getting started: https://docs.github.com/en/github/getting-started-with-github diff --git a/docs/iris/src/developers_guide/contributing_graphics_tests.rst b/docs/iris/src/developers_guide/contributing_graphics_tests.rst new file mode 100644 index 0000000000..a276f520d6 --- /dev/null +++ b/docs/iris/src/developers_guide/contributing_graphics_tests.rst @@ -0,0 +1,166 @@ +.. include:: ../common_links.inc + +.. _testing.graphics: + +Graphics tests +************** + +Iris may be used to create various forms of graphical output; to ensure +the output is consistent, there are automated tests to check against +known acceptable graphical output. See :ref:`developer_running_tests` for +more information. + +At present graphical tests are used in the following areas of Iris: + +* Module ``iris.tests.test_plot`` +* Module ``iris.tests.test_quickplot`` +* :ref:`sphx_glr_generated_gallery` plots contained in + ``docs/iris/gallery_tests``. + + +Challenges +========== + +Iris uses many dependencies that provide functionality, an example that +applies here is matplotlib_. For more information on the dependences, see +:ref:`installing_iris`. When there are updates to the matplotlib_ or a +dependency of matplotlib, this may result in a change in the rendered graphical +output. This means that there may be no changes to Iris_, but due to an +updated dependency any automated tests that compare a graphical output to a +known acceptable output may fail. The failure may also not be visually +perceived as it may be a simple pixel shift. + + +Testing strategy +================ + +The `Iris Travis matrix`_ defines multiple test runs that use +different versions of Python to ensure Iris is working as expected. + +To make this manageable, the ``iris.tests.IrisTest_nometa.check_graphic`` test +routine tests against multiple alternative **acceptable** results. It does +this using an image **hash** comparison technique which avoids storing +reference images in the Iris repository itself. + +This consists of: + + * The ``iris.tests.IrisTest_nometa.check_graphic`` function uses a perceptual + **image hash** of the outputs (see https://github.com/JohannesBuchner/imagehash) + as the basis for checking test results. + + * The hashes of known **acceptable** results for each test are stored in a + lookup dictionary, saved to the repo file + ``lib/iris/tests/results/imagerepo.json`` + (`link `_) . + + * An actual reference image for each hash value is stored in a *separate* + public repository https://github.com/SciTools/test-iris-imagehash. + + * The reference images allow human-eye assessment of whether a new output is + judged to be close enough to the older ones, or not. + + * The utility script ``iris/tests/idiff.py`` automates checking, enabling the + developer to easily compare proposed new **acceptable** result images + against the existing accepted reference images, for each failing test. + + +Reviewing failing tests +======================= + +When you find that a graphics test in the Iris testing suite has failed, +following changes in Iris or the run dependencies, this is the process +you should follow: + +#. Create a new, empty directory to store temporary image results, at the path + ``lib/iris/tests/result_image_comparison`` in your Iris repository checkout. + +#. **In your Iris repo root directory**, run the relevant (failing) tests + directly as python scripts, or by using a command such as:: + + python -m unittest discover paths/to/test/files + +#. In the ``iris/lib/iris/tests`` folder, run the command:: + + python idiff.py + + This will open a window for you to visually inspect + side-by-side **old**, **new** and **difference** images for each failed + graphics test. Hit a button to either :guilabel:`accept`, + :guilabel:`reject` or :guilabel:`skip` each new result. + + If the change is **accepted**: + + * the imagehash value of the new result image is added into the relevant + set of 'valid result hashes' in the image result database file, + ``tests/results/imagerepo.json`` + + * the relevant output file in ``tests/result_image_comparison`` is + renamed according to the image hash value, as ``.png``. + A copy of this new PNG file must then be added into the reference image + repository at https://github.com/SciTools/test-iris-imagehash + (See below). + + If a change is **skipped**: + + * no further changes are made in the repo. + + * when you run ``iris/tests/idiff.py`` again, the skipped choice will be + presented again. + + If a change is **rejected**: + + * the output image is deleted from ``result_image_comparison``. + + * when you run ``iris/tests/idiff.py`` again, the skipped choice will not + appear, unless the relevant failing test is re-run. + +#. **Now re-run the tests**. The **new** result should now be recognised and the + relevant test should pass. However, some tests can perform *multiple* + graphics checks within a single test case function. In those cases, any + failing check will prevent the following ones from being run, so a test + re-run may encounter further (new) graphical test failures. If that + happens, simply repeat the check-and-accept process until all tests pass. + + +Add your changes to Iris +======================== + +To add your changes to Iris, you need to make two pull requests (PR). + +#. The first PR is made in the ``test-iris-imagehash`` repository, at + https://github.com/SciTools/test-iris-imagehash. + + * First, add all the newly-generated referenced PNG files into the + ``images/v4`` directory. In your Iris repo, these files are to be found + in the temporary results folder ``iris/tests/result_image_comparison``. + + * Then, to update the file which lists available images, + ``v4_files_listing.txt``, run from the project root directory:: + + python recreate_v4_files_listing.py + + * Create a PR proposing these changes, in the usual way. + +#. The second PR is created in the Iris_ repository, and + should only include the change to the image results database, + ``tests/results/imagerepo.json``. + The description box of this pull request should contain a reference to + the matching one in ``test-iris-imagehash``. + +.. note:: + + The ``result_image_comparison`` folder is covered by a project + ``.gitignore`` setting, so those files *will not show up* in a + ``git status`` check. + +.. important:: + + The Iris pull-request will not test successfully in Travis until the + ``test-iris-imagehash`` pull request has been merged. This is because there + is an Iris_ test which ensures the existence of the reference images (uris) + for all the targets in the image results database. It will also fail + if you forgot to run ``recreate_v4_files_listing.py`` to update the + image-listing file in ``test-iris-imagehash``. + + +.. _Iris travis matrix: https://github.com/scitools/iris/blob/master/.travis.yml#L15 diff --git a/docs/iris/src/developers_guide/contributing_pull_request_checklist.rst b/docs/iris/src/developers_guide/contributing_pull_request_checklist.rst new file mode 100644 index 0000000000..b01f370ea2 --- /dev/null +++ b/docs/iris/src/developers_guide/contributing_pull_request_checklist.rst @@ -0,0 +1,60 @@ +.. include:: ../common_links.inc + +.. _pr_check: + +Pull request check list +======================= + +All pull request will be reviewed by a core developer who will manage the +process of merging. It is the responsibility of a developer submitting a +pull request to do their best to deliver a pull request which meets the +requirements of the project it is submitted to. + +The check list summarises criteria which will be checked before a pull request +is merged. Before submitting a pull request please consider this list. + + +#. **Provide a helpful description** of the Pull Request. This should include: + + * The aim of the change / the problem addressed / a link to the issue. + * How the change has been delivered. + +#. **Include a "What's New" entry**, if appropriate. + See :ref:`whats_new_contributions`. + +#. **Check all tests pass**. This includes existing tests and any new tests + added for any new functionality. For more information see + :ref:`developer_running_tests`. + +#. **Check all modified and new source files conform to the required** + :ref:`code_formatting`. + +#. **Check the source documentation been updated to explain all new or changed + features**. See :ref:`docstrings`. + +#. **Include code examples inside the docstrings where appropriate**. See + :ref:`contributing.documentation.testing`. + +#. **Check the documentation builds without warnings or errors**. See + :ref:`contributing.documentation.building` + +#. **Check for any new dependencies in the** `.travis.yml`_ **config file.** + +#. **Check for any new dependencies in the** `readthedocs.yml`_ **file**. This + file is used to build the documentation that is served from + https://scitools-iris.readthedocs.io/en/latest/ + +#. **Check for updates needed for supporting projects for test or example + data**. For example: + + * `iris-test-data`_ is a github project containing all the data to support + the tests. + * `iris-sample-data`_ is a github project containing all the data to support + the gallery and examples. + * `test-iris-imagehash`_ is a github project containing reference plot + images to support Iris :ref:`testing.graphics`. + + If new files are required by tests or code examples, they must be added to + the appropriate supporting project via a suitable pull-request. This pull + request should be referenced in the main Iris pull request and must be + accepted and merged before the Iris one can be. diff --git a/docs/iris/src/developers_guide/contributing_running_tests.rst b/docs/iris/src/developers_guide/contributing_running_tests.rst new file mode 100644 index 0000000000..cadf3710db --- /dev/null +++ b/docs/iris/src/developers_guide/contributing_running_tests.rst @@ -0,0 +1,93 @@ +.. include:: ../common_links.inc + +.. _developer_running_tests: + +Running the tests +***************** + +A prerequisite of running the tests is to have the Python environment +setup. For more information on this see :ref:`installing_from_source`. + +Many Iris tests will use data that may be defined in the test itself, however +this is not always the case as sometimes example files may be used. Due to +the size of some of the files used these are not kept in the Iris repository. +A separate repository under the `SciTools`_ organisation is used, see +https://github.com/SciTools/iris-test-data. + +In order to run the tests with **all** the test data you must clone the +``iris-test-data`` repository and then configure your shell to ensure the Iris +tests can find it by using the shell environment variable named +**OVERRIDE_TEST_DATA_REPOSITORY**. The example command below uses +``~/projects`` as the parent directory:: + + cd ~/projects + git clone git@github.com:SciTools/iris-test-data.git + export OVERRIDE_TEST_DATA_REPOSITORY=~/projects/iris-test-data/test_data + +All the Iris tests may be run from the root ``iris`` project directory via:: + + python setup.py test + +You can also run a specific test, the example below runs the tests for +mapping:: + + cd lib/iris/tests + python test_mapping.py + +When running the test directly as above you can view the command line options +using the commands ``python test_mapping.py -h`` or +``python test_mapping.py --help``. + +.. tip:: A useful command line option to use is ``-d``. This will display + matplotlib_ figures as the tests are run. For example:: + + python test_mapping.py -d + + You can also use the ``-d`` command line option when running all + the tests but this will take a while to run and will require the + manual closing of each of the figures for the tests to continue. + +The output from running the tests is verbose as it will run ~5000 separate +tests. Below is a trimmed example of the output:: + + running test + Running test suite(s): default + + Running test discovery on iris.tests with 2 processors. + test_circular_subset (iris.tests.experimental.regrid.test_regrid_area_weighted_rectilinear_src_and_grid.TestAreaWeightedRegrid) ... ok + test_cross_section (iris.tests.experimental.regrid.test_regrid_area_weighted_rectilinear_src_and_grid.TestAreaWeightedRegrid) ... ok + test_different_cs (iris.tests.experimental.regrid.test_regrid_area_weighted_rectilinear_src_and_grid.TestAreaWeightedRegrid) ... ok + ... + ... + test_ellipsoid (iris.tests.unit.experimental.raster.test_export_geotiff.TestProjection) ... SKIP: Test requires 'gdal'. + test_no_ellipsoid (iris.tests.unit.experimental.raster.test_export_geotiff.TestProjection) ... SKIP: Test requires 'gdal'. + ... + ... + test_slice (iris.tests.test_util.TestAsCompatibleShape) ... ok + test_slice_and_transpose (iris.tests.test_util.TestAsCompatibleShape) ... ok + test_transpose (iris.tests.test_util.TestAsCompatibleShape) ... ok + + ---------------------------------------------------------------------- + Ran 4762 tests in 238.649s + + OK (SKIP=22) + +There may be some tests that have been **skipped**. This is due to a Python +decorator being present in the test script that will intentionally skip a test +if a certain condition is not met. In the example output above there are +**22** skipped tests, at the point in time when this was run this was primarily +due to an experimental dependency not being present. + + +.. tip:: + + The most common reason for tests to be skipped is when the directory for the + ``iris-test-data`` has not been set which would shows output such as:: + + test_coord_coord_map (iris.tests.test_plot.Test1dScatter) ... SKIP: Test(s) require external data. + test_coord_coord (iris.tests.test_plot.Test1dScatter) ... SKIP: Test(s) require external data. + test_coord_cube (iris.tests.test_plot.Test1dScatter) ... SKIP: Test(s) require external data. + + All Python decorators that skip tests will be defined in + ``lib/iris/tests/__init__.py`` with a function name with a prefix of + ``skip_``. \ No newline at end of file diff --git a/docs/iris/src/developers_guide/tests.rst b/docs/iris/src/developers_guide/contributing_testing.rst similarity index 75% rename from docs/iris/src/developers_guide/tests.rst rename to docs/iris/src/developers_guide/contributing_testing.rst index 0322abfdba..375ad57003 100644 --- a/docs/iris/src/developers_guide/tests.rst +++ b/docs/iris/src/developers_guide/contributing_testing.rst @@ -1,25 +1,25 @@ -.. _developer_tests: +.. include:: ../common_links.inc -Testing -******* +.. _developer_test_categories: -The Iris tests may be run with ``python setup.py test`` which has a -command line utility included. -There are three categories of tests within Iris: +Test categories +*************** - - Unit tests - - Integration tests - - Legacy tests +There are two main categories of tests within Iris: + + - :ref:`testing.unit_test` + - :ref:`testing.integration` Ideally, all code changes should be accompanied by one or more unit -tests, and by zero or more integration tests. And where possible, new -tests should not be added to the legacy tests. +tests, and by zero or more integration tests. But if in any doubt about what tests to add or how to write them please feel free to submit a pull-request in any state and ask for assistance. +.. _testing.unit_test: + Unit tests ========== @@ -35,16 +35,21 @@ For example: All unit tests must be placed and named according to the following structure: + +.. _testing.classes: + Classes ------- + When testing a class all the tests must reside in the module: :literal:`lib/iris/tests/unit//test_.py` Within this test module each tested method must have one or more -corresponding test classes: -- Either: `Test_name_of_public_method` -- Or: `Test_name_of_public_method__aspect_of_method` +corresponding test classes, for example: + +* ``Test_`` +* ``Test___`` And within those test classes, the test methods must be named according to the aspect of the tested method which they address. @@ -79,15 +84,19 @@ Within that file the tests might look something like: ... +.. _testing.functions: + Functions --------- + When testing a function all the tests must reside in the module: :literal:`lib/iris/tests/unit//test_.py` -Within this test module there must be one or more test classes: -- Either: `Test` -- Or: `TestAspectOfFunction` +Within this test module there must be one or more test classes, for example: + +* ``Test`` +* ``TestAspectOfFunction`` And within those test classes, the test methods must be named according to the aspect of the tested function which they address. @@ -117,6 +126,8 @@ Within that file the tests might look something like: ... +.. _testing.integration: + Integration tests ================= @@ -124,21 +135,10 @@ Some code changes may require tests which exercise several units in order to demonstrate an important consequence of their interaction which may not be apparent when considering the units in isolation. -These tests must be placed in the `lib/iris/tests/integration` folder. +These tests must be placed in the ``lib/iris/tests/integration`` folder. Unlike unit tests, there is no fixed naming scheme for integration tests. But folders and files must be created as required to help developers locate relevant tests. It is recommended they are named according to the capabilities under test, e.g. -`metadata/test_pp_preservation.py`, and not named according to the -module(s) under test. - - -Graphics tests -================= -Certain Iris tests are based on checking plotted images. -This the only way of testing the modules :mod:`iris.plot` and -:mod:`iris.quickplot`, but is also used for some other legacy and integration- -style testcases. - -There are specific mechanisms for handling this. -See :ref:`developer_graphics_tests`. +``metadata/test_pp_preservation.py``, and not named according to the +module(s) under test. \ No newline at end of file diff --git a/docs/iris/src/developers_guide/contributing_testing_index.rst b/docs/iris/src/developers_guide/contributing_testing_index.rst new file mode 100644 index 0000000000..517111507b --- /dev/null +++ b/docs/iris/src/developers_guide/contributing_testing_index.rst @@ -0,0 +1,12 @@ +.. _testing: + +Testing +======= + +.. toctree:: + :maxdepth: 3 + + contributing_testing + contributing_graphics_tests + contributing_running_tests + contributing_ci_tests diff --git a/docs/iris/src/developers_guide/documenting/docstrings.rst b/docs/iris/src/developers_guide/documenting/docstrings.rst index 323d753a52..34ec790d03 100644 --- a/docs/iris/src/developers_guide/documenting/docstrings.rst +++ b/docs/iris/src/developers_guide/documenting/docstrings.rst @@ -4,39 +4,58 @@ Docstrings ========== -Guiding principle: Every public object in the Iris package should have an -appropriate docstring. +Every public object in the Iris package should have an appropriate docstring. +This is important as the docstrings are used by developers to understand +the code and may be read directly in the source or via the :ref:`Iris`. This document has been influenced by the following PEP's, * Attribute Docstrings :pep:`224` * Docstring Conventions :pep:`257` +For consistency always use: -For consistency, always use ``"""triple double quotes"""`` around docstrings. Use ``r"""raw triple double quotes"""`` if you use any backslashes in your docstrings. For Unicode docstrings, use ``u"""Unicode triple-quoted string"""``. +* ``"""triple double quotes"""`` around docstrings. +* ``r"""raw triple double quotes"""`` if you use any backslashes in your + docstrings. +* ``u"""Unicode triple-quoted string"""`` for Unicode docstrings -All docstrings should be written in rST (reStructuredText) markup; an rST guide follows this page. +All docstrings should be written in reST (reStructuredText) markup. See the +:ref:`reST_quick_start` for more detail. -There are two forms of docstrings: **single-line** and **multi-line** docstrings. +There are two forms of docstrings: **single-line** and **multi-line** +docstrings. Single-line docstrings ====================== -The single line docstring of an object must state the *purpose* of that object, known as the *purpose section*. This terse overview must be on one line and ideally no longer than 90 characters. + +The single line docstring of an object must state the **purpose** of that +object, known as the **purpose section**. This terse overview must be on one +line and ideally no longer than 80 characters. Multi-line docstrings ===================== -Multi-line docstrings must consist of at least a purpose section akin to the single-line docstring, followed by a blank line and then any other content, as described below. The entire docstring should be indented to the same level as the quotes at the docstring's first line. + +Multi-line docstrings must consist of at least a purpose section akin to the +single-line docstring, followed by a blank line and then any other content, as +described below. The entire docstring should be indented to the same level as +the quotes at the docstring's first line. Description ----------- -The multi-line docstring *description section* should expand on what was stated in the one line *purpose section*. The description section should try not to document *argument* and *keyword argument* details. Such information should be documented in the following *arguments and keywords section*. + +The multi-line docstring *description section* should expand on what was +stated in the one line *purpose section*. The description section should try +not to document *argument* and *keyword argument* details. Such information +should be documented in the following *arguments and keywords section*. Sample multi-line docstring --------------------------- + Here is a simple example of a standard docstring: .. literalinclude:: docstrings_sample_routine.py @@ -58,14 +77,24 @@ Additionally, a summary can be extracted automatically, which would result in: Documenting classes =================== -The class constructor should be documented in the docstring for its ``__init__`` or ``__new__`` method. Methods should be documented by their own docstring, not in the class header itself. -If a class subclasses another class and its behaviour is mostly inherited from that class, its docstring should mention this and summarise the differences. Use the verb "override" to indicate that a subclass method replaces a superclass method and does not call the superclass method; use the verb "extend" to indicate that a subclass method calls the superclass method (in addition to its own behaviour). +The class constructor should be documented in the docstring for its +``__init__`` or ``__new__`` method. Methods should be documented by their own +docstring, not in the class header itself. + +If a class subclasses another class and its behaviour is mostly inherited from +that class, its docstring should mention this and summarise the differences. +Use the verb "override" to indicate that a subclass method replaces a +superclass method and does not call the superclass method; use the verb +"extend" to indicate that a subclass method calls the superclass method +(in addition to its own behaviour). Attribute and property docstrings --------------------------------- -Here is a simple example of a class containing an attribute docstring and a property docstring: + +Here is a simple example of a class containing an attribute docstring and a +property docstring: .. literalinclude:: docstrings_attribute.py @@ -77,4 +106,5 @@ This would be rendered as: :members: :undoc-members: -.. note:: The purpose section of the property docstring **must** state whether the property is read-only. +.. note:: The purpose section of the property docstring **must** state whether + the property is read-only. diff --git a/docs/iris/src/developers_guide/documenting/index.rst b/docs/iris/src/developers_guide/documenting/index.rst deleted file mode 100644 index b30a16b2a6..0000000000 --- a/docs/iris/src/developers_guide/documenting/index.rst +++ /dev/null @@ -1,10 +0,0 @@ -======================= - Documentation in Iris -======================= - -.. toctree:: - :maxdepth: 2 - - docstrings.rst - rest_guide.rst - whats_new_contributions.rst diff --git a/docs/iris/src/developers_guide/documenting/rest_guide.rst b/docs/iris/src/developers_guide/documenting/rest_guide.rst index aadb5ffea4..bc34d16cd8 100644 --- a/docs/iris/src/developers_guide/documenting/rest_guide.rst +++ b/docs/iris/src/developers_guide/documenting/rest_guide.rst @@ -1,12 +1,18 @@ +.. include:: ../../common_links.inc + +.. _reST_quick_start: + ================ reST quick start ================ +`reST`_ is used to create the documentation for Iris_. It is used to author +all of the documentation content including use in docstrings where appropriate. +For more information see :ref:`docstrings`. -reST (http://en.wikipedia.org/wiki/ReStructuredText) is a lightweight markup -language intended to be highly readable in source format. This guide will -cover some of the more frequently used advanced reST markup syntaxes, for the -basics of reST the following links may be useful: +reST is a lightweight markup language intended to be highly readable in +source format. This guide will cover some of the more frequently used advanced +reST markup syntaxes, for the basics of reST the following links may be useful: * https://www.sphinx-doc.org/en/master/usage/restructuredtext/ * http://packages.python.org/an_example_pypi_project/sphinx.html @@ -35,4 +41,4 @@ syntax ``:py:class:`zipfile.ZipFile``` which will result in links such as :py:class:`zipfile.ZipFile` and :py:class:`numpy.ndarray`. - +.. _reST: http://en.wikipedia.org/wiki/ReStructuredText diff --git a/docs/iris/src/developers_guide/gitwash/branch_dropdown.png b/docs/iris/src/developers_guide/gitwash/branch_dropdown.png index 1bb7a57773..6d74f3d643 100644 Binary files a/docs/iris/src/developers_guide/gitwash/branch_dropdown.png and b/docs/iris/src/developers_guide/gitwash/branch_dropdown.png differ diff --git a/docs/iris/src/developers_guide/gitwash/configure_git.rst b/docs/iris/src/developers_guide/gitwash/configure_git.rst index fd3cf0db20..b958a683ee 100644 --- a/docs/iris/src/developers_guide/gitwash/configure_git.rst +++ b/docs/iris/src/developers_guide/gitwash/configure_git.rst @@ -1,8 +1,10 @@ +.. include:: links.inc + .. _configure-git: -=============== - Configure git -=============== +============= +Configure git +============= .. _git-config-basic: @@ -138,21 +140,17 @@ and it gives graph / text output something like this (but with color!):: * 6d8e1ee - (HEAD, origin/my-fancy-feature, my-fancy-feature) NF - a fancy file (45 minutes ago) [Matthew Brett] * d304a73 - (origin/placeholder, placeholder) Merge pull request #48 from hhuuggoo/master (2 weeks ago) [Jonathan Terhorst] - |\ + |\ | * 4aff2a8 - fixed bug 35, and added a test in test_bugfixes (2 weeks ago) [Hugo] - |/ + |/ * a7ff2e5 - Added notes on discussion/proposal made during Data Array Summit. (2 weeks ago) [Corran Webster] * 68f6752 - Initial implimentation of AxisIndexer - uses 'index_by' which needs to be changed to a call on an Axes object - this is all very sketchy right now. (2 weeks ago) [Corr * 376adbd - Merge pull request #46 from terhorst/master (2 weeks ago) [Jonathan Terhorst] - |\ + |\ | * b605216 - updated joshu example to current api (3 weeks ago) [Jonathan Terhorst] | * 2e991e8 - add testing for outer ufunc (3 weeks ago) [Jonathan Terhorst] | * 7beda5a - prevent axis from throwing an exception if testing equality with non-axis object (3 weeks ago) [Jonathan Terhorst] | * 65af65e - convert unit testing code to assertions (3 weeks ago) [Jonathan Terhorst] | * 956fbab - Merge remote-tracking branch 'upstream/master' (3 weeks ago) [Jonathan Terhorst] - | |\ + | |\ | |/ - -Thanks to Yury V. Zaytsev for posting it. - -.. include:: links.inc diff --git a/docs/iris/src/developers_guide/gitwash/development_workflow.rst b/docs/iris/src/developers_guide/gitwash/development_workflow.rst index dee06454a1..b67885e6bd 100644 --- a/docs/iris/src/developers_guide/gitwash/development_workflow.rst +++ b/docs/iris/src/developers_guide/gitwash/development_workflow.rst @@ -14,7 +14,8 @@ Workflow summary In what follows we'll refer to the upstream iris ``master`` branch, as "trunk". -* Don't use your ``master`` branch for anything. Consider deleting it. +* Don't use your ``master`` (that is on your fork) branch for anything. + Consider deleting it. * When you are starting a new set of changes, fetch any changes from trunk, and start a new *feature branch* from that. * Make a new branch for each separable set of changes |emdash| "one task, one @@ -22,7 +23,7 @@ In what follows we'll refer to the upstream iris ``master`` branch, as * Name your branch for the purpose of the changes - e.g. ``bugfix-for-issue-14`` or ``refactor-database-code``. * If you can possibly avoid it, avoid merging trunk or any other branches into - your feature branch while you are working. + your feature branch while you are working. * If you do find yourself merging from trunk, consider :ref:`rebase-on-trunk` * Ask on the `iris mailing list`_ if you get stuck. * Ask for code review! @@ -134,21 +135,21 @@ In more detail #. Check what the actual changes are with ``git diff`` (`git diff`_). #. Add any new files to version control ``git add new_file_name`` (see `git add`_). -#. To commit all modified files into the local copy of your repo,, do +#. To commit all modified files into the local copy of your repo, do ``git commit -am 'A commit message'``. Note the ``-am`` options to ``commit``. The ``m`` flag just signals that you're going to type a - message on the command line. The ``a`` flag |emdash| you can just take on - faith |emdash| or see `why the -a flag?`_ |emdash| and the helpful use-case - description in the `tangled working copy problem`_. The `git commit`_ manual - page might also be useful. + message on the command line. The ``a`` flag will automatically stage + all files that have been modified and deleted. #. To push the changes up to your forked repo on github, do a ``git push`` (see `git push`_). + Testing your changes ==================== -Once you are happy with your changes, work thorough the :ref:`pr_check` and make sure -your branch passes all the relevant tests. +Once you are happy with your changes, work thorough the :ref:`pr_check` and +make sure your branch passes all the relevant tests. + Ask for your changes to be reviewed or merged ============================================= @@ -188,9 +189,10 @@ Delete a branch on github # delete branch on github git push origin :my-unwanted-branch -(Note the colon ``:`` before ``test-branch``. See also: +Note the colon ``:`` before ``test-branch``. See also: http://github.com/guides/remove-a-remote-branch + Several people sharing a single repository ------------------------------------------ @@ -201,12 +203,14 @@ share it via github. First fork iris into your account, as from :ref:`forking`. Then, go to your forked repository github page, say -``http://github.com/your-user-name/iris`` +``http://github.com/your-user-name/iris``, select :guilabel:`Settings`, +:guilabel:`Manage Access` and then :guilabel:`Invite collaborator`. -Click on the 'Admin' button, and add anyone else to the repo as a -collaborator: +.. note:: For more information on sharing your repository see the + GitHub documentation on `Inviting collaborators`_. - .. image:: pull_button.png + +.. _Inviting collaborators: https://docs.github.com/en/github/setting-up-and-managing-your-github-user-account/inviting-collaborators-to-a-personal-repository Now all those people can do:: @@ -233,189 +237,18 @@ To see a linear list of commits for this branch:: git log -You can also look at the `network graph visualizer`_ for your github -repo. - Finally the :ref:`fancy-log` ``lg`` alias will give you a reasonable text-based graph of the repository. + .. _rebase-on-trunk: Rebasing on trunk ----------------- -Let's say you thought of some work you'd like to do. You -:ref:`update-mirror-trunk` and :ref:`make-feature-branch` called -``cool-feature``. At this stage trunk is at some commit, let's call it E. Now -you make some new commits on your ``cool-feature`` branch, let's call them A, B, -C. Maybe your changes take a while, or you come back to them after a while. In -the meantime, trunk has progressed from commit E to commit (say) G:: - - A---B---C cool-feature - / - D---E---F---G trunk - -At this stage you consider merging trunk into your feature branch, and you -remember that this here page sternly advises you not to do that, because the -history will get messy. Most of the time you can just ask for a review, and not -worry that trunk has got a little ahead. But sometimes, the changes in trunk -might affect your changes, and you need to harmonize them. In this situation -you may prefer to do a rebase. - -rebase takes your changes (A, B, C) and replays them as if they had been made to -the current state of ``trunk``. In other words, in this case, it takes the -changes represented by A, B, C and replays them on top of G. After the rebase, -your history will look like this:: - - A'--B'--C' cool-feature - / - D---E---F---G trunk - -See `rebase without tears`_ for more detail. - -To do a rebase on trunk:: - - # Update the mirror of trunk - git fetch upstream - # go to the feature branch - git checkout cool-feature - # make a backup in case you mess up - git branch tmp cool-feature - # rebase cool-feature onto trunk - git rebase --onto upstream/master upstream/master cool-feature - -In this situation, where you are already on branch ``cool-feature``, the last -command can be written more succinctly as:: - - git rebase upstream/master - -When all looks good you can delete your backup branch:: - - git branch -D tmp - -If it doesn't look good you may need to have a look at -:ref:`recovering-from-mess-up`. - -If you have made changes to files that have also changed in trunk, this may -generate merge conflicts that you need to resolve - see the `git rebase`_ man -page for some instructions at the end of the "Description" section. There is -some related help on merging in the git user manual - see `resolving a merge`_. - -.. _recovering-from-mess-up: - -Recovering from mess-ups ------------------------- - -Sometimes, you mess up merges or rebases. Luckily, in git it is -relatively straightforward to recover from such mistakes. - -If you mess up during a rebase:: - - git rebase --abort - -If you notice you messed up after the rebase:: - - # reset branch back to the saved point - git reset --hard tmp - -If you forgot to make a backup branch:: - - # look at the reflog of the branch - git reflog show cool-feature - - 8630830 cool-feature@{0}: commit: BUG: io: close file handles immediately - 278dd2a cool-feature@{1}: rebase finished: refs/heads/my-feature-branch onto 11ee694744f2552d - 26aa21a cool-feature@{2}: commit: BUG: lib: make seek_gzip_factory not leak gzip obj - ... - - # reset the branch to where it was before the botched rebase - git reset --hard cool-feature@{2} - -.. _rewriting-commit-history: - -Rewriting commit history ------------------------- - -.. note:: - - Do this only for your own feature branches. - -There's an embarrassing typo in a commit you made? Or perhaps the you -made several false starts you would like the posterity not to see. - -This can be done via *interactive rebasing*. - -Suppose that the commit history looks like this:: - - git log --oneline - eadc391 Fix some remaining bugs - a815645 Modify it so that it works - 2dec1ac Fix a few bugs + disable - 13d7934 First implementation - 6ad92e5 * masked is now an instance of a new object, MaskedConstant - 29001ed Add pre-nep for a copule of structured_array_extensions. - ... - -and ``6ad92e5`` is the last commit in the ``cool-feature`` branch. Suppose we -want to make the following changes: - -* Rewrite the commit message for ``13d7934`` to something more sensible. -* Combine the commits ``2dec1ac``, ``a815645``, ``eadc391`` into a single one. - -We do as follows:: - - # make a backup of the current state - git branch tmp HEAD - # interactive rebase - git rebase -i 6ad92e5 - -This will open an editor with the following text in it:: - - pick 13d7934 First implementation - pick 2dec1ac Fix a few bugs + disable - pick a815645 Modify it so that it works - pick eadc391 Fix some remaining bugs - - # Rebase 6ad92e5..eadc391 onto 6ad92e5 - # - # Commands: - # p, pick = use commit - # r, reword = use commit, but edit the commit message - # e, edit = use commit, but stop for amending - # s, squash = use commit, but meld into previous commit - # f, fixup = like "squash", but discard this commit's log message - # - # If you remove a line here THAT COMMIT WILL BE LOST. - # However, if you remove everything, the rebase will be aborted. - # - -To achieve what we want, we will make the following changes to it:: - - r 13d7934 First implementation - pick 2dec1ac Fix a few bugs + disable - f a815645 Modify it so that it works - f eadc391 Fix some remaining bugs - -This means that (i) we want to edit the commit message for -``13d7934``, and (ii) collapse the last three commits into one. Now we -save and quit the editor. - -Git will then immediately bring up an editor for editing the commit -message. After revising it, we get the output:: - - [detached HEAD 721fc64] FOO: First implementation - 2 files changed, 199 insertions(+), 66 deletions(-) - [detached HEAD 0f22701] Fix a few bugs + disable - 1 files changed, 79 insertions(+), 61 deletions(-) - Successfully rebased and updated refs/heads/my-feature-branch. - -and the history looks now like this:: - - 0f22701 Fix a few bugs + disable - 721fc64 ENH: Sophisticated feature - 6ad92e5 * masked is now an instance of a new object, MaskedConstant +For more information please see the +`official github documentation on git rebase`_. -If it went wrong, recovery is again possible as explained :ref:`above -`. +.. _official github documentation on git rebase: https://docs.github.com/en/github/using-git/about-git-rebase .. include:: links.inc diff --git a/docs/iris/src/developers_guide/gitwash/forking.rst b/docs/iris/src/developers_guide/gitwash/forking.rst new file mode 100644 index 0000000000..e10b8f84ca --- /dev/null +++ b/docs/iris/src/developers_guide/gitwash/forking.rst @@ -0,0 +1,36 @@ +.. include:: links.inc + +.. _forking: + +=================================== +Making your own copy (fork) of Iris +=================================== + +You need to do this only once. The instructions here are very similar +to the instructions at http://help.github.com/forking/, please see +that page for more detail. We're repeating some of it here just to give the +specifics for the `Iris`_ project, and to suggest some default names. + + +Set up and configure a github account +===================================== + +If you don't have a github account, go to the github page, and make one. + +You then need to configure your account to allow write access, see +the `generating sss keys for GitHub`_ help on `github help`_. + + +Create your own forked copy of Iris +=================================== + +#. Log into your github account. +#. Go to the `Iris`_ github home at `Iris github`_. +#. Click on the *fork* button: + + .. image:: forking_button.png + + Now, after a short pause, you should find yourself at the home page for + your own forked copy of `Iris`_. + + diff --git a/docs/iris/src/developers_guide/gitwash/forking_button.png b/docs/iris/src/developers_guide/gitwash/forking_button.png index d0e04134d4..1eb21051d1 100644 Binary files a/docs/iris/src/developers_guide/gitwash/forking_button.png and b/docs/iris/src/developers_guide/gitwash/forking_button.png differ diff --git a/docs/iris/src/developers_guide/gitwash/forking_hell.rst b/docs/iris/src/developers_guide/gitwash/forking_hell.rst deleted file mode 100644 index 4b591d7b0e..0000000000 --- a/docs/iris/src/developers_guide/gitwash/forking_hell.rst +++ /dev/null @@ -1,33 +0,0 @@ -.. _forking: - -====================================================== -Making your own copy (fork) of Iris -====================================================== - -You need to do this only once. The instructions here are very similar -to the instructions at http://help.github.com/forking/ |emdash| please see -that page for more detail. We're repeating some of it here just to give the -specifics for the `iris`_ project, and to suggest some default names. - -Set up and configure a github account -===================================== - -If you don't have a github account, go to the github page, and make one. - -You then need to configure your account to allow write access |emdash| see -the ``Generating SSH keys`` help on `github help`_. - -Create your own forked copy of `Iris`_ -====================================================== - -#. Log into your github account. -#. Go to the `iris`_ github home at `iris github`_. -#. Click on the *fork* button: - - .. image:: forking_button.png - - Now, after a short pause and some 'Hardcore forking action', you - should find yourself at the home page for your own forked copy of `iris`_. - -.. include:: links.inc - diff --git a/docs/iris/src/developers_guide/gitwash/git_development.rst b/docs/iris/src/developers_guide/gitwash/git_development.rst deleted file mode 100644 index 1b4398e132..0000000000 --- a/docs/iris/src/developers_guide/gitwash/git_development.rst +++ /dev/null @@ -1,14 +0,0 @@ -.. _git-development: - -===================== - Git for development -===================== - -.. toctree:: - :maxdepth: 2 - - forking_hell - set_up_fork - configure_git - development_workflow - maintainer_workflow diff --git a/docs/iris/src/developers_guide/gitwash/git_install.rst b/docs/iris/src/developers_guide/gitwash/git_install.rst deleted file mode 100644 index d63f188b2e..0000000000 --- a/docs/iris/src/developers_guide/gitwash/git_install.rst +++ /dev/null @@ -1,24 +0,0 @@ -.. _install-git: - -============= - Install git -============= - -Overview -======== - -================ =============================== -Debian / Ubuntu ``sudo apt-get install git`` -Fedora ``sudo yum install git`` -Windows Download and install msysGit_ -OS X Use the git-osx-installer_ -================ =============================== - -In detail -========= - -See the git page for the most recent information. - -Have a look at the github install help pages available from `github help`_ - -.. include:: links.inc diff --git a/docs/iris/src/developers_guide/gitwash/git_intro.rst b/docs/iris/src/developers_guide/gitwash/git_intro.rst index 486e1c6c08..dfb64da872 100644 --- a/docs/iris/src/developers_guide/gitwash/git_intro.rst +++ b/docs/iris/src/developers_guide/gitwash/git_intro.rst @@ -1,18 +1,15 @@ -============== - Introduction -============== +.. include:: links.inc -These pages describe a git_ and github_ workflow for the `iris`_ -project. +Introduction +============ -There are several different workflows here, for different ways of -working with *iris*. +These pages describe a git_ and github_ workflow for the `Iris`_ +project. This is not a comprehensive git reference, it's just a workflow for our own project. It's tailored to the github hosting service. You may well find better or quicker ways of getting stuff done with git, but these should get you started. -For general resources for learning git, see :ref:`git-resources`. - -.. include:: links.inc +.. tip:: Please see the official `git documentation`_ for a complete list of + git **commands** and **cheat sheets**. \ No newline at end of file diff --git a/docs/iris/src/developers_guide/gitwash/git_links.inc b/docs/iris/src/developers_guide/gitwash/git_links.inc index 28cae6a025..f219d013a6 100644 --- a/docs/iris/src/developers_guide/gitwash/git_links.inc +++ b/docs/iris/src/developers_guide/gitwash/git_links.inc @@ -8,25 +8,11 @@ __not_case_sensitive__, so only one target definition is needed for nipy, NIPY, Nipy, etc... -.. git stuff .. _git: http://git-scm.com/ .. _github: http://github.com .. _github help: http://help.github.com -.. _msysgit: http://code.google.com/p/msysgit/downloads/list -.. _git-osx-installer: http://code.google.com/p/git-osx-installer/downloads/list -.. _subversion: http://subversion.tigris.org/ -.. _pro git book: http://progit.org/ -.. _git svn crash course: http://git-scm.com/course/svn.html -.. _learn.github: http://learn.github.com/ -.. _network graph visualizer: http://github.com/blog/39-say-hello-to-the-network-graph-visualizer -.. _git user manual: http://schacon.github.com/git/user-manual.html -.. _git tutorial: http://schacon.github.com/git/gittutorial.html -.. _git community book: http://book.git-scm.com/ -.. _git ready: http://www.gitready.com/ -.. _git casts: http://www.gitcasts.com/ -.. _Fernando's git page: http://www.fperez.org/py4science/git.html -.. _git magic: http://www-cs-students.stanford.edu/~blynn/gitmagic/index.html -.. _git concepts: http://www.eecs.harvard.edu/~cduan/technical/git/ +.. _git documentation: https://git-scm.com/docs + .. _git clone: http://schacon.github.com/git/git-clone.html .. _git checkout: http://schacon.github.com/git/git-checkout.html .. _git commit: http://schacon.github.com/git/git-commit.html @@ -40,21 +26,8 @@ .. _git remote: http://schacon.github.com/git/git-remote.html .. _git rebase: http://schacon.github.com/git/git-rebase.html .. _git config: http://schacon.github.com/git/git-config.html -.. _why the -a flag?: http://www.gitready.com/beginner/2009/01/18/the-staging-area.html -.. _git staging area: http://www.gitready.com/beginner/2009/01/18/the-staging-area.html -.. _tangled working copy problem: http://tomayko.com/writings/the-thing-about-git -.. _git management: http://kerneltrap.org/Linux/Git_Management + .. _linux git workflow: http://www.mail-archive.com/dri-devel@lists.sourceforge.net/msg39091.html -.. _git parable: http://tom.preston-werner.com/2009/05/19/the-git-parable.html -.. _git foundation: http://matthew-brett.github.com/pydagogue/foundation.html .. _deleting master on github: http://matthew-brett.github.com/pydagogue/gh_delete_master.html -.. _rebase without tears: http://matthew-brett.github.com/pydagogue/rebase_without_tears.html -.. _resolving a merge: http://schacon.github.com/git/user-manual.html#resolving-a-merge -.. _ipython git workflow: http://mail.scipy.org/pipermail/ipython-dev/2010-October/006746.html - -.. other stuff -.. _python: http://www.python.org .. |emdash| unicode:: U+02014 - -.. vim: ft=rst diff --git a/docs/iris/src/developers_guide/gitwash/git_resources.rst b/docs/iris/src/developers_guide/gitwash/git_resources.rst deleted file mode 100644 index 6f05422771..0000000000 --- a/docs/iris/src/developers_guide/gitwash/git_resources.rst +++ /dev/null @@ -1,55 +0,0 @@ -.. _git-resources: - -============= -Git resources -============= - -Tutorials and summaries -======================= - -* `github help`_ has an excellent series of how-to guides. -* `learn.github`_ has an excellent series of tutorials -* The `git user manual`_ -* The `git tutorial`_ -* The `git community book`_ -* `git ready`_ |emdash| a nice series of tutorials -* `git casts`_ |emdash| video snippets giving git how-tos. -* `git magic`_ |emdash| extended introduction with intermediate detail -* The `git parable`_ is an easy read explaining the concepts behind git. -* `git foundation`_ expands on the `git parable`_. -* Fernando Perez' git page |emdash| `Fernando's git page`_ |emdash| many - links and tips -* A good but technical page on `git concepts`_ - -Advanced git workflow -===================== - -There are many ways of working with git; here are some posts on the -rules of thumb that other projects have come up with: - -* Linus Torvalds on `linux git workflow`_ . Summary; use the git tools - to make the history of your edits as clean as possible; merge from - upstream edits as little as possible in branches where you are doing - active development. - -Manual pages online -=================== - -You can get these on your own machine with (e.g) ``git help push`` or -(same thing) ``git push --help``, but, for convenience, here are the -online manual pages for some common commands: - -* `git add`_ -* `git branch`_ -* `git checkout`_ -* `git clone`_ -* `git commit`_ -* `git config`_ -* `git diff`_ -* `git log`_ -* `git pull`_ -* `git push`_ -* `git remote`_ -* `git status`_ - -.. include:: links.inc diff --git a/docs/iris/src/developers_guide/gitwash/index.rst b/docs/iris/src/developers_guide/gitwash/index.rst index a69515548a..ddb73d0a84 100644 --- a/docs/iris/src/developers_guide/gitwash/index.rst +++ b/docs/iris/src/developers_guide/gitwash/index.rst @@ -7,8 +7,8 @@ Working with *iris* source code :maxdepth: 2 git_intro - git_install - git_development - git_resources - + forking + set_up_fork + configure_git + development_workflow diff --git a/docs/iris/src/developers_guide/gitwash/known_projects.inc b/docs/iris/src/developers_guide/gitwash/known_projects.inc deleted file mode 100644 index 1761d975aa..0000000000 --- a/docs/iris/src/developers_guide/gitwash/known_projects.inc +++ /dev/null @@ -1,41 +0,0 @@ -.. Known projects - -.. PROJECTNAME placeholders -.. _PROJECTNAME: http://nipy.org -.. _`PROJECTNAME github`: https://github.com/nipy -.. _`PROJECTNAME mailing list`: https://mail.python.org/mailman/listinfo/neuroimaging - -.. numpy -.. _numpy: http://www.numpy.org -.. _`numpy github`: https://github.com/numpy/numpy -.. _`numpy mailing list`: http://mail.scipy.org/mailman/listinfo/numpy-discussion - -.. scipy -.. _scipy: https://www.scipy.org -.. _`scipy github`: https://github.com/scipy/scipy -.. _`scipy mailing list`: http://mail.scipy.org/mailman/listinfo/scipy-dev - -.. nipy -.. _nipy: http://nipy.org/nipy -.. _`nipy github`: https://github.com/nipy/nipy -.. _`nipy mailing list`: https://mail.python.org/mailman/listinfo/neuroimaging - -.. ipython -.. _ipython: https://ipython.org -.. _`ipython github`: https://github.com/ipython/ipython -.. _`ipython mailing list`: http://mail.scipy.org/mailman/listinfo/IPython-dev - -.. dipy -.. _dipy: http://nipy.org/dipy -.. _`dipy github`: https://github.com/Garyfallidis/dipy -.. _`dipy mailing list`: https://mail.python.org/mailman/listinfo/neuroimaging - -.. nibabel -.. _nibabel: http://nipy.org/nibabel -.. _`nibabel github`: https://github.com/nipy/nibabel -.. _`nibabel mailing list`: https://mail.python.org/mailman/listinfo/neuroimaging - -.. marsbar -.. _marsbar: http://marsbar.sourceforge.net -.. _`marsbar github`: https://github.com/matthew-brett/marsbar -.. _`MarsBaR mailing list`: https://lists.sourceforge.net/lists/listinfo/marsbar-users diff --git a/docs/iris/src/developers_guide/gitwash/links.inc b/docs/iris/src/developers_guide/gitwash/links.inc index 20f4dcfffd..1d67f20d3a 100644 --- a/docs/iris/src/developers_guide/gitwash/links.inc +++ b/docs/iris/src/developers_guide/gitwash/links.inc @@ -1,4 +1,3 @@ -.. compiling links file -.. include:: known_projects.inc -.. include:: this_project.inc +.. include:: ../../common_links.inc + .. include:: git_links.inc diff --git a/docs/iris/src/developers_guide/gitwash/maintainer_workflow.rst b/docs/iris/src/developers_guide/gitwash/maintainer_workflow.rst deleted file mode 100644 index b05be47611..0000000000 --- a/docs/iris/src/developers_guide/gitwash/maintainer_workflow.rst +++ /dev/null @@ -1,96 +0,0 @@ -.. _maintainer-workflow: - -################### -Maintainer workflow -################### - -This page is for maintainers |emdash| those of us who merge our own or other -peoples' changes into the upstream repository. - -Being as how you're a maintainer, you are completely on top of the basic stuff -in :ref:`development-workflow`. - -The instructions in :ref:`linking-to-upstream` add a remote that has read-only -access to the upstream repo. Being a maintainer, you've got read-write access. - -It's good to have your upstream remote have a scary name, to remind you that -it's a read-write remote:: - - git remote add upstream-rw git@github.com:SciTools/iris.git - git fetch upstream-rw - -******************* -Integrating changes -******************* - -Let's say you have some changes that need to go into trunk -(``upstream-rw/master``). - -The changes are in some branch that you are currently on. For example, you are -looking at someone's changes like this:: - - git remote add someone git://github.com/someone/iris.git - git fetch someone - git branch cool-feature --track someone/cool-feature - git checkout cool-feature - -So now you are on the branch with the changes to be incorporated upstream. The -rest of this section assumes you are on this branch. - -A few commits -============= - -If there are only a few commits, consider rebasing to upstream:: - - # Fetch upstream changes - git fetch upstream-rw - # rebase - git rebase upstream-rw/master - -Remember that, if you do a rebase, and push that, you'll have to close any -github pull requests manually, because github will not be able to detect the -changes have already been merged. - -A long series of commits -======================== - -If there are a longer series of related commits, consider a merge instead:: - - git fetch upstream-rw - git merge --no-ff upstream-rw/master - -The merge will be detected by github, and should close any related pull requests -automatically. - -Note the ``--no-ff`` above. This forces git to make a merge commit, rather than -doing a fast-forward, so that these set of commits branch off trunk then rejoin -the main history with a merge, rather than appearing to have been made directly -on top of trunk. - -Check the history -================= - -Now, in either case, you should check that the history is sensible and you have -the right commits:: - - git log --oneline --graph - git log -p upstream-rw/master.. - -The first line above just shows the history in a compact way, with a text -representation of the history graph. The second line shows the log of commits -excluding those that can be reached from trunk (``upstream-rw/master``), and -including those that can be reached from current HEAD (implied with the ``..`` -at the end). So, it shows the commits unique to this branch compared to trunk. -The ``-p`` option shows the diff for these commits in patch form. - -Push to trunk -============= - -:: - - git push upstream-rw my-new-feature:master - -This pushes the ``my-new-feature`` branch in this repository to the ``master`` -branch in the ``upstream-rw`` repository. - -.. include:: links.inc diff --git a/docs/iris/src/developers_guide/gitwash/pull_button.png b/docs/iris/src/developers_guide/gitwash/pull_button.png index e5031681b9..a088e19e95 100644 Binary files a/docs/iris/src/developers_guide/gitwash/pull_button.png and b/docs/iris/src/developers_guide/gitwash/pull_button.png differ diff --git a/docs/iris/src/developers_guide/gitwash/set_up_fork.rst b/docs/iris/src/developers_guide/gitwash/set_up_fork.rst index 172cbb2051..9dc6618c64 100644 --- a/docs/iris/src/developers_guide/gitwash/set_up_fork.rst +++ b/docs/iris/src/developers_guide/gitwash/set_up_fork.rst @@ -1,8 +1,10 @@ +.. include:: links.inc + .. _set-up-fork: -================== - Set up your fork -================== +================ +Set up your fork +================ First you follow the instructions for :ref:`forking`. @@ -23,7 +25,7 @@ Clone your fork #. Clone your fork to the local computer with ``git clone git@github.com:your-user-name/iris.git`` -#. Investigate. Change directory to your new repo: ``cd iris``. Then +#. Change directory to your new repo: ``cd iris``. Then ``git branch -a`` to show you all branches. You'll get something like:: @@ -35,7 +37,7 @@ Clone your fork What remote repository is ``remote/origin``? Try ``git remote -v`` to see the URLs for the remote. They will point to your github fork. - Now you want to connect to the upstream `iris github`_ repository, so + Now you want to connect to the upstream `Iris github`_ repository, so you can merge in changes from trunk. .. _linking-to-upstream: @@ -49,7 +51,7 @@ Linking your repository to the upstream repo git remote add upstream git://github.com/SciTools/iris.git ``upstream`` here is just the arbitrary name we're using to refer to the -main `iris`_ repository at `iris github`_. +main `Iris`_ repository at `Iris github`_. Note that we've used ``git://`` for the URL rather than ``git@``. The ``git://`` URL is read only. This means we that we can't accidentally @@ -57,12 +59,12 @@ Note that we've used ``git://`` for the URL rather than ``git@``. The use it to merge into our own code. Just for your own satisfaction, show yourself that you now have a new -'remote', with ``git remote -v show``, giving you something like:: +'remote', with ``git remote -v``, giving you something like:: upstream git://github.com/SciTools/iris.git (fetch) upstream git://github.com/SciTools/iris.git (push) origin git@github.com:your-user-name/iris.git (fetch) origin git@github.com:your-user-name/iris.git (push) -.. include:: links.inc + diff --git a/docs/iris/src/developers_guide/gitwash/this_project.inc b/docs/iris/src/developers_guide/gitwash/this_project.inc deleted file mode 100644 index 38219bf4b4..0000000000 --- a/docs/iris/src/developers_guide/gitwash/this_project.inc +++ /dev/null @@ -1,5 +0,0 @@ -.. iris -.. _`iris`: http://scitools.org.uk/iris -.. _`iris github`: http://github.com/SciTools/iris - -.. _`iris mailing list`: https://groups.google.com/forum/#!forum/scitools-iris diff --git a/docs/iris/src/developers_guide/gitwash_build.sh b/docs/iris/src/developers_guide/gitwash_build.sh deleted file mode 100755 index e1c4cdb2af..0000000000 --- a/docs/iris/src/developers_guide/gitwash_build.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env sh - - -# Generate the gitwash sub-directory. -echo -echo "Building gitwash ..." -echo -python gitwash_dumper.py --repo-name=iris --github-user=SciTools --gitwash-url=https://github.com/matthew-brett/gitwash.git --project-url=http://scitools.org.uk/iris --project-ml-url=https://groups.google.com/forum/#!forum/scitools-iris ./ iris diff --git a/docs/iris/src/developers_guide/gitwash_dumper.py b/docs/iris/src/developers_guide/gitwash_dumper.py deleted file mode 100644 index b450d1f40c..0000000000 --- a/docs/iris/src/developers_guide/gitwash_dumper.py +++ /dev/null @@ -1,233 +0,0 @@ -#!/usr/bin/env python -''' Checkout gitwash repo into directory and do search replace on name ''' - -import os -from os.path import join as pjoin -import shutil -import sys -import re -import glob -import fnmatch -import tempfile -from subprocess import call -from optparse import OptionParser - -verbose = False - - -def clone_repo(url, branch): - cwd = os.getcwd() - tmpdir = tempfile.mkdtemp() - try: - cmd = 'git clone %s %s' % (url, tmpdir) - call(cmd, shell=True) - os.chdir(tmpdir) - cmd = 'git checkout %s' % branch - call(cmd, shell=True) - except: - shutil.rmtree(tmpdir) - raise - finally: - os.chdir(cwd) - return tmpdir - - -def cp_files(in_path, globs, out_path): - try: - os.makedirs(out_path) - except OSError: - pass - out_fnames = [] - for in_glob in globs: - in_glob_path = pjoin(in_path, in_glob) - for in_fname in glob.glob(in_glob_path): - out_fname = in_fname.replace(in_path, out_path) - pth, _ = os.path.split(out_fname) - if not os.path.isdir(pth): - os.makedirs(pth) - shutil.copyfile(in_fname, out_fname) - out_fnames.append(out_fname) - return out_fnames - - -def filename_search_replace(sr_pairs, filename, backup=False): - ''' Search and replace for expressions in files - - ''' - with open(filename, 'rt') as in_fh: - in_txt = in_fh.read(-1) - out_txt = in_txt[:] - for in_exp, out_exp in sr_pairs: - in_exp = re.compile(in_exp) - out_txt = in_exp.sub(out_exp, out_txt) - if in_txt == out_txt: - return False - with open(filename, 'wt') as out_fh: - out_fh.write(out_txt) - if backup: - with open(filename + '.bak', 'wt') as bak_fh: - bak_fh.write(in_txt) - return True - - -def copy_replace(replace_pairs, - repo_path, - out_path, - cp_globs=('*',), - rep_globs=('*',), - renames = ()): - out_fnames = cp_files(repo_path, cp_globs, out_path) - renames = [(re.compile(in_exp), out_exp) for in_exp, out_exp in renames] - fnames = [] - for rep_glob in rep_globs: - fnames += fnmatch.filter(out_fnames, rep_glob) - if verbose: - print('\n'.join(fnames)) - for fname in fnames: - filename_search_replace(replace_pairs, fname, False) - for in_exp, out_exp in renames: - new_fname, n = in_exp.subn(out_exp, fname) - if n: - os.rename(fname, new_fname) - break - - -def make_link_targets(proj_name, - user_name, - repo_name, - known_link_fname, - out_link_fname, - url=None, - ml_url=None): - """ Check and make link targets - - If url is None or ml_url is None, check if there are links present for these - in `known_link_fname`. If not, raise error. The check is: - - Look for a target `proj_name`. - Look for a target `proj_name` + ' mailing list' - - Also, look for a target `proj_name` + 'github'. If this exists, don't write - this target into the new file below. - - If we are writing any of the url, ml_url, or github address, then write new - file with these links, of form: - - .. _`proj_name` - .. _`proj_name`: url - .. _`proj_name` mailing list: url - """ - with open(known_link_fname, 'rt') as link_fh: - link_contents = link_fh.readlines() - have_url = not url is None - have_ml_url = not ml_url is None - have_gh_url = None - for line in link_contents: - if not have_url: - match = re.match(r'..\s+_`%s`:\s+' % proj_name, line) - if match: - have_url = True - if not have_ml_url: - match = re.match(r'..\s+_`%s mailing list`:\s+' % proj_name, line) - if match: - have_ml_url = True - if not have_gh_url: - match = re.match(r'..\s+_`%s github`:\s+' % proj_name, line) - if match: - have_gh_url = True - if not have_url or not have_ml_url: - raise RuntimeError('Need command line or known project ' - 'and / or mailing list URLs') - lines = [] - if not url is None: - lines.append('.. _`%s`: %s\n' % (proj_name, url)) - if not have_gh_url: - gh_url = 'http://github.com/%s/%s\n' % (user_name, repo_name) - lines.append('.. _`%s github`: %s\n' % (proj_name, gh_url)) - if not ml_url is None: - lines.append('.. _`%s mailing list`: %s\n' % (proj_name, ml_url)) - if len(lines) == 0: - # Nothing to do - return - # A neat little header line - lines = ['.. %s\n' % proj_name] + lines - with open(out_link_fname, 'wt') as out_links: - out_links.writelines(lines) - - -USAGE = ''' - -If not set with options, the repository name is the same as the - -If not set with options, the main github user is the same as the -repository name.''' - - -GITWASH_CENTRAL = 'git://github.com/matthew-brett/gitwash.git' -GITWASH_BRANCH = 'master' - - -def main(): - parser = OptionParser() - parser.set_usage(parser.get_usage().strip() + USAGE) - parser.add_option("--repo-name", dest="repo_name", - help="repository name - e.g. nitime", - metavar="REPO_NAME") - parser.add_option("--github-user", dest="main_gh_user", - help="github username for main repo - e.g fperez", - metavar="MAIN_GH_USER") - parser.add_option("--gitwash-url", dest="gitwash_url", - help="URL to gitwash repository - default %s" - % GITWASH_CENTRAL, - default=GITWASH_CENTRAL, - metavar="GITWASH_URL") - parser.add_option("--gitwash-branch", dest="gitwash_branch", - help="branch in gitwash repository - default %s" - % GITWASH_BRANCH, - default=GITWASH_BRANCH, - metavar="GITWASH_BRANCH") - parser.add_option("--source-suffix", dest="source_suffix", - help="suffix of ReST source files - default '.rst'", - default='.rst', - metavar="SOURCE_SUFFIX") - parser.add_option("--project-url", dest="project_url", - help="URL for project web pages", - default=None, - metavar="PROJECT_URL") - parser.add_option("--project-ml-url", dest="project_ml_url", - help="URL for project mailing list", - default=None, - metavar="PROJECT_ML_URL") - (options, args) = parser.parse_args() - if len(args) < 2: - parser.print_help() - sys.exit() - out_path, project_name = args - if options.repo_name is None: - options.repo_name = project_name - if options.main_gh_user is None: - options.main_gh_user = options.repo_name - repo_path = clone_repo(options.gitwash_url, options.gitwash_branch) - try: - copy_replace((('PROJECTNAME', project_name), - ('REPONAME', options.repo_name), - ('MAIN_GH_USER', options.main_gh_user)), - repo_path, - out_path, - cp_globs=(pjoin('gitwash', '*'),), - rep_globs=('*.rst',), - renames=(('\.rst$', options.source_suffix),)) - make_link_targets(project_name, - options.main_gh_user, - options.repo_name, - pjoin(out_path, 'gitwash', 'known_projects.inc'), - pjoin(out_path, 'gitwash', 'this_project.inc'), - options.project_url, - options.project_ml_url) - finally: - shutil.rmtree(repo_path) - - -if __name__ == '__main__': - main() diff --git a/docs/iris/src/developers_guide/gitwash_get.sh b/docs/iris/src/developers_guide/gitwash_get.sh deleted file mode 100755 index c61b406603..0000000000 --- a/docs/iris/src/developers_guide/gitwash_get.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/usr/bin/env sh - - -# Get the latest gitwash_dumper.py from GitHub. -echo "Downloading latest gitwash_dumper.py from GitHub ..." -echo -curl -O https://raw.github.com/matthew-brett/gitwash/master/gitwash_dumper.py diff --git a/docs/iris/src/developers_guide/graphics_tests.rst b/docs/iris/src/developers_guide/graphics_tests.rst deleted file mode 100644 index 8f63bd3381..0000000000 --- a/docs/iris/src/developers_guide/graphics_tests.rst +++ /dev/null @@ -1,151 +0,0 @@ -:orphan: - -.. _developer_graphics_tests: - -Graphics tests -************** - -The only practical way of testing plotting functionality is to check actual -output plots. For this, a basic 'graphics test' assertion operation is -provided in the method :meth:`iris.tests.IrisTest.check_graphic` : This -tests plotted output for a match against a stored reference. A -"graphics test" is any test which employs this. - -At present, such tests include the testing for modules ``iris.tests.test_plot`` -and ``iris.tests.test_quickplot``, all output plots from the gallery -contained in ``docs/iris/gallery_tests``, and a few other 'legacy' style tests -as described in :ref:`developer_tests` -. -It is conceivable that new 'graphics tests' of this sort can still be added. -However, as graphics tests are inherently "integration" style rather than true -unit tests, results can differ with the installed versions of dependent -libraries (see below), so this is not recommended except where no alternative -is practical. - -Testing actual plot results introduces some significant difficulties: - - * Graphics tests are inherently 'integration' style tests, so results will - often vary with the versions of key dependencies, i.e. the exact versions of - third-party modules which are installed : Obviously, results will depend on - the matplotlib version, but they can also depend on numpy and other - installed packages. - * Although it seems possible in principle to accommodate 'small' result changes - by distinguishing plots which are 'nearly the same' from those which are - 'significantly different', in practice no *automatic* scheme for this can be - perfect : That is, any calculated tolerance in output matching will allow - some changes which a human would judge as a significant error. - * Storing a variety of alternative 'acceptable' results as reference images - can easily lead to uncontrolled increases in the size of the repository, - given multiple independent sources of variation. - - -Graphics testing strategy -========================= - -In the Iris Travis matrix, and over time, graphics tests must run with -multiple versions of Python, and of key dependencies such as matplotlib. -To make this manageable, the "check_graphic" test routine tests against -multiple alternative 'acceptable' results. It does this using an image "hash" -comparison technique which avoids storing reference images in the Iris -repository itself, to avoid space problems. - -This consists of : - - * The 'check_graphic' function uses a perceptual 'image hash' of the outputs - (see https://github.com/JohannesBuchner/imagehash) as the basis for checking - test results. - * The hashes of known 'acceptable' results for each test are stored in a - lookup dictionary, saved to the repo file - ``lib/iris/tests/results/imagerepo.json`` . - * An actual reference image for each hash value is stored in a *separate* - public repository : https://github.com/SciTools/test-iris-imagehash . - * The reference images allow human-eye assessment of whether a new output is - judged to be 'close enough' to the older ones, or not. - * The utility script ``iris/tests/idiff.py`` automates checking, enabling the - developer to easily compare proposed new 'acceptable' result images against the - existing accepted reference images, for each failing test. - - -How to add new 'Acceptable' result images to existing tests -=========================================================== - -When you find that a graphics test in the Iris testing suite has failed, -following changes in Iris or the run dependencies, this is the process -you should follow: - -#. Create a new, empty directory to store temporary image results, at the path - ``lib/iris/tests/result_image_comparison`` in your Iris repository checkout. - -#. **In your Iris repo root directory**, run the relevant (failing) tests - directly as python scripts, or by using a command such as - ``python -m unittest discover paths/to/test/files``. - -#. **In the** ``iris/lib/iris/tests`` **folder**, run the command: ``python idiff.py``. - This will open a window for you to visually inspect side-by-side 'old', 'new' - and 'difference' images for each failed graphics test. - Hit a button to either "accept", "reject" or "skip" each new result ... - - * If the change is *"accepted"* : - - * the imagehash value of the new result image is added into the relevant - set of 'valid result hashes' in the image result database file, - ``tests/results/imagerepo.json`` ; - - * the relevant output file in ``tests/result_image_comparison`` is - renamed according to the image hash value, as ``.png``. - A copy of this new PNG file must then be added into the reference image - repository at https://github.com/SciTools/test-iris-imagehash. - (See below). - - * If a change is *"skipped"* : - - * no further changes are made in the repo. - - * when you run idiff again, the skipped choice will be presented again. - - * If a change is *"rejected"* : - - * the output image is deleted from ``result_image_comparison``. - - * when you run idiff again, the skipped choice will not appear, unless - and until the relevant failing test is re-run. - -#. Now re-run the tests. The 'new' result should now be recognised and the - relevant test should pass. However, some tests can perform *multiple* graphics - checks within a single test case function : In those cases, any failing - check will prevent the following ones from being run, so a test re-run may - encounter further (new) graphical test failures. If that happens, simply - repeat the check-and-accept process until all tests pass. - -#. To add your changes to Iris, you need to make two pull requests : - - * (1) The first PR is made in the test-iris-imagehash repository, at - https://github.com/SciTools/test-iris-imagehash. - - * First, add all the newly-generated referenced PNG files into the - ``images/v4`` directory. In your Iris repo, these files are to be found - in the temporary results folder ``iris/tests/result_image_comparison``. - - .. Note:: - - The ``result_image_comparison`` folder is covered by a project - ``.gitignore`` setting, so those files *will not show up* in a - ``git status`` check. - - * Then, run ``python recreate_v4_files_listing.py``, to update the file - which lists available images, ``v4_files_listing.txt``. - - * Create a PR proposing these changes, in the usual way. - - * (2) The second PR is created in the Iris repository, and - should only include the change to the image results database, - ``tests/results/imagerepo.json`` : - The description box of this pull request should contain a reference to - the matching one in test-iris-imagehash. - -Note: the Iris pull-request will not test out successfully in Travis until the -test-iris-imagehash pull request has been merged : This is because there is -an Iris test which ensures the existence of the reference images (uris) for all -the targets in the image results database. N.B. likewise, it will *also* fail -if you forgot to run ``recreate_v4_files_listing.py`` to update the image-listing -file in test-iris-imagehash. diff --git a/docs/iris/src/developers_guide/pulls.rst b/docs/iris/src/developers_guide/pulls.rst deleted file mode 100644 index 26b55d693a..0000000000 --- a/docs/iris/src/developers_guide/pulls.rst +++ /dev/null @@ -1,107 +0,0 @@ -.. _pr_check: - - -Pull request check list -*********************** - -A pull request to a SciTools project master should be ready to merge into the -master branch. - -All pull request will be reviewed by a core developer who will manage the -process of merging. It is the responsibility of a developer submitting a -pull request to do their best to deliver a pull request which meets the -requirements of the project it is submitted to. - -The check list summarises criteria which will be checked before a pull request -is merged. Before submitting a pull request please consider this list. - - -The Iris check list -==================== - -* Have you provided a helpful description of the Pull Request? - I.E. what has changed and why. This should include: - - * the aim of the change ; the problem addressed ; a link to the issue. - * how the change has been delivered. - * a "What's New" entry - see :ref:`whats_new_contributions`. - -* Do all the tests pass locally? - - * The Iris tests may be run with ``python setup.py test`` which has a command - line utility included. - -* Have new tests been provided for all additional functionality? - -* Do all modified and new source files conform to the required - :ref:`iris_code_format`? - -* Do all modified and new source files have a correct copyright header? - - * a standard Iris test checks that all source files include a copyright - message (see "iris.tests.test_coding_standards.TestLicenseHeaders"). - -* Has the documentation been updated to explain all new or changed features? - - * refer to the developer guide on :ref:`docstrings` - -* Have code examples been provided inside docstrings, where relevant? - - * these are strongly recommended as concrete (working) examples always - considerably enhance the documentation. - - * live test code can be included in docstrings. - - * See for example :data:`iris.cube.Cube.data` - * Details at http://www.sphinx-doc.org/en/stable/ext/doctest.html - - * The documentation tests may be run with ``make doctest``, from within the - ``./docs/iris`` subdirectory. - -* Have you provided a "what's new" contribution? - - * this should be done for **all** changes - see :ref:`whats_new_contributions` - -* Does the documentation build without errors? - - * The documentation is built using ``make html`` in ``./docs/iris``. - -* Do the documentation and code-example tests pass? - - * Run with ``make doctest`` and ``make gallerytest``, from within the subdirectory - ``./docs/iris``. - * note that code examples must *not* raise deprecations. This is now checked - and will result in an error. - When an existing code example encounters a deprecation, it must be fixed. - -* Has the travis file been updated to reflect any dependency updates? - - * ``./.travis.yml`` is used to manage the continuous integration testing. - * the files ``./conda-requirements.yml`` and - ``./minimal-conda-requirements.yml`` are used to define the software - environments used, using the conda_ package manager. - -* Have you provided updates to supporting projects for test or example data? - - * the following separate repos are used to manage larger files used by tests - and code examples : - - * iris-test-data_ is a github project containing all the data to support the - tests. - * iris-sample-data_ is a github project containing all the data to support - the gallery and examples. - * test-iris-imagehash_ is a github project containing reference plot images - to support iris graphics tests : see :ref:`developer_graphics_tests`. - - * If new files are required by tests or code examples, they must be added to - the appropriate supporting project via a suitable pull-request. - This new 'supporting pull request' should be referenced in the main Iris - pull request, and must be accepted and merged before the Iris one can be. - - -.. _PEP8: http://www.python.org/dev/peps/pep-0008/ -.. _python-pep8: https://pypi.python.org/pypi/pep8 -.. _conda: https://docs.conda.io/en/latest/ -.. _iris-test-data: https://github.com/SciTools/iris-test-data -.. _iris-sample-data: https://github.com/SciTools/iris-sample-data -.. _test-iris-imagehash: https://github.com/SciTools/test-iris-imagehash diff --git a/docs/iris/src/index.rst b/docs/iris/src/index.rst index 5343391332..51bbfc32af 100644 --- a/docs/iris/src/index.rst +++ b/docs/iris/src/index.rst @@ -57,6 +57,7 @@ Iris is part of SciTools, for more information see https://scitools.org.uk/. .. toctree:: :maxdepth: 1 :caption: User Guide + :name: userguide_index userguide/index userguide/iris_cubes @@ -74,17 +75,18 @@ Iris is part of SciTools, for more information see https://scitools.org.uk/. userguide/code_maintenance +.. _developers_guide: + .. toctree:: :maxdepth: 1 :caption: Developers Guide + :name: development_index - developers_guide/contributing_documentation - developers_guide/documenting/index + developers_guide/contributing_getting_involved developers_guide/gitwash/index - developers_guide/code_format - developers_guide/pulls - developers_guide/tests - developers_guide/deprecations + developers_guide/contributing_documentation + developers_guide/contributing_codebase_index + developers_guide/contributing_changes developers_guide/release generated/api/iris diff --git a/docs/iris/src/installing.rst b/docs/iris/src/installing.rst index 9b8a647f0d..ec2b4f1875 100644 --- a/docs/iris/src/installing.rst +++ b/docs/iris/src/installing.rst @@ -82,6 +82,8 @@ the command:: python setup.py test +For more information see :ref:`developer_running_tests`. + Custom site configuration ------------------------- diff --git a/docs/iris/src/spelling_allow.txt b/docs/iris/src/spelling_allow.txt index 6ef4134699..ed883ac3bf 100644 --- a/docs/iris/src/spelling_allow.txt +++ b/docs/iris/src/spelling_allow.txt @@ -6,6 +6,7 @@ Args Autoscale Biggus CF +CI Cartopy Checklist Color @@ -103,6 +104,7 @@ centrepoints cf cftime chunksizes +ci clabel cmap cmpt @@ -176,6 +178,7 @@ idiff ieee ifunc imagehash +inc init inline inplace @@ -187,6 +190,7 @@ io isosurfaces iterable jja +jupyter kwarg kwargs landsea @@ -313,6 +317,7 @@ th timepoint timestamp timesteps +todo tol tos traceback @@ -353,4 +358,4 @@ xx xxx zeroth zlev -zonal \ No newline at end of file +zonal diff --git a/docs/iris/src/techpapers/change_management.rst b/docs/iris/src/techpapers/change_management.rst index d09237a4bf..ab45fe7926 100644 --- a/docs/iris/src/techpapers/change_management.rst +++ b/docs/iris/src/techpapers/change_management.rst @@ -1,3 +1,5 @@ +.. include:: ../common_links.inc + :orphan: .. _change_management: diff --git a/docs/iris/src/whatsnew/2.3.rst b/docs/iris/src/whatsnew/2.3.rst index 914d86fda2..ec6bb3254b 100644 --- a/docs/iris/src/whatsnew/2.3.rst +++ b/docs/iris/src/whatsnew/2.3.rst @@ -252,7 +252,7 @@ Documentation in the user guide, demonstrating how to load data within a specified date range. -* Added notes to the :func:`iris.load` documentation, and the userguide +* Added notes to the :func:`iris.load` documentation, and the user guide :ref:`loading_iris_cubes` chapter, emphasizing that the *order* of the cubes returned by an iris load operation is effectively random and unstable, and should not be relied on. diff --git a/docs/iris/src/whatsnew/latest.rst b/docs/iris/src/whatsnew/latest.rst index 7aba2967ea..cec413e841 100644 --- a/docs/iris/src/whatsnew/latest.rst +++ b/docs/iris/src/whatsnew/latest.rst @@ -190,7 +190,7 @@ Documentation now automatically checked on GitHub PRs, replacing the older, unittest-based "iris.tests.test_coding_standards.TestCodeFormat". Black provides automatic code format correction for most IDEs. See the new developer guide section on - :ref:`iris_code_format`. (:pull:`3518`) + :ref:`code_formatting`. (:pull:`3518`) * `@tkknight`_ and `@trexfeathers`_ refreshed the :ref:`whats_new_contributions` for the :ref:`iris_whatsnew`. This includes always creating the ``latest``