Update the documentation following the PyPI changes

doc/development/cli_internals.rst

@@ -7,15 +7,14 @@ The `cotainr` command line interface is designed around a subcommand for each ma
 
 The CLI is build using :mod:`argparse`.
 
-The main way to invoke the CLI (sub)commands in an HPC environment is via the `bin/cotainr` script, e.g.
+The main way to invoke the CLI (sub)commands is via the `cotainr` entrypoint,
 
 .. code-block:: console
 
-    $ ./bin/cotainr build <positional_arg> <non-positional args>
-    $ ./bin/cotainr info
+    $ cotainr build <positional_arg> <non-positional args>
+    $ cotainr info
 
-The CLI (sub)commands may also be executed via the `cotainr` entrypoint that is installed when installing the package;
-substitute ``cotainr`` instead of ``./bin/cotainr``.
+This assumes that `cotainr` is installed via :code:`pip install cotainr`. Alternatively, the CLI (sub)commands may also be executed directly via the `bin/cotainr` executable in an HPC environment by substitute ``./bin/cotainr`` instead of ``cotainr``.
 
 Implementation of command line interface
 ----------------------------------------

doc/development/documentation.rst

@@ -9,22 +9,26 @@ The `cotainr` reference documentation is what you are reading right now. It cons
 
 Building the HTML version
 -------------------------
-The HTML version of the documentation may be built by running the following commands from the `doc` folder:
+The HTML version of the documentation can be built locally for development. First install the required dependencies specified in `pyproject.toml`. This can be done with `uv <https://docs.astral.sh/uv/>`_,
 
 .. code-block:: console
 
-    $ make apidoc
-    $ make relnotes
-    $ make html
+    $ uv sync --group docs
 
-The HTML output is available in the `doc/_build` folder. The :code:`make apidoc` command generates reStructuredText files for the API reference documentation. These files include :code:`sphinx.ext.autodoc` directives that automatically generate the API reference documentation from the docstrings of the modules/classes/functions/... in `cotainr`. The :code:`make relnotes` command generates the list of release notes from the markdown files in the `doc/release_notes` folder.
+the documentation may then be built by running the following `make <https://www.gnu.org/software/make/>`_ commands in the `doc` folder
 
-The HTML version of the documentation is based on the `PyData Spinx Theme <https://pydata-sphinx-theme.readthedocs.io/>`_ and its design and layout is heavily inspired by the `NumPy <https://numpy.org/doc/stable/>`_, `SciPy <https://docs.scipy.org/doc/scipy/>`_, and `Pandas <https://pandas.pydata.org/docs>`_ documentations - a big shout-out to the people who created those documentation designs!
+.. code-block:: console
 
-Dependencies
-~~~~~~~~~~~~
+    $ uv run make apidoc
+    $ uv run make relnotes
+    $ uv run make html
 
-In order to build the HTML version of the documentation, you must have the Python packages listed in the `docs` extra.
-You can use ``pip install -e .[docs]`` to install the required packages.
+The HTML output is available in the `doc/_build` folder and can be inspected in a browser such as firefox,
+
+.. code-block:: console
 
-Also, in order to run the above :code:`make` commands, you muse have `make <https://www.gnu.org/software/make/>`_ installed.
+   $ firefox _build/html/index.html
+
+The :code:`make apidoc` command generates reStructuredText files for the API reference documentation. These files include :code:`sphinx.ext.autodoc` directives that automatically generate the API reference documentation from the docstrings of the modules/classes/functions/... in `cotainr`. The :code:`make relnotes` command generates the list of release notes from the markdown files in the `doc/release_notes` folder.
+
+The HTML version of the documentation is based on the `PyData Spinx Theme <https://pydata-sphinx-theme.readthedocs.io/>`_ and its design and layout is heavily inspired by the `NumPy <https://numpy.org/doc/stable/>`_, `SciPy <https://docs.scipy.org/doc/scipy/>`_, and `Pandas <https://pandas.pydata.org/docs>`_ documentations - a big shout-out to the people who created those documentation designs!

doc/development/releasing.rst

@@ -5,6 +5,19 @@ Releasing a new version
 
 Most of the process related to releasing a new version of `cotainr` is automated. However a few things, as detailed on this page, must still be done to initiate the release process.
 
+Release process
+---------------
+In order to release a new version of `cotainr`, one must:
+
+1. Update the :attr:`cotainr.__version__` string in `cotainr/__init__.py`.
+2. Update the docs version switcher list by running the `doc/create_switcher.py` script.
+3. Create a new branch and create new release notes in the format `YYYY.MM.MINOR.md` in the folder `doc/release_notes` based on the template `doc/release_notes/release_note.md.template` and fill in the release notes.
+4. Open a new pull request to merge the release notes into main and assert successful :ref:`CI tests <continuous_integration>` of the branch. Then merge it into the main branch.
+5. Tag the main branch locally using :code:`git tag YYYY.MM.MINOR` and push the tag to the GitHub repository :code:`git push origin tag YYYY.MM.MINOR`. This triggers the :ref:`CD on release <continuous_integration>` GitHub action.
+6. Review and verify that :ref:`CD on release <continuous_integration>` creates a GitHub release. Then review and approve the TestPyPI and PyPI deployments. Finally review that `readthedocs` is updated correctly, i.e. assert that the `stable <https://cotainr.readthedocs.io/en/stable>`_ and `latest <https://cotainr.readthedocs.io/en/latest>`_ docs versions points to the newly released version.
+
+.. _version-scheme:
+
 Versioning scheme
 -----------------
 A `CalVer <https://calver.org/>`_ versioning scheme is used for `cotainr`, more specifically:
@@ -18,15 +31,3 @@ with:
 - :code:`YYYY` - full year - 2022, 2023, ...
 - :code:`MM` - short month - 1, 2 ... 11, 12
 - :code:`MINOR` - zero-indexed counter specific to :code:`YYYY.MM` - 0, 1, ... - incremented for each version released in a given year/month.
-
-Release process
----------------
-In order to release a new version of `cotainr`, one must:
-
-1. Update the :attr:`cotainr.__version__` string in `cotianr/__init__.py`.
-2. Update the version switcher list by running the `doc/create_switcher.py` script.
-3. Create the associated release notes, i.e. create a `YYYY.MM.MINOR.md` file in the `doc/release_notes` folder based on the template `doc/release_notes/release_note.md.template`.
-4. Update the project README.md with new version in EasyBuild script (and possibly new `system.json` content).
-5. When all of these changes have been merged into *main* branch create a git tag (:code:`git tag YYYY.MM.MINOR`) for the version on the *main* branch and push it to the GitHub repository.
-6. Create a release on Github from the tag. Use the tag as release title and the release notes as description in MD-format.
-7. Assert that the :ref:`CD setup <continuous_delivery>` finishes correctly by checking that `readthedocs` is updated correctly (fx. that `stable <https://cotainr.readthedocs.io/en/stable>`_ and `latest <https://cotainr.readthedocs.io/en/latest>`_ points to the latest version and that the version is available by itself as well). If the version does not show up all builds from Read the Docs can be found under `versions <https://readthedocs.org/projects/cotainr/versions/>`_.

doc/development/test_suite_ci_cd.rst

@@ -8,16 +8,20 @@ Test suite & CI/CD
 The test suite
 --------------
 
-The `cotainr` test suite is implemented using `pytest <https://docs.pytest.org/>`_ and uses the `pytest-cov <https://docs.pytest.org/>`_ plugin for reporting test coverage. The entire test suite is run from the repository root directory by issuing:
+The `cotainr` test suite is implemented using `pytest <https://docs.pytest.org/>`_ and uses the `pytest-cov <https://docs.pytest.org/>`_ plugin for reporting test coverage. In order to run the test suite locally, first set up a developer environment by installing the test dependencies declared in the `cotainr pyproject.toml file <https://github.com/DeiC-HPC/cotainr/blob/main/pyproject.toml>`_. This can be done using `uv <https://docs.astral.sh/uv/>`_ for setting up the environment:
 
 .. code-block:: console
 
-    $ pytest
+    $ uv sync --group tests
+
+Alternatively, you can also install the (default) `dev` group which contains the full development environment, including the `tests` group.
+
+Once the development environment has been installed, simply run `pytest` from the repository root directory, e.g. using uv:
+
+.. code-block:: console
+
+    $ uv run pytest
 
-Dependencies
-~~~~~~~~~~~~
-In order to run the tests, you must have the Python packages listed in the `test` extra.
-You can use ``pip install -e .[tests]`` to install the required packages.
 
 Pytest marks
 ~~~~~~~~~~~~
@@ -61,6 +65,8 @@ Imports in test modules are based on the following conventions:
 - Sub-package specific fixtures are explicitly imported using relative imports, e.g. :code:`from ..container.data import data_cached_ubuntu_sif` in `tests/pack/test_conda_install.py`.
 - Fixtures defined in `tests/conftest.py` are not explicitly imported (they are implicitly imported by `pytest``). Thus, if a fixture is used, but not imported, in a test module, `tests/conftest.py` is the only module in which it can (or at least should) be defined.
 
+.. _continuous_integration:
+
 Continuous Integration (CI)
 ---------------------------
 Continuous Integration (CI) is handled via `GitHub Actions <https://docs.github.com/en/actions>`_ in the `cotainr` GitHub repository https://github.com/DeiC-HPC/cotainr/actions. The tests run on the GitHub-hosted `ubuntu-latest <https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources>`_ runner. When running the CI test `matrix <https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs>`_, we differentiate between the following (meta)versions of dependencies:
@@ -82,12 +88,27 @@ The following CI `workflows <https://docs.github.com/en/actions/using-workflows/
   Lint and formatting checks (as described in the :ref:`style guide <style_guide>`) are also run and enforced.
 
 
-.. _continuous_delivery:
-
 Continuous Delivery (CD)
 ------------------------
 Continuous Delivery (CD) is handled partly via `GitHub Actions <https://docs.github.com/en/actions>`_, partly via the a `Read the Docs webhook integration <https://docs.readthedocs.io/en/stable/continuous-deployment.html>`_ to the `cotainr` GitHub repository: https://github.com/DeiC-HPC/cotainr/.
 
+CD workflows
+~~~~~~~~~~~~
+The following CD `workflow <https://docs.github.com/en/actions/using-workflows/about-workflows>`_ is implemented:
+
+- `CD_release.yml <https://github.com/DeiC-HPC/cotainr/actions/workflows/CD_release.yml>`_: Creates GitHub and PyPI releases when new tags following the :ref:`versioning scheme <version-scheme>` are committed to the main branch.
+
+  The GitHub release job is run independently and does not have deployment protection rules as it can easily be undone by first removing the release through the GitHub UI and then remove the tag if something goes wrong.
+
+  The PyPI release process goes as following:
+
+  - Build the Python Wheel
+  - Publish to TestPyPI index
+  - In a clean environment, download and install from TestPyPI and run basic CLI functionality
+  - publish to PyPI.
+
+  The testPyPI and PyPI index locations are both implemented as `GitHub environments <https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-deployments/managing-environments-for-deployment>`_ attached to the DeiC-HPC account. These environments have deployment protection rules which require review from a member of the HPC-developers team before the action is executed. This ensures protection against accidental tag pushes which is needed since removal of releases from TestPyPI and PyPI is difficult.
+
 Read the Docs continuous documentation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 The :ref:`HTML documentation <building_the_html_docs>` served at http://cotainr.readthedocs.io is automatically built by Read the Docs based on the configuration in the `.readthedocs.yml configuration file <https://github.com/DeiC-HPC/cotainr/blob/main/.readthedocs.yml>`_.
@@ -108,4 +129,4 @@ Currently, the following scheduled `workflows <https://docs.github.com/en/action
 
 - `SCHED_docs_linkcheck <https://github.com/DeiC-HPC/cotainr/actions/workflows/SCHED_docs_linkcheck.yml>`_: Builds the documentation and checks for any broken hyperlinks.
 
-Additionally, we currently also schedule the `CI_pull_requests.yml <https://github.com/DeiC-HPC/cotainr/actions/workflows/CI_pull_request.yml>`_ workflow which tests the most recent point releases of Python (as provided by GitHub Actions) as well as the most recent Conda version. Ideally, this should be separated into its own workflow that also includes the *latest* versions of Python and Singularity/Apptainer in the test matrix.
+Additionally, we currently also schedule the `CI_pull_requests.yml <https://github.com/DeiC-HPC/cotainr/actions/workflows/CI_pull_request.yml>`_ workflow which tests the most recent point release of Python in the :ref:`CI test matrix <continuous_integration>`.

doc/getting_started.rst

@@ -5,23 +5,18 @@ Getting Started
 
 `cotainr` makes it easy to build `Singularity`_/`Apptainer`_ containers for certain :ref:`use cases <use_cases>`. If this is your first time learning about `cotainr`, we encourage you to read the :ref:`motivation for using cotainr <why_cotainr>`.
 
-In order to get started using `cotainr`, you need to download and install the `cotainr` source code. Please note that cotainr only supports x86_64 and ARM64/AArch64 architectures. Alternative, you may be in luck that you are using an HPC system on which `cotainr` is already installed.
+In order to get started, you can install `cotainr` directly using pip.
 
-.. grid:: 1
-    :gutter: 4
+.. code-block:: console
 
-    .. grid-item-card:: Getting the `cotainr` source code
-        :class-card: sd-rounded-0 sd-shadow-sm
+    $ pip install cotainr
 
-        All releases of `cotainr` are available on GitHub: https://github.com/DeiC-HPC/cotainr/releases
-
-        .. dropdown:: Installation instructions
-            :animate: fade-in
-            :color: secondary
+Additionally, you need to have `Singularity`_ or `Apptainer`_ installed. More details about dependencies may be found in the :ref:`User Guide <cotainr_dependencies>`.
 
-            `cotainr` only runs on Linux and requires that `Python`_ >=3.9 as well as `Singularity`_ >=3.7.4 [#]_ or `Apptainer`_ >=1.0.0 is installed on the system. More details about dependencies may be found in the :ref:`User Guide <cotainr_dependencies>`.
+Alternatively, if you are using an HPC system, you may be in luck that `cotainr` is already installed on the system. Refer to the instructions below for getting started in that case.
 
-            To install `cotainr`, download and unpack the source code. Then add the :code:`cotainr/bin` directory to your :code:`PATH` to get access to the :ref:`cotainr command line interface <command_line_interface>`.
+.. grid:: 1
+    :gutter: 4
 
     .. grid-item-card:: Using `cotainr` on HPC systems where it is already installed
 
@@ -50,6 +45,3 @@ In order to get started using `cotainr`, you need to download and install the `c
 .. _Apptainer: https://apptainer.org/
 .. _Python: https://www.python.org/
 .. _Singularity: https://sylabs.io/singularity/
-
-.. rubric:: Footnotes
-.. [#] As of version 3.8.0, Sylabs have changed the name of their version to SingularityCE. However, we only officially support their version from 3.9.2.