Update CI tools and dependencies (#24)

* Adapt to pydantic v2.4, fastapi 0.103

* Let pydantic return a SchemaError instead of InconsistentIamcType

* Use pandera.DataFrameModel instead of pandera.SchemaModel

* Clean up changes in API layer

* Make creation information optional in iamc/variable
* Make id and name required for unit
* Revert bulk_* changes to use json_encoder of api.DataFrame again
* NOTE: this option will be deprecated in the future,
* pydantic serialization decorators should be used instead

* Clean up changes in data/db to enable tests again

* Revert use of Annotated in iamc/datapoint/filter
* Make sqla_model a proper ClassVar (rather than private attr)

* Set field.alias only when it does not equal field.name

* Update version of ruff

* Set field alias in Field directly

* Use httpx-preferred content to send bytes/raw content/json st

* Use alias for model_ fields that conflict with pydantic protected namespace

* Use variable rather than unmapped class reference to resolve SAwarning

* Delete unused import

* Bump mypy and poetry versions

* Use pre-commit over lint-GHA

* Runs mypy, black, ruff
* NOTE: needs to be run in the repo s.t. poetry detects the correct venv
* Remove separate lint.yaml action

* Bump sphinx version

* Update docstrings according to new sphinx version

* Delete duplicate docs files

* TEMPORARY Use own fork for sphinxcontrib-openapi workaround

* Bump dask,sqlalchemy,pre-commit versions

* restore filter framework api

* fix union types with filter subojects

* fix code style

* fix formatting error

* Remove outdated comments

* Update various things:

* Bump fastapi,black,ruff,mypy versions
* Update pre-commit-config accordingly
* Update doc/source/openapi-v1.json

* Include pre-commit, ruff, etc in DEVELOPING.md

---------

Co-authored-by: Max Wolschlager <[email protected]>

Author: glatterf42

.github/workflows/lint.yaml

@@ -46,7 +46,6 @@ jobs:
           virtualenvs-create: true
           virtualenvs-in-project: true
           installer-parallel: true
-
       #----------------------------------------------
       #       load cached venv if cache exists
       #----------------------------------------------
@@ -74,3 +73,46 @@ jobs:
         run: |
           source .venv/bin/activate
           pytest --cov-report xml:.coverage.xml --cov-report term --cov=ixmp4 -rsx --benchmark-skip
+
+  pre-commit:
+    name: Code quality
+    runs-on: ubuntu-latest
+    steps:
+      #----------------------------------------------
+      #       check-out repo and set-up python
+      #----------------------------------------------
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+      #----------------------------------------------
+      #  -----  install & configure poetry  -----
+      #----------------------------------------------
+      - name: Install Poetry
+        uses: snok/install-poetry@v1
+        with:
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+          installer-parallel: true
+      #----------------------------------------------
+      #       load cached venv if cache exists
+      #----------------------------------------------
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v2
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
+      #----------------------------------------------
+      # install dependencies if cache does not exist
+      #----------------------------------------------
+      - name: Install dependencies
+        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+        run: poetry install --no-interaction --no-root --with dev,server
+      #----------------------------------------------
+      # install your root project, if required
+      #----------------------------------------------
+      - name: Install library
+        run: poetry install --no-interaction
+      #----------------------------------------------
+      # run pre-commit/(mypy, black, ruff)
+      #----------------------------------------------
+      - uses: pre-commit/[email protected]

.github/workflows/pytest.yaml

@@ -0,0 +1,15 @@
+repos:
+- repo: https://github.com/pre-commit/mirrors-mypy
+  rev: v1.6.1
+  hooks:
+  - id: mypy
+    entry: bash -c "poetry run mypy ."
+    language: system
+- repo: https://github.com/psf/black
+  rev: 23.10.0
+  hooks:
+  - id: black
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  rev: v0.1.0
+  hooks:
+  - id: ruff

.pre-commit-config.yaml

@@ -317,3 +317,53 @@ It is overwritten on the fly by the poetry-dynamic-versioning plugin.
    (without the rc<N>).
 1. Check that the "Publish to PyPI and TestPyPI" GitHub action passed and that the
   distributions are published on https://pypi.org/project/ixmp4/ .
+
+## Contributing
+
+Contributions to the code are always welcome! Please make sure your code follows our 
+code style so that the style is consistent. Each PR will be checked by a Code Quality 
+test that examines compliance with black, ruff, and mypy. 
+
+### Running pre-commit locally
+
+We use [pre-commit](https://pre-commit.com/) to check the code style. You can install 
+pre-commit locally by installing ixmp4 with the optional `dev` group. Running
+
+```bash
+pre-commit install
+```
+
+will set pre-commit up to run on every `git commit`. Per default, pre-commit will run 
+on changed files, but if you want to run it on all files, you can run
+
+```bash
+pre-commit run --all-files
+```
+
+If you only want certain hooks to run, choose from `ruff`, `black`, and `mypy` as 
+`hook-ids` and run
+
+```bash
+pre-commit run <hook-ids> --all-files
+```
+
+### Ensuring compliance
+
+Whether you run pre-commit locally or see it on your PR for the first time, the checks 
+are the same. You can, of course, run the code style tools manually. From within the 
+ixmp4 directory, this would look similar to this:
+
+```bash
+black .
+mypy .
+ruff check .
+
+# Or to enable ruff's automic fixes
+ruff check --fix .
+```
+
+However, it is easy to forget running these commands manually. Therefore, we recommend 
+setting your editor up to run at least 
+[black](https://black.readthedocs.io/en/stable/integrations/editors.html) and 
+[ruff](https://docs.astral.sh/ruff/usage/#vs-code) automatically whenever you hit 
+`save`. A few minutes of configuration will save you time and nerves later on.

DEVELOPING.md

@@ -10,6 +10,7 @@ IAMC Data
    :members:
    :undoc-members:
    :show-inheritance:
+   :noindex:
 
 
 .. automodule:: ixmp4.core.iamc.repository

doc/source/devs/ixmp4.core/iamc.rst

@@ -6,9 +6,10 @@ Core API (ixmp4.core)
 
    platform
    run
+   model
+   scenario
    iamc
    meta
    region
    unit
    exceptions
-

doc/source/devs/ixmp4.core/modules.rst

@@ -1,6 +1,6 @@
 
 Scenarios
-=======
+=========
 
 .. toctree::
    :maxdepth: 1

doc/source/devs/ixmp4.core/scenario.rst

@@ -44,7 +44,7 @@ def login(
     ),
 ):
     try:
-        auth = ManagerAuth(username, password, settings.manager_url)
+        auth = ManagerAuth(username, password, str(settings.manager_url))
         user = auth.get_user()
         utils.good(f"Successfully authenticated as user '{user.username}'.")
         if typer.confirm(