diff --git a/Makefile b/Makefile index 1ad9735a..65862258 100644 --- a/Makefile +++ b/Makefile @@ -4,10 +4,10 @@ PYTHON=venv/bin/python3 help: @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' -.PHONY: setup-dev -setup-dev: ## Setup development environment +.PHONY: setup +setup: ## Setup development environment python3 -m venv venv - . venv/bin/activate && pip install ".[tests]" + . venv/bin/activate && pip --no-cache-dir install ".[tests]" @echo "To activate the virtual environment, run:" @echo "source venv/bin/activate" @@ -22,8 +22,9 @@ clean: ## Removes build and test artifacts .PHONY: build-whl -build-whl: setup-dev ## Build installable whl file +build-whl: setup ## Build installable whl file cd dev + rm -rf dev/include/* python3 -m build --outdir dev/include/ .PHONY: docker-run diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md new file mode 100644 index 00000000..1edda9d7 --- /dev/null +++ b/docs/contributing/howto.md @@ -0,0 +1,207 @@ +# Contributing Guide + +All contributions, bug reports, bug fixes, documentation improvements, and enhancements are welcome. + +All contributors and maintainers to this project should abide by the [Contributor Code of Conduct](code_of_conduct.md). + +Learn more about the contributors' roles in [the Roles page](roles.md). + +This document describes how to contribute to DAG Factory, covering: + +- Overview of how to contribute +- How to set up the local development environment +- Running tests +- Pre-commit and linting +- Authoring the documentation +- Releasing + +## Overview of how to contribute + +To contribute to the DAG Factory project: + +1. Please create a [GitHub Issue](https://github.com/astronomer/dag-factory/issues) describing a bug, enhancement, or feature request. +2. Open a branch off of the `main` branch and create a Pull Request into the `main` branch from your feature branch. +3. Link your issue to the pull request. +4. After you complete development on your feature branch, request a review. A maintainer will merge your PR after all reviewers approve it. + +## Set up a local development environment + +### Requirements + +* [Git](https://git-scm.com/) +* [Python](https://www.python.org/) <= 3.12 (due to dependencies, such as ``google-re2`` not supporting Python 3.13 yet) +* [Hatch](https://hatch.pypa.io/latest/) + +Clone the **DAG Factory** repository and change the current working directory to the repo's root directory: + +```bash +git clone https://github.com/astronomer/dag-factory.git +cd dag-factory/ +``` + +After cloning the project, there are two options for setting up the local development environment: + +* Use a Python virtual environment, or +* Use Docker + +### Using a Python virtual environment for local development + +1. Install the project dependencies: + +```bash +make setup +``` + +2. Activate the local python environment: + +```bash +source venv/bin/activate +``` + +3. Set [Apache Airflow®](https://airflow.apache.org/) home to the ``dev/``, so you can see DAG Factory example DAGs. +Disable loading Airflow standard example DAGs: + +```bash +export AIRFLOW_HOME=$(pwd)/dev/ +export AIRFLOW__CORE__LOAD_EXAMPLES=false +``` + +Then, run Airflow in standalone mode; the command below will create a new user (if it does not exist) and run the necessary Airflow component (webserver, scheduler and triggered): + +> Note: By default, Airflow will use sqlite as a database; you can override this by setting the variable ``AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`` to the SQL connection string. + +```bash +airflow standalone +``` + +After Airflow is running, you can access the Airflow UI at ``http://localhost:8080``. + +> Note: whenever you want to start the development server, you need to activate the ``virtualenv`` and set the ``environment variables`` + +### Use Docker for local development + +It is also possible to build the development environment using [Docker](https://www.docker.com/products/docker-desktop/): + +```bash +make docker-run +``` + +After the sandbox is running, you can access the Airflow UI at ``http://localhost:8080``. + +This approach builds a DAG Factory wheel, so if there are code changes, you must stop and restart the containers: + +```bash +make docker-stop +``` + +## Testing application with hatch + +The tests are developed using PyTest and run using hatch. + +The [pyproject. toml](https://github.com/astronomer/dag-factory/blob/main/pyproject.toml) file currently defines a matrix of supported versions of Python and Airflow against which a user can run the tests. + +### Run unit tests + +To run unit tests using Python 3.10 and Airflow 2.5, use the following: + +```bash +hatch run tests.py3.10-2.5:test-cov +``` + +It is also possible to run the tests using all the matrix combinations, by using: + +```bash +hatch run tests:test-cov +``` + +### Run integration tests + +> Note: these tests create local Python virtual environments in a hatch-managed directory. +> They also use the user-defined `AIRFLOW_HOME`, overriding any pre-existing `airflow.cfg` and `airflow.db` files. + +First, set the following environment variables: + +```bash +export AIRFLOW_HOME=$(pwd)/dev/ +export CONFIG_ROOT_DIR=`pwd`"/dev/dags" +export PYTHONPATH=dev/dags:$PYTHONPATH +``` + +To run the integration tests using Python 3.9 and Airflow 2.9, use + +```bash +hatch run tests.py3.9-2.9:test-integration-setup +hatch run tests.py3.9-2.9:test-integration +``` + +## Pre-Commit and linting + +We use pre-commit to run several checks on the code before committing. To install pre-commit hooks, run: + +```bash +pre-commit install +``` + +To run the checks manually, run the following: + +```bash +pre-commit run --all-files +``` + +Pre-commit runs several static checks, including Black and Ruff. It is also possible to run them using ``hatch``: + +```bash +hatch run tests.py3.9-2.9:static-check +``` + +## Write docs + +We use Markdown to author DAG Factory documentation. + +Similar to running tests, we also use hatch to manage the documentation. + +To build the documentation locally: + +```bash +hatch run docs:build +``` + +To serve the documentation locally in `localhost:8080`: + +```bash +hatch run docs:serve +``` + +To release the documentation with the current project version and set it to the latest: + +```bash +hatch run docs:release +``` + +## Releasing + +We currently use [hatch](https://github.com/pypa/hatch) for building and distributing ``dag-factory``. + +We use GitHub actions to create and deploy new releases. To create a new release, update the latest release version. + +It is possible to update the version either by using hatch: + +> Note: You can update the version in several different ways. To learn more, check out the [hatch docs](https://hatch.pypa.io/latest/version/#updating). + +```bash +hatch version minor +``` + +Or by manually updating the value of `__version__` in `dagfactory/__init__.py`. + +Make sure the [CHANGELOG file](https://github.com/astronomer/dag-factory/blob/main/CHANGELOG.md) is up-to-date. + +Create a release using the [GitHub UI](https://github.com/astronomer/dag-factory/releases/new). GitHub will update the package directly to [PyPI](https://pypi.org/project/dag-factory/). + +If you're a [project maintainer in PyPI](https://pypi.org/project/dag-factory/), it is also possible to create a release manually, +by authenticating to PyPI and running the commands: + +```bash +hatch build +hatch publish +``` diff --git a/docs/contributing/roles.md b/docs/contributing/roles.md index f3a7c01f..744dc3d3 100644 --- a/docs/contributing/roles.md +++ b/docs/contributing/roles.md @@ -3,7 +3,7 @@ Contributors are welcome and are greatly appreciated! Every little bit helps, and we give credit to them. This document aims to explain the current roles in the DAG Factory project. -For more information, check the contributing docs. +For more information, check the [contributing docs](howto.md). ## Contributors @@ -41,7 +41,7 @@ General prerequisites that we look for in all candidates: 1. Consistent contribution over last few months 2. Visibility on discussions on the Slack channel or GitHub issues/discussions -3. Contributions to community health and project's sustainability for the long-term -4. Understands the project's contributors' guidelines. +3. Contributes to community health and project's sustainability for the long-term +4. Understands the project's [contributors' guidelines](howto.md). Astronomer is responsible and accountable for releasing new versions of DAG Factory in [PyPI](https://pypi.org/project/dag-factory/), following the [milestones](https://github.com/astronomer/dag-factory/milestones). Astronomer has the right to grant and revoke write access permissions to the project's official repository for any reason it sees fit. diff --git a/docs/index.md b/docs/index.md index 47ffaa00..493a608d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -34,7 +34,7 @@ Having trouble? We'd like to help! DAG Factory is an Open-Source project. Learn about its development process and about how you can contribute: -* Contributing to DAG Factory +* [Contributing to DAG Factory](contributing/howto.md) * [Github repository](https://github.com/astronomer/dag-factory/) ## License diff --git a/pyproject.toml b/pyproject.toml index cb4c4418..d6a32a3d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -40,9 +40,10 @@ dependencies = [ [project.optional-dependencies] tests = [ "apache-airflow-providers-slack", + "build", "pytest>=6.0", "pytest-cov", - "pre-commit" + "pre-commit", ] ######################################