From b8ff9e586616b3ef135e3553e5a75afae80fd9d0 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 13:49:55 +0000 Subject: [PATCH 01/33] Add how to contribute docs --- Makefile | 11 +- docs/contributing/howto.md | 205 +++++++++++++++++++++++++++++++++++++ pyproject.toml | 3 +- 3 files changed, 213 insertions(+), 6 deletions(-) create mode 100644 docs/contributing/howto.md diff --git a/Makefile b/Makefile index 1ad9735a..2412164f 100644 --- a/Makefile +++ b/Makefile @@ -4,10 +4,10 @@ PYTHON=venv/bin/python3 help: @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' -.PHONY: setup-dev -setup-dev: ## Setup development environment - python3 -m venv venv - . venv/bin/activate && pip install ".[tests]" +.PHONY: setup +setup: ## Setup development environment + python3.12 -m venv venv + . venv/bin/activate && pip --no-cache-dir install ".[tests]" @echo "To activate the virtual environment, run:" @echo "source venv/bin/activate" @@ -22,8 +22,9 @@ clean: ## Removes build and test artifacts .PHONY: build-whl -build-whl: setup-dev ## Build installable whl file +build-whl: setup ## Build installable whl file cd dev + rm -rf dev/include/* python3 -m build --outdir dev/include/ .PHONY: docker-run diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md new file mode 100644 index 00000000..77bf08d6 --- /dev/null +++ b/docs/contributing/howto.md @@ -0,0 +1,205 @@ +# Contributing Guide + +All contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome. + +As contributors and maintainers to this project, you are expected to abide by the +[Contributor Code of Conduct](code_of_conduct.md). + +Learn more about the contributors roles in [this page](roles.md). + +This document describes how to contribute to DAG Factory, covering: + +1. Overview in how to contribute +2. How to set up the local development environment +3. Running tests +4. Pre-commit and linting +5. Authoring the documentation +6. Releasing + +## 1. Overview in how to contribute + +To contribute to the DAG Factory project: + +1. Please create a `GitHub Issue `_ describing a bug, enhancement or feature request +2. Open a branch off of the ``main`` branch and create a Pull Request into the ``main`` branch from your feature branch +3. Link your issue to the pull request +4. Once the development is complete on your feature branch, request a review, and it will be merged once approved. + +## 2. Setup a local development environment + +### Requirements + +* [Git](https://git-scm.com/) +* [Python](https://www.python.org/) <= 3.12 (due to dependencies, such as ``google-re2`` not supporting Python 3.13 yet) + +Clone the **DAG Factory** repository and change the current working directory to the repo's root directory: + +```bash +git clone https://github.com/astronomer/dag-factory.git +cd dag-factory/ +``` + +Once the project is cloned, there are two options for development: + +* Use a Python virtual environment, or +* Use Docker + +### a) Using a Python virtual environment for local development + +Install the project dependencies: + +```bash +make setup +``` + +Activate the local python environment: + +```bash +source venv/bin/activate +``` + +Set airflow home to the ``dev/``, so you can see DAG Factory example DAGs. +Disabled loading Airflow standard example DAGs: + +```bash +export AIRFLOW_HOME=$(pwd)/dev/ +export AIRFLOW__CORE__LOAD_EXAMPLES=false +``` + +Then, run Airflow in standalone mode, command below will create a new user (if not exist) and run necessary airflow component (webserver, scheduler and triggerer): + +> Note: By default, Airflow will use sqlite as database, you can overwrite this by set variable ``AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`` to the sql connection string. + +```bash +airflow standalone +``` + +Once Airflow is up, you can access the Airflow UI at ``http://localhost:8080``. + +> Note: whenever you want to start the development server, you need to activate the ``virtualenv`` and set the ``environment variables`` + +### b) Using Docker for local development + +It is also possible to build the development environment using [Docker](https://www.docker.com/products/docker-desktop/): + +```bash +make docker-run +``` + +Once the sandbox is up, you can access the Airflow UI at ``http://localhost:8080``. + +This approach builds a DAG Factory wheel, so if there are code changes, you'll have to stop and restart the containers: + +```bash +make docker-stop +``` + +## 3. Testing application with hatch + +We currently use `hatch `_ for building and distributing ``dag-factory``. + +The tool can also be used for local development. The `pyproject.toml `_ file currently defines a matrix of supported versions of Python and Airflow for which a user can run the tests against. + +For instance, to run the tests using Python 3.10 and `Apache Airflow® `_ 2.5, use the following: + +### a) Running unit tests + +```bash +hatch run tests.py3.10-2.5:test-cov +``` + +It is also possible to run the tests using all the matrix combinations, by using: + +```bash +hatch run tests:test-cov +``` + +### b) Running integration tests + +> Note: these tests will create local Python virtual environments in a hatch managed directory +> They will also use the user-defined `AIRFLOW_HOME`, overriding any pre-existing `airflow.cfg` and `airflow.db` files + +First, set the following environment variables: + +```bash +export AIRFLOW_HOME=$(pwd)/dev/ +export CONFIG_ROOT_DIR=`pwd`"/dev/dags" +export PYTHONPATH=dev/dags:$PYTHONPATH +``` + +To run the integration tests using Python 3.9 and Airflow 2.9, use + +```bash +hatch run tests.py3.9-2.9:test-integration-setup +hatch run tests.py3.9-2.9:test-integration +``` + +## 4. Pre-Commit and linting + +We use pre-commit to run a number of checks on the code before committing. To install pre-commit, run: + +```bash +pre-commit install +``` + +To run the checks manually, run: + +```bash +pre-commit run --all-files +``` + +Pre-commit runs several static checks, including Black amd Ruff. They can also be run using ``hatch``: + +```bash +hatch run tests.py3.9-2.9:static-check +``` + +## 5. Authoring the documentation + +We use Markdown to author DAG Factory documentation. + +Similar to running tests, we also use hatch to manage the documentation. + +To build the documentation locally: + +```bash +hatch run docs:build +``` + +To serve the documentation locally in `localhost:8080`: + +```bash +hatch run docs:serve +``` + +To release the documentation with the current project version, and set it to the latest: + +```bash +hatch run docs:release +``` + +## 6. Releasing + +We use GitHub actions to create and deploy new releases. To create a new release, update the new release version. + +This can be done either by using hatch: + +> Note: You can update the version in a few different ways. Check out the `hatch docs `_ to learn more. + +```bash +hatch version minor +``` + +Or by manually updating the value of `__version__` in `dagfactory/__init__.py`. + +Make sure the [CHANGELOG file](https://github.com/astronomer/dag-factory/blob/main/CHANGELOG.md) is up-to-date. + +Create a release using the [GitHub UI](https://github.com/astronomer/dag-factory/releases/new). The release will be automatically deployed to PyPI. + +If you're a [project maintainer in PyPI](https://pypi.org/project/dag-factory/), it is also possible to create a release manually, +by authenticating to PyPI and running the commands: + +```bash +hatch build +hatch publish +``` diff --git a/pyproject.toml b/pyproject.toml index cb4c4418..d6a32a3d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -40,9 +40,10 @@ dependencies = [ [project.optional-dependencies] tests = [ "apache-airflow-providers-slack", + "build", "pytest>=6.0", "pytest-cov", - "pre-commit" + "pre-commit", ] ###################################### From 365e95cbd807cbc88193379e3a5b320aba52508a Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 14:01:52 +0000 Subject: [PATCH 02/33] Add how to contribute docs --- docs/contributing/howto.md | 49 ++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 23 deletions(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 77bf08d6..ed029ede 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -1,29 +1,29 @@ # Contributing Guide -All contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome. +All contributions, bug reports, bug fixes, documentation improvements, and enhancements are welcome. -As contributors and maintainers to this project, you are expected to abide by the +All contributors and maintainers to this project should abide by the [Contributor Code of Conduct](code_of_conduct.md). -Learn more about the contributors roles in [this page](roles.md). +Learn more about the contributors' roles in [this page](roles.md). This document describes how to contribute to DAG Factory, covering: -1. Overview in how to contribute +1. Overview of how to contribute 2. How to set up the local development environment 3. Running tests 4. Pre-commit and linting 5. Authoring the documentation 6. Releasing -## 1. Overview in how to contribute +## 1. Overview of how to contribute To contribute to the DAG Factory project: 1. Please create a `GitHub Issue `_ describing a bug, enhancement or feature request 2. Open a branch off of the ``main`` branch and create a Pull Request into the ``main`` branch from your feature branch 3. Link your issue to the pull request -4. Once the development is complete on your feature branch, request a review, and it will be merged once approved. +4. Once the development is complete on your feature branch, request a review, and a maintainer will merge it once approved. ## 2. Setup a local development environment @@ -31,6 +31,7 @@ To contribute to the DAG Factory project: * [Git](https://git-scm.com/) * [Python](https://www.python.org/) <= 3.12 (due to dependencies, such as ``google-re2`` not supporting Python 3.13 yet) +* [Hatch](https://hatch.pypa.io/latest/) Clone the **DAG Factory** repository and change the current working directory to the repo's root directory: @@ -39,7 +40,7 @@ git clone https://github.com/astronomer/dag-factory.git cd dag-factory/ ``` -Once the project is cloned, there are two options for development: +After cloning the project, there are two options for setting up the local development environment: * Use a Python virtual environment, or * Use Docker @@ -58,7 +59,7 @@ Activate the local python environment: source venv/bin/activate ``` -Set airflow home to the ``dev/``, so you can see DAG Factory example DAGs. +Set [Apache Airflow®](https://airflow.apache.org/) home to the ``dev/``, so you can see DAG Factory example DAGs. Disabled loading Airflow standard example DAGs: ```bash @@ -66,9 +67,9 @@ export AIRFLOW_HOME=$(pwd)/dev/ export AIRFLOW__CORE__LOAD_EXAMPLES=false ``` -Then, run Airflow in standalone mode, command below will create a new user (if not exist) and run necessary airflow component (webserver, scheduler and triggerer): +Then, run Airflow in standalone mode; the command below will create a new user (if it does not exist) and run the necessary Airflow component (webserver, scheduler and triggered): -> Note: By default, Airflow will use sqlite as database, you can overwrite this by set variable ``AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`` to the sql connection string. +> Note: By default, Airflow will use sqlite as a database; you can override this by setting the variable ``AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`` to the SQL connection string. ```bash airflow standalone @@ -96,14 +97,14 @@ make docker-stop ## 3. Testing application with hatch -We currently use `hatch `_ for building and distributing ``dag-factory``. +The tests are developed using PyTest and run using hatch. -The tool can also be used for local development. The `pyproject.toml `_ file currently defines a matrix of supported versions of Python and Airflow for which a user can run the tests against. - -For instance, to run the tests using Python 3.10 and `Apache Airflow® `_ 2.5, use the following: +The [pyproject. toml](https://github.com/astronomer/dag-factory/blob/main/pyproject.toml) file currently defines a matrix of supported versions of Python and Airflow against which a user can run the tests. ### a) Running unit tests +For instance, to run unit tests using Python 3.10 and Airflow 2.5, use the following: + ```bash hatch run tests.py3.10-2.5:test-cov ``` @@ -116,7 +117,7 @@ hatch run tests:test-cov ### b) Running integration tests -> Note: these tests will create local Python virtual environments in a hatch managed directory +> Note: these tests will create local Python virtual environments in a hatch-managed directory > They will also use the user-defined `AIRFLOW_HOME`, overriding any pre-existing `airflow.cfg` and `airflow.db` files First, set the following environment variables: @@ -136,19 +137,19 @@ hatch run tests.py3.9-2.9:test-integration ## 4. Pre-Commit and linting -We use pre-commit to run a number of checks on the code before committing. To install pre-commit, run: +We use pre-commit to run several checks on the code before committing. To install pre-commit, run: ```bash pre-commit install ``` -To run the checks manually, run: +To run the checks manually, run the following: ```bash pre-commit run --all-files ``` -Pre-commit runs several static checks, including Black amd Ruff. They can also be run using ``hatch``: +Pre-commit runs several static checks, including Black and Ruff. It is also possible to run them using ``hatch``: ```bash hatch run tests.py3.9-2.9:static-check @@ -172,7 +173,7 @@ To serve the documentation locally in `localhost:8080`: hatch run docs:serve ``` -To release the documentation with the current project version, and set it to the latest: +To release the documentation with the current project version and set it to the latest: ```bash hatch run docs:release @@ -180,11 +181,13 @@ hatch run docs:release ## 6. Releasing -We use GitHub actions to create and deploy new releases. To create a new release, update the new release version. +We currently use [hatch](https://github.com/pypa/hatch) for building and distributing ``dag-factory``. + +We use GitHub actions to create and deploy new releases. To create a new release, update the latest release version. -This can be done either by using hatch: +It is possible to update the version either by using hatch: -> Note: You can update the version in a few different ways. Check out the `hatch docs `_ to learn more. +> Note: You can update the version in several different ways. To learn more, check out the [hatch docs](https://hatch.pypa.io/latest/version/#updating). ```bash hatch version minor @@ -194,7 +197,7 @@ Or by manually updating the value of `__version__` in `dagfactory/__init__.py`. Make sure the [CHANGELOG file](https://github.com/astronomer/dag-factory/blob/main/CHANGELOG.md) is up-to-date. -Create a release using the [GitHub UI](https://github.com/astronomer/dag-factory/releases/new). The release will be automatically deployed to PyPI. +Create a release using the [GitHub UI](https://github.com/astronomer/dag-factory/releases/new). GitHub will update the package directly to [PyPI](https://pypi.org/project/dag-factory/). If you're a [project maintainer in PyPI](https://pypi.org/project/dag-factory/), it is also possible to create a release manually, by authenticating to PyPI and running the commands: From b6b62787ad178c4b1545d36cb75e49f6b5bb577a Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 14:14:22 +0000 Subject: [PATCH 03/33] Update contributing guidelines --- docs/index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/index.md b/docs/index.md index f4b92754..e5b1dfa9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -7,12 +7,12 @@ Everything you need to know about how to build Apache Airflow® workflows using Are you new to DAG Factory? This is the place to start! * DAG Factory at a glance - * [Quickstart with Airflow standalone](getting-started/quick-start-airflow-standalone.md) - * [Quickstart with Astro CLI](getting-started/quick-start-astro-cli.md) + * [Quickstart with Airflow standalone](getting-started/quick-start-airflow-standalone.md) + * [Quickstart with Astro CLI](getting-started/quick-start-astro-cli.md) * Install guide * [Using YAML instead of Python](./comparison/index.md) - * [Traditional Airflow Operators](./comparison/traditional_operators.md) - * [TaskFlow API](./comparison/taskflow_api.md) + * [Traditional Airflow Operators](./comparison/traditional_operators.md) + * [TaskFlow API](./comparison/taskflow_api.md) ## Features @@ -34,7 +34,7 @@ Having trouble? We'd like to help! DAG Factory is an Open-Source project. Learn about its development process and about how you can contribute: -* Contributing to DAG Factory +* [Contributing to DAG Factory](contributing/howto.md) * [Github repository](https://github.com/astronomer/dag-factory/) ## License From f93ff4d4ce70562338fe06df3d8220effd2ba90b Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 14:15:09 +0000 Subject: [PATCH 04/33] Remove 3.12 pin from Makefile --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 2412164f..65862258 100644 --- a/Makefile +++ b/Makefile @@ -6,7 +6,7 @@ help: .PHONY: setup setup: ## Setup development environment - python3.12 -m venv venv + python3 -m venv venv . venv/bin/activate && pip --no-cache-dir install ".[tests]" @echo "To activate the virtual environment, run:" @echo "source venv/bin/activate" From 6e2fe6326e1b82fdb62e798a5f9f033e8cc325d2 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 14:27:08 +0000 Subject: [PATCH 05/33] Add links to roles.md document referring the howto.md --- docs/contributing/roles.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/contributing/roles.md b/docs/contributing/roles.md index f3a7c01f..9832e474 100644 --- a/docs/contributing/roles.md +++ b/docs/contributing/roles.md @@ -3,7 +3,7 @@ Contributors are welcome and are greatly appreciated! Every little bit helps, and we give credit to them. This document aims to explain the current roles in the DAG Factory project. -For more information, check the contributing docs. +For more information, check the [contributing docs](howto.md). ## Contributors @@ -42,6 +42,6 @@ General prerequisites that we look for in all candidates: 1. Consistent contribution over last few months 2. Visibility on discussions on the Slack channel or GitHub issues/discussions 3. Contributions to community health and project's sustainability for the long-term -4. Understands the project's contributors' guidelines. +4. Understands the project's [contributors' guidelines](howto.md). Astronomer is responsible and accountable for releasing new versions of DAG Factory in [PyPI](https://pypi.org/project/dag-factory/), following the [milestones](https://github.com/astronomer/dag-factory/milestones). Astronomer has the right to grant and revoke write access permissions to the project's official repository for any reason it sees fit. From 0ba5ad9126c06b56badcd65ee94030cd8a6600a0 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:13:45 +0000 Subject: [PATCH 06/33] Update docs/contributing/howto.md Co-authored-by: Pankaj Koti --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index ed029ede..d9ac2f34 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -20,7 +20,7 @@ This document describes how to contribute to DAG Factory, covering: To contribute to the DAG Factory project: -1. Please create a `GitHub Issue `_ describing a bug, enhancement or feature request +1. Please create a [GitHub Issue](https://github.com/astronomer/dag-factory/issues) describing a bug, enhancement or feature request 2. Open a branch off of the ``main`` branch and create a Pull Request into the ``main`` branch from your feature branch 3. Link your issue to the pull request 4. Once the development is complete on your feature branch, request a review, and a maintainer will merge it once approved. From e50c9787e3d3b7e8ee3c626b4af248cdd6f3e4e5 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:13:52 +0000 Subject: [PATCH 07/33] Update docs/contributing/howto.md Co-authored-by: Pankaj Koti --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index d9ac2f34..2d93c619 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -60,7 +60,7 @@ source venv/bin/activate ``` Set [Apache Airflow®](https://airflow.apache.org/) home to the ``dev/``, so you can see DAG Factory example DAGs. -Disabled loading Airflow standard example DAGs: +Disable loading Airflow standard example DAGs: ```bash export AIRFLOW_HOME=$(pwd)/dev/ From 3347a4bbfc660921c03f97cc703797835f4ae76e Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:14:01 +0000 Subject: [PATCH 08/33] Update docs/contributing/howto.md Co-authored-by: Pankaj Koti --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 2d93c619..48ac0dd3 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -137,7 +137,7 @@ hatch run tests.py3.9-2.9:test-integration ## 4. Pre-Commit and linting -We use pre-commit to run several checks on the code before committing. To install pre-commit, run: +We use pre-commit to run several checks on the code before committing. To install pre-commit hooks, run: ```bash pre-commit install From 37954953544417f546add8346f9f06f2c347acbd Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:14:30 +0000 Subject: [PATCH 09/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 48ac0dd3..2be4ea89 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -2,8 +2,7 @@ All contributions, bug reports, bug fixes, documentation improvements, and enhancements are welcome. -All contributors and maintainers to this project should abide by the -[Contributor Code of Conduct](code_of_conduct.md). +All contributors and maintainers to this project should abide by the [Contributor Code of Conduct](code_of_conduct.md). Learn more about the contributors' roles in [this page](roles.md). From 6590985c5b34586553e8bcf15de4aa746c930810 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:14:38 +0000 Subject: [PATCH 10/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 2be4ea89..0bbc364a 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -4,7 +4,7 @@ All contributions, bug reports, bug fixes, documentation improvements, and enhan All contributors and maintainers to this project should abide by the [Contributor Code of Conduct](code_of_conduct.md). -Learn more about the contributors' roles in [this page](roles.md). +Learn more about the contributors' roles in [the Roles page](roles.md). This document describes how to contribute to DAG Factory, covering: From 6a8df2794d09b44b3bb70246947d5cd12a0d03d1 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:15:18 +0000 Subject: [PATCH 11/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 0bbc364a..808ae553 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -20,7 +20,7 @@ This document describes how to contribute to DAG Factory, covering: To contribute to the DAG Factory project: 1. Please create a [GitHub Issue](https://github.com/astronomer/dag-factory/issues) describing a bug, enhancement or feature request -2. Open a branch off of the ``main`` branch and create a Pull Request into the ``main`` branch from your feature branch +2. Open a branch off of the `main` branch and create a Pull Request into the `main` branch from your feature branch. 3. Link your issue to the pull request 4. Once the development is complete on your feature branch, request a review, and a maintainer will merge it once approved. From f921398420a53c9d33b6c49e2be4b83a50c4ec65 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:15:33 +0000 Subject: [PATCH 12/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 808ae553..200ac8d0 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -21,8 +21,8 @@ To contribute to the DAG Factory project: 1. Please create a [GitHub Issue](https://github.com/astronomer/dag-factory/issues) describing a bug, enhancement or feature request 2. Open a branch off of the `main` branch and create a Pull Request into the `main` branch from your feature branch. -3. Link your issue to the pull request -4. Once the development is complete on your feature branch, request a review, and a maintainer will merge it once approved. +3. Link your issue to the pull request. +4. After you complete development on your feature branch, request a review. A maintainer will merge your PR after all reviewers approve it. ## 2. Setup a local development environment From a97df08cc060e1995fbc86f28d5df3fa19c5e920 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:15:45 +0000 Subject: [PATCH 13/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 200ac8d0..2fd0cfb0 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -116,8 +116,8 @@ hatch run tests:test-cov ### b) Running integration tests -> Note: these tests will create local Python virtual environments in a hatch-managed directory -> They will also use the user-defined `AIRFLOW_HOME`, overriding any pre-existing `airflow.cfg` and `airflow.db` files +> Note: these tests create local Python virtual environments in a hatch-managed directory. +> They also use the user-defined `AIRFLOW_HOME`, overriding any pre-existing `airflow.cfg` and `airflow.db` files. First, set the following environment variables: From 7ae56217eae1d1841bba345d2a9bb44d052f4192 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:16:26 +0000 Subject: [PATCH 14/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 2fd0cfb0..d6cfbc1c 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -24,7 +24,7 @@ To contribute to the DAG Factory project: 3. Link your issue to the pull request. 4. After you complete development on your feature branch, request a review. A maintainer will merge your PR after all reviewers approve it. -## 2. Setup a local development environment +## Set up a local development environment ### Requirements From 9f48478d9548c15c7cc55377a25feb6786813c9b Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:16:36 +0000 Subject: [PATCH 15/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index d6cfbc1c..17712000 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -178,7 +178,7 @@ To release the documentation with the current project version and set it to the hatch run docs:release ``` -## 6. Releasing +## Releasing We currently use [hatch](https://github.com/pypa/hatch) for building and distributing ``dag-factory``. From 5682c22a4520a4e6abc3aa89196d74ece850af73 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:16:50 +0000 Subject: [PATCH 16/33] Update docs/contributing/roles.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/roles.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/roles.md b/docs/contributing/roles.md index 9832e474..744dc3d3 100644 --- a/docs/contributing/roles.md +++ b/docs/contributing/roles.md @@ -41,7 +41,7 @@ General prerequisites that we look for in all candidates: 1. Consistent contribution over last few months 2. Visibility on discussions on the Slack channel or GitHub issues/discussions -3. Contributions to community health and project's sustainability for the long-term +3. Contributes to community health and project's sustainability for the long-term 4. Understands the project's [contributors' guidelines](howto.md). Astronomer is responsible and accountable for releasing new versions of DAG Factory in [PyPI](https://pypi.org/project/dag-factory/), following the [milestones](https://github.com/astronomer/dag-factory/milestones). Astronomer has the right to grant and revoke write access permissions to the project's official repository for any reason it sees fit. From 3d53e6b455db88730cbd0ba2b707fe838975fee2 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:16:58 +0000 Subject: [PATCH 17/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 17712000..a326b0ce 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -154,7 +154,7 @@ Pre-commit runs several static checks, including Black and Ruff. It is also poss hatch run tests.py3.9-2.9:static-check ``` -## 5. Authoring the documentation +## Write docs We use Markdown to author DAG Factory documentation. From 7c2d21f3573fad976f45859695038c77a7352a47 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:17:05 +0000 Subject: [PATCH 18/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index a326b0ce..3ad5fd53 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -134,7 +134,7 @@ hatch run tests.py3.9-2.9:test-integration-setup hatch run tests.py3.9-2.9:test-integration ``` -## 4. Pre-Commit and linting +## Pre-Commit and linting We use pre-commit to run several checks on the code before committing. To install pre-commit hooks, run: From f6122da55098f16752826abb978c790f484fb96d Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:17:43 +0000 Subject: [PATCH 19/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 3ad5fd53..420d91c4 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -15,7 +15,7 @@ This document describes how to contribute to DAG Factory, covering: 5. Authoring the documentation 6. Releasing -## 1. Overview of how to contribute +## Overview of how to contribute To contribute to the DAG Factory project: From b2e21ddc7d9be376de77effe775874a114149e2f Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:17:53 +0000 Subject: [PATCH 20/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 420d91c4..454108ae 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -8,12 +8,12 @@ Learn more about the contributors' roles in [the Roles page](roles.md). This document describes how to contribute to DAG Factory, covering: -1. Overview of how to contribute -2. How to set up the local development environment -3. Running tests -4. Pre-commit and linting -5. Authoring the documentation -6. Releasing +- Overview of how to contribute +- How to set up the local development environment +- Running tests +- Pre-commit and linting +- Authoring the documentation +- Releasing ## Overview of how to contribute From 57f40f3e03d24669fbf51027afd9f600cbc105fb Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:18:03 +0000 Subject: [PATCH 21/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 454108ae..4574473c 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -44,7 +44,7 @@ After cloning the project, there are two options for setting up the local develo * Use a Python virtual environment, or * Use Docker -### a) Using a Python virtual environment for local development +### Using a Python virtual environment for local development Install the project dependencies: From 456045bbb385b9663ca9685a148bbc58b86c44ec Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:18:24 +0000 Subject: [PATCH 22/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 4574473c..13a55745 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -46,7 +46,7 @@ After cloning the project, there are two options for setting up the local develo ### Using a Python virtual environment for local development -Install the project dependencies: +1. Install the project dependencies: ```bash make setup From c349df4150f585c365c8da88f75c7763facd0be6 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:18:31 +0000 Subject: [PATCH 23/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 13a55745..144f7cd7 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -52,7 +52,7 @@ After cloning the project, there are two options for setting up the local develo make setup ``` -Activate the local python environment: +2. Activate the local python environment: ```bash source venv/bin/activate From 343cbe72f6b0e6b5f0adffb5544b9a6881cff5f9 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:18:37 +0000 Subject: [PATCH 24/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 144f7cd7..d4888416 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -58,7 +58,7 @@ make setup source venv/bin/activate ``` -Set [Apache Airflow®](https://airflow.apache.org/) home to the ``dev/``, so you can see DAG Factory example DAGs. +3. Set [Apache Airflow®](https://airflow.apache.org/) home to the ``dev/``, so you can see DAG Factory example DAGs. Disable loading Airflow standard example DAGs: ```bash From 2048068d25cdb7d9bc07909033c9d672c195282c Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:18:48 +0000 Subject: [PATCH 25/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index d4888416..78d4d6fd 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -74,7 +74,7 @@ Then, run Airflow in standalone mode; the command below will create a new user ( airflow standalone ``` -Once Airflow is up, you can access the Airflow UI at ``http://localhost:8080``. +After Airflow is running, you can access the Airflow UI at ``http://localhost:8080``. > Note: whenever you want to start the development server, you need to activate the ``virtualenv`` and set the ``environment variables`` From 5a4a310f67c6510f79f817fe41f8892d17c1dad8 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:18:55 +0000 Subject: [PATCH 26/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 78d4d6fd..a5b401e1 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -78,7 +78,7 @@ After Airflow is running, you can access the Airflow UI at ``http://localhost:80 > Note: whenever you want to start the development server, you need to activate the ``virtualenv`` and set the ``environment variables`` -### b) Using Docker for local development +### Use Docker for local development It is also possible to build the development environment using [Docker](https://www.docker.com/products/docker-desktop/): From 119adf39a774e012209561efcf5e4e2684dfa99c Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:19:02 +0000 Subject: [PATCH 27/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index a5b401e1..f8fc5455 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -86,7 +86,7 @@ It is also possible to build the development environment using [Docker](https:// make docker-run ``` -Once the sandbox is up, you can access the Airflow UI at ``http://localhost:8080``. +After the sandbox is running, you can access the Airflow UI at ``http://localhost:8080``. This approach builds a DAG Factory wheel, so if there are code changes, you'll have to stop and restart the containers: From 170803f1e1c3b3a201101248569bbaf3ac43d6c1 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:19:09 +0000 Subject: [PATCH 28/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index f8fc5455..9c948249 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -88,7 +88,7 @@ make docker-run After the sandbox is running, you can access the Airflow UI at ``http://localhost:8080``. -This approach builds a DAG Factory wheel, so if there are code changes, you'll have to stop and restart the containers: +This approach builds a DAG Factory wheel, so if there are code changes, you must stop and restart the containers: ```bash make docker-stop From 5d5e3076d5d82e663eaf29792295af203d6544ce Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:19:16 +0000 Subject: [PATCH 29/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 9c948249..44317132 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -94,7 +94,7 @@ This approach builds a DAG Factory wheel, so if there are code changes, you must make docker-stop ``` -## 3. Testing application with hatch +## Testing application with hatch The tests are developed using PyTest and run using hatch. From 0dba4015a0ee9ecff8a12781afde22d65974f5f8 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:19:24 +0000 Subject: [PATCH 30/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 44317132..9d4c3296 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -100,7 +100,7 @@ The tests are developed using PyTest and run using hatch. The [pyproject. toml](https://github.com/astronomer/dag-factory/blob/main/pyproject.toml) file currently defines a matrix of supported versions of Python and Airflow against which a user can run the tests. -### a) Running unit tests +### Run unit tests For instance, to run unit tests using Python 3.10 and Airflow 2.5, use the following: From 9198680d1ea95353e168aefb31658abc888e39bb Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:19:33 +0000 Subject: [PATCH 31/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 9d4c3296..83a7b9d5 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -102,7 +102,7 @@ The [pyproject. toml](https://github.com/astronomer/dag-factory/blob/main/pyproj ### Run unit tests -For instance, to run unit tests using Python 3.10 and Airflow 2.5, use the following: +To run unit tests using Python 3.10 and Airflow 2.5, use the following: ```bash hatch run tests.py3.10-2.5:test-cov From 05d1c9eab7cac441b85bfa6aa2cdce4205e068a1 Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:19:39 +0000 Subject: [PATCH 32/33] Update docs/contributing/howto.md Co-authored-by: Laura Zdanski <25642903+lzdanski@users.noreply.github.com> --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 83a7b9d5..6acc0c3b 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -114,7 +114,7 @@ It is also possible to run the tests using all the matrix combinations, by using hatch run tests:test-cov ``` -### b) Running integration tests +### Run integration tests > Note: these tests create local Python virtual environments in a hatch-managed directory. > They also use the user-defined `AIRFLOW_HOME`, overriding any pre-existing `airflow.cfg` and `airflow.db` files. From af863fd3650affc14e8823669d9887b0589d068c Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Mon, 6 Jan 2025 16:21:01 +0000 Subject: [PATCH 33/33] Update docs/contributing/howto.md --- docs/contributing/howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/howto.md b/docs/contributing/howto.md index 6acc0c3b..1edda9d7 100644 --- a/docs/contributing/howto.md +++ b/docs/contributing/howto.md @@ -19,7 +19,7 @@ This document describes how to contribute to DAG Factory, covering: To contribute to the DAG Factory project: -1. Please create a [GitHub Issue](https://github.com/astronomer/dag-factory/issues) describing a bug, enhancement or feature request +1. Please create a [GitHub Issue](https://github.com/astronomer/dag-factory/issues) describing a bug, enhancement, or feature request. 2. Open a branch off of the `main` branch and create a Pull Request into the `main` branch from your feature branch. 3. Link your issue to the pull request. 4. After you complete development on your feature branch, request a review. A maintainer will merge your PR after all reviewers approve it.