Skip to content

Latest commit

 

History

History
460 lines (340 loc) · 25.7 KB

STATIC_CODE_CHECKS.rst

File metadata and controls

460 lines (340 loc) · 25.7 KB

The static code checks in Airflow are used to verify that the code meets certain quality standards. All the static code checks can be run through pre-commit hooks.

The pre-commit hooks perform all the necessary installation when you run them for the first time. See the table below to identify which pre-commit checks require the Breeze Docker images.

You can also run some static code checks via Breeze environment using available bash scripts.

Pre-commit hooks help speed up your local development cycle and place less burden on the CI infrastructure. Consider installing the pre-commit hooks as a necessary prerequisite.

The pre-commit hooks by default only check the files you are currently working on and make them fast. Yet, these checks use exactly the same environment as the CI tests use. So, you can be sure your modifications will also work for CI if they pass pre-commit hooks.

We have integrated the fantastic pre-commit framework in our development workflow. To install and use it, you need at least Python 3.6 locally.

It is the best to use pre-commit hooks when you have your local virtualenv for Airflow activated since then pre-commit hooks and other dependencies are automatically installed. You can also install the pre-commit hooks manually using pip install.

pip install pre-commit

After installation, pre-commit hooks are run automatically when you commit the code and they will only run on the files that you change during your commit, so they are usually pretty fast and do not slow down your iteration speed on your changes. There are also ways to disable the pre-commits temporarily when you commit your code with --no-verify switch or skip certain checks that you find to much disturbing your local workflow. See `Available pre-commit checks<#available-pre-commit-checks>`_ and Using pre-commit

Note

Additional prerequisites might be needed

The pre-commit hooks use several external linters that need to be installed before pre-commit is run. Each of the checks installs its own environment, so you do not need to install those, but there are some checks that require locally installed binaries. On Linux, you typically install them with sudo apt install, on macOS - with brew install.

The current list of prerequisites is limited to xmllint:

  • on Linux, install via sudo apt install libxml2-utils
  • on macOS, install via brew install libxml2

Some pre-commit hooks also require the Docker Engine to be configured as the static checks are executed in the Docker environment (See table in the Available pre-commit checks<#available-pre-commit-checks>`_ . You should build the images locally before installing pre-commit checks as described in BREEZE.rst.

Sometimes your image is outdated and needs to be rebuilt because some dependencies have been changed. In such cases, the Docker-based pre-commit will inform you that you should rebuild the image.

In case you do not have your local images built, the pre-commit hooks fail and provide instructions on what needs to be done.

To turn on pre-commit checks for commit operations in git, enter:

pre-commit install

To install the checks also for pre-push operations, enter:

pre-commit install -t pre-push

For details on advanced usage of the install method, use:

pre-commit install --help

This table lists pre-commit hooks used by Airflow. The Breeze column indicates which hooks require Breeze Docker images to be installed locally.

Note

Disabling particular checks

In case you have a problem with running particular pre-commit check you can still continue using the benefits of having pre-commit installed, with some of the checks disabled. In order to disable checks you need to set SKIP environment variable to coma-separated list of checks to skip. For example when you want to skip all checks that require Breeze Docker image to be installed, you should be able to do it by setting export SKIP=bat-in-container-tests,build,flake8,mypy. You can also add this to your .bashrc or .zshrc if you do not want to set it manually every time you enter the terminal.

Checks Description Breeze
airflow-config-yaml Checks that airflow config YAML is 1-1 with the code  
airflow-providers-available Checks that providers are properly declared by extras  
airflow-provider-yaml-files-ok Checks that providers YAML files are valid  
base-operator Checks that BaseOperator is imported properly  
bats-tests Runs BATS bash unit tests  
bats-in-container-tests Run in Breeze container bats tests
black Runs Black (the uncompromising Python code formatter)  
blacken-docs Run black on python code blocks in documentation files  
boring-cyborg Checks for Boring Cyborg configuration consistency  
build Builds image for mypy, flake8
build-providers-dependencies Regenerates the JSON file with cross-provider dependencies  
changelog-duplicates Checks for duplicate changelog entries  
check-apache-license Checks compatibility with Apache License requirements  
check-builtin-literals Require literal syntax when initializing Python builtin types  
check-executables-have-shebangs Checks that executables have shebang  
check-extras-order Checks that extras in Dockerfile are sorted  
check-hooks-apply Checks which hooks are applicable to the repository  
check-integrations Checks if integration list is synchronized in code  
check-merge-conflicts Checks that merge conflicts are not being committed  
check-xml Checks XML files with xmllint  
daysago-import-check Checks if daysago is properly imported  
debug-statements Detects accidentally committed debug statements  
detect-private-key Detects if private key is added to the repository  
doctoc Refreshes the table of contents for MD files  
dont-use-safe-filter Don't use safe in templates  
no-providers-in-core-examples Don't use providers imports in core example DAGs  
no-relative-imports Use absolute imports, not relative  
end-of-file-fixer Makes sure that there is an empty line at the end.  
fix-encoding-pragma Removes encoding header from python files  
pyupgrade Runs pyupgrade  
flake8 Runs flake8
flynt Runs flynt  
codespell Checks for common misspellings in files.  
forbid-tabs Fails if tabs are used in the project  
helm-lint Verifies if helm lint passes for the chart  
identity Prints inputs to the static check hooks for troubleshooting  
incorrect-use-of-LoggingMixin Checks if LoggingMixin is properly imported  
insert-license Adds licenses for most file types  
isort Sorts imports in python files  
json-schema Checks that the files complies with the JSON Schema spec  
language-matters Check for language that we do not accept as community  
lint-dockerfile Lints a dockerfile  
lint-openapi Lints openapi specification  
markdownlint Lints Markdown files  
mermaid Generates diagrams from mermaid files  
mixed-line-ending Detects if mixed line ending is used (r vs. rn)  
mypy Runs mypy
pre-commit-descriptions Check if all pre-commits are described in docs  
pre-commit-hook-names Check that hook names are not overly long  
pretty-format-json Formats json files  
provide-create-sessions Make sure provide-session and create-session imports are OK  
providers-changelogs Updates documentation for providers changelogs  
providers-subpackages-init-file Check that providers' subpackages __init__.py files are there  
providers-init-file Check that provider's __init__.py file is removed  
provider-yamls Checks that provider.yaml files have the correct content  
pydevd Check for accidentally committed pydevd statements  
pydocstyle Runs pydocstyle  
python-no-log-warn Checks if there are no deprecate log warn  
restrict-start_date 'start_date' should not be in default_args in example_dags  
rst-backticks Checks if RST files use double backticks for code  
setup-order Checks for an order of dependencies in setup.py  
setup-extra-packages Checks if all the libraries in setup.py are listed in docs  
shellcheck Checks shell files with shellcheck  
sort-in-the-wild Sort INTHEWILD.md alphabetically  
sort-spelling-wordlist Sort alphabetically and uniquify spelling_wordlist.txt  
stylelint Checks CSS files with stylelint  
trailing-whitespace Removes trailing whitespace at end of line  
ui-lint Static checks of airflow/ui/ folder  
update-breeze-file Update output of breeze command in BREEZE.rst  
update-extras Updates extras in the documentation  
update-local-yml-file Updates mounts in local.yml file  
update-setup-cfg-file Update setup.cfg file with all licenses  
update-versions Updates latest versions in the documentation  
verify-db-migrations-documented Verify DB Migrations have been documented  
www-lint Static checks of js in airflow/www/static/js/ folder  
yesqa Removes unnecessary noqa statements  
yamllint Checks YAML files with yamllint  

After installation, pre-commit hooks are run automatically when you commit the code. But you can run pre-commit hooks manually as needed.

  • Run all checks on your staged files by using:
pre-commit run
  • Run only mypy check on your staged files by using:
pre-commit run mypy
  • Run only mypy checks on all files by using:
pre-commit run mypy --all-files
  • Run all checks on all files by using:
pre-commit run --all-files
  • Run all checks only on files modified in the last locally available commit in your checked out branch:
pre-commit run --source=HEAD^ --origin=HEAD
  • Show files modified automatically by pre-commit when pre-commits automatically fix errors
pre-commit run --show-diff-on-failure
  • Skip one or more of the checks by specifying a comma-separated list of checks to skip in the SKIP variable:
SKIP=mypy,flake8,build pre-commit run --all-files

You can always skip running the tests by providing --no-verify flag to the git commit command.

To check other usage types of the pre-commit framework, see Pre-commit website.

The static code checks can be launched using the Breeze environment.

You run the static code checks via ./breeze static-check or commands.

You can see the list of available static checks either via --help flag or by using the autocomplete option. Note that the all static check runs all configured static checks.

Run the mypy check for the currently staged changes:

./breeze static-check mypy

Run the mypy check for all files:

./breeze static-check mypy -- --all-files

Run the flake8 check for the tests.core.py file with verbose output:

./breeze static-check flake8 -- --files tests/core.py --verbose

Run the flake8 check for the tests.core package with verbose output:

./breeze static-check mypy -- --files tests/hooks/test_druid_hook.py

Run all tests for the currently staged files:

./breeze static-check all

Run all tests for all files:

./breeze static-check all -- --all-files

Run all tests for last commit :

./breeze static-check all -- --from-ref HEAD^ --to-ref HEAD

The license check is run via a separate script and a separate Docker image containing the Apache RAT verification tool that checks for Apache-compatibility of licenses within the codebase. It does not take pre-commit parameters as extra arguments.

./breeze static-check licenses

You can trigger the static checks from the host environment, without entering the Docker container. To do this, run the following scripts:

The scripts may ask you to rebuild the images, if needed.

You can force rebuilding the images by deleting the .build directory. This directory keeps cached information about the images already built and you can safely delete it if you want to start from scratch.

After documentation is built, the HTML results are available in the docs/_build/html folder. This folder is mounted from the host so you can access those files on your host as well.

If you are already in the Breeze Docker environment (by running the ./breeze command), you can also run the same static checks via run_scripts:

  • Mypy: ./scripts/in_container/run_mypy.sh airflow tests
  • Flake8: ./scripts/in_container/run_flake8.sh
  • License check: ./scripts/in_container/run_check_licence.sh
  • Documentation: ./scripts/in_container/run_docs_build.sh

In all static check scripts, both in the container and host versions, you can also pass a module/file path as parameters of the scripts to only check selected modules or files. For example:

In the Docker container:

./scripts/in_container/run_mypy.sh ./airflow/example_dags/

or

./scripts/in_container/run_mypy.sh ./airflow/example_dags/test_utils.py

On the host:

./scripts/ci/static_checks/mypy.sh ./airflow/example_dags/
./scripts/ci/static_checks/mypy.sh ./airflow/example_dags/test_utils.py