Optional pydantic by pixie79 · Pull Request #736 · astronomer/astronomer-cosmos

pixie79 · 2023-12-02T14:57:57Z

Description

Converts pydantic to being optional, as it is required for Airflow 2.6 but not others.

Related Issue(s)

Closes #725
Closes #654

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

netlify · 2023-12-02T14:58:03Z

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name	Link
🔨 Latest commit	`6849341`
🔍 Latest deploy log	https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/656dc93f30178b00080e716f

pixie79 · 2023-12-04T10:50:14Z

@tatiana @jlaneve, Tests passing with Pydantic now marked as optional, as you ok to approve this PR? - thanks

pixie79 · 2023-12-04T11:49:59Z

I don’t think the failures here have anything to do with Pydantic now ?
As they relate to cache being removed in the main branch

tatiana · 2023-12-04T12:43:14Z

@pixie79 thanks! I fixed the issues in the main branch, and rebase your branch. Once the tests pass in your PR, I'll merge it.

codecov · 2023-12-04T13:02:12Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (e9950d7) 92.99% compared to head (6849341) 92.99%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #736   +/-   ##
=======================================
  Coverage   92.99%   92.99%           
=======================================
  Files          55       55           
  Lines        2313     2313           
=======================================
  Hits         2151     2151           
  Misses        162      162

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@perttus

Features * Add ProfileMapping for Vertica by @perttus in #540 and #688 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add support to select using (some) graph operators when using LoadMode.CUSTOM and LoadMode.DBT_MANIFEST by @tatiana in #728 * Add cosmos/propagate_logs Airflow config support for disabling log pr… by @agreenburg in #648 * Add operator_args full_refresh as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ProjectConfig by @jbandoro in #735 Enhancements * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to dbt_packages when dbt_deps is False when using LoadMode.DBT_LS by @DanMawdsleyBA in #730 * Support no profile_config for ExecutionMode.KUBERNETES and ExecutionMode.DOCKER by @MrBones757 and @tatiana in #681 and #731 * Add aws_session_token for Athena mapping by @benjamin-awd in #663 Others * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 * Speed up integration tests by @jbandoro in #732

@perttus

Features * Add ProfileMapping for Vertica by @perttus in #540 and #688 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add support to select using (some) graph operators when using LoadMode.CUSTOM and LoadMode.DBT_MANIFEST by @tatiana in #728 * Add cosmos/propagate_logs Airflow config support for disabling log pr… by @agreenburg in #648 * Add operator_args full_refresh as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ProjectConfig by @jbandoro in #735 Enhancements * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to dbt_packages when dbt_deps is False when using LoadMode.DBT_LS by @DanMawdsleyBA in #730 * Support no profile_config for ExecutionMode.KUBERNETES and ExecutionMode.DOCKER by @MrBones757 and @tatiana in #681 and #731 * Add aws_session_token for Athena mapping by @benjamin-awd in #663 Others * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 * Speed up integration tests by @jbandoro in #732

@DanMawdsleyBA

**Features** * Add `ProfileMapping` for Snowflake encrypted private key path by @ivanstillfront in #608 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in #616 * Add support to select using (some) graph operators when using `LoadMode.CUSTOM` and `LoadMode.DBT_MANIFEST` by @tatiana in #728 * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in #648 * Add operator_args ``full_refresh`` as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in #735 **Enhancements** * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to `dbt_packages` when `dbt_deps` is False when using `LoadMode.DBT_LS` by @DanMawdsleyBA in #730 * Support no `profile_config` for `ExecutionMode.KUBERNETES` and `ExecutionMode.DOCKER` by @MrBones757 and @tatiana in #681 and #731 * Add `aws_session_token` for Athena mapping by @benjamin-awd in #663 **Others** * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 * Speed up integration tests by @jbandoro in #732

@woogakoki

**Features** * Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in #733 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)). * Add support to select using (some) graph operators when using ``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in #728 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude)) * Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in #755, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)). * Add ``ProfileMapping`` for Vertica by @perttus in #540, #688 and #741, as ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)). * Add ``ProfileMapping`` for Snowflake encrypted private key path by @ivanstillfront in #608, as ([documentation]( https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)). * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro in #616, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)). * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in #648, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)). * Add operator_args ``full_refresh`` as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in #735 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)). * Support disabling event tracking when using Cosmos profile mapping by @jbandoro in #768, ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)). **Enhancements** * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in #730 * Add ``aws_session_token`` for Athena mapping by @benjamin-awd in #663 * Retrieve temporary credentials from ``conn_id`` for Athena by @octiva in #758 * Extend ``DbtDocsLocalOperator`` with static flag by @joppevos in #759 **Bug fixes** * Remove Pydantic upper version restriction so Cosmos can be used with Airflow 2.8 by @jlaneve in #772 **Others** * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Speed up integration tests by @jbandoro in #732 * Fix README quickstart link in by @RNHTTR in #776 * Add package location to work with hatchling 1.19.0 by @jbandoro in #761 * Fix type check error in ``DbtKubernetesBaseOperator.build_env_args`` by @jbandoro in #766 * Improve ``DBT_MANIFEST`` documentation by @dwreeves in #757 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 and #779 * pre-commit updates in #775, #770, #762

@woogakoki

**Features** * Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in astronomer#733 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)). * Add support to select using (some) graph operators when using ``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in astronomer#728 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude)) * Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in astronomer#755, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)). * Add ``ProfileMapping`` for Vertica by @perttus in astronomer#540, astronomer#688 and astronomer#741, as ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)). * Add ``ProfileMapping`` for Snowflake encrypted private key path by @ivanstillfront in astronomer#608, as ([documentation]( https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)). * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in astronomer#649 * Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro in astronomer#616, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)). * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in astronomer#648, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)). * Add operator_args ``full_refresh`` as a templated field by @joppevos in astronomer#623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in astronomer#735 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)). * Support disabling event tracking when using Cosmos profile mapping by @jbandoro in astronomer#768, ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)). **Enhancements** * Make Pydantic an optional dependency by @pixie79 in astronomer#736 * Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in astronomer#730 * Add ``aws_session_token`` for Athena mapping by @benjamin-awd in astronomer#663 * Retrieve temporary credentials from ``conn_id`` for Athena by @octiva in astronomer#758 * Extend ``DbtDocsLocalOperator`` with static flag by @joppevos in astronomer#759 **Bug fixes** * Remove Pydantic upper version restriction so Cosmos can be used with Airflow 2.8 by @jlaneve in astronomer#772 **Others** * Replace flake8 for Ruff by @joppevos in astronomer#743 * Reduce code complexity to 8 by @joppevos in astronomer#738 * Speed up integration tests by @jbandoro in astronomer#732 * Fix README quickstart link in by @RNHTTR in astronomer#776 * Add package location to work with hatchling 1.19.0 by @jbandoro in astronomer#761 * Fix type check error in ``DbtKubernetesBaseOperator.build_env_args`` by @jbandoro in astronomer#766 * Improve ``DBT_MANIFEST`` documentation by @dwreeves in astronomer#757 * Update conflict matrix between Airflow and dbt versions by @tatiana in astronomer#731 and astronomer#779 * pre-commit updates in astronomer#775, astronomer#770, astronomer#762

initially designated Pydantic as an optional dependency. However, a subsequent pull request, #794, introduced an implementation that requires Pydantic to be a mandatory dependency for astronomer-cosmos. This pull request addresses the bug encountered while running airflow db init, as described in issue #936, by enforcing Pydantic as a required dependency. Furthermore, it's worth noting that Pydantic version 2 has known issues with Airflow versions prior to 2.7.0. Given that the minimum Airflow version supported by astronomer-cosmos is 2.3, we have restricted the Pydantic version to be less than 2. Additionally, a comment has been added indicating that we may remove this upper bound once Airflow 2.7 becomes the minimum required version for astronomer-cosmos. closes: #936

The pull request #736 initially designated Pydantic as an optional dependency. However, a subsequent pull request, #794 as an [import](https://github.com/astronomer/astronomer-cosmos/pull/794/files#diff-bd3fa47d7a9b96d7bb365f3ba3b60eaf0b20e06a48b28814cf6f6e6fb64d4da6R12), introduced an implementation that requires Pydantic to be a mandatory dependency for astronomer-cosmos. This pull request addresses the bug encountered while running `airflow db init`, as described in #936, by enforcing Pydantic as a required dependency. Closes: #936

Converts pydantic to being optional, as it is required for Airflow 2.6 but not others. Closes astronomer#725 Closes astronomer#654

@DanMawdsleyBA

**Features** * Add `ProfileMapping` for Snowflake encrypted private key path by @ivanstillfront in astronomer#608 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in astronomer#649 * Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in astronomer#616 * Add support to select using (some) graph operators when using `LoadMode.CUSTOM` and `LoadMode.DBT_MANIFEST` by @tatiana in astronomer#728 * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in astronomer#648 * Add operator_args ``full_refresh`` as a templated field by @joppevos in astronomer#623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in astronomer#735 **Enhancements** * Make Pydantic an optional dependency by @pixie79 in astronomer#736 * Create a symbolic link to `dbt_packages` when `dbt_deps` is False when using `LoadMode.DBT_LS` by @DanMawdsleyBA in astronomer#730 * Support no `profile_config` for `ExecutionMode.KUBERNETES` and `ExecutionMode.DOCKER` by @MrBones757 and @tatiana in astronomer#681 and astronomer#731 * Add `aws_session_token` for Athena mapping by @benjamin-awd in astronomer#663 **Others** * Replace flake8 for Ruff by @joppevos in astronomer#743 * Reduce code complexity to 8 by @joppevos in astronomer#738 * Update conflict matrix between Airflow and dbt versions by @tatiana in astronomer#731 * Speed up integration tests by @jbandoro in astronomer#732

@woogakoki

**Features** * Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in astronomer#733 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)). * Add support to select using (some) graph operators when using ``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in astronomer#728 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude)) * Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in astronomer#755, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)). * Add ``ProfileMapping`` for Vertica by @perttus in astronomer#540, astronomer#688 and astronomer#741, as ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)). * Add ``ProfileMapping`` for Snowflake encrypted private key path by @ivanstillfront in astronomer#608, as ([documentation]( https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)). * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in astronomer#649 * Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro in astronomer#616, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)). * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in astronomer#648, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)). * Add operator_args ``full_refresh`` as a templated field by @joppevos in astronomer#623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in astronomer#735 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)). * Support disabling event tracking when using Cosmos profile mapping by @jbandoro in astronomer#768, ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)). **Enhancements** * Make Pydantic an optional dependency by @pixie79 in astronomer#736 * Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in astronomer#730 * Add ``aws_session_token`` for Athena mapping by @benjamin-awd in astronomer#663 * Retrieve temporary credentials from ``conn_id`` for Athena by @octiva in astronomer#758 * Extend ``DbtDocsLocalOperator`` with static flag by @joppevos in astronomer#759 **Bug fixes** * Remove Pydantic upper version restriction so Cosmos can be used with Airflow 2.8 by @jlaneve in astronomer#772 **Others** * Replace flake8 for Ruff by @joppevos in astronomer#743 * Reduce code complexity to 8 by @joppevos in astronomer#738 * Speed up integration tests by @jbandoro in astronomer#732 * Fix README quickstart link in by @RNHTTR in astronomer#776 * Add package location to work with hatchling 1.19.0 by @jbandoro in astronomer#761 * Fix type check error in ``DbtKubernetesBaseOperator.build_env_args`` by @jbandoro in astronomer#766 * Improve ``DBT_MANIFEST`` documentation by @dwreeves in astronomer#757 * Update conflict matrix between Airflow and dbt versions by @tatiana in astronomer#731 and astronomer#779 * pre-commit updates in astronomer#775, astronomer#770, astronomer#762

The pull request astronomer#736 initially designated Pydantic as an optional dependency. However, a subsequent pull request, astronomer#794 as an [import](https://github.com/astronomer/astronomer-cosmos/pull/794/files#diff-bd3fa47d7a9b96d7bb365f3ba3b60eaf0b20e06a48b28814cf6f6e6fb64d4da6R12), introduced an implementation that requires Pydantic to be a mandatory dependency for astronomer-cosmos. This pull request addresses the bug encountered while running `airflow db init`, as described in astronomer#936, by enforcing Pydantic as a required dependency. Closes: astronomer#936

jlaneve and others added 6 commits November 6, 2023 15:38

Remove Pydantic dependency

534926e

Merge branch 'main' into remove-pydantic

685b71f

add pydantic only for apache-airflow 2.6

c4d3fb2

fix pyproject

959b4f6

Merge branch 'main' into remove-pydantic

64a2b31

Merge branch 'remove-pydantic' into remove-pydantic

45e45f8

pixie79 requested a review from a team as a code owner December 2, 2023 14:57

pixie79 requested a review from a team December 2, 2023 14:57

dosubot Bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Dec 2, 2023

pixie79 had a problem deploying to external December 2, 2023 14:58 — with GitHub Actions Error

dosubot Bot added the area:dependencies Related to dependencies, like Python packages, library versions, etc label Dec 2, 2023

add pydantic as a dependency for 2.6

ebe0a9d

pixie79 temporarily deployed to external December 2, 2023 17:24 — with GitHub Actions Inactive

move override

8fb6ea9

pixie79 temporarily deployed to external December 4, 2023 10:43 — with GitHub Actions Inactive

Merge branch 'main' into remove-pydantic

ca76ea5

tatiana temporarily deployed to external December 4, 2023 11:30 — with GitHub Actions Inactive

Merge branch 'main' into remove-pydantic

6849341

tatiana temporarily deployed to external December 4, 2023 12:42 — with GitHub Actions Inactive

tatiana approved these changes Dec 4, 2023

View reviewed changes

tatiana merged commit abeb15c into astronomer:main Dec 4, 2023

tatiana added this to the 1.3.0 milestone Dec 4, 2023

tatiana mentioned this pull request Dec 7, 2023

Release 1.3.0a3 #753

Merged

tatiana mentioned this pull request Jan 4, 2024

Release 1.3.0 #781

Merged

pankajkoti mentioned this pull request May 3, 2024

Make Pydantic a required dependency #939

Merged

2 tasks

arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024

Optional pydantic (astronomer#736)

7456681

Converts pydantic to being optional, as it is required for Airflow 2.6 but not others. Closes astronomer#725 Closes astronomer#654

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optional pydantic#736

Optional pydantic#736
tatiana merged 10 commits into
astronomer:mainfrom
pixie79:remove-pydantic

pixie79 commented Dec 2, 2023

Uh oh!

netlify Bot commented Dec 2, 2023 •

edited

Loading

Uh oh!

pixie79 commented Dec 4, 2023

Uh oh!

pixie79 commented Dec 4, 2023

Uh oh!

tatiana commented Dec 4, 2023

Uh oh!

codecov Bot commented Dec 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pixie79 commented Dec 2, 2023

Description

Related Issue(s)

Checklist

Uh oh!

netlify Bot commented Dec 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Uh oh!

pixie79 commented Dec 4, 2023

Uh oh!

pixie79 commented Dec 4, 2023

Uh oh!

tatiana commented Dec 4, 2023

Uh oh!

codecov Bot commented Dec 4, 2023

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

netlify Bot commented Dec 2, 2023 •

edited

Loading