Skip to content

Support Snowflake encrypted private key path#608

Merged
tatiana merged 2 commits into
astronomer:mainfrom
metzlar:snowflake-encrypted-pk-path
Oct 20, 2023
Merged

Support Snowflake encrypted private key path#608
tatiana merged 2 commits into
astronomer:mainfrom
metzlar:snowflake-encrypted-pk-path

Conversation

@metzlar
Copy link
Copy Markdown
Contributor

@metzlar metzlar commented Oct 17, 2023

Description

This adds support for Snowflake Airflow Connections with encrypted private key paths via extra.private_key_file and password configuration.

Related Issue(s)

closes #607

Breaking Change?

None

Checklist

  • [ x ] I have made corresponding changes to the documentation (if required)
  • [ x ] I have added tests that prove my fix is effective or that my feature works

@metzlar metzlar requested a review from a team as a code owner October 17, 2023 23:46
@metzlar metzlar requested a review from a team October 17, 2023 23:46
@netlify
Copy link
Copy Markdown

netlify Bot commented Oct 17, 2023 •
edited
Loading

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name Link
🔨 Latest commit ab71a31
🔍 Latest deploy log https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/65315987dd727600080315bd

@metzlar metzlar force-pushed the snowflake-encrypted-pk-path branch from 40e25b8 to a90316a Compare October 17, 2023 23:47
@metzlar metzlar temporarily deployed to external October 17, 2023 23:47 — with GitHub Actions Inactive
jlaneve
jlaneve reviewed Oct 18, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@jlaneve jlaneve left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for opening this so quickly! love the direction this is headed - I wonder if there's a way to converge this new profile mapping with the existing SnowflakePrivateKeyPemProfileMapping, did you try that by chance? I haven't looked into it yet so it may not be possible, was just curious

@codecov
Copy link
Copy Markdown

codecov Bot commented Oct 18, 2023 •
edited
Loading

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (c82e6bc) 93.21% compared to head (ab71a31) 93.28%.
Report is 1 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #608      +/-   ##
==========================================
+ Coverage   93.21%   93.28%   +0.07%     
==========================================
  Files          53       54       +1     
  Lines        2079     2116      +37     
==========================================
+ Hits         1938     1974      +36     
- Misses        141      142       +1
Files Coverage Δ
cosmos/profiles/__init__.py 96.55% <100.00%> (+0.12%) ⬆️
cosmos/profiles/snowflake/__init__.py 100.00% <100.00%> (ø)
cosmos/profiles/snowflake/user_pass.py 97.05% <100.00%> (+0.50%) ⬆️
...os/profiles/snowflake/user_encrypted_privatekey.py 96.66% <96.66%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tatiana tatiana temporarily deployed to external October 18, 2023 20:33 — with GitHub Actions Inactive
@metzlar
Copy link
Copy Markdown
Contributor Author

metzlar commented Oct 18, 2023 •
edited
Loading

thanks for opening this so quickly! love the direction this is headed - I wonder if there's a way to converge this new profile mapping with the existing SnowflakePrivateKeyPemProfileMapping, did you try that by chance? I haven't looked into it yet so it may not be possible, was just curious

@jlaneve I did try this, the existing SnowflakePrivateKeyPemProfileMapping maps to private_key_content but I needed private_key_path (and passhprase) The mapping class became much more complicated when I tried to make it work for both scenarios. (Also because Airflow Snowlake connection re-purposes password for the key passphrase)

tatiana
tatiana approved these changes Oct 19, 2023
View reviewed changes
Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution and addressing @jlaneve comments, @ivanstillfront ! We can merge this PR once the checks pass. Since it's a new feature, we'll release it as part of Cosmos 1.3.

@tatiana tatiana added this to the 1.3.0 milestone Oct 19, 2023
@tatiana tatiana temporarily deployed to external October 19, 2023 10:50 — with GitHub Actions Inactive
@tatiana
Copy link
Copy Markdown
Collaborator

tatiana commented Oct 19, 2023

@ivanstillfront please, could you add tests covering these lines pointed out by codecov?

Screenshot 2023-10-19 at 14 29 29

Once this is done, we can merge this PR.

@metzlar metzlar force-pushed the snowflake-encrypted-pk-path branch from 79eac78 to f0e4ccf Compare October 19, 2023 16:29
@metzlar metzlar temporarily deployed to external October 19, 2023 16:32 — with GitHub Actions Inactive
@tatiana tatiana merged commit b7381c8 into astronomer:main Oct 20, 2023
tatiana added a commit that referenced this pull request Oct 25, 2023
@tatiana
Release 1.3.0a1
c0a86d4 
Features

* Add ProfileMapping for Vertica by @perttus in #540
* Add ProfileMapping for Snowflake encrypted private key path by @ivanstillfront in #608
tatiana added a commit that referenced this pull request Oct 25, 2023
@tatiana
Release 1.3.0a1
ea4b6bf 
Features

* Add `ProfileMapping` for Vertica by @perttus in #540
* Add `ProfileMapping` for Snowflake encrypted private key path by @ivanstillfront in #608
* Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in #616
tatiana added a commit that referenced this pull request Oct 25, 2023
@tatiana
Release 1.3.0a1
136dfed 
Features

* Add ProfileMapping for Vertica by @perttus in #540
* Add ProfileMapping for Snowflake encrypted private key path by @ivanstillfront in #608
* Add DbtDocsGCSOperator for uploading dbt docs to GCS by @jbandoro in #616
@tatiana tatiana mentioned this pull request Oct 25, 2023
Merged
tatiana added a commit that referenced this pull request Oct 27, 2023
@tatiana
Release 1.3.0a1 (#628)
c1a1406 
Features

* Add `ProfileMapping` for Vertica by @perttus in #540
* Add `ProfileMapping` for Snowflake encrypted private key path by
@ivanstillfront in #608
* Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in
#616
@tatiana
Copy link
Copy Markdown
Collaborator

tatiana commented Oct 27, 2023

@ivanstillfront , this change is available in the 1.3.0a1 pre-release of Cosmos:

Thank you very much for your contribution!

LennartKloppenburg pushed a commit to LennartKloppenburg/astronomer-cosmos that referenced this pull request Nov 9, 2023
@metzlar @LennartKloppenburg
Add ProfileMapping for Snowflake encrypted private key path (astronom…
08782fc 
…er#608)

Add support for Snowflake Airflow Connections with encrypted
private key paths via `extra.private_key_file` and `password`
configuration.

Closes: astronomer#607
@tatiana tatiana mentioned this pull request Nov 23, 2023
Merged
@tatiana tatiana mentioned this pull request Dec 7, 2023
Merged
jbandoro pushed a commit that referenced this pull request Dec 7, 2023
@tatiana
Release 1.3.0a3 (#753)
b1cdc3c 
**Features**

* Add `ProfileMapping` for Snowflake encrypted private key path by
@ivanstillfront in #608
* Add support for Snowflake encrypted private key environment variable
by @DanMawdsleyBA in #649
* Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in
#616
* Add support to select using (some) graph operators when using
`LoadMode.CUSTOM` and `LoadMode.DBT_MANIFEST` by @tatiana in #728
* Add cosmos/propagate_logs Airflow config support for disabling log
propagation by @agreenburg in #648
* Add operator_args ``full_refresh`` as a templated field by @joppevos
in #623
* Expose environment variables and dbt variables in ``ProjectConfig`` by
@jbandoro in #735

**Enhancements**

* Make Pydantic an optional dependency by @pixie79 in #736
* Create a symbolic link to `dbt_packages` when `dbt_deps` is False when
using `LoadMode.DBT_LS` by @DanMawdsleyBA in #730
* Support no `profile_config` for `ExecutionMode.KUBERNETES` and
`ExecutionMode.DOCKER` by @MrBones757 and @tatiana in #681 and #731
* Add `aws_session_token` for Athena mapping by @benjamin-awd in #663

**Others**

* Replace flake8 for Ruff by @joppevos in #743
* Reduce code complexity to 8 by @joppevos in #738
* Update conflict matrix between Airflow and dbt versions by @tatiana in
#731
* Speed up integration tests by @jbandoro in #732
@tatiana tatiana mentioned this pull request Jan 4, 2024
Merged
tatiana added a commit that referenced this pull request Jan 4, 2024
@tatiana
Release 1.3.0 (#781)
0de8bdd 
**Features**

* Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in #733
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)).
* Add support to select using (some) graph operators when using
``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in #728
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude))
* Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in
#755,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)).
* Add ``ProfileMapping`` for Vertica by @perttus in #540, #688 and #741,
as
([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)).
* Add ``ProfileMapping`` for Snowflake encrypted private key path by
@ivanstillfront in #608, as ([documentation](
https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)).
* Add support for Snowflake encrypted private key environment variable
by @DanMawdsleyBA in #649
* Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro
in #616,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)).
* Add cosmos/propagate_logs Airflow config support for disabling log
propagation by @agreenburg in #648,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)).
* Add operator_args ``full_refresh`` as a templated field by @joppevos
in #623
* Expose environment variables and dbt variables in ``ProjectConfig`` by
@jbandoro in #735
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)).
* Support disabling event tracking when using Cosmos profile mapping by
@jbandoro in #768,
([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)).

**Enhancements**

* Make Pydantic an optional dependency by @pixie79 in #736
* Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False
when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in #730
* Add ``aws_session_token`` for Athena mapping by @benjamin-awd in #663
* Retrieve temporary credentials from ``conn_id`` for Athena by @octiva
in #758
* Extend ``DbtDocsLocalOperator`` with static flag by @joppevos  in #759

**Bug fixes**

* Remove Pydantic upper version restriction so Cosmos can be used with
Airflow 2.8 by @jlaneve in #772

**Others**

* Replace flake8 for Ruff by @joppevos in #743
* Reduce code complexity to 8 by @joppevos in #738
* Speed up integration tests by @jbandoro in #732
* Fix README quickstart link in by @RNHTTR in #776
* Add package location to work with hatchling 1.19.0 by @jbandoro in
#761
* Fix type check error in ``DbtKubernetesBaseOperator.build_env_args``
by @jbandoro in #766
* Improve ``DBT_MANIFEST`` documentation by @dwreeves in #757
* Update conflict matrix between Airflow and dbt versions by @tatiana in
#731 and #779
* pre-commit updates in #775, #770, #762
ykuc pushed a commit to ykuc/astronomer-cosmos that referenced this pull request Jan 11, 2024
@tatiana @YuriyK90
Release 1.3.0 (astronomer#781)
3aa9331 
**Features**

* Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in astronomer#733
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)).
* Add support to select using (some) graph operators when using
``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in astronomer#728
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude))
* Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in
astronomer#755,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)).
* Add ``ProfileMapping`` for Vertica by @perttus in astronomer#540, astronomer#688 and astronomer#741,
as
([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)).
* Add ``ProfileMapping`` for Snowflake encrypted private key path by
@ivanstillfront in astronomer#608, as ([documentation](
https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)).
* Add support for Snowflake encrypted private key environment variable
by @DanMawdsleyBA in astronomer#649
* Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro
in astronomer#616,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)).
* Add cosmos/propagate_logs Airflow config support for disabling log
propagation by @agreenburg in astronomer#648,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)).
* Add operator_args ``full_refresh`` as a templated field by @joppevos
in astronomer#623
* Expose environment variables and dbt variables in ``ProjectConfig`` by
@jbandoro in astronomer#735
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)).
* Support disabling event tracking when using Cosmos profile mapping by
@jbandoro in astronomer#768,
([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)).

**Enhancements**

* Make Pydantic an optional dependency by @pixie79 in astronomer#736
* Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False
when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in astronomer#730
* Add ``aws_session_token`` for Athena mapping by @benjamin-awd in astronomer#663
* Retrieve temporary credentials from ``conn_id`` for Athena by @octiva
in astronomer#758
* Extend ``DbtDocsLocalOperator`` with static flag by @joppevos  in astronomer#759

**Bug fixes**

* Remove Pydantic upper version restriction so Cosmos can be used with
Airflow 2.8 by @jlaneve in astronomer#772

**Others**

* Replace flake8 for Ruff by @joppevos in astronomer#743
* Reduce code complexity to 8 by @joppevos in astronomer#738
* Speed up integration tests by @jbandoro in astronomer#732
* Fix README quickstart link in by @RNHTTR in astronomer#776
* Add package location to work with hatchling 1.19.0 by @jbandoro in
astronomer#761
* Fix type check error in ``DbtKubernetesBaseOperator.build_env_args``
by @jbandoro in astronomer#766
* Improve ``DBT_MANIFEST`` documentation by @dwreeves in astronomer#757
* Update conflict matrix between Airflow and dbt versions by @tatiana in
astronomer#731 and astronomer#779
* pre-commit updates in astronomer#775, astronomer#770, astronomer#762
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
@tatiana @arojasb3
Release 1.3.0a2 (astronomer#711)
d7246de 
Features

* Add ProfileMapping for Vertica by @perttus in astronomer#540 and astronomer#688
* Add ProfileMapping for Snowflake encrypted private key path by
@ivanstillfront in astronomer#608
* Add support for Snowflake encrypted private key environment variable
by @DanMawdsleyBA in astronomer#649
* Add DbtDocsGCSOperator for uploading dbt docs to GCS by @jbandoro in
astronomer#616

Others

Rebased on changes released on 1.2.5 (1.3.0a1 was based on 1.2.4)
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
@tatiana @arojasb3
Release 1.3.0a3 (astronomer#753)
5038c2c 
**Features**

* Add `ProfileMapping` for Snowflake encrypted private key path by
@ivanstillfront in astronomer#608
* Add support for Snowflake encrypted private key environment variable
by @DanMawdsleyBA in astronomer#649
* Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in
astronomer#616
* Add support to select using (some) graph operators when using
`LoadMode.CUSTOM` and `LoadMode.DBT_MANIFEST` by @tatiana in astronomer#728
* Add cosmos/propagate_logs Airflow config support for disabling log
propagation by @agreenburg in astronomer#648
* Add operator_args ``full_refresh`` as a templated field by @joppevos
in astronomer#623
* Expose environment variables and dbt variables in ``ProjectConfig`` by
@jbandoro in astronomer#735

**Enhancements**

* Make Pydantic an optional dependency by @pixie79 in astronomer#736
* Create a symbolic link to `dbt_packages` when `dbt_deps` is False when
using `LoadMode.DBT_LS` by @DanMawdsleyBA in astronomer#730
* Support no `profile_config` for `ExecutionMode.KUBERNETES` and
`ExecutionMode.DOCKER` by @MrBones757 and @tatiana in astronomer#681 and astronomer#731
* Add `aws_session_token` for Athena mapping by @benjamin-awd in astronomer#663

**Others**

* Replace flake8 for Ruff by @joppevos in astronomer#743
* Reduce code complexity to 8 by @joppevos in astronomer#738
* Update conflict matrix between Airflow and dbt versions by @tatiana in
astronomer#731
* Speed up integration tests by @jbandoro in astronomer#732
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
@tatiana @arojasb3
Release 1.3.0 (astronomer#781)
0fe721e 
**Features**

* Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in astronomer#733
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)).
* Add support to select using (some) graph operators when using
``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in astronomer#728
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude))
* Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in
astronomer#755,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)).
* Add ``ProfileMapping`` for Vertica by @perttus in astronomer#540, astronomer#688 and astronomer#741,
as
([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)).
* Add ``ProfileMapping`` for Snowflake encrypted private key path by
@ivanstillfront in astronomer#608, as ([documentation](
https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)).
* Add support for Snowflake encrypted private key environment variable
by @DanMawdsleyBA in astronomer#649
* Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro
in astronomer#616,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)).
* Add cosmos/propagate_logs Airflow config support for disabling log
propagation by @agreenburg in astronomer#648,
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)).
* Add operator_args ``full_refresh`` as a templated field by @joppevos
in astronomer#623
* Expose environment variables and dbt variables in ``ProjectConfig`` by
@jbandoro in astronomer#735
([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)).
* Support disabling event tracking when using Cosmos profile mapping by
@jbandoro in astronomer#768,
([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)).

**Enhancements**

* Make Pydantic an optional dependency by @pixie79 in astronomer#736
* Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False
when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in astronomer#730
* Add ``aws_session_token`` for Athena mapping by @benjamin-awd in astronomer#663
* Retrieve temporary credentials from ``conn_id`` for Athena by @octiva
in astronomer#758
* Extend ``DbtDocsLocalOperator`` with static flag by @joppevos  in astronomer#759

**Bug fixes**

* Remove Pydantic upper version restriction so Cosmos can be used with
Airflow 2.8 by @jlaneve in astronomer#772

**Others**

* Replace flake8 for Ruff by @joppevos in astronomer#743
* Reduce code complexity to 8 by @joppevos in astronomer#738
* Speed up integration tests by @jbandoro in astronomer#732
* Fix README quickstart link in by @RNHTTR in astronomer#776
* Add package location to work with hatchling 1.19.0 by @jbandoro in
astronomer#761
* Fix type check error in ``DbtKubernetesBaseOperator.build_env_args``
by @jbandoro in astronomer#766
* Improve ``DBT_MANIFEST`` documentation by @dwreeves in astronomer#757
* Update conflict matrix between Airflow and dbt versions by @tatiana in
astronomer#731 and astronomer#779
* pre-commit updates in astronomer#775, astronomer#770, astronomer#762
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jlaneve jlaneve jlaneve left review comments

@tatiana tatiana tatiana approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

1.3.0

Development

Successfully merging this pull request may close these issues.

Snowflake auth encrypted key path not supported

3 participants

@metzlar @tatiana @jlaneve