Add support to base64 encoded pem in Snowflake profiles by brunocmartins · Pull Request #1801 · astronomer/astronomer-cosmos

brunocmartins · 2025-06-09T21:41:31Z

Description

Starting from apache-airflow-providers-snowflake version 6.3, the provider expects the
private_key_content field to be base64 encoded instead of plain text PEM. This change
adds decoding of base64 encoded keys while preserving backwards compatibility with older
versions that use plain text PEM.

Related Issue(s)

Closes #1798

Breaking Change?

None

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

…ider >= 6.3 Starting from `apache-airflow-providers-snowflake` version 6.3, the provider expects the `private_key_content` field to be base64 encoded instead of plain text PEM. This change adds decoding of base64 encoded keys while preserving backwards compatibility with older versions that use plain text PEM.

netlify · 2025-06-09T21:41:44Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`4672f8d`
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/68541b365a699400086c53b7

tatiana

This looks great, @brunocmartins. Thank you for reporting and proposing a fix for the problem, which is backwards-compatible.

Please, could you add a unit test to cover this new use case? It would be fine if you created a test for _decode_private_key_content that checks the scenario where a base64 value is being passed.

The tests for this module are in:
https://github.com/astronomer/astronomer-cosmos/blob/main/tests/profiles/snowflake/test_snowflake_user_encrypted_privatekey_file.py

Move function to SnowflakeBaseProfileMapping to avoid duplicating code on SnowflakePrivateKeyPemProfileMapping and SnowflakeEncryptedPrivateKeyPemProfileMapping

…mProfileMapping and SnowflakePrivateKeyPemProfileMapping

codecov · 2025-06-19T14:40:44Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.01%. Comparing base (f8331a2) to head (4672f8d).
Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1801   +/-   ##
=======================================
  Coverage   98.00%   98.01%           
=======================================
  Files          85       85           
  Lines        5266     5279   +13     
=======================================
+ Hits         5161     5174   +13     
  Misses        105      105

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tatiana

Excellent work, thanks a lot for fixing the issue and adding tests, @brunocmartins !

Starting from `apache-airflow-providers-snowflake` version 6.3, the [provider expects the `private_key_content` field to be base64 encoded](https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/changelog.html#id2) instead of plain text PEM. This change adds decoding of base64 encoded keys while preserving backwards compatibility with older versions that use plain text PEM. ## Related Issue(s) Closes #1798 ## Breaking Change? None ## Checklist - [x] I have made corresponding changes to the documentation (if required) - [x] I have added tests that prove my fix is effective or that my feature works --------- Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com> (cherry picked from commit 05ca4e8)

@pankajkoti

**Bug Fixes** * Fix task instance ``try_number`` attribute for Airflow 3 compatibility by @pankajkoti in #1781 * Fix rendered template override logic when ``should_store_compiled_sql=False`` to restore pre-refactor behaviour by @pankajkoti in #1777 * Fix ``ProfileConfig`` in GCP Cloud Run job execution mode by @ramonvermeulen in #1783 * Fix dbt Docs page height by @1cadumagalhaes in #1793 * Add support to base64 encoded pem in Snowflake profiles by @brunocmartins in #1801 * Allow to disable owner inheritance from dbt into airflow DAG owners by @CorsettiS in #1787 * Fix Kubernetes Pod Operator conversion of ``container_resources`` to ``resources`` by @johnhoran in #1821 * Fix ``dbt deps`` with project level variables by @AlexandrKhabarov in #1822 * Fix source freshness warnings in kubernetes execution mode by @Pawel-Drabczyk in #1859 * Fix: Harden DbtNode against null config/meta by @pankajkoti in #1877 * Fix cache behaviour when DAG name contains "." by @tatiana in #1908 **Documentation** * Fix ``contributing.rst`` docs by @tatiana in #1785 * Fix docs rendering in Airflow 3 Compatibility by @pankajastro in #1790 * Fix typo in ``selecting-excluding.rst`` by @msshroff in #1814 * Update testing behavior file with ``ExecutionMode.KUBERNETES`` by @LuigiCerone in #1813 * Add step to fork repo in contributing guide by @pankajastro in #1808 * Fix ``depends_on`` attribute by @benedikt-buchert in #1837 * Fix character name by @ThePsyjo in #1860 * Update suggested MWAA startup script by @jaklan in #1884 * Make implementation & docs consistent regarding ``use_dataset_airflow3_uri_standard`` by @Anti0ff in #1878 **Others** * Set retries to 0 in example DAGs by @pankajkoti in #1782 * Fix ``test_async_example_dag_without_setup_task`` tests by @pankajastro in #1788 * Fix test hash value for Darwin when using Py 3.12.10 by @tatiana in #1786 * Upgrade Python and Airflow used to run MyPy checks by @tatiana in #1796 * Assert example DAGs' ``DagRunState`` and fix issues by @pankajkoti and @tatiana in #1778 * Update the conflict matrix to include AF 2.10, 2.11 & 3.0 and dbt 1.9 & 1.10 by @tatiana in #1820 * Fix broken CI due to Pydantic conflicts by @tatiana in #1809 * Drop Python 3.8 Support by @pankajastro in #1852 * Add Airflow 2.11 to the test matrix by @tatiana in #1807 * Require Authorize for all jobs on pull requests from external contributors in CI by @pankajkoti in #1861 * Leverage Trusted Publisher Management when publishing PyPI package by @tatiana in #1862 * CI: Add back accidentally deleted python-version matrix for running unit tests by @pankajkoti in #1872 * Remove commented code and fix mypy failures by @pankajkoti in #1876 * Add Zizmor analysis GitHub action by @pankajkoti in #1870 * Catch FlushError on Datasets for Airflow 2.11 dags test by @pankajkoti in #1880 * Add deprecation warning for ``LoadMode.CUSTOM`` parser by @duongphannamhung in #1885 * CI: Add GitHub CodeQL analysis workflow (codeql.yml) by @pankajkoti in #1871 * Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in #1890 * Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in #1889 * Resolve Zizmor error alerts for unpinned image references; mark alert for pull_request_target ignored by @pankajkoti in #1888 * Fix broken CI ``tests.py3.11-2.8-1.9:test-integration-setup`` by @tatiana in #1902 * Add dbt-core 1.10 to test matrix by @tatiana in #1767 * Pin package dbt-databricks by @pankajastro in #1909 * Enable matrix test entry for dbt-1.9, python-3.9 and airflow-3.0 tests in CI by @pankajastro in #1900 * Pre-commit updates: #1779, #1795, #1800, #1857, #1863, #1869, #1892, #1901 * Dependabot updates: #1904 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

@pankajkoti

**Bug Fixes** * Fix task instance ``try_number`` attribute for Airflow 3 compatibility by @pankajkoti in #1781 * Fix rendered template override logic when ``should_store_compiled_sql=False`` to restore pre-refactor behaviour by @pankajkoti in #1777 * Fix ``ProfileConfig`` in GCP Cloud Run job execution mode by @ramonvermeulen in #1783 * Fix dbt Docs page height by @1cadumagalhaes in #1793 * Add support to base64 encoded pem in Snowflake profiles by @brunocmartins in #1801 * Allow to disable owner inheritance from dbt into airflow DAG owners by @CorsettiS in #1787 * Fix Kubernetes Pod Operator conversion of ``container_resources`` to ``resources`` by @johnhoran in #1821 * Fix ``dbt deps`` with project level variables by @AlexandrKhabarov in #1822 * Fix source freshness warnings in kubernetes execution mode by @Pawel-Drabczyk in #1859 * Fix: Harden DbtNode against null config/meta by @pankajkoti in #1877 * Fix cache behaviour when DAG name contains "." by @tatiana in #1908 **Documentation** * Fix ``contributing.rst`` docs by @tatiana in #1785 * Fix docs rendering in Airflow 3 Compatibility by @pankajastro in #1790 * Fix typo in ``selecting-excluding.rst`` by @msshroff in #1814 * Update testing behavior file with ``ExecutionMode.KUBERNETES`` by @LuigiCerone in #1813 * Add step to fork repo in contributing guide by @pankajastro in #1808 * Fix ``depends_on`` attribute by @benedikt-buchert in #1837 * Fix character name by @ThePsyjo in #1860 * Update suggested MWAA startup script by @jaklan in #1884 * Make implementation & docs consistent regarding ``use_dataset_airflow3_uri_standard`` by @Anti0ff in #1878 **Others** * Set retries to 0 in example DAGs by @pankajkoti in #1782 * Fix ``test_async_example_dag_without_setup_task`` tests by @pankajastro in #1788 * Fix test hash value for Darwin when using Py 3.12.10 by @tatiana in #1786 * Upgrade Python and Airflow used to run MyPy checks by @tatiana in #1796 * Assert example DAGs' ``DagRunState`` and fix issues by @pankajkoti and @tatiana in #1778 * Update the conflict matrix to include AF 2.10, 2.11 & 3.0 and dbt 1.9 & 1.10 by @tatiana in #1820 * Fix broken CI due to Pydantic conflicts by @tatiana in #1809 * Drop Python 3.8 Support by @pankajastro in #1852 * Add Airflow 2.11 to the test matrix by @tatiana in #1807 * Require Authorize for all jobs on pull requests from external contributors in CI by @pankajkoti in #1861 * Leverage Trusted Publisher Management when publishing PyPI package by @tatiana in #1862 * CI: Add back accidentally deleted python-version matrix for running unit tests by @pankajkoti in #1872 * Remove commented code and fix mypy failures by @pankajkoti in #1876 * Add Zizmor analysis GitHub action by @pankajkoti in #1870 * Catch FlushError on Datasets for Airflow 2.11 dags test by @pankajkoti in #1880 * Add deprecation warning for ``LoadMode.CUSTOM`` parser by @duongphannamhung in #1885 * CI: Add GitHub CodeQL analysis workflow (codeql.yml) by @pankajkoti in #1871 * Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in #1890 * Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in #1889 * Resolve Zizmor error alerts for unpinned image references; mark alert for pull_request_target ignored by @pankajkoti in #1888 * Fix broken CI ``tests.py3.11-2.8-1.9:test-integration-setup`` by @tatiana in #1902 * Add dbt-core 1.10 to test matrix by @tatiana in #1767 * Pin package dbt-databricks by @pankajastro in #1909 * Enable matrix test entry for dbt-1.9, python-3.9 and airflow-3.0 tests in CI by @pankajastro in #1900 * Pre-commit updates: #1779, #1795, #1800, #1857, #1863, #1869, #1892, #1901 * Dependabot updates: #1904 Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

brunocmartins added 3 commits June 9, 2025 17:51

Adjust code to pass hatch cov tests

7415acc

Remove unused logging

a15ded9

brunocmartins had a problem deploying to external June 9, 2025 21:41 — with GitHub Actions Error

brunocmartins marked this pull request as ready for review June 10, 2025 00:06

dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. profile:snowflake Related to Snowflake ProfileConfig labels Jun 10, 2025

Remove redundancy

3d6a31b

brunocmartins temporarily deployed to external June 10, 2025 00:08 — with GitHub Actions Inactive

tatiana reviewed Jun 10, 2025

View reviewed changes

brunocmartins added 2 commits June 10, 2025 13:57

Move _decode_private_key_content function to SnowflakeBaseProfileMapping

fc15641

Move function to SnowflakeBaseProfileMapping to avoid duplicating code on SnowflakePrivateKeyPemProfileMapping and SnowflakeEncryptedPrivateKeyPemProfileMapping

Add test to support base64 decode from SnowflakeEncryptedPrivateKeyPe…

8ad6a46

…mProfileMapping and SnowflakePrivateKeyPemProfileMapping

dosubot Bot removed the size:S This PR changes 10-29 lines, ignoring generated files. label Jun 10, 2025

brunocmartins had a problem deploying to external June 10, 2025 18:14 — with GitHub Actions Error

dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jun 10, 2025

Change private_key_passphrase to correct location

3c0cbdc

brunocmartins temporarily deployed to external June 10, 2025 18:18 — with GitHub Actions Inactive

pankajastro reviewed Jun 16, 2025

View reviewed changes

Comment thread cosmos/profiles/snowflake/user_encrypted_privatekey_env_variable.py

pankajastro approved these changes Jun 19, 2025

View reviewed changes

Merge branch 'main' into add-support-to-base64-encoded-snowflake-pem

4672f8d

tatiana temporarily deployed to external June 19, 2025 14:14 — with GitHub Actions Inactive

tatiana changed the title ~~Add support to base64 encoded snowflake pem~~ Add support to base64 encoded pem in Snowflake profiles Jun 19, 2025

tatiana approved these changes Jun 19, 2025

View reviewed changes

tatiana merged commit 05ca4e8 into astronomer:main Jun 19, 2025
85 checks passed

pankajkoti added this to the Cosmos 1.10.2 milestone Aug 7, 2025

tatiana mentioned this pull request Aug 7, 2025

Release 1.10.2 #1911

Merged

pankajkoti mentioned this pull request Aug 8, 2025

Release 1.10.2 #1912

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support to base64 encoded pem in Snowflake profiles#1801

Add support to base64 encoded pem in Snowflake profiles#1801
tatiana merged 8 commits into
astronomer:mainfrom
brunocmartins:add-support-to-base64-encoded-snowflake-pem

brunocmartins commented Jun 9, 2025 •

edited

Loading

Uh oh!

netlify Bot commented Jun 9, 2025 •

edited

Loading

Uh oh!

tatiana left a comment

Uh oh!

Uh oh!

codecov Bot commented Jun 19, 2025 •

edited

Loading

Uh oh!

tatiana left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

brunocmartins commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Breaking Change?

Checklist

Uh oh!

netlify Bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov Bot commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

brunocmartins commented Jun 9, 2025 •

edited

Loading

netlify Bot commented Jun 9, 2025 •

edited

Loading

codecov Bot commented Jun 19, 2025 •

edited

Loading