add DatabricksOauthProfileMapping profile by CorsettiS · Pull Request #1091 · astronomer/astronomer-cosmos

CorsettiS · 2024-07-13T20:17:34Z

Description

As of now, Token-based auth is the only possible way to access databricks from cosmos, even though dbt-databricks support OAuth-based access as can be seen here .

This PR aims to enhance the support offered by cosmos to databricks. The final goal is to close #1085

I have tested locally and got a positive result.

Breaking Change?

No

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

add DatabricksClientProfileMapping

…hub.com/CorsettiS/astronomer-cosmos into rename-to-DatabricksOauthProfileMapping

netlify · 2024-07-13T20:17:49Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`02cad9f`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/6692e0e014b64a000894d8cd

netlify · 2024-07-13T20:18:06Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`998908b`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/6694d9a6f96f9c0008f69055

codecov · 2024-07-13T21:12:52Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.25%. Comparing base (1f27638) to head (998908b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1091      +/-   ##
==========================================
+ Coverage   96.23%   96.25%   +0.01%     
==========================================
  Files          63       64       +1     
  Lines        3270     3284      +14     
==========================================
+ Hits         3147     3161      +14     
  Misses        123      123

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pankajastro

Thank you for your contribution! It looks great. I just posted a few minor comments

Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>

@pankajkoti

New Features * Add support for loading manifest from cloud stores using Airflow Object Storage by @pankajkoti in #1109 * Cache ``package-lock.yml`` file by @pankajastro in #1086 * Support persisting the ``LoadMode.VIRTUALENV`` directory by @tatiana in #1079 * Add support to store and fetch ``dbt ls`` cache in remote stores by @pankajkoti in #1147 * Add default source nodes rendering by @arojasb3 in #1107 * Add Teradata ``ProfileMapping`` by @sc250072 in #1077 Enhancements * Add ``DatabricksOauthProfileMapping`` profile by @CorsettiS in #1091 * Use ``dbt ls`` as the default parser when ``profile_config`` is provided by @pankajastro in #1101 * Add task owner to dbt operators by @wornjs in #1082 * Extend Cosmos custom selector to support + when using paths and tags by @mvictoria in #1150 * Simplify logging by @dwreeves in #1108 Bug fixes * Fix Teradata ``ProfileMapping`` target invalid issue by @sc250072 in #1088 * Fix empty tag in case of custom parser by @pankajastro in #1100 * Fix ``dbt deps`` of ``LoadMode.DBT_LS`` should use ``ProjectConfig.dbt_vars`` by @tatiana in #1114 * Fix import handling by lazy loading hooks introduced in PR #1109 by @dwreeves in #1132 * Fix Airflow 2.10 regression and add Airflow 2.10 in test matrix by @pankajastro in #1162 Docs * Fix typo in azure-container-instance docs by @pankajastro in #1106 * Use Airflow trademark as it has been registered by @pankajastro in #1105 Others * Run some example DAGs in Kubernetes execution mode in CI by @pankajastro in #1127 * Install requirements.txt by default during dev env spin up by @@CorsettiS in #1099 * Remove ``DbtGraph.current_version`` dead code by @tatiana in #1111 * Disable test for Airflow-2.5 and Python-3.11 combination in CI by @pankajastro in #1124 * Pre-commit hook updates in #1074, #1113, #1125, #1144, #1154, #1167 --------- Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>

@corsettigyg

[Giovanni Corsetti Silva](https://www.linkedin.com/in/giovanni-corsetti/) (@corsettigyg @CorsettiS @giovannicorsetti) is a Core Data Platform Data Engineer at [Get Your Guide](https://www.getyourguide.com/). He's a Brazilian based in Berlin (Germany) and has been an early adopter of Cosmos, using it regularly throughout his career. Not only has he been using Cosmos since the early stages, but he has consistently improved Cosmos since April 2023: <img width="494" height="249" alt="Screenshot 2025-09-17 at 16 39 49" src="https://github.com/user-attachments/assets/ff9b0055-7df6-47a4-8efa-83a30d5aee99" /> ] m. , <img width="492" height="252" alt="Screenshot 2025-09-17 at 16 40 11" src="https://github.com/user-attachments/assets/8402d883-eff4-46aa-9ae7-afe35bd39a6e" /> <img width="499" height="256" alt="Screenshot 2025-09-17 at 16 39 23" src="https://github.com/user-attachments/assets/c6b84245-aae1-46e6-99e3-552553619dd9" /> His contributions include new features, enhancements, bug fixes and also improvements on the local dev setup, as it can see in some of the many contributions he's done to the project so far: * #1787 * #1761 * #1695 * #1693 * #1663 * #1571 * #1449 * #1099 * #1091 * #419 * #402 * #355 * #354 * #225 Additionally, he has been interacting with users in the #airflow-dbt Slack channel in a very collaborative and supportive way. We want to promote him as a Cosmos committer and maintainer for all these, recognising his constant efforts and achievements towards our community. Thank you very much, @corsettigyg! From this moment, you have write permissions on the repo: <img width="884" height="93" alt="Screenshot 2025-09-18 at 14 34 07" src="https://github.com/user-attachments/assets/abcc8c42-87cf-4b32-a7be-81127190d164" />

CorsettiS and others added 9 commits July 13, 2024 20:24

fix CI

a471817

scratch (first step to client creation)

d5884a0

test permissions

166da62

test permissions

c044ddf

add tests and fix syntax

23fc0ed

revert files added for testing only

eb7023e

python black

6fbbdde

Merge pull request #1 from CorsettiS/create-client-databricks-profile

3c0b3e0

add DatabricksClientProfileMapping

rename profile

02cad9f

dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jul 13, 2024

CorsettiS had a problem deploying to external July 13, 2024 20:17 — with GitHub Actions Error

pre-commit-ci Bot and others added 2 commits July 13, 2024 20:17

🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

cbe1e2d

Merge branch 'rename-to-DatabricksOauthProfileMapping' of https://git…

2fc0ebd

…hub.com/CorsettiS/astronomer-cosmos into rename-to-DatabricksOauthProfileMapping

CorsettiS closed this Jul 13, 2024

pre-commit-ci Bot had a problem deploying to external July 13, 2024 20:18 Error

dosubot Bot added area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc profile:databricks Related to Databricks ProfileConfig labels Jul 13, 2024

CorsettiS reopened this Jul 13, 2024

CorsettiS changed the title ~~Rename to databricks oauth profile mapping~~ add DatabricksOauthProfileMapping profile Jul 13, 2024

CorsettiS temporarily deployed to external July 13, 2024 20:18 — with GitHub Actions Inactive

pre-commit-ci Bot and others added 2 commits July 13, 2024 20:25

🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

0ce4b4a

fix CI (2)

eb7af6d

pre-commit-ci Bot had a problem deploying to external July 13, 2024 20:25 Error

CorsettiS had a problem deploying to external July 13, 2024 20:31 — with GitHub Actions Error

🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

bf9a0e7

pre-commit-ci Bot temporarily deployed to external July 13, 2024 20:31 Inactive

pankajastro reviewed Jul 15, 2024

View reviewed changes

Comment thread cosmos/profiles/databricks/client.py

Comment thread cosmos/profiles/databricks/client.py Outdated

Comment thread tests/profiles/databricks/test_dbr_client.py Outdated

Comment thread tests/profiles/databricks/test_dbr_client.py Outdated

rename file to oauth

3bdbc95

corsettigyg had a problem deploying to external July 15, 2024 08:07 — with GitHub Actions Error

CorsettiS and others added 2 commits July 15, 2024 10:09

Update tests/profiles/databricks/test_dbr_client.py

a7e0f49

Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>

Update tests/profiles/databricks/test_dbr_client.py

c4c6e76

Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>

CorsettiS had a problem deploying to external July 15, 2024 08:10 — with GitHub Actions Error

fix doc string

b947403

corsettigyg had a problem deploying to external July 15, 2024 08:10 — with GitHub Actions Error

rename file

998908b

corsettigyg temporarily deployed to external July 15, 2024 08:11 — with GitHub Actions Inactive

pankajastro approved these changes Jul 15, 2024

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Jul 15, 2024

pankajastro added this to the Cosmos 1.6.0 milestone Jul 17, 2024

pankajastro merged commit 954fa34 into astronomer:main Jul 18, 2024

pankajkoti mentioned this pull request Jul 31, 2024

Release 1.6.0 #1080

Merged

tatiana mentioned this pull request Sep 18, 2025

Promote @corsettigyg to committer #1985

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add DatabricksOauthProfileMapping profile#1091

add DatabricksOauthProfileMapping profile#1091
pankajastro merged 19 commits into
astronomer:mainfrom
CorsettiS:rename-to-DatabricksOauthProfileMapping

CorsettiS commented Jul 13, 2024 •

edited

Loading

Uh oh!

netlify Bot commented Jul 13, 2024 •

edited

Loading

Uh oh!

netlify Bot commented Jul 13, 2024 •

edited

Loading

Uh oh!

codecov Bot commented Jul 13, 2024 •

edited

Loading

Uh oh!

pankajastro left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

CorsettiS commented Jul 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Breaking Change?

Checklist

Uh oh!

netlify Bot commented Jul 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

netlify Bot commented Jul 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

codecov Bot commented Jul 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pankajastro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CorsettiS commented Jul 13, 2024 •

edited

Loading

netlify Bot commented Jul 13, 2024 •

edited

Loading

netlify Bot commented Jul 13, 2024 •

edited

Loading

codecov Bot commented Jul 13, 2024 •

edited

Loading