Add support render for versioned dbt models.#516
Conversation
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
tatiana
left a comment
There was a problem hiding this comment.
@binhnq94, thanks a lot for your contribution!
I will still read the docs to understand more about aliases and versions, but I have two initial questions:
- Would there be any benefit in introducing a new attribute to
DbtNodecalledaliasinstead of overloading the propertynamewith two possible sources? We could also introduce anidentifierif it helped. What are the pros and cons of overridingname? - When we render dbt nodes as Airflow nodes, what would be the benefit of rendering models using aliases instead of their names?
Codecov ReportAll modified lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #516 +/- ##
=======================================
Coverage 92.83% 92.83%
=======================================
Files 49 49
Lines 1983 1983
=======================================
Hits 1841 1841
Misses 142 142
☔ View full report in Codecov by Sentry. |
|
c107c07 to
80be88d
Compare
|
@binhnq94 let's aim to have this change released as part of 1.2. Please, let me know if you'd like any further support on anything - I'm looking forward to merging this change. |
67eda36 to
ce3d0eb
Compare
8e15561 to
deb9964
Compare
|
@tatiana I have already fixed the fail in integrate tests (sorry, last time I only run model_version dag only.) |
**Features** * Add support to model versioning available since dbt 1.6 by @binhnq94 in #516 * Add AWS Athena profile mapping by @benjamin-awd in #578 * Support customizing how dbt nodes are converted to Airflow by @tatiana in #503 * Make the arg ``dbt_project_path`` in the ``ProjectConfig`` optional by @MrBones757 in #581 **Bug fixes** * Fix Cosmos custom selector to support filtering a single model by @jlaneve and @harels in #576 * Fix using ``GoogleCloudServiceAccountDictProfileMapping`` together with ``LoadMethod.DBT_LS`` by @joppevos in #587 * Fix using the ``full_refresh`` argument in projects that contain tests by @EgorSemenov and @tatiana in #590 * Stop creating symbolic links for ``dbt_packages`` (solves ``LocalExecutor`` concurrency issue) by @tatiana in #600 **Others** * Docs: add reference to original Jaffle Shop project by @erdos2n in #583 * Docs: retries & note about DagBag error by @TJaniF in #592 * pre-commit updates in #575 and #585

Description
Model version is supported by dbt-core >= 1.5.
As cosmos v1.1.0a1, it not support to render task for model have versions.
In this PR I propose to use
node_dict["alias"]asDbtNode.nameinstead ofnode_dict["name"]ifnode_dict["resource_type"]==model.This PR also help cosmos render the names of nodes to be correctly the same as
identifierif users config alias or custom thegenerate_alias_namemacro. Refer: alias document and thegenerate_alias_nameimplementation.Related Issue(s)
#260
Breaking Change?
When model have version config, each model version will create a database relation with alias
<model_name>_v<v>, for examples:dim_customers_v1,dim_customers_v2...So I expected that cosmos will created run/test tasks for each versions, for example: dim_customers_v1_run, dim_customers_v2_run, dim_customers_v1_test, dim_customers_v2_test...
The second expectation is the dag will render right depedences, for example: stg_customers_v1 >> customers_v1, stg_customers_v2 >> customer_v2...
Checklist