fix-issue-2393: Support rendering of DbtResourceType.EXPOSURE#2576
fix-issue-2393: Support rendering of DbtResourceType.EXPOSURE#2576jroachgolf84 wants to merge 1 commit into
fix-issue-2393: Support rendering of DbtResourceType.EXPOSURE#2576Conversation
tatiana
left a comment
There was a problem hiding this comment.
Hey @jroachgolf84 , thanks so much for working on this — really excited to see your contribution!
Since Cosmos v1.12.0, Cosmos has stopped rendering exposures when users try to run this DAG. Cosmos used to handle this dynamically, without requiring explicit logic for exposures via this feature:
https://astronomer.github.io/astronomer-cosmos/guides/translate_dbt_to_airflow/dag-customization.html
This is an example of a DAG that used to work in previous versions:
https://github.com/astronomer/astronomer-cosmos/blob/main/dev/dags/example_cosmos_sources.py
In my opinion, issue #2393 uncovers more than just the exposure node problem. My impression is that Cosmos has stopped supporting the previous level of pluggability it offered. This could potentially apply to analysis and metrics nodes, which may be available in the manifest.json or dbt ls and users would like to handle somehow.
Ideally, the fix would not be specific to exposures but would instead restore users' ability to resolve dynamically resolved dbt node types, as before.
I believe the regression may have been introduced by one of these changes:
#2080
#1759
Would you be open to proposing an alternative change that addresses the root cause? It could live in a separate PR. With this we could discuss the approaches and decide on the solution.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2576 +/- ##
=======================================
Coverage 98.05% 98.05%
=======================================
Files 103 103
Lines 7601 7605 +4
=======================================
+ Hits 7453 7457 +4
Misses 148 148 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
@tatiana - thanks for the feedback! I'd be open to either re-working this PR to include the changes you mentioned. OR, I'd also be open to merging this PR with this logic and opening up a new one to address the other points you mentioned. What would you like to do? |
|
@jroachgolf84 my preference would be with another PR, so we can compare the approaches and make a decision on how to proceed |
|
@tatiana, sounds good - I'll open up another PR to address this broader scope. |
|
Closing this PR in favor of #2608. |
#2608) Per issue #2393, `DbtResourceType.EXPOSURE` nodes defined in `render_config.node_converters` are silently ignored during DAG rendering, producing the following warning instead: ``` {graph.py:426} WARNING - Unavailable conversion function for <DbtResourceType.EXPOSURE> (node <exposure.{project_name}.{node_name}>). Define a converter function using render_config.node_converters. ``` Previously, #2576 was opened to address this only for `DbtResourceType.EXPOSURE`s. However, @tatiana called out that this functionality was actually deprecated for more than just Exposures. Thus, this PR was created to ensure that all node types with `node_converters` specified by the user are properly rendered using their desired function. ## Testing This was tested with the addition of two new unit-tests (as well as the successful execution of previous unit-tests). There were: - `test_generate_task_or_group_with_dynamic_node_type_and_converter` - `test_generate_task_or_group_with_dynamic_node_type_no_converter_returns_none` These changes were also tested E2E using the `example_cosmos_sources` DAG. Before this change was made, the source and exposure node types were not rendered in the UI, despite being explicitly added to the `RenderConfig.node_converters`. Following the change, those nodes were properly rendered. See the image below. <img width="1724" height="836" alt="image" src="https://github.com/user-attachments/assets/fcab57fe-9962-48eb-bea5-43d49a0f2159" /> Closes: #2393
Description
Per issue #2393,
DbtResourceType.EXPOSUREnodes defined inrender_config.node_convertersare silently ignored during DAG rendering, producing the following warning instead:This PR ensures that exposures are properly rendered in the Airflow graph (as an
EmptyOperator) with the appropriate metadata.Related Issue(s)
closes: #2393
Breaking Change?
No, this is not a breaking change.
Testing
These changes were tested via a unit-test (
test_create_task_metadata_exposure) as well as E2E by running Airflow with Docker.Checklist