Views V2 #954

benc-db · 2025-03-04T23:38:05Z

Description

Next step in the V2 materialization saga, this time Views. The big deal with views is adding the ability to update previously deployed views with ALTER rather than replacement, so that they keep the same grants, and UC id (which is how create or replace already works for tables). The other change is that now replace.sql is doing more heavy lifting, simplifying what would otherwise be a fairly complicated materialization.

Probably need more tests, but want to get the ideas out in front of reviewers

Checklist

I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

…g statements

dbt/adapters/databricks/impl.py

benc-db · 2025-03-04T23:39:46Z

dbt/adapters/databricks/impl.py

@@ -138,7 +139,9 @@ class DatabricksConfig(AdapterConfig):
    target_alias: Optional[str] = None
    source_alias: Optional[str] = None
    merge_with_schema_evolution: Optional[bool] = None
-    safe_table_create: Optional[bool] = None
+    use_safer_relation_operations: Optional[bool] = None
+    incremental_apply_config_changes: Optional[bool] = None


This was previously missing

dbt/adapters/databricks/impl.py

benc-db · 2025-03-04T23:41:28Z

dbt/adapters/databricks/impl.py

@@ -918,3 +923,30 @@ def _describe_relation(
            results["information_schema.tags"] = adapter.execute_macro("fetch_tags", kwargs=kwargs)
        results["show_tblproperties"] = adapter.execute_macro("fetch_tbl_properties", kwargs=kwargs)
        return results
+
+
+class ViewAPI(RelationAPIBase[ViewConfig]):


We need to use something similar to MV/ST and Incremental to check for changes so that we can update views via ALTER.

benc-db · 2025-03-04T23:41:45Z

dbt/adapters/databricks/relation.py

@@ -91,6 +91,10 @@ def is_materialized_view(self) -> bool:
    def is_streaming_table(self) -> bool:
        return self.type == DatabricksRelationType.StreamingTable

+    @property


this was missing from a previous PR

benc-db · 2025-03-04T23:42:13Z

dbt/adapters/databricks/relation_configs/base.py

    def get_changeset(self, existing: Self) -> Optional[DatabricksRelationChangeSet]:
        """Get the changeset that must be applied to the existing relation to make it match the
        current state of the dbt project. If no changes are required, this method should return
        None.
        """
+        changes: dict[str, DatabricksComponentConfig] = {}


Defining a base since the incremental and view versions of this method are very similar

benc-db · 2025-03-04T23:43:03Z

dbt/adapters/databricks/relation_configs/view.py

+
+    def get_changeset(self, existing: Self) -> Optional[DatabricksRelationChangeSet]:
+        changeset = super().get_changeset(existing)
+        if changeset and "comment" in changeset.changes:


For some reason there is no mechanism in Databricks to alter the comment on a view ;(. Therefore, even if you want me to update the view via alter, if you update the description, we'll need to replace the view.

benc-db · 2025-03-04T23:44:13Z

dbt/include/databricks/macros/adapters/columns.sql

@@ -56,7 +56,7 @@

 {% macro databricks__get_columns_in_query(select_sql) %}
  {{ log("Getting column information via empty query")}}
-  {% call statement('get_columns_in_query', fetch_result=True, auto_begin=False) -%}
+  {% call statement('get_columns_in_query', fetch_result=True) -%}


auto_begin doesn't do anything in Databricks

benc-db · 2025-03-04T23:44:35Z

dbt/include/databricks/macros/adapters/persist_docs.sql

@@ -4,7 +4,7 @@
      {% set comment = column['description'] %}
      {% set escaped_comment = comment | replace('\'', '\\\'') %}
      {% set comment_query %}
-        alter table {{ relation.render()|lower }} change column {{ api.Column.get_name(column) }} comment '{{ escaped_comment }}';
+        COMMENT ON COLUMN {{ relation.render()|lower }}.{{ api.Column.get_name(column) }} IS '{{ escaped_comment }}';


This form works with tables and views

benc-db · 2025-03-04T23:45:01Z

dbt/include/databricks/macros/etc/statement.sql

@@ -15,3 +15,17 @@

  {%- endif -%}
 {%- endmacro %}
+
+{% macro execute_multiple_statements(statements) %}


Refactoring out of MV/ST because now we have a third place that we need this pattern.

benc-db · 2025-03-04T23:45:21Z

dbt/include/databricks/macros/materializations/incremental/incremental.sql

@@ -15,7 +15,7 @@
  {% if adapter.behavior.use_materialization_v2 %}
    {{ log("USING V2 MATERIALIZATION") }}
    {#-- Set vars --#}
-    {% set safe_create = config.get('safe_table_create', True) | as_bool  %}
+    {% set safe_create = config.get('use_safer_relation_operations', True) | as_bool  %}


renamed this flag since now I want to use it for views too.

benc-db · 2025-03-04T23:45:48Z

dbt/include/databricks/macros/materializations/materialized_view.sql

@@ -32,7 +32,7 @@

        -- get config options
        {% set on_configuration_change = config.get('on_configuration_change') %}
-        {% set configuration_changes = get_materialized_view_configuration_changes(existing_relation, config) %}
+        {% set configuration_changes = get_configuration_changes(existing_relation) %}


Was able to make this macro more generic too.

benc-db · 2025-03-04T23:46:47Z

dbt/include/databricks/macros/relations/create_backup.sql

@@ -1,10 +1,9 @@
 {%- macro create_backup(relation) -%}
-    -- get the standard backup name
-    {% set backup_relation = make_backup_relation(relation, relation.type) %}
+  -- get the standard backup name


working on consistent tab size of 2

benc-db · 2025-03-04T23:47:26Z

dbt/include/databricks/macros/relations/replace.sql

 {% endmacro %}

 {% macro databricks__get_replace_sql(existing_relation, target_relation, sql) %}
-    {# /* use a create or replace statement if possible */ #}
+  {# /* if safe_relation_replace, prefer renaming */ #}


Updated this macro per replace_flow.md. Now works for views as well as MV/ST

benc-db added 12 commits February 26, 2025 09:34

wip

1d4ea2e

cleanup view spacing

88e677d

remove unneeded events

b0c2bdf

fixing tests

d1a892e

remove wpm test that has been flaky for a long time and depends on lo…

c44b70e

…g statements

fix failing test

01e8df4

changelog

27686af

wip

cc30c83

wip:

672bb4a

rewritten to take advantage of replace flow

20f134d

more minor changes

a7690a3

Merge branch '1.10.latest' into views_v2

c938e85

benc-db requested review from andrefurlan-db and rcypher-databricks as code owners March 4, 2025 23:38

benc-db commented Mar 4, 2025

View reviewed changes

dbt/adapters/databricks/impl.py Outdated Show resolved Hide resolved

benc-db commented Mar 4, 2025

View reviewed changes

dbt/adapters/databricks/impl.py Outdated Show resolved Hide resolved

benc-db commented Mar 4, 2025

View reviewed changes

benc-db requested review from eric-wang-1990 and jackyhu-db March 4, 2025 23:48

benc-db requested a review from alexguo-db March 4, 2025 23:48

skip test that raises exception intentionally, fix other tests

ab93fd6

benc-db had a problem deploying to azure-prod March 5, 2025 18:06 — with GitHub Actions Failure

benc-db temporarily deployed to azure-prod March 5, 2025 18:06 — with GitHub Actions Inactive

benc-db had a problem deploying to azure-prod March 5, 2025 18:06 — with GitHub Actions Failure

separating hive tests

0495d85

benc-db had a problem deploying to azure-prod March 5, 2025 20:46 — with GitHub Actions Failure

benc-db temporarily deployed to azure-prod March 5, 2025 20:46 — with GitHub Actions Inactive

fix minor issues

68c841d

benc-db temporarily deployed to azure-prod March 6, 2025 00:03 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Views V2 #954

Views V2 #954

benc-db commented Mar 4, 2025

benc-db Mar 4, 2025

benc-db Mar 4, 2025

benc-db Mar 4, 2025

benc-db Mar 4, 2025

benc-db Mar 4, 2025

benc-db Mar 4, 2025

benc-db Mar 4, 2025

benc-db Mar 4, 2025

benc-db Mar 4, 2025

benc-db Mar 4, 2025

benc-db Mar 4, 2025

benc-db Mar 4, 2025

Views V2 #954

Are you sure you want to change the base?

Views V2 #954

Conversation

benc-db commented Mar 4, 2025

Description

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment