Skip to content

Commit

Permalink
ADAP-2: Materialized Views (#7239)
Browse files Browse the repository at this point in the history
* changie

* ADAP-387: Stub materialized view as a materialization (#7211)

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* updating to feature main

---------

Co-authored-by: Matthew McKnight <[email protected]>

* ADAP-387: reverting db_api implementation (#7322)

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* updating to feature main

* removing db_api and strategies directories in favor of matching current materialization setups

* macro name change

* revert to current approach for materializations

* added tests

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* typo

* moved materialized view tests into adapter test framework

* add enum to relation for comparison in jinja

---------

Co-authored-by: Mike Alfare <[email protected]>

* ADAP-391: Add configuration change option (#7272)

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* move up pre-existing drop of backup

* change relation type to view to be consistent

* add base test case

* fix jinja exception message expression, basic test passing

* align names between postgres and default to same convention

* init set of Enum for config

* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core

* add on_configuration_change to unit test expected values

* make suggested name change to Enum class

* add on_configuration_change to some integration tests

* add on_configuration_change to expected_manifest to pass functional tests

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* moved materialized view tests into adapter test framework

* add alter materialized view macro

* change class name, and config setup

* play with field setup for on_configuration_change

* add method for default selection in enum class

* renamed get_refresh_data_in_materialized_view_sql to align with experimental package

* changed expected values to default string

* added in `on_configuration_change` setting

* change ignore to skip

* updated default option for on_configuration_change on NodeConfig

* removed explicit calls to enum values

* add test setup for testing fail config option

* updated `config_updates` to `configuration_changes` to align with `on_configuration_change` name

* setup configuration change framework

* skipped tests that are expected to fail without adapter implementation

* cleaned up log checks

---------

Co-authored-by: Mike Alfare <[email protected]>

* ADAP-388: Stub materialized view as a materialization - postgres (#7244)

* move the body of the default macros into the postgres implementation, throw errors if the default is used, indicating that materialized views have not been implemented for that adapter

---------

Co-authored-by: Matthew McKnight <[email protected]>

* ADAP-402: Add configuration change option - postgres (#7334)

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* added materialized view stubs and test

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* updated postgres to use the new macros structure

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* init copy of pr 387 to begin 391 implementation

* init set of Enum for config

* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core

* remove postgres-specific materialization in favor of core default materialization

* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies

* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer

* add on_configuration_change to unit test expected values

* make suggested name change to Enum class

* add on_configuration_change to some integretion tests

* add on_configuration_change to expected_manifest to pass functuional tests

* removing db_api and strategies directories in favor of matching current materialization setups

* macro name change

* revert to current approach for materializations

* revert to current approach for materializations

* added tests

* move materialized view logic into the `/materializations` directory in line with `dbt-core`

* moved default macros in `dbt-core` into `dbt-postgres`

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* moved materialized view tests into adapter test framework

* updated materialized view tests to use adapter test framework

* add alter materialized view macro

* add alter materialized view macro

* change class name, and config setup

* change class name, and config setup

* play with field setup for on_configuration_change

* add method for default selection in enum class

* renamed get_refresh_data_in_materialized_view_sql to align with experimental package

* changed expected values to default string

* added in `on_configuration_change` setting

* change ignore to skip

* added in `on_configuration_change` setting

* updated default option for on_configuration_change on NodeConfig

* updated default option for on_configuration_change on NodeConfig

* fixed list being passed as string bug

* removed explicit calls to enum values

* removed unneeded test class

* fixed on_configuration_change to be picked up appropriately

* add test setup for testing fail config option

* remove breakpoint, uncomment tests

* update skip scenario to use empty strings

* update skip scenario to avoid using sql at all, remove extra whitespace in some templates

* push up initial addition of indexes for mv macro

* push slight change up

* reverting alt macro and moving the do create_index call to be more in line with other materializations

* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402

# Conflicts:
#	core/dbt/contracts/graph/model_config.py
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
#	plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
#	tests/adapter/dbt/tests/adapter/materialized_views/base.py
#	tests/functional/materializations/test_materialized_view.py

* merge feature branch into story branch

* merge feature branch into story branch

* added indexes into the workflow

* fix error in jinja that caused print error

* working on test messaging and skipping tests that might not fit quite into current system

* add drop and show macros for indexes

* add drop and show macros for indexes

* add logic to determine the indexes to create or drop

* pulled index updates through the workflow properly

* convert configuration changes to fixtures, implement index changes into tests

* created Model dataclass for readability, added column to swap index columns for testing

* fixed typo

---------

Co-authored-by: Matthew McKnight <[email protected]>

* ADAP-395: Implement native materialized view DDL (#7336)

* changie

* changie

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources

* remove unneeded return statement, rename directory

* remove unneeded ()

* responding to some pr feedback

* adjusting order of events for mv base work

* move up prexisting drop of backup

* change relatiion type to view to be consistent

* add base test case

* fix jinja exeception message expression, basic test passing

* added materialized view stubs and test

* response to feedback, removeal of refresh infavor of combined create_as, etc.

* updated postgres to use the new macros structure

* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)

* remove stratgey to limit need for now

* remove unneeded story level changelog entry

* add strategies to condtional in place of old macros

* macro name fix

* rename refresh macro in api level

* align names between postgres and default to same convention

* align names between postgres and default to same convention

* change a create call to full refresh

* pull adapter rename into strategy, add backup_relation as optional arg

* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic

* init copy of pr 387 to begin 391 implementation

* updating to feature main

* updating to feature main

* init set of Enum for config

* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core

* remove postgres-specific materialization in favor of core default materialization

* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies

* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer

* add on_configuration_change to unit test expected values

* make suggested name change to Enum class

* add on_configuration_change to some integretion tests

* add on_configuration_change to expected_manifest to pass functuional tests

* removing db_api and strategies directories in favor of matching current materialization setups

* macro name change

* revert to current approach for materializations

* revert to current approach for materializations

* added tests

* move materialized view logic into the `/materializations` directory in line with `dbt-core`

* moved default macros in `dbt-core` into `dbt-postgres`

* added `is_materialized_view` to `BaseRelation`

* updated materialized view stored value to snake case

* typo

* moved materialized view tests into adapter test framework

* updated materialized view tests to use adapter test framework

* add alter materialized view macro

* add alter materialized view macro

* added basic sql to default macros, added postgres-specific sql for alter scenario, stubbed a test case for index update

* change class name, and config setup

* change class name, and config setup

* play with field setup for on_configuration_change

* add method for default selection in enum class

* renamed get_refresh_data_in_materialized_view_sql to align with experimental package

* changed expected values to default string

* added in `on_configuration_change` setting

* change ignore to skip

* added in `on_configuration_change` setting

* updated default option for on_configuration_change on NodeConfig

* updated default option for on_configuration_change on NodeConfig

* fixed list being passed as string bug

* fixed list being passed as string bug

* removed explicit calls to enum values

* removed explicit calls to enum values

* removed unneeded test class

* fixed on_configuration_change to be picked up appropriately

* add test setup for testing fail config option

* remove breakpoint, uncomment tests

* update skip scenario to use empty strings

* update skip scenario to avoid using sql at all, remove extra whitespace in some templates

* push up initial addition of indexes for mv macro

* push slight change up

* reverting alt macro and moving the do create_index call to be more in line with other materializations

* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402

# Conflicts:
#	core/dbt/contracts/graph/model_config.py
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
#	plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
#	tests/adapter/dbt/tests/adapter/materialized_views/base.py
#	tests/functional/materializations/test_materialized_view.py

* merge feature branch into story branch

* merge feature branch into story branch

* added indexes into the workflow

* fix error in jinja that caused print error

* working on test messaging and skipping tests that might not fit quite into current system

* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-395

# Conflicts:
#	core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
#	plugins/postgres/dbt/include/postgres/macros/adapters.sql
#	plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
#	tests/adapter/dbt/tests/adapter/materialized_views/test_on_configuration_change.py
#	tests/functional/materializations/test_materialized_view.py

* moved postgres implemention into plugin directory

* update index methods to align with the configuration update macro

* added native ddl to postgres macros

* removed extra docstring

* updated references to View, now references MaterializedView

* decomposed materialization into macros

* refactor index create statement parser, add exceptions for unexpected formats

* swapped conditional to check for positive state

* removed skipped test now that materialized view is being used

* return the results and logs of the run so that additional checks can be applied at the adapter level, add check for refresh to a test

* add check for indexes in particular for apply on configuration scenario

* removed extra argument

* add materialized views to get_relations / list_relations

* typos in index change logic

* moved full refresh check inside the build sql step

---------

Co-authored-by: Matthew McKnight <[email protected]>

* removing returns from tests to stop logs from printing

* moved test cases into postgres tests, left non-test functionality in base as new methods or fixtures

* fixed overwrite issue, simplified assertion method

* updated import order to standard

* fixed test import paths

* updated naming convention for proper test collection with the test runner

* still trying to make the test runner happy

* rewrite index updates to use a better source in Postgres

* break out a large test suite as a separate run

* update `skip` and `fail` scenarios with more descriptive results

* typo

* removed call to skip status

* reverting `exceptions_jinja.py`

* added FailFastError back, the right way

* removed PostgresIndex in favor of the already existing PostgresIndexConfig, pulled it into its own file to avoid circular imports

* removed assumed models in method calls, removed odd insert records and replaced with get row count

* fixed index issue, removed some indirection in testing

* made test more readable

* remove the "apply" from the tests and put it on the base as the default

* generalized assertion for reuse with dbt-snowflake, fixed bug in record count utility

* fixed type to be more generic to accommodate adapters with their own relation types

* fixed all the broken index stuff

* updated on_configuration_change to use existing patterns

* updated on_configuration_change to use existing patterns

* reflected update in tests and materialization logic

* reflected update in tests and materialization logic

* reverted the change to create a config object from the option object, using just the option object now

* reverted the change to create a config object from the option object, using just the option object now

* modelled database objects to support monitoring all configuration changes

* updated "skip" to "continue", throw an error on non-implemented macro defaults

* updated "skip" to "continue", throw an error on non-implemented macro defaults

* updated "skip" to "continue", throw an error on non-implemented macro defaults

* updated "skip" to "continue", throw an error on non-implemented macro defaults

* reverted centralized framework, retained a few reusable base classes

* updated names to be more consistent

* readability updates

* added readme specifying that `relation_configs` only supports materialized views for now

---------

Co-authored-by: Matthew McKnight <[email protected]>
Co-authored-by: Matthew McKnight <[email protected]>
  • Loading branch information
3 people authored Jun 7, 2023
1 parent f4253da commit 2d23782
Show file tree
Hide file tree
Showing 39 changed files with 1,366 additions and 8 deletions.
6 changes: 6 additions & 0 deletions .changes/unreleased/Features-20230329-120313.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: Add support for materialized views
time: 2023-03-29T12:03:13.862041-04:00
custom:
Author: mikealfare McKnight-42
Issue: "6911"
8 changes: 8 additions & 0 deletions core/dbt/adapters/base/relation.py
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,10 @@ def is_cte(self) -> bool:
def is_view(self) -> bool:
return self.type == RelationType.View

@property
def is_materialized_view(self) -> bool:
return self.type == RelationType.MaterializedView

@classproperty
def Table(cls) -> str:
return str(RelationType.Table)
Expand All @@ -344,6 +348,10 @@ def View(cls) -> str:
def External(cls) -> str:
return str(RelationType.External)

@classproperty
def MaterializedView(cls) -> str:
return str(RelationType.MaterializedView)

@classproperty
def get_relation_type(cls) -> Type[RelationType]:
return RelationType
Expand Down
25 changes: 25 additions & 0 deletions core/dbt/adapters/relation_configs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# RelationConfig
This package serves as an initial abstraction for managing the inspection of existing relations and determining
changes on those relations. It arose from the materialized view work and is currently only supporting
materialized views for Postgres and Redshift as well as dynamic tables for Snowflake. There are three main
classes in this package.

## RelationConfigBase
This is a very small class that only has a `from_dict()` method and a default `NotImplementedError()`. At some
point this could be replaced by a more robust framework, like `mashumaro` or `pydantic`.

## RelationConfigChange
This class inherits from `RelationConfigBase` ; however, this can be thought of as a separate class. The subclassing
merely points to the idea that both classes would likely inherit from the same class in a `mashumaro` or
`pydantic` implementation. This class is much more restricted in attribution. It should really only
ever need an `action` and a `context`. This can be though of as being analogous to a web request. You need to
know what you're doing (`action`: 'create' = GET, 'drop' = DELETE, etc.) and the information (`context`) needed
to make the change. In our scenarios, the context tends to be an instance of `RelationConfigBase` corresponding
to the new state.

## RelationConfigValidationMixin
This mixin provides optional validation mechanics that can be applied to either `RelationConfigBase` or
`RelationConfigChange` subclasses. A validation rule is a combination of a `validation_check`, something
that should evaluate to `True`, and an optional `validation_error`, an instance of `DbtRuntimeError`
that should be raised in the event the `validation_check` fails. While optional, it's recommended that
the `validation_error` be provided for clearer transparency to the end user.
12 changes: 12 additions & 0 deletions core/dbt/adapters/relation_configs/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
from dbt.adapters.relation_configs.config_base import ( # noqa: F401
RelationConfigBase,
RelationResults,
)
from dbt.adapters.relation_configs.config_change import ( # noqa: F401
RelationConfigChangeAction,
RelationConfigChange,
)
from dbt.adapters.relation_configs.config_validation import ( # noqa: F401
RelationConfigValidationMixin,
RelationConfigValidationRule,
)
44 changes: 44 additions & 0 deletions core/dbt/adapters/relation_configs/config_base.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
from dataclasses import dataclass
from typing import Union, Dict

import agate
from dbt.utils import filter_null_values


"""
This is what relation metadata from the database looks like. It's a dictionary because there will be
multiple grains of data for a single object. For example, a materialized view in Postgres has base level information,
like name. But it also can have multiple indexes, which needs to be a separate query. It might look like this:
{
"base": agate.Row({"table_name": "table_abc", "query": "select * from table_def"})
"indexes": agate.Table("rows": [
agate.Row({"name": "index_a", "columns": ["column_a"], "type": "hash", "unique": False}),
agate.Row({"name": "index_b", "columns": ["time_dim_a"], "type": "btree", "unique": False}),
])
}
"""
RelationResults = Dict[str, Union[agate.Row, agate.Table]]


@dataclass(frozen=True)
class RelationConfigBase:
@classmethod
def from_dict(cls, kwargs_dict) -> "RelationConfigBase":
"""
This assumes the subclass of `RelationConfigBase` is flat, in the sense that no attribute is
itself another subclass of `RelationConfigBase`. If that's not the case, this should be overriden
to manually manage that complexity.
Args:
kwargs_dict: the dict representation of this instance
Returns: the `RelationConfigBase` representation associated with the provided dict
"""
return cls(**filter_null_values(kwargs_dict)) # type: ignore

@classmethod
def _not_implemented_error(cls) -> NotImplementedError:
return NotImplementedError(
"This relation type has not been fully configured for this adapter."
)
23 changes: 23 additions & 0 deletions core/dbt/adapters/relation_configs/config_change.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Hashable

from dbt.adapters.relation_configs.config_base import RelationConfigBase
from dbt.dataclass_schema import StrEnum


class RelationConfigChangeAction(StrEnum):
alter = "alter"
create = "create"
drop = "drop"


@dataclass(frozen=True, eq=True, unsafe_hash=True)
class RelationConfigChange(RelationConfigBase, ABC):
action: RelationConfigChangeAction
context: Hashable # this is usually a RelationConfig, e.g. IndexConfig, but shouldn't be limited

@property
@abstractmethod
def requires_full_refresh(self) -> bool:
raise self._not_implemented_error()
57 changes: 57 additions & 0 deletions core/dbt/adapters/relation_configs/config_validation.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
from dataclasses import dataclass
from typing import Set, Optional

from dbt.exceptions import DbtRuntimeError


@dataclass(frozen=True, eq=True, unsafe_hash=True)
class RelationConfigValidationRule:
validation_check: bool
validation_error: Optional[DbtRuntimeError]

@property
def default_error(self):
return DbtRuntimeError(
"There was a validation error in preparing this relation config."
"No additional context was provided by this adapter."
)


@dataclass(frozen=True)
class RelationConfigValidationMixin:
def __post_init__(self):
self.run_validation_rules()

@property
def validation_rules(self) -> Set[RelationConfigValidationRule]:
"""
A set of validation rules to run against the object upon creation.
A validation rule is a combination of a validation check (bool) and an optional error message.
This defaults to no validation rules if not implemented. It's recommended to override this with values,
but that may not always be necessary.
Returns: a set of validation rules
"""
return set()

def run_validation_rules(self):
for validation_rule in self.validation_rules:
try:
assert validation_rule.validation_check
except AssertionError:
if validation_rule.validation_error:
raise validation_rule.validation_error
else:
raise validation_rule.default_error
self.run_child_validation_rules()

def run_child_validation_rules(self):
for attr_value in vars(self).values():
if hasattr(attr_value, "validation_rules"):
attr_value.run_validation_rules()
if isinstance(attr_value, set):
for member in attr_value:
if hasattr(member, "validation_rules"):
member.run_validation_rules()
6 changes: 6 additions & 0 deletions core/dbt/context/exceptions_jinja.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
RelationWrongTypeError,
ContractError,
ColumnTypeMissingError,
FailFastError,
)


Expand Down Expand Up @@ -107,6 +108,10 @@ def column_type_missing(column_names) -> NoReturn:
raise ColumnTypeMissingError(column_names)


def raise_fail_fast_error(msg, node=None) -> NoReturn:
raise FailFastError(msg, node=node)


# Update this when a new function should be added to the
# dbt context's `exceptions` key!
CONTEXT_EXPORTS = {
Expand All @@ -131,6 +136,7 @@ def column_type_missing(column_names) -> NoReturn:
relation_wrong_type,
raise_contract_error,
column_type_missing,
raise_fail_fast_error,
]
}

Expand Down
27 changes: 24 additions & 3 deletions core/dbt/contracts/graph/model_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,17 @@
from enum import Enum
from itertools import chain
from typing import Any, List, Optional, Dict, Union, Type, TypeVar, Callable

from dbt.dataclass_schema import (
dbtClassMixin,
ValidationError,
register_pattern,
StrEnum,
)
from dbt.contracts.graph.unparsed import AdditionalPropertiesAllowed, Docs
from dbt.contracts.graph.utils import validate_color
from dbt.exceptions import DbtInternalError, CompilationError
from dbt.contracts.util import Replaceable, list_str
from dbt.exceptions import DbtInternalError, CompilationError
from dbt import hooks
from dbt.node_types import NodeType

Expand Down Expand Up @@ -189,6 +191,16 @@ class Severity(str):
register_pattern(Severity, insensitive_patterns("warn", "error"))


class OnConfigurationChangeOption(StrEnum):
Apply = "apply"
Continue = "continue"
Fail = "fail"

@classmethod
def default(cls) -> "OnConfigurationChangeOption":
return cls.Apply


@dataclass
class ContractConfig(dbtClassMixin, Replaceable):
enforced: bool = False
Expand Down Expand Up @@ -287,11 +299,17 @@ def same_contents(cls, unrendered: Dict[str, Any], other: Dict[str, Any]) -> boo
return False
return True

# This is used in 'add_config_call' to created the combined config_call_dict.
# This is used in 'add_config_call' to create the combined config_call_dict.
# 'meta' moved here from node
mergebehavior = {
"append": ["pre-hook", "pre_hook", "post-hook", "post_hook", "tags"],
"update": ["quoting", "column_types", "meta", "docs", "contract"],
"update": [
"quoting",
"column_types",
"meta",
"docs",
"contract",
],
"dict_key_append": ["grants"],
}

Expand Down Expand Up @@ -445,6 +463,9 @@ class NodeConfig(NodeAndTestConfig):
# sometimes getting the Union order wrong, causing serialization failures.
unique_key: Union[str, List[str], None] = None
on_schema_change: Optional[str] = "ignore"
on_configuration_change: OnConfigurationChangeOption = field(
default_factory=OnConfigurationChangeOption.default
)
grants: Dict[str, Any] = field(
default_factory=dict, metadata=MergeBehavior.DictKeyAppend.meta()
)
Expand Down
2 changes: 1 addition & 1 deletion core/dbt/contracts/relation.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ class RelationType(StrEnum):
Table = "table"
View = "view"
CTE = "cte"
MaterializedView = "materializedview"
MaterializedView = "materialized_view"
External = "external"


Expand Down
18 changes: 18 additions & 0 deletions core/dbt/include/global_project/macros/adapters/indexes.sql
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,21 @@
{% endif %}
{% endfor %}
{% endmacro %}


{% macro get_drop_index_sql(relation, index_name) -%}
{{ adapter.dispatch('get_drop_index_sql', 'dbt')(relation, index_name) }}
{%- endmacro %}

{% macro default__get_drop_index_sql(relation, index_name) -%}
{{ exceptions.raise_compiler_error("`get_drop_index_sql has not been implemented for this adapter.") }}
{%- endmacro %}


{% macro get_show_indexes_sql(relation) -%}
{{ adapter.dispatch('get_show_indexes_sql', 'dbt')(relation) }}
{%- endmacro %}

{% macro default__get_show_indexes_sql(relation) -%}
{{ exceptions.raise_compiler_error("`get_show_indexes_sql has not been implemented for this adapter.") }}
{%- endmacro %}
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{% macro get_alter_materialized_view_as_sql(
relation,
configuration_changes,
sql,
existing_relation,
backup_relation,
intermediate_relation
) %}
{{- log('Applying ALTER to: ' ~ relation) -}}
{{- adapter.dispatch('get_alter_materialized_view_as_sql', 'dbt')(
relation,
configuration_changes,
sql,
existing_relation,
backup_relation,
intermediate_relation
) -}}
{% endmacro %}


{% macro default__get_alter_materialized_view_as_sql(
relation,
configuration_changes,
sql,
existing_relation,
backup_relation,
intermediate_relation
) %}
{{ exceptions.raise_compiler_error("Materialized views have not been implemented for this adapter.") }}
{% endmacro %}
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{% macro get_create_materialized_view_as_sql(relation, sql) -%}
{{- log('Applying CREATE to: ' ~ relation) -}}
{{- adapter.dispatch('get_create_materialized_view_as_sql', 'dbt')(relation, sql) -}}
{%- endmacro %}


{% macro default__get_create_materialized_view_as_sql(relation, sql) -%}
{{ exceptions.raise_compiler_error("Materialized views have not been implemented for this adapter.") }}
{% endmacro %}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{% macro get_materialized_view_configuration_changes(existing_relation, new_config) %}
/* {#
It's recommended that configuration changes be formatted as follows:
{"<change_category>": [{"action": "<name>", "context": ...}]}
For example:
{
"indexes": [
{"action": "drop", "context": "index_abc"},
{"action": "create", "context": {"columns": ["column_1", "column_2"], "type": "hash", "unique": True}},
],
}
Either way, `get_materialized_view_configuration_changes` needs to align with `get_alter_materialized_view_as_sql`.
#} */
{{- log('Determining configuration changes on: ' ~ existing_relation) -}}
{%- do return(adapter.dispatch('get_materialized_view_configuration_changes', 'dbt')(existing_relation, new_config)) -%}
{% endmacro %}


{% macro default__get_materialized_view_configuration_changes(existing_relation, new_config) %}
{{ exceptions.raise_compiler_error("Materialized views have not been implemented for this adapter.") }}
{% endmacro %}
Loading

0 comments on commit 2d23782

Please sign in to comment.