Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* changie * ADAP-387: Stub materialized view as a materialization (#7211) * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * remove unneeded return statement, rename directory * remove unneeded () * responding to some pr feedback * adjusting order of events for mv base work * move up prexisting drop of backup * change relatiion type to view to be consistent * add base test case * fix jinja exeception message expression, basic test passing * response to feedback, removeal of refresh infavor of combined create_as, etc. * swapping to api layer and stratgeies for default implementation (basing off postgres, redshift) * remove stratgey to limit need for now * remove unneeded story level changelog entry * add strategies to condtional in place of old macros * macro name fix * rename refresh macro in api level * align names between postgres and default to same convention * align names between postgres and default to same convention * change a create call to full refresh * pull adapter rename into strategy, add backup_relation as optional arg * minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic * updating to feature main --------- Co-authored-by: Matthew McKnight <[email protected]> * ADAP-387: reverting db_api implementation (#7322) * changie * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * remove unneeded return statement, rename directory * remove unneeded () * responding to some pr feedback * adjusting order of events for mv base work * move up prexisting drop of backup * change relatiion type to view to be consistent * add base test case * fix jinja exeception message expression, basic test passing * response to feedback, removeal of refresh infavor of combined create_as, etc. * swapping to api layer and stratgeies for default implementation (basing off postgres, redshift) * remove stratgey to limit need for now * remove unneeded story level changelog entry * add strategies to condtional in place of old macros * macro name fix * rename refresh macro in api level * align names between postgres and default to same convention * change a create call to full refresh * pull adapter rename into strategy, add backup_relation as optional arg * minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic * updating to feature main * removing db_api and strategies directories in favor of matching current materialization setups * macro name change * revert to current approach for materializations * added tests * added `is_materialized_view` to `BaseRelation` * updated materialized view stored value to snake case * typo * moved materialized view tests into adapter test framework * add enum to relation for comparison in jinja --------- Co-authored-by: Mike Alfare <[email protected]> * ADAP-391: Add configuration change option (#7272) * changie * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * move up pre-existing drop of backup * change relation type to view to be consistent * add base test case * fix jinja exception message expression, basic test passing * align names between postgres and default to same convention * init set of Enum for config * work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core * add on_configuration_change to unit test expected values * make suggested name change to Enum class * add on_configuration_change to some integration tests * add on_configuration_change to expected_manifest to pass functional tests * added `is_materialized_view` to `BaseRelation` * updated materialized view stored value to snake case * moved materialized view tests into adapter test framework * add alter materialized view macro * change class name, and config setup * play with field setup for on_configuration_change * add method for default selection in enum class * renamed get_refresh_data_in_materialized_view_sql to align with experimental package * changed expected values to default string * added in `on_configuration_change` setting * change ignore to skip * updated default option for on_configuration_change on NodeConfig * removed explicit calls to enum values * add test setup for testing fail config option * updated `config_updates` to `configuration_changes` to align with `on_configuration_change` name * setup configuration change framework * skipped tests that are expected to fail without adapter implementation * cleaned up log checks --------- Co-authored-by: Mike Alfare <[email protected]> * ADAP-388: Stub materialized view as a materialization - postgres (#7244) * move the body of the default macros into the postgres implementation, throw errors if the default is used, indicating that materialized views have not been implemented for that adapter --------- Co-authored-by: Matthew McKnight <[email protected]> * ADAP-402: Add configuration change option - postgres (#7334) * changie * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * remove unneeded return statement, rename directory * remove unneeded () * responding to some pr feedback * adjusting order of events for mv base work * move up prexisting drop of backup * change relatiion type to view to be consistent * add base test case * fix jinja exeception message expression, basic test passing * added materialized view stubs and test * response to feedback, removeal of refresh infavor of combined create_as, etc. * updated postgres to use the new macros structure * swapping to api layer and stratgeies for default implementation (basing off postgres, redshift) * remove stratgey to limit need for now * remove unneeded story level changelog entry * add strategies to condtional in place of old macros * macro name fix * rename refresh macro in api level * align names between postgres and default to same convention * change a create call to full refresh * pull adapter rename into strategy, add backup_relation as optional arg * minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic * init copy of pr 387 to begin 391 implementation * init set of Enum for config * work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core * remove postgres-specific materialization in favor of core default materialization * update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies * add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer * add on_configuration_change to unit test expected values * make suggested name change to Enum class * add on_configuration_change to some integretion tests * add on_configuration_change to expected_manifest to pass functuional tests * removing db_api and strategies directories in favor of matching current materialization setups * macro name change * revert to current approach for materializations * revert to current approach for materializations * added tests * move materialized view logic into the `/materializations` directory in line with `dbt-core` * moved default macros in `dbt-core` into `dbt-postgres` * added `is_materialized_view` to `BaseRelation` * updated materialized view stored value to snake case * moved materialized view tests into adapter test framework * updated materialized view tests to use adapter test framework * add alter materialized view macro * add alter materialized view macro * change class name, and config setup * change class name, and config setup * play with field setup for on_configuration_change * add method for default selection in enum class * renamed get_refresh_data_in_materialized_view_sql to align with experimental package * changed expected values to default string * added in `on_configuration_change` setting * change ignore to skip * added in `on_configuration_change` setting * updated default option for on_configuration_change on NodeConfig * updated default option for on_configuration_change on NodeConfig * fixed list being passed as string bug * removed explicit calls to enum values * removed unneeded test class * fixed on_configuration_change to be picked up appropriately * add test setup for testing fail config option * remove breakpoint, uncomment tests * update skip scenario to use empty strings * update skip scenario to avoid using sql at all, remove extra whitespace in some templates * push up initial addition of indexes for mv macro * push slight change up * reverting alt macro and moving the do create_index call to be more in line with other materializations * Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402 # Conflicts: # core/dbt/contracts/graph/model_config.py # core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql # plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql # tests/adapter/dbt/tests/adapter/materialized_views/base.py # tests/functional/materializations/test_materialized_view.py * merge feature branch into story branch * merge feature branch into story branch * added indexes into the workflow * fix error in jinja that caused print error * working on test messaging and skipping tests that might not fit quite into current system * add drop and show macros for indexes * add drop and show macros for indexes * add logic to determine the indexes to create or drop * pulled index updates through the workflow properly * convert configuration changes to fixtures, implement index changes into tests * created Model dataclass for readability, added column to swap index columns for testing * fixed typo --------- Co-authored-by: Matthew McKnight <[email protected]> * ADAP-395: Implement native materialized view DDL (#7336) * changie * changie * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * remove unneeded return statement, rename directory * remove unneeded () * responding to some pr feedback * adjusting order of events for mv base work * move up prexisting drop of backup * change relatiion type to view to be consistent * add base test case * fix jinja exeception message expression, basic test passing * added materialized view stubs and test * response to feedback, removeal of refresh infavor of combined create_as, etc. * updated postgres to use the new macros structure * swapping to api layer and stratgeies for default implementation (basing off postgres, redshift) * remove stratgey to limit need for now * remove unneeded story level changelog entry * add strategies to condtional in place of old macros * macro name fix * rename refresh macro in api level * align names between postgres and default to same convention * align names between postgres and default to same convention * change a create call to full refresh * pull adapter rename into strategy, add backup_relation as optional arg * minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic * init copy of pr 387 to begin 391 implementation * updating to feature main * updating to feature main * init set of Enum for config * work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core * remove postgres-specific materialization in favor of core default materialization * update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies * add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer * add on_configuration_change to unit test expected values * make suggested name change to Enum class * add on_configuration_change to some integretion tests * add on_configuration_change to expected_manifest to pass functuional tests * removing db_api and strategies directories in favor of matching current materialization setups * macro name change * revert to current approach for materializations * revert to current approach for materializations * added tests * move materialized view logic into the `/materializations` directory in line with `dbt-core` * moved default macros in `dbt-core` into `dbt-postgres` * added `is_materialized_view` to `BaseRelation` * updated materialized view stored value to snake case * typo * moved materialized view tests into adapter test framework * updated materialized view tests to use adapter test framework * add alter materialized view macro * add alter materialized view macro * added basic sql to default macros, added postgres-specific sql for alter scenario, stubbed a test case for index update * change class name, and config setup * change class name, and config setup * play with field setup for on_configuration_change * add method for default selection in enum class * renamed get_refresh_data_in_materialized_view_sql to align with experimental package * changed expected values to default string * added in `on_configuration_change` setting * change ignore to skip * added in `on_configuration_change` setting * updated default option for on_configuration_change on NodeConfig * updated default option for on_configuration_change on NodeConfig * fixed list being passed as string bug * fixed list being passed as string bug * removed explicit calls to enum values * removed explicit calls to enum values * removed unneeded test class * fixed on_configuration_change to be picked up appropriately * add test setup for testing fail config option * remove breakpoint, uncomment tests * update skip scenario to use empty strings * update skip scenario to avoid using sql at all, remove extra whitespace in some templates * push up initial addition of indexes for mv macro * push slight change up * reverting alt macro and moving the do create_index call to be more in line with other materializations * Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402 # Conflicts: # core/dbt/contracts/graph/model_config.py # core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql # plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql # tests/adapter/dbt/tests/adapter/materialized_views/base.py # tests/functional/materializations/test_materialized_view.py * merge feature branch into story branch * merge feature branch into story branch * added indexes into the workflow * fix error in jinja that caused print error * working on test messaging and skipping tests that might not fit quite into current system * Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-395 # Conflicts: # core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql # plugins/postgres/dbt/include/postgres/macros/adapters.sql # plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql # tests/adapter/dbt/tests/adapter/materialized_views/test_on_configuration_change.py # tests/functional/materializations/test_materialized_view.py * moved postgres implemention into plugin directory * update index methods to align with the configuration update macro * added native ddl to postgres macros * removed extra docstring * updated references to View, now references MaterializedView * decomposed materialization into macros * refactor index create statement parser, add exceptions for unexpected formats * swapped conditional to check for positive state * removed skipped test now that materialized view is being used * return the results and logs of the run so that additional checks can be applied at the adapter level, add check for refresh to a test * add check for indexes in particular for apply on configuration scenario * removed extra argument * add materialized views to get_relations / list_relations * typos in index change logic * moved full refresh check inside the build sql step --------- Co-authored-by: Matthew McKnight <[email protected]> * removing returns from tests to stop logs from printing * moved test cases into postgres tests, left non-test functionality in base as new methods or fixtures * fixed overwrite issue, simplified assertion method * updated import order to standard * fixed test import paths * updated naming convention for proper test collection with the test runner * still trying to make the test runner happy * rewrite index updates to use a better source in Postgres * break out a large test suite as a separate run * update `skip` and `fail` scenarios with more descriptive results * typo * removed call to skip status * reverting `exceptions_jinja.py` * added FailFastError back, the right way * removed PostgresIndex in favor of the already existing PostgresIndexConfig, pulled it into its own file to avoid circular imports * removed assumed models in method calls, removed odd insert records and replaced with get row count * fixed index issue, removed some indirection in testing * made test more readable * remove the "apply" from the tests and put it on the base as the default * generalized assertion for reuse with dbt-snowflake, fixed bug in record count utility * fixed type to be more generic to accommodate adapters with their own relation types * fixed all the broken index stuff * updated on_configuration_change to use existing patterns * updated on_configuration_change to use existing patterns * reflected update in tests and materialization logic * reflected update in tests and materialization logic * reverted the change to create a config object from the option object, using just the option object now * reverted the change to create a config object from the option object, using just the option object now * modelled database objects to support monitoring all configuration changes * updated "skip" to "continue", throw an error on non-implemented macro defaults * updated "skip" to "continue", throw an error on non-implemented macro defaults * updated "skip" to "continue", throw an error on non-implemented macro defaults * updated "skip" to "continue", throw an error on non-implemented macro defaults * reverted centralized framework, retained a few reusable base classes * updated names to be more consistent * readability updates * added readme specifying that `relation_configs` only supports materialized views for now --------- Co-authored-by: Matthew McKnight <[email protected]> Co-authored-by: Matthew McKnight <[email protected]>
- Loading branch information
3 people
authored
Jun 7, 2023
1 parent
f4253da
commit 2d23782