model versions #7287

MichelleArk · 2023-04-06T14:14:40Z

resolves #7263

Description

Splits out ModelPatchParser and UnparsedModelUpdate for model-specific parsing. Also introduces VersionedTestBlock.
Returns a ParseResult instead of a list of test blocks from base parser parse method
Updates node.refs from List[List[str]] to List[RefArgs]

TODO

Handle disabled node lookup for versioned nodes
Partial parsing for versioned blocks
fully overridable UnparsedVersions tests
new integration + unit tests

Checklist

I have read the contributing guide and understand what's expected of me
I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
I have run changie new to create a changelog entry

🎩

schema.yml:

models:
  - name: dim_customers
    latest_version: 2
    config:
      materialized: table
      test: test
    tests:
      - unique:
          column_name: "location_id"
    columns:
      - name: customer_id
        description: This is the primary key
        data_type: int
      - name: location_id
        description: id of location
        data_type: int
        tests: 
          - not_null
    versions:
      - v: 3
        columns: 
          - include: '*'
            exclude: ['location_id']
          - name: extra
        defined_in: arbitrary_file_name
        tests: []
      - v: 2
        config: 
          materialized: view
        columns:
          - include: '*'

models/dim_customers_v2.sql

select 1 as location_id

models/arbitrary_file_name.sql

SELECT 1 as a
UNION ALL 
SELECT null as a

models/test_version.sql

select * from {{ ref('dim_customers', v=3) }}
UNION ALL 
select * from {{ ref('dim_customers') }}

MichelleArk · 2023-04-10T20:05:43Z

core/dbt/parser/schemas.py

+                source_file.append_patch(
+                    versioned_model_patch.yaml_key, versioned_model_node.unique_id
+                )
+            self.manifest.rebuild_ref_lookup()


TODO: look into whether this can be more granular and moved into the loop for a performance improvement

MichelleArk · 2023-04-10T20:15:19Z

core/dbt/contracts/graph/nodes.py

@@ -405,6 +427,11 @@ def patch(self, patch: "ParsedNodePatch"):
        self.created_at = time.time()
        self.description = patch.description
        self.columns = patch.columns
+


consider moving this to parse_patch code on PatchParser

gshank

Looks really good! I just had one question about "build_model_str"...

core/dbt/parser/generic_test_builders.py

jtcohen6

I found a small partial parsing bug, and you just fixed it.

That's ... really it! I've done my best to break this, and it's been holding up well. A handful of comments, none of them a blocker to merging.

Nice work :)

core/dbt/context/providers.py

core/dbt/contracts/graph/unparsed.py

core/dbt/include/global_project/macros/get_custom_name/get_custom_alias.sql

core/dbt/node_types.py

jtcohen6 · 2023-04-11T18:45:43Z

core/dbt/contracts/graph/nodes.py

+        # TODO: version, is_latest_version, and access are specific to ModelNodes, consider splitting out to ModelNode
+        if self.resource_type != NodeType.Model:


When would we expect these validation warnings to come up in practice (if ever)? e.g. I've tried adding versions / version attributes to a seed patch, and I don't see any warnings

core/dbt/parser/schemas.py

MichelleArk · 2023-04-11T23:30:27Z

Added support for ref resolution for python models in this commit (thank you @jtcohen6!!). Updated adapter tests as well, and verified those work in Snowflake (BigQuery skips these tests, Spark CI is broken - but there is no adapter-specific code so this should work across all adapters if it works for Snowflake)

MichelleArk · 2023-04-12T13:41:24Z

Latest change adds a latest_version attribute to the ModelNode, and switches is_latest_version to a property: 6e49ea5. Was considering this during spiking but couldn't find a reason to keep latest_version on ModelNode at that point. Keeping latest_version on the model directly makes a whole lot of sense to implement new version selector method (#7330), so I've added it in this PR.

resolves #3194 [Page preview](https://deploy-preview-3195--docs-getdbt-com.netlify.app/docs/collaborate/govern/model-versions#how-to-create-a-new-version-of-a-model) ## What are you changing in this pull request and why? Update code examples for `defined_in` and `exclude` to align with the implementation (and code examples) in [dbt-labs/dbt-core#7287](dbt-labs/dbt-core#7287): - e27c2da Example using `defined_in` property - 9da3e00 Fix the example to `exclude` column(s) - d0f5396 Use postgres-specific data types (since it is a neutral database platform and its data types can be easily ported) ## Checklist - [x] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) and [About versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) so my content adheres to these guidelines. - [x] Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch."

dbeatty10

I found some un-submitted review comments relating to changie 😅

dbeatty10 · 2023-04-06T19:20:29Z

.changes/unreleased/Features-20230406-101019.yaml

+time: 2023-04-06T10:10:19.794672-04:00
+custom:
+  Author: michelleark
+  Issue: '#7263'


Does Issue: '#7263' work?

Or does it need to be like the following?

Suggested change

Issue: '#7263'

Issue: "7263"

To close the loop, Issue: '#7263' didn't work:

https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md

dbt-core/CHANGELOG.md

Line 47 in dee5e70

- model versions ([##7263](https://github.com/dbt-labs/dbt-core/issues/#7263))

first pass - model versions

2d61cbb

cla-bot bot added the cla:yes label Apr 6, 2023

MichelleArk added 3 commits April 6, 2023 16:07

NodeVersion alias, config/meta merging

c652fb4

ref with version/v keyword arg

93d406e

Merge branch 'main' into 6866/model-versions2

61df807

MichelleArk changed the title ~~first pass - model versions~~ model versions Apr 10, 2023

MichelleArk added 2 commits April 10, 2023 10:51

default test=None for UnparsedVersion

2cca5d5

partial parsing for versions

915331b

MichelleArk commented Apr 10, 2023

View reviewed changes

MichelleArk added 3 commits April 10, 2023 23:56

unit tests, functional tests for partial parsing, misc cleanup

eed6de3

disabled versioned nodes

8bd22ed

replace '.' with '_' in version for get_custom_alias

b17971e

MichelleArk marked this pull request as ready for review April 11, 2023 16:12

MichelleArk requested review from a team as code owners April 11, 2023 16:12

MichelleArk requested review from nathaniel-may, emmyoop, gshank and peterallenwebb and removed request for nathaniel-may April 11, 2023 16:12

gshank approved these changes Apr 11, 2023

View reviewed changes

core/dbt/parser/generic_test_builders.py Outdated Show resolved Hide resolved

TestVerifyArtifactsVersions

717c491

jtcohen6 reviewed Apr 11, 2023

View reviewed changes

partial parsing - account for modified versioned source files

5ca3bd0

MichelleArk force-pushed the 6866/model-versions2 branch from b5ba176 to 5ca3bd0 Compare April 11, 2023 19:03

more explicit handling of ModelUpdate with no version in build_model_str

51ec615

gshank reviewed Apr 11, 2023

View reviewed changes

core/dbt/parser/schemas.py Outdated Show resolved Hide resolved

include all base columns in versioned models by default

f3bd959

MichelleArk added 3 commits April 11, 2023 16:23

Merge branch 'main' into 6866/model-versions2

dc5c241

fix merge - constraint validation on unversioned models

3b0e7da

versioned ref for python models

1b33416

MichelleArk force-pushed the 6866/model-versions2 branch from cffd825 to 1b33416 Compare April 11, 2023 21:28

emmyoop mentioned this pull request Apr 11, 2023

Add most recent dbt-docs changes #7328

Merged

merge meta within config

bb79e71

add latest_version attribute to ModelNode, is_latest_version as property

6e49ea5

MichelleArk merged commit c7ebc89 into main Apr 12, 2023

MichelleArk deleted the 6866/model-versions2 branch April 12, 2023 13:50

jtcohen6 mentioned this pull request Apr 13, 2023

[CT-2409] [Bug] FQN selection for versioned models in subdirectory #7348

Closed

This was referenced Apr 16, 2023

Fix syntax examples for model versions dbt-labs/docs.getdbt.com#3194

Closed

Fix syntax examples for model versions dbt-labs/docs.getdbt.com#3195

Merged

MichelleArk mentioned this pull request Apr 20, 2023

fix v0 ref resolution #7415

Merged

6 tasks

dbeatty10 reviewed Apr 20, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

model versions #7287

model versions #7287

MichelleArk commented Apr 6, 2023 •

edited

Loading

MichelleArk Apr 10, 2023

MichelleArk Apr 10, 2023

gshank left a comment

jtcohen6 left a comment

jtcohen6 Apr 11, 2023 •

edited

Loading

MichelleArk commented Apr 11, 2023

MichelleArk commented Apr 12, 2023

dbeatty10 left a comment

dbeatty10 Apr 6, 2023

dbeatty10 Apr 20, 2023

		# TODO: version, is_latest_version, and access are specific to ModelNodes, consider splitting out to ModelNode
		if self.resource_type != NodeType.Model:

model versions #7287

model versions #7287

Conversation

MichelleArk commented Apr 6, 2023 • edited Loading

Description

TODO

Checklist

MichelleArk Apr 10, 2023

Choose a reason for hiding this comment

MichelleArk Apr 10, 2023

Choose a reason for hiding this comment

gshank left a comment

Choose a reason for hiding this comment

jtcohen6 left a comment

Choose a reason for hiding this comment

jtcohen6 Apr 11, 2023 • edited Loading

Choose a reason for hiding this comment

MichelleArk commented Apr 11, 2023

MichelleArk commented Apr 12, 2023

dbeatty10 left a comment

Choose a reason for hiding this comment

dbeatty10 Apr 6, 2023

Choose a reason for hiding this comment

dbeatty10 Apr 20, 2023

Choose a reason for hiding this comment

MichelleArk commented Apr 6, 2023 •

edited

Loading

jtcohen6 Apr 11, 2023 •

edited

Loading