[CT-2734] Add metricflow_time_spine to the semantic manifest.

[Jstein77]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

We need to create a time spine model in the users dbt project to support cumulative metrics (full context here: https://www.notion.so/dbtlabs/Time-spine-for-SL-Beta-7c10850fcaf44d47ba0cf23b0a03f4b4). This will be a model that is fully managed by dbt. The user shouldn't have to think about configuring this model or need to write a seperate semantic manifest config.

We should automatically add this model to the semantic manifest, even if there is no semantic model config, so metricflow can use it for constructing cumulative metrics.

Describe alternatives you've considered

The user would create a semantic manifest file manually. This feels like unnecessary friction for since this model is an internal MetricFlow model, and not something a user should have to think about configuring.

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

[marcodamore]

Link to spec: https://github.com/dbt-labs/dbt-semantic-interfaces/pull/95/files/db47347be2568f58a015b952fd7cbbae7651f5aa#diff-bcadf38027b48517e9746238936cd9be52c0e28e6c74603d1eb11583a3157a9f

[jtcohen6]

I understand the current plan here is:

• For v1.6: Document how users can define this model themselves
• For v1.7: Find an elegant-but-not-too-magical way to automatically create this model, fully managed by dbt — perhaps using an approach similar to what we'd do for a "latest" pointer to a versioned model ([CT-2468] [Feature] For versioned models, automatically create view/clone of latest version in unsuffixed database location #7442)

We should automatically add this model to the semantic manifest, even if there is no semantic model config, so metricflow can use it for constructing cumulative metrics.

Question: Should dbt automatically create this model in every project? Even projects that are not using the Semantic Layer (defining semantic models / metrics) at all?

[Jstein77]

I love the idea of using similar logic for finding the latest versioned model. Thanks for surfacing that discussion!

Question: Should dbt automatically create this model in every project? Even projects that are not using the Semantic Layer (defining semantic models / metrics) at all?

I don't think we should create the time spine if the project doesn't have semantic models for two reasons:

1. We're creating an extra table/view in the users warehouse that they don't need. It would be confusing to see this pop up if you're just using dbt for transformations.
2. Although it's probably negligible, we're adding extra latency and compute to your project when we don't need to be.

Ideally we're smart enough to know if a project has semantic models, and only create the time spine table if they do.

[marcodamore]

Link to DSI changes task: https://linear.app/dbt-labs/issue/SEM-342/project-key-in-semantic-manifest

[marcodamore]

Also, here's the schema from DSI for the project config: https://github.com/dbt-labs/dbt-semantic-interfaces/pull/95/files#diff-5c4c38db4204c3cf53b401e057f98e3686d3aa2d13ff5c391bd5036097115f21R258

[gshank]

From the standup meeting what I gathered was that the first step was for users to create this model themselves, from documentation. Is that correct? Is there documentation of what that would look like?

[Jstein77]

@gshank That is correct! Here is the documentation https://github.com/dbt-labs/docs.getdbt.com/pull/3392/files#diff-526134beefd89cf44f090a6108f73da6cc639c7e0e6a5d35fa639a55057f9c7c

[gshank]

Do we want the whole model stored in the SemanticManifest? Where? It's neither a semantic_model nor a metric... Or do we just want the relation name in config?

[gshank]

I have code in the attached pull request to check for the "metricflow_time_spine" model and put the information in the semantic manifest config. But that doesn't get the table created. Do we want to tell people to always run that model first? Add it as a "depends_on.nodes" to all of the semantic models?

We need some way to get the table actually created...