-
Notifications
You must be signed in to change notification settings - Fork 944
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
combine and simplify sq and exports #5470
Merged
Merged
Changes from 12 commits
Commits
Show all changes
43 commits
Select commit
Hold shift + click to select a range
5f3014c
consolidate
mirnawong1 6a42672
updates
mirnawong1 e0d3c2a
remove buildling
mirnawong1 1dba52a
simplify
mirnawong1 4aa4afe
swap
mirnawong1 1de4604
clarify
mirnawong1 9763506
add to
mirnawong1 6266752
Merge branch 'current' into combine-sq-exports
mirnawong1 0cc0685
Update website/docs/docs/build/saved-queries.md
mirnawong1 f953a34
Merge branch 'current' into combine-sq-exports
mirnawong1 506f80a
Update website/docs/docs/use-dbt-semantic-layer/exports.md
mirnawong1 300576c
rephrase
mirnawong1 5528a84
add configs
mirnawong1 3f6fec7
hide
mirnawong1 108408f
updates
mirnawong1 fcfb88f
Merge branch 'current' into combine-sq-exports
mirnawong1 aed4d2a
update to 1.7
mirnawong1 63c538b
Update website/docs/docs/build/saved-queries.md
mirnawong1 eccb743
Merge branch 'current' into combine-sq-exports
mirnawong1 cfa1083
Merge branch 'current' into combine-sq-exports
mirnawong1 44bd166
Update website/docs/docs/build/saved-queries.md
mirnawong1 5ed3e7b
Update exports.md
mirnawong1 996ed55
Update saved-queries.md
mirnawong1 b358577
Update website/docs/docs/build/saved-queries.md
mirnawong1 3b813a5
Update website/docs/docs/build/saved-queries.md
mirnawong1 42babec
Update exports.md
mirnawong1 2e1d4b4
Update saved-queries.md
mirnawong1 3e9bade
Update exports.md
mirnawong1 01531ae
fix example
mirnawong1 79a943b
add links
mirnawong1 caa07d3
updates
mirnawong1 6abc906
Update saved-queries.md
mirnawong1 ab1429f
Update saved-queries.md
mirnawong1 32d2a31
Merge branch 'current' into combine-sq-exports
mirnawong1 dcb5b80
Update saved-queries.md
mirnawong1 c93364d
quigley's updates
mirnawong1 cf37743
Merge branch 'current' into combine-sq-exports
mirnawong1 61ed76b
Merge branch 'current' into combine-sq-exports
mirnawong1 b9b7661
Merge branch 'current' into combine-sq-exports
mirnawong1 4013e16
Merge branch 'current' into combine-sq-exports
mirnawong1 e11f062
Merge branch 'current' into combine-sq-exports
mirnawong1 cec421c
Update website/docs/docs/use-dbt-semantic-layer/exports.md
mirnawong1 8fe8e4b
Update exports.md
mirnawong1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,19 +6,66 @@ sidebar_label: "Saved queries" | |
tags: [Metrics, Semantic Layer] | ||
--- | ||
|
||
Saved queries are a way to save commonly used queries in MetricFlow. You can group metrics, dimensions, and filters that are logically related into a saved query. | ||
Saved queries are a way to save commonly used queries in MetricFlow. You can group metrics, dimensions, and filters that are logically related into a saved query. Saved queries is a node and visible in the dbt <Term id="dag" />. | ||
|
||
### Exports and saved queries comparison | ||
Saved queries serve as the foundational building block, allowing you to [configure exports](#configure-exports) in your saved query configuration. Exports takes this functionality a step further by enabling you to [schedule and write saved queries](/docs/use-dbt-semantic-layer/exports) directly within your data platform using [dbt Cloud's job scheduler](/docs/deploy/job-scheduler). | ||
|
||
Saved queries are distinct from [exports](/docs/use-dbt-semantic-layer/exports), which schedule and execute saved queries using [dbt Cloud's job scheduler](/docs/deploy/job-scheduler). The following table compares the features and usage of exports and saved queries: | ||
## Parameters | ||
|
||
To create a saved query, refer to the following table parameters. | ||
|
||
:::tip | ||
Note that we use the double colon (::) to indicate whether a parameter is nested within another parameter. So for example, `query_params::metrics` means the `metrics` parameter is nested under `query_params`. | ||
::: | ||
<!-- For versions 1.8 and higher --> | ||
<VersionBlock firstVersion="1.8"> | ||
|
||
| Feature | Exports | <div style={{width:'250px, text-align: center'}}>Saved queries</div> | | ||
| ----------- | ----------- | ---------------- | | ||
| **Availability** | Available on dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) plans on dbt versions 1.7 or newer.| Available in both dbt Core and dbt Cloud. | | ||
| **Purpose** | To write saved queries in your data platform and expose metrics and dimensions as a view or table. | To define and manage common Semantic Layer queries in YAML, which includes metrics and dimensions. | | ||
| **Usage** | Automatically runs saved queries and writes them within your data platform. Exports count towards [queried metrics](/docs/cloud/billing#what-counts-as-a-queried-metric) usage. <br /><br />**Example**: Creating a weekly aggregated table for active user metrics, automatically updated and stored in the data platform. | Used for organizing and reusing common MetricFlow queries within dbt projects.<br /><br /><br />**Example**: Group related metrics together for better organization, and include commonly used dimensions and filters. | For materializing query results in the data platform. | | ||
| **Integration** | Must have the dbt Semantic Layer configured in your dbt project.<br /><br />Tightly integrated with the [MetricFlow Server](/docs/use-dbt-semantic-layer/sl-architecture#components) and dbt Cloud's job scheduler. | Integrated into the dbt <Term id="dag" /> and managed alongside other dbt nodes. | | ||
| **Configuration** | Defined within the `saved_queries` configuration. Set up within the dbt Cloud environment and job scheduler settings. | Defined in YAML format within dbt project files. | | ||
| Parameter | Type | Required | Description | | ||
|-------|---------|----------|----------------| | ||
| `name` | String | Required | Name of the saved query object. | | ||
| `description` | String | Required | A description of the saved query. | | ||
| `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. | | ||
| `config` | String | Required | A config section for any parameters specifying the saved query. | | ||
| `config::cache` | String | Optional | A boolean to specify if a saved query should be used to populate the cache. Accepts `true` or `false`. Defaults to `false` | | ||
| `query_params` | Structure | Required | Contains the query parameters. | | ||
| `query_params::metrics` | List or String | Optional | A list of the metrics to be used in the query as specified in the command line interface. | | ||
| `query_params::group_by` | List or String | Optional | A list of the Entities and Dimensions to be used in the query, which include the `Dimension` or `TimeDimension`. | | ||
| `query_params::where` | List or String | Optional | A list of strings that may include the `Dimension` or `TimeDimension` objects. | | ||
| `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. | | ||
| `exports::name` | String | Required | Name of the export object. | | ||
| `exports::config` | List or Structure | Required | A config section for any parameters specifying the export. | | ||
| `exports::config::export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. | | ||
| `exports::config::schema` | String | Optional | The schema for creating the table or view. This option cannot be used for caching. | | ||
| `exports::config::alias` | String | Optional | The table alias used to write to the table or view. This option cannot be used for caching. | | ||
|
||
</VersionBlock> | ||
|
||
<!-- For versions 1.7 and lower--> | ||
<VersionBlock lastVersion="1.7"> | ||
|
||
| Parameter | Type | Required | Description | | ||
|-------|---------|----------|----------------| | ||
| `name` | String | Required | Name of the saved query object. | | ||
| `description` | String | Required | A description of the saved query. | | ||
| `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. | | ||
| `query_params` | Structure | Required | Contains the query parameters. | | ||
| `query_params::metrics` | List or String | Optional | Metrics nested with the `query_params`: a list of the metrics to be used in the query as specified in the command line interface. | | ||
| `query_params::group_by` | List or String | Optional | Grouping nested with the `query_params`: a list of the Entities and Dimensions to be used in the query, which include the `Dimension` or `TimeDimension`. | | ||
| `query_params::where` | List or String | Optional | Conditions nested with the `query_params`: a list of strings that may include the `Dimension` or `TimeDimension` objects. | | ||
| `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. | | ||
| `exports::name` | String | Required | Name of export object, nested within `exports`. | | ||
| `exports::config` | List or Structure | Required | A config section for any parameters specifying the export, nested within `exports`. | | ||
| `exports::config::export_as` | String | Required | Specifies the type of export: table, view, or upcoming cache options. Nested within `exports` and `config`. | | ||
| `exports::config::schema` | String | Optional | Schema for creating the table or view, not applicable for caching. Nested within `exports` and `config`. | | ||
| `exports::config::alias` | String | Optional | Table alias used to write to the table or view. This option can't be used for caching. Nested within `exports` and `config`. | | ||
|
||
</VersionBlock> | ||
|
||
If you use multiple metrics in a saved query, then you will only be able to reference the common dimensions these metrics share in the `group_by` or `where` clauses. Use the entity name prefix with the Dimension object, like `Dimension('user__ds')`. | ||
|
||
mirnawong1 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
## Configure saved query | ||
|
||
Use saved queries to define and manage common Semantic Layer queries in YAML, including metrics and dimensions. Saved queries enable you to organize and reuse common MetricFlow queries within dbt projects. For example, you can group related metrics together for better organization, and include commonly used dimensions and filters. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Include dbt_project.yml config (saved-queries) blurb |
||
|
||
All metrics in a saved query need to use the same dimensions in the `group_by` or `where` clauses. The following is an example of a saved query: | ||
|
||
|
@@ -36,19 +83,19 @@ saved_queries: | |
cache: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add file name semantic model |
||
enabled: true # Or false if you want it disabled by default | ||
query_params: | ||
metrics: | ||
- simple_metric | ||
group_by: | ||
- "Dimension('user__ds')" | ||
where: | ||
- "{{ Dimension('user__ds', 'DAY') }} <= now()" | ||
- "{{ Dimension('user__ds', 'DAY') }} >= '2023-01-01'" | ||
metrics: | ||
- simple_metric | ||
group_by: | ||
- "Dimension('user__ds')" | ||
where: | ||
- "{{ Dimension('user__ds', 'DAY') }} <= now()" | ||
- "{{ Dimension('user__ds', 'DAY') }} >= '2023-01-01'" | ||
exports: | ||
- name: my_export | ||
config: | ||
alias: my_export_alias | ||
export_as: table | ||
schema: my_export_schema_name | ||
- name: my_export | ||
config: | ||
alias: my_export_alias | ||
export_as: table | ||
schema: my_export_schema_name | ||
``` | ||
</File> | ||
|
||
|
@@ -65,73 +112,65 @@ saved_queries: | |
description: "{{ doc('saved_query_description') }}" | ||
label: Test saved query | ||
query_params: | ||
metrics: | ||
- simple_metric | ||
group_by: | ||
- "Dimension('user__ds')" | ||
where: | ||
- "{{ Dimension('user__ds', 'DAY') }} <= now()" | ||
- "{{ Dimension('user__ds', 'DAY') }} >= '2023-01-01'" | ||
metrics: | ||
- simple_metric | ||
group_by: | ||
- "Dimension('user__ds')" | ||
where: | ||
- "{{ Dimension('user__ds', 'DAY') }} <= now()" | ||
- "{{ Dimension('user__ds', 'DAY') }} >= '2023-01-01'" | ||
exports: | ||
- name: my_export | ||
config: | ||
alias: my_export_alias | ||
export_as: table | ||
schema: my_export_schema_name | ||
- name: my_export | ||
config: | ||
alias: my_export_alias | ||
export_as: table | ||
schema: my_export_schema_name | ||
``` | ||
</File> | ||
</VersionBlock> | ||
|
||
## Parameters | ||
## Configure exports | ||
|
||
To define a saved query, refer to the following parameters: | ||
Exports are an additional configuration added to a saved query. They define _how_ to write a saved query, along with the schema and table name. | ||
|
||
<!-- For versions 1.8 and higher --> | ||
<VersionBlock firstVersion="1.8"> | ||
Once you've configured your saved query and set the foundation block, you can now configure exports in the `saved_queries` YAML configuration file (the same file as your metric definitions). This will also allow you to [run exports](#run-exports) automatically within your data platform using [dbt Cloud's job scheduler](/docs/deploy/job-scheduler). | ||
|
||
| Parameter | Type | Required | Description | | ||
|-------|---------|----------|----------------| | ||
| `name` | String | Required | Name of the saved query object. | | ||
| `description` | String | Required | A description of the saved query. | | ||
| `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. | | ||
| `config` | String | Required | A config section for any parameters specifying the saved query. | | ||
| `config::cache` | String | Optional | A boolean to specify if a saved query should be used to populate the cache. Accepts `true` or `false`. Defaults to `false` | | ||
| `query_params` | Structure | Required | Contains the query parameters. | | ||
| `query_params::metrics` | List or String | Optional | A list of the metrics to be used in the query as specified in the command line interface. | | ||
| `query_params::group_by` | List or String | Optional | A list of the Entities and Dimensions to be used in the query, which include the `Dimension` or `TimeDimension`. | | ||
| `query_params::where` | List or String | Optional | A list of strings that may include the `Dimension` or `TimeDimension` objects. | | ||
| `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. | | ||
| `exports::name` | String | Required | Name of the export object. | | ||
| `exports::config` | List or Structure | Required | A config section for any parameters specifying the export. | | ||
| `exports::config::export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. | | ||
| `exports::config::schema` | String | Optional | The schema for creating the table or view. This option cannot be used for caching. | | ||
| `exports::config::alias` | String | Optional | The table alias used to write to the table or view. This option cannot be used for caching. | | ||
The following is an example of a saved query with an export: | ||
|
||
</VersionBlock> | ||
<File name='semantic_model.yml'> | ||
|
||
<!-- For versions 1.7 and lower--> | ||
<VersionBlock lastVersion="1.7"> | ||
```yaml | ||
saved_queries: | ||
- name: order_metrics | ||
description: Relevant order metrics | ||
query_params: | ||
metrics: | ||
- orders | ||
- large_order | ||
- food_orders | ||
- order_total | ||
group_by: | ||
- Entity('order_id') | ||
- TimeDimension('metric_time', 'day') | ||
- Dimension('customer__customer_name') | ||
- ... # Additional group_by | ||
where: | ||
- "{{TimeDimension('metric_time')}} > current_timestamp - interval '1 week'" | ||
- ... # Additional where clauses | ||
exports: | ||
- name: order_metrics | ||
config: | ||
export_as: table # Options available: table, view | ||
schema: YOUR_SCHEMA # Optional - defaults to deployment schema | ||
alias: SOME_TABLE_NAME # Optional - defaults to Export name | ||
``` | ||
</File> | ||
|
||
| Parameter | Type | Required | Description | | ||
|-------|---------|----------|----------------| | ||
| `name` | String | Required | Name of the saved query object. | | ||
| `description` | String | Required | A description of the saved query. | | ||
| `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. | | ||
| `query_params` | Structure | Required | Contains the query parameters. | | ||
| `query_params::metrics` | List or String | Optional | A list of the metrics to be used in the query as specified in the command line interface. | | ||
| `query_params::group_by` | List or String | Optional | A list of the Entities and Dimensions to be used in the query, which include the `Dimension` or `TimeDimension`. | | ||
| `query_params::where` | List or String | Optional | A list of strings that may include the `Dimension` or `TimeDimension` objects. | | ||
| `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. | | ||
| `exports::name` | String | Required | Name of the export object. | | ||
| `exports::config` | List or Structure | Required | A config section for any parameters specifying the export. | | ||
| `exports::config::export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. | | ||
| `exports::config::schema` | String | Optional | The schema for creating the table or view. This option cannot be used for caching. | | ||
| `exports::config::alias` | String | Optional | The table alias used to write to the table or view. This option cannot be used for caching. | | ||
## Run exports | ||
|
||
</VersionBlock> | ||
Once you've configured exports, you can now take things a step further by running exports to automatically write saved queries within your data platform using [dbt Cloud's job scheduler](/docs/deploy/job-scheduler). This feature is only available with the [dbt Cloud's Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl). | ||
|
||
All metrics in a saved query need to use the same dimensions in the `group_by` or `where` clauses. | ||
When using the `Dimension` object, prepend the semantic model name, for example `Dimension('user__ds')` | ||
For more information on how to run exports, refer to the [Exports](/docs/use-dbt-semantic-layer/exports) documentation. | ||
|
||
## Related docs | ||
|
||
|
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this really required or optional? m
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Jordan confirmed it's optional