Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

combine and simplify sq and exports #5470

Merged
merged 43 commits into from
May 16, 2024
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
5f3014c
consolidate
mirnawong1 May 8, 2024
6a42672
updates
mirnawong1 May 8, 2024
e0d3c2a
remove buildling
mirnawong1 May 8, 2024
1dba52a
simplify
mirnawong1 May 8, 2024
4aa4afe
swap
mirnawong1 May 8, 2024
1de4604
clarify
mirnawong1 May 8, 2024
9763506
add to
mirnawong1 May 8, 2024
6266752
Merge branch 'current' into combine-sq-exports
mirnawong1 May 8, 2024
0cc0685
Update website/docs/docs/build/saved-queries.md
mirnawong1 May 8, 2024
f953a34
Merge branch 'current' into combine-sq-exports
mirnawong1 May 9, 2024
506f80a
Update website/docs/docs/use-dbt-semantic-layer/exports.md
mirnawong1 May 9, 2024
300576c
rephrase
mirnawong1 May 9, 2024
5528a84
add configs
mirnawong1 May 9, 2024
3f6fec7
hide
mirnawong1 May 9, 2024
108408f
updates
mirnawong1 May 9, 2024
fcfb88f
Merge branch 'current' into combine-sq-exports
mirnawong1 May 9, 2024
aed4d2a
update to 1.7
mirnawong1 May 9, 2024
63c538b
Update website/docs/docs/build/saved-queries.md
mirnawong1 May 9, 2024
eccb743
Merge branch 'current' into combine-sq-exports
mirnawong1 May 9, 2024
cfa1083
Merge branch 'current' into combine-sq-exports
mirnawong1 May 10, 2024
44bd166
Update website/docs/docs/build/saved-queries.md
mirnawong1 May 10, 2024
5ed3e7b
Update exports.md
mirnawong1 May 10, 2024
996ed55
Update saved-queries.md
mirnawong1 May 10, 2024
b358577
Update website/docs/docs/build/saved-queries.md
mirnawong1 May 10, 2024
3b813a5
Update website/docs/docs/build/saved-queries.md
mirnawong1 May 10, 2024
42babec
Update exports.md
mirnawong1 May 10, 2024
2e1d4b4
Update saved-queries.md
mirnawong1 May 10, 2024
3e9bade
Update exports.md
mirnawong1 May 10, 2024
01531ae
fix example
mirnawong1 May 13, 2024
79a943b
add links
mirnawong1 May 13, 2024
caa07d3
updates
mirnawong1 May 13, 2024
6abc906
Update saved-queries.md
mirnawong1 May 13, 2024
ab1429f
Update saved-queries.md
mirnawong1 May 13, 2024
32d2a31
Merge branch 'current' into combine-sq-exports
mirnawong1 May 13, 2024
dcb5b80
Update saved-queries.md
mirnawong1 May 14, 2024
c93364d
quigley's updates
mirnawong1 May 14, 2024
cf37743
Merge branch 'current' into combine-sq-exports
mirnawong1 May 14, 2024
61ed76b
Merge branch 'current' into combine-sq-exports
mirnawong1 May 14, 2024
b9b7661
Merge branch 'current' into combine-sq-exports
mirnawong1 May 14, 2024
4013e16
Merge branch 'current' into combine-sq-exports
mirnawong1 May 15, 2024
e11f062
Merge branch 'current' into combine-sq-exports
mirnawong1 May 16, 2024
cec421c
Update website/docs/docs/use-dbt-semantic-layer/exports.md
mirnawong1 May 16, 2024
8fe8e4b
Update exports.md
mirnawong1 May 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
192 changes: 116 additions & 76 deletions website/docs/docs/build/saved-queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,67 @@ sidebar_label: "Saved queries"
tags: [Metrics, Semantic Layer]
---

Saved queries are a way to save commonly used queries in MetricFlow. You can group metrics, dimensions, and filters that are logically related into a saved query.
Saved queries are a way to save commonly used queries in MetricFlow. You can group metrics, dimensions, and filters that are logically related into a saved query. Saved queries is a node and visible in the dbt <Term id="dag" />.

### Exports and saved queries comparison
Saved queries serve as the foundational building block, allowing you to [configure exports](#configure-exports) in your saved query configuration. Exports takes this functionality a step further by enabling you to [schedule and write saved queries](/docs/use-dbt-semantic-layer/exports) directly within your data platform using [dbt Cloud's job scheduler](/docs/deploy/job-scheduler).

Saved queries are distinct from [exports](/docs/use-dbt-semantic-layer/exports), which schedule and execute saved queries using [dbt Cloud's job scheduler](/docs/deploy/job-scheduler). The following table compares the features and usage of exports and saved queries:
## Parameters

To create a saved query, refer to the following table parameters.

:::tip
Note that we use the double colon (::) to indicate whether a parameter is nested within another parameter. So for example, `query_params::metrics` means the `metrics` parameter is nested under `query_params`.
:::
<!-- For versions 1.8 and higher -->
<VersionBlock firstVersion="1.8">

| Parameter | Type | Required | Description |
|-------|---------|----------|----------------|
| `name` | String | Required | Name of the saved query object. |
| `description` | String | Required | A description of the saved query. |
| `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. |
| `config` | String | Required | A config section for any parameters specifying the saved query. |
| `config::cache` | String | Optional | A boolean to specify if a saved query should be used to populate the cache. Accepts `true` or `false`. Defaults to `false` |
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this really required or optional? m

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jordan confirmed it's optional

| `query_params` | Structure | Required | Contains the query parameters. |
| `query_params::metrics` | List or String | Optional | A list of the metrics to be used in the query as specified in the command line interface. |
| `query_params::group_by` | List or String | Optional | A list of the Entities and Dimensions to be used in the query, which include the `Dimension` or `TimeDimension`. |
| `query_params::where` | List or String | Optional | A list of strings that may include the `Dimension` or `TimeDimension` objects. |
| `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. |
| `exports::name` | String | Required | Name of the export object. |
| `exports::config` | List or Structure | Required | A config section for any parameters specifying the export. |
| `exports::config::export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. |
| `exports::config::schema` | String | Optional | The schema for creating the table or view. This option cannot be used for caching. |
| `exports::config::alias` | String | Optional | The table alias used to write to the table or view. This option cannot be used for caching. |

</VersionBlock>

<!-- For versions 1.7 and lower-->
<VersionBlock lastVersion="1.7">

| Parameter | Type | Required | Description |
|-------|---------|----------|----------------|
| `name` | String | Required | Name of the saved query object. |
| `description` | String | Required | A description of the saved query. |
| `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. |
| `query_params` | Structure | Required | Contains the query parameters. |
| `query_params::metrics` | List or String | Optional | Metrics nested with the `query_params`: a list of the metrics to be used in the query as specified in the command line interface. |
| `query_params::group_by` | List or String | Optional | Grouping nested with the `query_params`: a list of the Entities and Dimensions to be used in the query, which include the `Dimension` or `TimeDimension`. |
| `query_params::where` | List or String | Optional | Conditions nested with the `query_params`: a list of strings that may include the `Dimension` or `TimeDimension` objects. |
| `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. |
| `exports::name` | String | Required | Name of export object, nested within `exports`. |
| `exports::config` | List or Structure | Required | A config section for any parameters specifying the export, nested within `exports`. |
| `exports::config::export_as` | String | Required | Specifies the type of export: table, view, or upcoming cache options. Nested within `exports` and `config`. |
| `exports::config::schema` | String | Optional | Schema for creating the table or view, not applicable for caching. Nested within `exports` and `config`. |
| `exports::config::alias` | String | Optional | Table alias used to write to the table or view. This option can't be used for caching. Nested within `exports` and `config`. |

| Feature | Exports | <div style={{width:'250px, text-align: center'}}>Saved queries</div> |
| ----------- | ----------- | ---------------- |
| **Availability** | Available on dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) plans on dbt versions 1.7 or newer.| Available in both dbt Core and dbt Cloud. |
| **Purpose** | To write saved queries in your data platform and expose metrics and dimensions as a view or table. | To define and manage common Semantic Layer queries in YAML, which includes metrics and dimensions. |
| **Usage** | Automatically runs saved queries and writes them within your data platform. Exports count towards [queried metrics](/docs/cloud/billing#what-counts-as-a-queried-metric) usage. <br /><br />**Example**: Creating a weekly aggregated table for active user metrics, automatically updated and stored in the data platform. | Used for organizing and reusing common MetricFlow queries within dbt projects.<br /><br /><br />**Example**: Group related metrics together for better organization, and include commonly used dimensions and filters. | For materializing query results in the data platform. |
| **Integration** | Must have the dbt Semantic Layer configured in your dbt project.<br /><br />Tightly integrated with the [MetricFlow Server](/docs/use-dbt-semantic-layer/sl-architecture#components) and dbt Cloud's job scheduler. | Integrated into the dbt <Term id="dag" /> and managed alongside other dbt nodes. |
| **Configuration** | Defined within the `saved_queries` configuration. Set up within the dbt Cloud environment and job scheduler settings. | Defined in YAML format within dbt project files. |
</VersionBlock>

All metrics in a saved query need to use the same dimensions in the `group_by` or `where` clauses.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
Use the semantic model name prefix with the Dimension object, like `Dimension('user__ds')`.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
## Configure saved query

Use saved queries to define and manage common Semantic Layer queries in YAML, including metrics and dimensions. Saved queries enable you to organize and reuse common MetricFlow queries within dbt projects. For example, you can group related metrics together for better organization, and include commonly used dimensions and filters.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include dbt_project.yml config (saved-queries) blurb


All metrics in a saved query need to use the same dimensions in the `group_by` or `where` clauses. The following is an example of a saved query:

Expand All @@ -36,19 +84,19 @@ saved_queries:
cache:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add file name semantic model

enabled: true # Or false if you want it disabled by default
query_params:
metrics:
- simple_metric
group_by:
- "Dimension('user__ds')"
where:
- "{{ Dimension('user__ds', 'DAY') }} <= now()"
- "{{ Dimension('user__ds', 'DAY') }} >= '2023-01-01'"
metrics:
- simple_metric
group_by:
- "Dimension('user__ds')"
where:
- "{{ Dimension('user__ds', 'DAY') }} <= now()"
- "{{ Dimension('user__ds', 'DAY') }} >= '2023-01-01'"
exports:
- name: my_export
config:
alias: my_export_alias
export_as: table
schema: my_export_schema_name
- name: my_export
config:
alias: my_export_alias
export_as: table
schema: my_export_schema_name
```
</File>

Expand All @@ -65,73 +113,65 @@ saved_queries:
description: "{{ doc('saved_query_description') }}"
label: Test saved query
query_params:
metrics:
- simple_metric
group_by:
- "Dimension('user__ds')"
where:
- "{{ Dimension('user__ds', 'DAY') }} <= now()"
- "{{ Dimension('user__ds', 'DAY') }} >= '2023-01-01'"
metrics:
- simple_metric
group_by:
- "Dimension('user__ds')"
where:
- "{{ Dimension('user__ds', 'DAY') }} <= now()"
- "{{ Dimension('user__ds', 'DAY') }} >= '2023-01-01'"
exports:
- name: my_export
config:
alias: my_export_alias
export_as: table
schema: my_export_schema_name
- name: my_export
config:
alias: my_export_alias
export_as: table
schema: my_export_schema_name
```
</File>
</VersionBlock>

## Parameters
## Configure exports

To define a saved query, refer to the following parameters:
Exports are an additional configuration added to a saved query. They define _how_ to write a saved query, along with the schema and table name.

<!-- For versions 1.8 and higher -->
<VersionBlock firstVersion="1.8">
Once you've configured your saved query and set the foundation block, you can now configure exports in the `saved_queries` YAML configuration file (the same file as your metric definitions). This will also allow you to [run exports](#run-exports) automatically within your data platform using [dbt Cloud's job scheduler](/docs/deploy/job-scheduler).

| Parameter | Type | Required | Description |
|-------|---------|----------|----------------|
| `name` | String | Required | Name of the saved query object. |
| `description` | String | Required | A description of the saved query. |
| `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. |
| `config` | String | Required | A config section for any parameters specifying the saved query. |
| `config::cache` | String | Optional | A boolean to specify if a saved query should be used to populate the cache. Accepts `true` or `false`. Defaults to `false` |
| `query_params` | Structure | Required | Contains the query parameters. |
| `query_params::metrics` | List or String | Optional | A list of the metrics to be used in the query as specified in the command line interface. |
| `query_params::group_by` | List or String | Optional | A list of the Entities and Dimensions to be used in the query, which include the `Dimension` or `TimeDimension`. |
| `query_params::where` | List or String | Optional | A list of strings that may include the `Dimension` or `TimeDimension` objects. |
| `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. |
| `exports::name` | String | Required | Name of the export object. |
| `exports::config` | List or Structure | Required | A config section for any parameters specifying the export. |
| `exports::config::export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. |
| `exports::config::schema` | String | Optional | The schema for creating the table or view. This option cannot be used for caching. |
| `exports::config::alias` | String | Optional | The table alias used to write to the table or view. This option cannot be used for caching. |
The following is an example of a saved query with an export:

</VersionBlock>
<File name='semantic_model.yml'>

<!-- For versions 1.7 and lower-->
<VersionBlock lastVersion="1.7">
```yaml
saved_queries:
- name: order_metrics
description: Relevant order metrics
query_params:
metrics:
- orders
- large_order
- food_orders
- order_total
group_by:
- Entity('order_id')
- TimeDimension('metric_time', 'day')
- Dimension('customer__customer_name')
- ... # Additional group_by
where:
- "{{TimeDimension('metric_time')}} > current_timestamp - interval '1 week'"
- ... # Additional where clauses
exports:
- name: order_metrics
config:
export_as: table # Options available: table, view
schema: YOUR_SCHEMA # Optional - defaults to deployment schema
alias: SOME_TABLE_NAME # Optional - defaults to Export name
```
</File>

| Parameter | Type | Required | Description |
|-------|---------|----------|----------------|
| `name` | String | Required | Name of the saved query object. |
| `description` | String | Required | A description of the saved query. |
| `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. |
| `query_params` | Structure | Required | Contains the query parameters. |
| `query_params::metrics` | List or String | Optional | A list of the metrics to be used in the query as specified in the command line interface. |
| `query_params::group_by` | List or String | Optional | A list of the Entities and Dimensions to be used in the query, which include the `Dimension` or `TimeDimension`. |
| `query_params::where` | List or String | Optional | A list of strings that may include the `Dimension` or `TimeDimension` objects. |
| `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. |
| `exports::name` | String | Required | Name of the export object. |
| `exports::config` | List or Structure | Required | A config section for any parameters specifying the export. |
| `exports::config::export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. |
| `exports::config::schema` | String | Optional | The schema for creating the table or view. This option cannot be used for caching. |
| `exports::config::alias` | String | Optional | The table alias used to write to the table or view. This option cannot be used for caching. |
## Run exports

</VersionBlock>
Once you've configured exports, you can now take things a step further by running exports to automatically write saved queries within your data platform using [dbt Cloud's job scheduler](/docs/deploy/job-scheduler). This feature is only available with the [dbt Cloud's Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl).

All metrics in a saved query need to use the same dimensions in the `group_by` or `where` clauses.
When using the `Dimension` object, prepend the semantic model name, for example `Dimension('user__ds')`
For more information on how to run exports, refer to the [Exports](/docs/use-dbt-semantic-layer/exports)documentation.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

## Related docs

Expand Down
Loading
Loading