Skip to content

Commit

Permalink
Staging environment (#5267)
Browse files Browse the repository at this point in the history
## What are you changing in this pull request and why?

Beta documentation for staging

## Checklist
<!--
Uncomment when publishing docs for a prerelease version of dbt:
- [ ] Add versioning components, as described in [Versioning
Docs](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-entire-pages)
- [ ] Add a note to the prerelease version [Migration
Guide](https://github.com/dbt-labs/docs.getdbt.com/tree/current/website/docs/docs/dbt-versions/core-upgrade)
-->
- [ ] Review the [Content style
guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md)
so my content adheres to these guidelines.
- [ ] For [docs
versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning),
review how to [version a whole
page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version)
and [version a block of
content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content).
- [ ] Add a checklist item for anything that needs to happen before this
PR is merged, such as "needs technical review" or "change base branch."

Adding or removing pages (delete if not applicable):
- [ ] Add/remove page in `website/sidebars.js`
- [ ] Provide a unique filename for new pages
- [ ] Add an entry for deleted pages in `website/vercel.json`
- [ ] Run link testing locally with `npm run build` to update the links
that point to deleted pages
  • Loading branch information
matthewshaver authored Apr 12, 2024
2 parents d41f914 + 39a6163 commit 63f080f
Show file tree
Hide file tree
Showing 4 changed files with 59 additions and 11 deletions.
54 changes: 49 additions & 5 deletions website/docs/docs/deploy/deploy-environments.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,18 @@ A dbt Cloud project can have multiple deployment environments, providing you the
To learn different approaches to managing dbt Cloud environments and recommendations for your organization's unique needs, read [dbt Cloud environment best practices](/guides/set-up-ci).
:::

This page reviews the different types of environments and how to configure your deployment environment in dbt Cloud.
Learn more about development vs. deployment environments in [dbt Cloud Environments](/docs/dbt-cloud-environments).

import CloudEnvInfo from '/snippets/_cloud-environments-info.md';
There are three types of deployment environments that serve different needs:
- **Production**: Environment for transforming data and building pipelines for production use.
- **Staging**<Lifecycle status='beta' />: Environment for working with production tools while limiting access to production data.
- **General**: General use environment for deployment development.

<CloudEnvInfo setup={'/snippets/_cloud-environments-info.md'} />
We highly recommend using the `Production` environment type for the final, source of truth deployment data. There can be only one environment marked for final production workflows and we don't recommend using a `General` environment for this purpose.

## Create a deployment environment

To create a new dbt Cloud development environment, navigate to **Deploy** -> **Environments** and then click **Create Environment**. Select **Deployment** as the environment type.
To create a new dbt Cloud deployment environment, navigate to **Deploy** -> **Environments** and then click **Create Environment**. Select **Deployment** as the environment type. The option will be greyed out if you already have a development environment.

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/create-deploy-env.jpg" width="85%" title="Navigate to Deploy -> Environments to create a deployment environment" />

Expand All @@ -38,7 +41,8 @@ In dbt Cloud, each project can have one designated deployment environment, which

For Semantic Layer-eligible customers, the next section of environment settings is the Semantic Layer configurations. [The Semantic Layer setup guide](/docs/use-dbt-semantic-layer/setup-sl) has the most up-to-date setup instructions!

### Deployment connection

## Deployment connection

:::info Warehouse Connections

Expand Down Expand Up @@ -186,6 +190,46 @@ This section allows you to determine the credentials that should be used when co
</WHCode>


## Staging environment <Lifecycle status='beta' />

:::note
Currently in limited availability beta. Contact support or your account team if you're interested in beta access.
:::

Staging environments are useful ways to grant developers access to deployment workflows and tools while controlling access to production data. They are configured with their own long-living branch (for example, `staging`) that may be very similar to `main` in many ways while potentially limiting the data the developers can access.

Ideally, the workflows would move upstream from the Development environment -> Staging environment -> Production environment with developer branches feeding into the staging branch, then ultimately `main`. In many cases, the `main` and `staging` branches will be identical after a merge and remain until the next batch of changes from the `development` branches are ready to be elevated. We recommend setting branch protection rules on `staging` similar to `main`.

### Create a staging environment

In the dbt Cloud, navigate to **Deploy** -> **Environments** and then click **Create Environment**. Select **Deployment** as the environment type. The option will be greyed out if you already have a development environment.

<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/create-staging-environment.png" width="85%" title="Create a staging environment" />


Follow the steps outlined in [deployment credentials](#deployment-connection) to complete the remainder of the environment setup.

We recommend that the data warehouse credentials be for a dedicated user or service principal.

### Why use a staging environment

There are two primary motivations for using a Staging environment:
1. An additional validation layer before changes are deployed into Production. You can deploy, test, and explore your dbt models in Staging.
2. Clear isolation between development workflows and production data. It enables developers to work in metadata-powered ways, using features like deferral and cross-project references, without accessing data in production deployments.

:::info Coming soon: environment-level permissions
Provide developers with the ability to create, edit, and trigger ad hoc jobs in the Staging environment, while keeping the Production environment locked down.
:::

Let's say you have `Project B` downstream of `Project A` with cross-project refs configured in the models. When developers work in the IDE for `Project B`, cross-project refs will resolve to the Staging environment of `Project A`, rather than production. You'll get the same results with those refs when jobs are run in the Staging environment. Only the Production environment will reference the Production data, keeping the data and access isolated without needing separate projects.

If `Project B` also has a Staging deployment, then references to unbuilt upstream models within `Project B` will resolve to that environment, using [deferral](/docs/cloud/about-cloud-develop-defer), rather than resolving to the models in Production. This saves developers time and warehouse spend, while preserving clear separation of environments.

Finally, the Staging environment has its own view in [dbt Explorer](/docs/collaborate/explore-projects), giving you a full view of your prod and pre-prod data.

<Lightbox src="/img/docs/collaborate/dbt-explorer/explore-staging-env.png" width="85%" title="Explore in a staging environment" />


## Related docs

- [dbt Cloud environment best practices](/guides/set-up-ci)
Expand Down
16 changes: 10 additions & 6 deletions website/snippets/_cloud-environments-info.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,19 @@
## Types of environments

In dbt Cloud, there are two types of environments:
- Deployment environment &mdash; Determines the settings used when jobs created within that environment are executed.
- Development environment &mdash; Determines the settings used in the dbt Cloud IDE or dbt Cloud CLI, for that particular project.
- **Deployment environment** &mdash; Determines the settings used when jobs created within that environment are executed.<br></br>
Types of deployment environments:
- General
- Staging <Lifecycle status='beta' />
- Production
- **Development environment** &mdash; Determines the settings used in the dbt Cloud IDE or dbt Cloud CLI, for that particular project.

Each dbt Cloud project can only have a single development environment but can have any number of deployment environments.

| | Development Environments | Deployment Environments |
| --- | --- | --- |
| Determines settings for | dbt Cloud IDE or dbt Cloud CLI | dbt Cloud Job runs |
| How many can I have in my project? | 1 | Any number |
|| Development | Staging <b></b> <Lifecycle status='beta' /> | Deployment |
|------| --- | --- | --- |
| **Determines settings for** | dbt Cloud IDE or dbt Cloud CLI | dbt Cloud Job runs | dbt Cloud Job runs |
| **How many can I have in my project?** | 1 | Any number | Any number |

:::note
For users familiar with development on dbt Core, each environment is roughly analogous to an entry in your `profiles.yml` file, with some additional information about your repository to ensure the proper version of code is executed. More info on dbt core environments [here](/docs/core/dbt-core-environments).
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 63f080f

Please sign in to comment.