[DOCS-1166][DOCS-880][DOCS-1167][DOCS-1168] Update automation docs #1084
+629
−720
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Deploying docs with
|
| Latest commit: |
8f2a6f6
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://effe2b53.docodile.pages.dev |
| Branch Preview URL: | https://automation-updates.docodile.pages.dev |
ssisk
reviewed
Feb 7, 2025
mdlinville
commented
Feb 11, 2025
mdlinville
commented
Feb 11, 2025
mdlinville
commented
Feb 11, 2025
Co-authored-by: Stephen Sisk <[email protected]>
mdlinville
commented
Feb 11, 2025
noaleetz
reviewed
Mar 25, 2025
| @@ -17,4 +17,5 @@ W&B Core provides capabilities across the entire ML lifecycle. With W&B Core, yo | |||
| - Explore and evaluate data and metrics using [interactive, configurable visualizations]({{< relref "/guides/models/tables/" >}}). | |||
| - [Document and share]({{< relref "./reports/" >}}) insights across the entire organization by generating live reports in digestible, visual formats that are easily understood by non-technical stakeholders. | |||
| - [Query and create visualizations of your data]({{< relref "/guides/models/app/features/panels/query-panels/" >}}) that serve your custom needs. | |||
| - Configure custom automations that trigger key workflows for [model CI/CD]({{< relref "/guides/core/automations/model-registry-automations.md" >}}). | |||
| - [Protect sensitive strings using secrets]({{< relref "/guides/core/secrets.md" >}}). | |||
| - Configure custom automations that trigger key workflows for [model CI/CD]({{< relref "/guides/core/automations/" >}}). | |||
There was a problem hiding this comment.
Suggested change
| - Configure custom automations that trigger key workflows for [model CI/CD]({{< relref "/guides/core/automations/" >}}). | |
| - Configure automations that trigger key workflows for [model CI/CD]({{< relref "/guides/core/automations/" >}}). |
noaleetz
reviewed
Mar 25, 2025
| @@ -86,6 +85,6 @@ You can pass a custom path into the `root` [parameter]({{< relref "/ref/python/a | |||
|
|
|||
| ## Next steps | |||
| * Learn how to [version]({{< relref "./create-a-new-artifact-version.md" >}}) and [update]({{< relref "./update-an-artifact.md" >}}) artifacts. | |||
| * Learn how to trigger downstream workflows in response to changes to your artifacts with [artifact automation]({{< relref "/guides/core/automations/project-scoped-automations/" >}}). | |||
| * Learn how to trigger downstream workflows or notify a Slack channel in response to changes to your artifacts or registered models with [automations]({{< relref "/guides/core/automations/" >}}). | |||
There was a problem hiding this comment.
Suggested change
| * Learn how to trigger downstream workflows or notify a Slack channel in response to changes to your artifacts or registered models with [automations]({{< relref "/guides/core/automations/" >}}). | |
| * Learn how to trigger downstream workflows or notify a Slack channel in response to changes to your artifacts with [automations]({{< relref "/guides/core/automations/" >}}). |
noaleetz
reviewed
Mar 25, 2025
| @@ -7,3 +7,34 @@ title: Automations | |||
| weight: 4 | |||
| --- | |||
|
|
|||
| This page describes _automations_ in W&B. [Create an automation]({{< relref "create-automations/" >}}) to trigger workflow steps, such as automated model testing and deployment, based on an event in W&B, such as when an [artifact]({{< relref "/guides/core/artifacts" >}}) is created or changed or a [registered model is changed]({{< relref "/guides/core/registry/" >}}). | |||
There was a problem hiding this comment.
Suggested change
| This page describes _automations_ in W&B. [Create an automation]({{< relref "create-automations/" >}}) to trigger workflow steps, such as automated model testing and deployment, based on an event in W&B, such as when an [artifact]({{< relref "/guides/core/artifacts" >}}) is created or changed or a [registered model is changed]({{< relref "/guides/core/registry/" >}}). | |
| This page describes _automations_ in W&B. [Create an automation]({{< relref "create-automations/" >}}) to trigger workflow steps, such as automated model testing and deployment, based on an event in W&B, such as when an [artifact]({{< relref "/guides/core/artifacts" >}}) artifact version is created ({{< relref "/guides/core/registry/" >}}). |
noaleetz
reviewed
Mar 25, 2025
| @@ -7,3 +7,34 @@ title: Automations | |||
| weight: 4 | |||
| --- | |||
|
|
|||
| This page describes _automations_ in W&B. [Create an automation]({{< relref "create-automations/" >}}) to trigger workflow steps, such as automated model testing and deployment, based on an event in W&B, such as when an [artifact]({{< relref "/guides/core/artifacts" >}}) is created or changed or a [registered model is changed]({{< relref "/guides/core/registry/" >}}). | |||
|
|
|||
| For example, an automation can post to a Slack channel when a new artifact is created, or run a webhook to trigger automated testing when a model has the `test` label applied. | |||
There was a problem hiding this comment.
Suggested change
| For example, an automation can post to a Slack channel when a new artifact is created, or run a webhook to trigger automated testing when a model has the `test` label applied. | |
| For example, an automation can post to a Slack channel when a new version is created, or send a webhook to trigger automated testing when the `production` alias is added to an artifact. |
noaleetz
reviewed
Mar 25, 2025
| An automation can run when a specific [event]({{< relref "automation-events.md" >}}) occurs in a registry or project. | ||
|
|
||
| For an artifact in [Registry]({{< relref "/guides/core/registry/">}}), you can configure an automation to run: | ||
| - When a new artifact is linked to a collection. For example, trigger testing and validation workflows for new models. |
There was a problem hiding this comment.
Suggested change
| - When a new artifact is linked to a collection. For example, trigger testing and validation workflows for new models. | |
| - When a new artifact version is linked to a collection. For example, trigger testing and validation workflows for new candidate models. |
noaleetz
reviewed
Mar 25, 2025
|
|
||
| For an artifact in [Registry]({{< relref "/guides/core/registry/">}}), you can configure an automation to run: | ||
| - When a new artifact is linked to a collection. For example, trigger testing and validation workflows for new models. | ||
| - When an alias is added to a new version of an artifact. For example, trigger a deployment workflow when the `deploy` alias is added to a model version. |
There was a problem hiding this comment.
Suggested change
| - When an alias is added to a new version of an artifact. For example, trigger a deployment workflow when the `deploy` alias is added to a model version. | |
| - When an alias is added to an artifact version. For example, trigger a deployment workflow when the `deploy` alias is added to a model version. |
noaleetz
reviewed
Mar 25, 2025
| - When an alias is added to a new version of an artifact. For example, trigger a deployment workflow when the `deploy` alias is added to a model version. | ||
|
|
||
| For an artifact in a project, you can configure an automation to run: | ||
| - When a new version of an artifact is created. For example, start a training job when a new version of a dataset artifact is created. |
There was a problem hiding this comment.
Suggested change
| - When a new version of an artifact is created. For example, start a training job when a new version of a dataset artifact is created. | |
| - When a new version is added to an artifact. For example, start a training job when a new version of a dataset artifact is added to a given collection. |
noaleetz
reviewed
Mar 25, 2025
|
|
||
| For an artifact in a project, you can configure an automation to run: | ||
| - When a new version of an artifact is created. For example, start a training job when a new version of a dataset artifact is created. | ||
| - When a new artifact is linked to a project or collection. For example, trigger testing and validation workflows for new models. |
There was a problem hiding this comment.
Suggested change
| - When a new artifact is linked to a project or collection. For example, trigger testing and validation workflows for new models. |
this is the same event as above
noaleetz
reviewed
Mar 25, 2025
| For an artifact in a project, you can configure an automation to run: | ||
| - When a new version of an artifact is created. For example, start a training job when a new version of a dataset artifact is created. | ||
| - When a new artifact is linked to a project or collection. For example, trigger testing and validation workflows for new models. | ||
| - When an alias is added to a new version of an artifact. For example, trigger a deployment workflow when the `deploy` alias is added to a model version. |
There was a problem hiding this comment.
Suggested change
| - When an alias is added to a new version of an artifact. For example, trigger a deployment workflow when the `deploy` alias is added to a model version. | |
| - When an alias is added to an artifact version. For example, trigger a PII redaction workflow when the alias "redaction" is added to a dataset artifact. |
noaleetz
reviewed
Mar 25, 2025
| To [create an automation]({{< relref "create-automations/" >}}), you: | ||
|
|
||
| 1. If required, configure [secrets]({{< relref "/guides/core/secrets.md" >}}) for sensitive strings the automation requires, such as access tokens, passwords, or sensitive configuration details. | ||
| 1. Authorize W&B to post to Slack or run the webhook on your behalf. This is required once per Slack channel or webhook. |
There was a problem hiding this comment.
Suggested change
| 1. Authorize W&B to post to Slack or run the webhook on your behalf. This is required once per Slack channel or webhook. | |
| 1. Authorize W&B to post to Slack or run the webhook on your behalf. This is required once per Slack channel or webhook. |
can you explain more what this step means? Is this authorization the user has to do in W&B or in the webhook provider?
noaleetz
reviewed
Mar 25, 2025
| 1. If required, configure [secrets]({{< relref "/guides/core/secrets.md" >}}) for sensitive strings the automation requires, such as access tokens, passwords, or sensitive configuration details. | ||
| 1. Authorize W&B to post to Slack or run the webhook on your behalf. This is required once per Slack channel or webhook. | ||
| 1. In the project or registry, create the automation: | ||
| 1. Grant it access to any secrets you created for it. |
There was a problem hiding this comment.
Suggested change
| 1. Grant it access to any secrets you created for it. | |
| 1. Grant it access to any secrets you created for it. |
Also not following what this step is - what does it mean to grant access to a secret?
noaleetz
reviewed
Mar 25, 2025
| 1. Authorize W&B to post to Slack or run the webhook on your behalf. This is required once per Slack channel or webhook. | ||
| 1. In the project or registry, create the automation: | ||
| 1. Grant it access to any secrets you created for it. | ||
| 1. Define the [event]({{< relref "#automation-events" >}}) to watch for, such as when a new artifact or version is created or linked. |
There was a problem hiding this comment.
Suggested change
| 1. Define the [event]({{< relref "#automation-events" >}}) to watch for, such as when a new artifact or version is created or linked. | |
| 1. Define the [event]({{< relref "#automation-events" >}}) to watch for, such as when a new artifact version is added. |
noaleetz
reviewed
Mar 25, 2025
|
|
||
| ### Scopes | ||
| You create a Registry automation in these scopes: | ||
| - A [Registry]({{< relref "/guides/core/registry/">}}) and all of its collections (the default). |
There was a problem hiding this comment.
Suggested change
| - A [Registry]({{< relref "/guides/core/registry/">}}) and all of its collections (the default). | |
| - A [Registry]({{< relref "/guides/core/registry/">}}) and all of its collections (the default). This automation listens for the event taking place on any collection in the given registry. |
What does an automation being applied to a registry mean? An automation applied to a registry == an automation applied to all of the collections. Here you say "A Registry and all of its collection" which is a bit confusing. Adding an automation to a registry IS basically bulk adding it to all collections in there (and all future collections added).
noaleetz
reviewed
Mar 25, 2025
| ### Events | ||
| These events can trigger a Registry automation: | ||
| - **Linking a new artifact to a collection**: Test and validate new models or datasets when they are registered. | ||
| - **Adding a new alias to a version of an artifact**: Trigger a special step of your workflow when a new artifact version has a specific alias applied. For example, deploy a model when it has the `deploy` alias applied. |
There was a problem hiding this comment.
Suggested change
| - **Adding a new alias to a version of an artifact**: Trigger a special step of your workflow when a new artifact version has a specific alias applied. For example, deploy a model when it has the `deploy` alias applied. | |
| - **Adding a new alias to a version of an artifact**: Trigger a special step of your workflow when a new artifact version has a specific alias applied. For example, deploy a model when it has the `production` alias applied. |
aliases typically are a model lifecycle state, and the automation an alias might trigger is the one who's action will move the model into this state.
noaleetz
reviewed
Mar 25, 2025
| - **Linking a new artifact to a collection**: Test and validate new models or datasets when they are registered. | ||
| - **Adding a new alias to a version of an artifact**: Trigger a special step of your workflow when a new artifact version has a specific alias applied. For example, deploy a model when it has the `deploy` alias applied. | ||
|
|
||
| ## Project |
There was a problem hiding this comment.
can you add a gif or screenshot pointing the user to where they can find this scope?
noaleetz
reviewed
Mar 25, 2025
| --- | ||
| An automation can run when a specific event occurs at a given scope, either a registry or a project. This page lists the events that can trigger an automation at each scope. | ||
|
|
||
| ## Registry |
There was a problem hiding this comment.
can you add a gif or screenshot pointing the user to where they can find this scope?
noaleetz
reviewed
Mar 25, 2025
| ### Events | ||
| These events can trigger a project automation: | ||
|
|
||
| - **Linking a new artifact**: Test and validate new models, datasets or dataset automatically. |
There was a problem hiding this comment.
Suggested change
| - **Linking a new artifact**: Test and validate new models, datasets or dataset automatically. | |
| - **Adding a new artifact version to a project collection**: Apply recurring actions to each version of an artifact. For example, start a training job when a new dataset artifact version is created. |
There is no linking happening in the project. Someone links an artifact in project into the registry, but that is for a registry-scoped automation.
noaleetz
reviewed
Mar 25, 2025
| These events can trigger a project automation: | ||
|
|
||
| - **Linking a new artifact**: Test and validate new models, datasets or dataset automatically. | ||
| - **Creating a new version of an artifact**: Apply recurring actions to each version of an artifact. For example, start a training job when a new dataset artifact version is created. |
There was a problem hiding this comment.
Suggested change
| - **Creating a new version of an artifact**: Apply recurring actions to each version of an artifact. For example, start a training job when a new dataset artifact version is created. |
same as above
noaleetz
reviewed
Mar 25, 2025
|
|
||
| - **Linking a new artifact**: Test and validate new models, datasets or dataset automatically. | ||
| - **Creating a new version of an artifact**: Apply recurring actions to each version of an artifact. For example, start a training job when a new dataset artifact version is created. | ||
| - **Adding a new alias to a version of an artifact**: Trigger a special step of your workflow when a new artifact version in a project or collection has a specific label or alias applied. For example, run a series of downstream processing steps when an artifact has the `test-set-quality-check` alias applied. |
There was a problem hiding this comment.
Suggested change
| - **Adding a new alias to a version of an artifact**: Trigger a special step of your workflow when a new artifact version in a project or collection has a specific label or alias applied. For example, run a series of downstream processing steps when an artifact has the `test-set-quality-check` alias applied. | |
| - **Adding a new alias to a version of an artifact**: Trigger a special step of your workflow when a new artifact version in a project or collection has a specific alias applied. For example, run a series of downstream processing steps when an artifact has the `test-set-quality-check` alias applied. |
new version and alias are 2 separate events. The artifact doesn't need to be new when adding the alias to it / trigger the action.
We don't have the concept of labels.
noaleetz
reviewed
Mar 26, 2025
| - To create a Slack automation, you must have permission to post to the Slack instance and channel you select. | ||
|
|
||
| ## Create an automation | ||
| Create an automation from the project or registry's **Automations** tab. At a high level, to create an automation, follow these steps: |
There was a problem hiding this comment.
can we add the relevant hyperlinks to project / registry to point the user to resources on scope?
noaleetz
reviewed
Mar 26, 2025
| ## Create an automation | ||
| Create an automation from the project or registry's **Automations** tab. At a high level, to create an automation, follow these steps: | ||
|
|
||
| 1. If necessary, [create a W&B secret]({{< relref "/guides/core/secrets.md" >}}) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your team settings. Secrets are most commonly used in webhook automation. |
There was a problem hiding this comment.
Suggested change
| 1. If necessary, [create a W&B secret]({{< relref "/guides/core/secrets.md" >}}) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your team settings. Secrets are most commonly used in webhook automation. | |
| 1. If necessary, [create a W&B secret]({{< relref "/guides/core/secrets.md" >}}) for each sensitive string required by the automation, such as an access or bearer token, or API key. Secrets are most commonly used in webhook automation, where they allow you to securely pass credentials or tokens needed to trigger external services without exposing them in plain text. |
There was a problem hiding this comment.
you grant it access to any secrets it requires for authentication
What does mean? How does one grant authentication?
noaleetz
reviewed
Mar 26, 2025
| ## Create an automation | ||
| Create an automation from the project or registry's **Automations** tab. At a high level, to create an automation, follow these steps: | ||
|
|
||
| 1. If necessary, [create a W&B secret]({{< relref "/guides/core/secrets.md" >}}) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your team settings. Secrets are most commonly used in webhook automation. |
There was a problem hiding this comment.
Suggested change
| 1. If necessary, [create a W&B secret]({{< relref "/guides/core/secrets.md" >}}) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your team settings. Secrets are most commonly used in webhook automation. | |
| 1. If necessary, [create a W&B secret]({{< relref "/guides/core/secrets.md" >}}) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your **team settings**. Secrets are most commonly used in webhook automation. |
noaleetz
reviewed
Mar 26, 2025
| Create an automation from the project or registry's **Automations** tab. At a high level, to create an automation, follow these steps: | ||
|
|
||
| 1. If necessary, [create a W&B secret]({{< relref "/guides/core/secrets.md" >}}) for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your team settings. Secrets are most commonly used in webhook automation. | ||
| 1. Configure the webhook or Slack integration to authorize W&B to post to Slack or run the webhook on your behalf. A single integration can be used by multiple automations. When you create the integration, you grant it access to any secrets it requires for authentication. Automation integrations are defined in your team settings. |
There was a problem hiding this comment.
Suggested change
| 1. Configure the webhook or Slack integration to authorize W&B to post to Slack or run the webhook on your behalf. A single integration can be used by multiple automations. When you create the integration, you grant it access to any secrets it requires for authentication. Automation integrations are defined in your team settings. | |
| 1. Configure the webhook or Slack integration to authorize W&B to post to Slack or run the webhook on your behalf. A single integration can be used by multiple automations. When you create the integration, you grant it access to any secrets it requires for authentication. Automation integrations are defined in your team settings. |
I believe the authorization is specific to slack - some webhook providers don't require any authorization outside of using a secret. I think parts of this section should be in the Slack specific instructions.
noaleetz
reviewed
Mar 26, 2025
| - To create a registry automation, you must have access to the registry. Refer to [Configure Registry access]({{< relref "/guides/core/registry/configure_registry.md#registry-roles" >}}). | ||
| - To create a Slack automation, you must have permission to post to the Slack instance and channel you select. | ||
|
|
||
| ## Create an automation |
There was a problem hiding this comment.
Can we structure this section a bit differently? It feels like we're trying cram some unrelated info into the sequence of steps, i.e authorization, what secrets are used for.
If you are a team admin - you can optionally go to team settings and create:
secret
webhook or slack notification
If you are not, the above doesn't apply to you and you:
- Go to the scope you want to create the automation in
- Configure the event
- Configure the action (using existing admin-created things such as secrets, webhooks)
For some of the other details, like reminding people to authorize W&B to send webhooks to a given provider, we should put that in the slack, webhook sections respectively as relevant.
noaleetz
reviewed
Mar 26, 2025
| @@ -72,7 +72,7 @@ Depending on your use case, explore the following resources to get started with | |||
| * Read the [models walkthrough]({{< relref "./walkthrough.md" >}}) for a step-by-step outline of the W&B Python SDK commands you could use to create, track, and use a dataset artifact. | |||
| * Learn about: | |||
| * [Protected models and access control]({{< relref "./access_controls.md" >}}). | |||
| * [How to connect the Model Registry to CI/CD processes]({{< relref "/guides/core/automations/model-registry-automations.md" >}}). | |||
| * [How to connect the Model Registry to CI/CD processes]({{< relref "/guides/core/automations/" >}}). | |||
There was a problem hiding this comment.
Suggested change
| * [How to connect the Model Registry to CI/CD processes]({{< relref "/guides/core/automations/" >}}). | |
| * [How to connect Registry to CI/CD processes]({{< relref "/guides/core/automations/" >}}). |
noaleetz
reviewed
Mar 26, 2025
| weight: 1 | ||
| --- | ||
|
|
||
| W&B Secret Manager allows you to securely and centrally store, manage, and inject _secrets_, which are sensitive strings such as passwords, API keys, or certificates. W&B Secret Manager removes the need to add sensitive strings directly to your code or to the configuration of an [automation]({{< relref "/guides/core/automations/" >}}). |
There was a problem hiding this comment.
Suggested change
| W&B Secret Manager allows you to securely and centrally store, manage, and inject _secrets_, which are sensitive strings such as passwords, API keys, or certificates. W&B Secret Manager removes the need to add sensitive strings directly to your code or to the configuration of an [automation]({{< relref "/guides/core/automations/" >}}). | |
| W&B Secret Manager allows you to securely and centrally store, manage, and inject _secrets_, which are sensitive strings such as access tokens, bearer tokens, API keys, or passwords. W&B Secret Manager removes the need to add sensitive strings directly to your code or when configuring a webhook's header or payload({{< relref "/guides/core/automations/" >}}). |
noaleetz
reviewed
Mar 26, 2025
| * Secrets are included as a core part of W&B, including in [W&B Server deployments]({{< relref "/guides/hosting/" >}}) that you host in Azure, GCP, or AWS. Connect with your W&B account team to discuss how you can use secrets in W&B if you use a different deployment type. | ||
| * In W&B Server, you are responsible for configuring security measures that satisfy your security needs. | ||
|
|
||
| - W&B strongly recommends that you store secrets in a W&B instance of a cloud secrets manager provided by AWS, GCP, or Azure, which are configured with advanced security capabilities. |
There was a problem hiding this comment.
Suggested change
| - W&B strongly recommends that you store secrets in a W&B instance of a cloud secrets manager provided by AWS, GCP, or Azure, which are configured with advanced security capabilities. | |
| - W&B strongly recommends that you store secrets in a W&B instance of a cloud provider's secrets manager provided by AWS, GCP, or Azure, which are configured with advanced security capabilities. |
noaleetz
reviewed
Mar 26, 2025
| ## Add a secret | ||
| To add a secret: | ||
|
|
||
| 1. If necessary, generate the sensitive string in the webhook's service. For example, generate an API key or set a password. If necessary, save the sensitive string securely, such as in a password manager. |
There was a problem hiding this comment.
Suggested change
| 1. If necessary, generate the sensitive string in the webhook's service. For example, generate an API key or set a password. If necessary, save the sensitive string securely, such as in a password manager. | |
| 1. If the receiving service requires it to authenticate incoming webhooks, generate the required token or API key. If necessary, save the sensitive string securely, such as in a password manager. |
reworded a bit as "webhook's service" is a bit confusing.
noaleetz
requested changes
Mar 26, 2025
There was a problem hiding this comment.
See comments in-line.
I think it might be helpful to add some kind of diagram showing the event, action pair options. I attached one Raven originally created but we would want to update it to remove launch jobs and add notification.
It would also be great if we added some more gifs, especially to the multi-step workflows that require you to navigate across parts of the app, like creating a webhook in team settings.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update all possible links and references to old Model Registry
Consolidate project-scoped and model registry automations, new Slack notification flow, ability to edit automations
Out of scope:
In technical review
Preview links for major changes: