Skip to content

scheduled-job: keep the Slack channel token out of state via ephemeral + write-only - #37

Merged
jwbron merged 2 commits into
mainfrom
jwies/scheduled-job-channel-ids
Jul 21, 2026
Merged

scheduled-job: keep the Slack channel token out of state via ephemeral + write-only#37
jwbron merged 2 commits into
mainfrom
jwies/scheduled-job-channel-ids

Conversation

@jwbron

@jwbron jwbron commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

The scheduled-job module reads Slack__API_token_for_alertlib with a data "google_secret_manager_secret_version" to build each job's Slack notification channel. Terraform state stores the full data-source response, including secret_data, so the token value sits in every consumer's state and every saved plan file; this is how the token was exposed by the committed-tfplan incident (see Khan/actions#274 and the consumer migrations).

Change

The channel stays fully Terraform-managed; the leak is fixed with the ephemeral/write-only mechanism (https://developer.hashicorp.com/terraform/language/manage-sensitive-data/ephemeral):

  • the token is read with ephemeral "google_secret_manager_secret_version" (never persisted to state or plans), and
  • written to the channel via the write-only sensitive_labels.auth_token_wo + auth_token_wo_version arguments, added to google_monitoring_notification_channel in provider 7.19.0.

Rotation becomes explicit: after adding a new secret version, bump the new slack_token_rotation variable; the next apply re-reads the latest version and rewrites the channel token. (The old data-source design also only picked up new versions on the next plan, so nothing is lost; the bump makes the cutover reviewable.)

An earlier revision of this PR added a notification_channel_ids input for pre-created channels; that is dropped in favor of keeping channel management in Terraform.

Requirements and rollout

  • Terraform >= 1.11 (write-only arguments) and hashicorp/google >= 7.19.0; the module's required_providers now enforces both.
  • Suggest releasing as scheduled-job-v0.4.0.
  • Consumer follow-ups (culture-cron, beep-boop, internal-webserver; five jobs total, all currently embedding the token in state): bump the module pin, and bump the google provider pin from 7.8.0 to >= 7.19.0 in the two stacks that pin exactly (no committed lock files, so that is the whole change). The adoption plan should show the channel's sensitive labels updating in place; after apply plus a state refresh, the token value is gone from live state.
  • Rotating the alertlib token (rotation runbook item Improve scheduled function module: replace versioning with hash-based naming and add retry configuration #4) is still required regardless, since historical plan files already leaked it; after consumers adopt this version, the runbook's Improve scheduled function module: replace versioning with hash-based naming and add retry configuration #4 mechanics change from "drift plan" to "bump slack_token_rotation in each repo".

jwbron added 2 commits July 20, 2026 21:25
…otification channels

Reading Slack__API_token_for_alertlib with a data source persists the
token value into Terraform state (state stores the full data-source
response) and from there into every saved plan file; this is how the
token was exposed by the committed-tfplan incident.

New notification_channel_ids input: pass full resource names of
pre-created Cloud Monitoring channels and the alert policies use them
directly; the data source and module-managed channel are skipped
entirely, so Terraform never touches the token. The token-based path
remains as a deprecated default for backward compatibility and should
be removed in the next major version.

Suggest releasing as scheduled-job-v0.4.0.
…raform-managed via ephemeral + write-only

Reworks the previous commit per review: instead of offloading channel
creation to pre-created channels, keep the Slack notification channel
fully module-managed and fix the leak with the mechanism built for it.
The token is read with an ephemeral resource and written via the
write-only sensitive_labels.auth_token_wo argument (google provider
7.19.0+), so the value never touches Terraform state or plan files.
Rotation is an explicit slack_token_rotation counter bump.

Requires Terraform >= 1.11 and hashicorp/google >= 7.19.0; consumers
currently pin 7.8.0 and bump the provider when they adopt v0.4.0.
@jwbron jwbron changed the title scheduled-job: accept pre-created notification channels scheduled-job: keep the Slack channel token out of state via ephemeral + write-only Jul 21, 2026
@nathanjd

Copy link
Copy Markdown
Member

An earlier revision of this PR added a notification_channel_ids input for pre-created channels; that is dropped in favor of keeping channel management in Terraform.

This seems like useless information.

@nathanjd nathanjd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simple enough. Thanks!

@jwbron
jwbron merged commit d9ef7c5 into main Jul 21, 2026
2 checks passed
@jwbron
jwbron deleted the jwies/scheduled-job-channel-ids branch July 21, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants