terraform: store binary plans in GCS instead of committing them

.changeset/terraform-plan-gcs-transport.md

@@ -0,0 +1,16 @@
+---
+"generate-terraform-plan": major
+"apply-terraform-plan": major
+---
+
+Stop committing the Terraform binary plan file to git. A binary plan embeds a full copy of the Terraform state, including sensitive values in cleartext, and the tip-delete cleanup never removed it from git history.
+
+generate-terraform-plan now uploads the binary plan to a GCS object keyed by a slug of the terraform path plus the plan's commit SHA, and the plan PR commits only the redacted plan text plus a small pointer file (default `tfplan.commit`) recording that SHA. apply-terraform-plan reads the pointer, rebuilds the object path from its own inputs, downloads the exact plan that was reviewed, applies it, and best-effort deletes the object during cleanup.
+
+Breaking changes: both actions gain a `plan_bucket` input (required for the plan-PR/apply flow; not needed for PR-comment-only or auto_approve usage) and must be upgraded together. Consumers need a GCS bucket that the plan workflow's service account can write objects to and the apply workflow's service account can read (and ideally delete) objects from, plus a lifecycle rule to expire plans that are never applied (e.g. superseded plan PRs).
+
+Consumer workflows must also update their trigger paths: apply workflows typically trigger on `push` with a `paths` filter on `tfplan.binary`, which is never committed anymore. Point that filter (and any `!tfplan.binary` excludes in plan workflows) at the pointer file (default `tfplan.commit`) instead, or apply will never run.
+
+Security model for the bucket: a binary plan contains the same sensitive data as the Terraform state itself, so treat the plan bucket exactly like the state bucket. Create it in the same project, enable uniform bucket-level access and public access prevention, and grant object access only to the CI service accounts (no human-facing grants). Note that project-level IAM (owners/editors, storage admins) inherits access to any bucket in the project; that audience can already read the state bucket, so the plan bucket exposes nothing new to it.
+
+Note: this only stops new leaks. Any `tfplan.binary` still committed at a repo's tip should be deleted, and histories that ever contained committed binary plans still need scrubbing and rotation of any secrets present in the embedded state.

actions/apply-terraform-plan/action.yml

@@ -1,7 +1,9 @@
-# This workflow applies checked-in Terraform plans created by the `terraform-plan` workflow
+# This workflow applies Terraform plans created by the `terraform-plan` workflow
 # and can be called from any repository. It handles:
-# - Checking for plan files (unless auto_approve is enabled)
+# - Checking for the committed plan pointer/text files (unless auto_approve is enabled)
 # - Authenticating with GCP
+# - Downloading the binary plan from GCS (binary plans embed a full copy of the
+#   Terraform state, including sensitive values, so they are never committed)
 # - Applying the Terraform plan (or running a blind apply with -auto-approve)
 # - Cleaning up plan files
 # - Commenting on PRs for success/failure
@@ -20,6 +22,14 @@ inputs:
     description: "Path to the plan text file (relative to terraform_path)"
     required: false
     default: "tfplan.txt"
+  plan_pointer_path:
+    description: "Path to the committed pointer file recording the commit SHA whose uploaded binary plan should be applied (relative to terraform_path)"
+    required: false
+    default: "tfplan.commit"
+  plan_bucket:
+    description: "GCS bucket, optionally with a path prefix, that the generate-terraform-plan action uploaded the binary plan to. Must match that action's plan_bucket input. Required unless auto_approve is 'true'. The service account needs permission to read (and, for cleanup, delete) objects in this bucket. A binary plan contains the same sensitive data as the Terraform state, so restrict this bucket exactly like the state bucket: CI service accounts only, no human-facing grants."
+    required: false
+    default: ""
   gcloud_project:
     description: "Google Cloud project ID"
     required: true
@@ -68,7 +78,7 @@ runs:
         if [ "${{ inputs.auto_approve }}" == "true" ]; then
           echo "Auto-approve mode enabled, skipping plan file check"
           echo "skip_apply=false" >> "$GITHUB_OUTPUT"
-        elif [ ! -f "${{ inputs.terraform_path }}/${{ inputs.plan_file_path }}" ] || [ ! -f "${{ inputs.terraform_path }}/${{ inputs.plan_text_path }}" ]; then
+        elif [ ! -f "${{ inputs.terraform_path }}/${{ inputs.plan_pointer_path }}" ] || [ ! -f "${{ inputs.terraform_path }}/${{ inputs.plan_text_path }}" ]; then
           echo "Plan files not found, skipping apply steps"
           echo "skip_apply=true" >> "$GITHUB_OUTPUT"
         else
@@ -87,6 +97,34 @@ runs:
         workload_identity_provider: ${{ inputs.workload_identity_provider }}
         service_account: ${{ inputs.service_account }}
 
+    # generate-terraform-plan uploads the binary plan to GCS (keyed by a slug
+    # of the terraform path plus the commit SHA the plan was generated from)
+    # and commits only a pointer file, because the binary embeds a full copy
+    # of the Terraform state, including sensitive values in cleartext.
+    # Rebuilding the object path here from our own plan_bucket input, rather
+    # than trusting a URI committed to the repo, means a tampered pointer can
+    # at worst select another plan previously uploaded for this same stack.
+    - name: Download binary plan from GCS
+      if: steps.check_files.outputs.skip_apply != 'true' && inputs.auto_approve != 'true'
+      id: download
+      shell: bash
+      working-directory: ${{ inputs.terraform_path }}
+      run: |
+        if [ -z "${{ inputs.plan_bucket }}" ]; then
+          echo "The plan_bucket input is required to download the binary plan uploaded by generate-terraform-plan." >&2
+          exit 1
+        fi
+        plan_sha=$(tr -d '[:space:]' < "${{ inputs.plan_pointer_path }}")
+        if ! printf '%s' "$plan_sha" | grep -Eq '^[0-9a-f]{40}$'; then
+          echo "Pointer file ${{ inputs.plan_pointer_path }} does not contain a valid commit SHA: '$plan_sha'" >&2
+          exit 1
+        fi
+        # Slug the terraform path the same way generate-terraform-plan does.
+        slug=$(printf '%s' "${{ inputs.terraform_path }}" | tr -c 'A-Za-z0-9._-' '-' | sed -E 's/-+/-/g; s/^-//; s/-$//')
+        object="gs://${{ inputs.plan_bucket }}/${slug}/${plan_sha}/${{ inputs.plan_file_path }}"
+        gcloud storage cp "$object" "${{ inputs.plan_file_path }}"
+        echo "object=$object" >> "$GITHUB_OUTPUT"
+
     - name: Terraform Init
       if: steps.check_files.outputs.skip_apply != 'true'
       shell: bash
@@ -106,9 +144,10 @@ runs:
         fi
       continue-on-error: true
 
-    # Once the plan is applied, we remove the checked-in plan files to clean up.
-    # This prevents stale plans from being applied and ensures that each
-    # apply job has a fresh plan generated for it.
+    # Once the plan is applied, we remove the committed pointer and plan text
+    # files to clean up, and delete the uploaded binary plan from GCS. This
+    # prevents stale plans from being applied and ensures that each apply job
+    # has a fresh plan generated for it.
     #
     # The base branch (typically `master`/`main`) is protected and rejects
     # direct pushes, so rather than committing and pushing the cleanup directly
@@ -129,7 +168,17 @@ runs:
       run: |
         # create-pull-request stages these deletions itself via add-paths; a
         # plain rm is more robust than `git rm -f` when a file is already gone.
-        rm -f "${{ inputs.terraform_path }}/${{ inputs.plan_file_path }}" "${{ inputs.terraform_path }}/${{ inputs.plan_text_path }}"
+        # The downloaded binary is untracked but removed too, so nothing later
+        # in the job can pick up an already-applied plan.
+        rm -f "${{ inputs.terraform_path }}/${{ inputs.plan_pointer_path }}" "${{ inputs.terraform_path }}/${{ inputs.plan_text_path }}" "${{ inputs.terraform_path }}/${{ inputs.plan_file_path }}"
+
+        # Best-effort delete of the uploaded binary plan; a failure (e.g. a
+        # service account without delete permission) shouldn't fail the run
+        # since the apply already succeeded. A bucket lifecycle rule should
+        # expire anything this misses, including plans that are never applied.
+        if ! gcloud storage rm "${{ steps.download.outputs.object }}"; then
+          echo "Warning: failed to delete ${{ steps.download.outputs.object }}; relying on the bucket lifecycle rule." >&2
+        fi
 
         # Scope the cleanup branch to this terraform stack. If two apply jobs
         # for different stacks run concurrently in the same repo, a shared
@@ -154,12 +203,12 @@ runs:
         body: |
           This PR was automatically generated by the `${{ github.workflow }}` workflow after the Terraform plan was applied.
 
-          It removes the checked-in plan files so that a stale plan can't be re-applied and the next run generates a fresh plan. It is merged automatically to complete the apply cycle.
+          It removes the committed plan pointer and plan text files so that a stale plan can't be re-applied and the next run generates a fresh plan. It is merged automatically to complete the apply cycle.
 
-          Plan file: `${{ inputs.terraform_path }}/${{ inputs.plan_file_path }}`
+          Plan pointer file: `${{ inputs.terraform_path }}/${{ inputs.plan_pointer_path }}`
           Plaintext plan file: `${{ inputs.terraform_path }}/${{ inputs.plan_text_path }}`
         add-paths: |
-          ${{ inputs.terraform_path }}/${{ inputs.plan_file_path }}
+          ${{ inputs.terraform_path }}/${{ inputs.plan_pointer_path }}
           ${{ inputs.terraform_path }}/${{ inputs.plan_text_path }}
 
     - name: Merge plan-file cleanup PR

actions/generate-terraform-plan/action.yml

@@ -3,7 +3,10 @@
 # - Checking for plan files
 # - Authenticating with GCP
 # - Generating a Terraform plan
-# - Handling PR creation/comments
+# - Uploading the binary plan to GCS (binary plans embed a full copy of the
+#   Terraform state, including sensitive values, so they are never committed)
+# - Handling PR creation/comments (the PR contains only the redacted plan
+#   text and a pointer file recording which uploaded plan to apply)
 # - Commenting on PRs for success/failure
 # All environment variables starting with TF_VAR_ will be automatically picked up by Terraform
 # Example: TF_VAR_project_id, TF_VAR_region, etc.
@@ -23,6 +26,14 @@ inputs:
     description: "Filename for the text plan file"
     required: false
     default: "tfplan.txt"
+  plan_file_pointer:
+    description: "Filename for the committed pointer file that records the commit SHA whose uploaded binary plan should be applied"
+    required: false
+    default: "tfplan.commit"
+  plan_bucket:
+    description: "GCS bucket, optionally with a path prefix (e.g. 'my-bucket/terraform-plans'), where the binary plan is uploaded. Required when enable_pr_creation is 'true'. Must match the apply-terraform-plan action's plan_bucket input. The service account needs permission to create objects in this bucket. A binary plan contains the same sensitive data as the Terraform state, so restrict this bucket exactly like the state bucket: CI service accounts only, no human-facing grants."
+    required: false
+    default: ""
   workload_identity_provider:
     description: "Workload Identity Provider for Google Cloud authentication"
     required: true
@@ -132,6 +143,42 @@ runs:
           echo "has_changes=true" >> "$GITHUB_OUTPUT"
         fi
 
+    # A binary plan file embeds a full copy of the Terraform state, including
+    # sensitive values in cleartext, regardless of what the plan changes.
+    # Committing it would put that state into git history permanently, so we
+    # upload it to an access-controlled GCS bucket instead and commit only a
+    # small pointer file recording the commit SHA the plan was generated from.
+    # The apply-terraform-plan action rebuilds the object path from its own
+    # plan_bucket input plus that SHA and downloads the exact plan that was
+    # reviewed. Plans that are never applied (e.g. superseded plan PRs) leave
+    # objects behind; configure a lifecycle rule on the bucket to expire them.
+    #
+    # The gcloud CLI is preinstalled on GitHub-hosted runners and picks up the
+    # credentials exported by the auth step above.
+    - name: Upload binary plan to GCS
+      if: inputs.enable_pr_creation == 'true' && github.ref_name == inputs.pr_creation_branch && steps.plan.outputs.has_changes == 'true'
+      id: upload
+      shell: bash
+      working-directory: ${{ inputs.terraform_path }}
+      run: |
+        if [ -z "${{ inputs.plan_bucket }}" ]; then
+          echo "The plan_bucket input is required when enable_pr_creation is 'true': the binary plan is uploaded to GCS instead of being committed." >&2
+          exit 1
+        fi
+        # Slug the terraform path the same way apply-terraform-plan does so
+        # concurrent stacks in one repo can't collide on an object path.
+        slug=$(printf '%s' "${{ inputs.terraform_path }}" | tr -c 'A-Za-z0-9._-' '-' | sed -E 's/-+/-/g; s/^-//; s/-$//')
+        object="gs://${{ inputs.plan_bucket }}/${slug}/${{ github.sha }}/${{ inputs.plan_file_binary }}"
+        gcloud storage cp "${{ inputs.plan_file_binary }}" "$object"
+        rm -f "${{ inputs.plan_file_binary }}"
+        # The pointer deliberately holds only the commit SHA, not a full
+        # gs:// URI: apply-terraform-plan reconstructs the object path from
+        # its own workflow-configured inputs, so a tampered pointer can at
+        # worst select another plan previously uploaded for this same stack.
+        printf '%s\n' "${{ github.sha }}" > "${{ inputs.plan_file_pointer }}"
+        echo "object=$object" >> "$GITHUB_OUTPUT"
+        echo "Uploaded binary plan to $object"
+
     - name: Get Last Deployed Commit
       if: steps.plan.outputs.has_changes == 'true' && github.ref_name == inputs.pr_creation_branch && inputs.enable_pr_creation == 'true'
       id: last_deployed
@@ -198,8 +245,10 @@ runs:
 
           Note, this plan reflects the current state of the `${{ inputs.pr_creation_branch }}` branch and may include changes from multiple contributors.
 
-          Plan file: `${{ inputs.terraform_path }}/${{ inputs.plan_file_binary }}`
           Plaintext plan file: `${{ inputs.terraform_path }}/${{ inputs.plan_file_text }}`
+          Binary plan: `${{ steps.upload.outputs.object }}`
+
+          The binary plan is stored in GCS rather than committed because it embeds a full copy of the Terraform state, including sensitive values. The committed pointer file `${{ inputs.terraform_path }}/${{ inputs.plan_file_pointer }}` records which uploaded plan the apply workflow will download.
 
           ## Changes Since Last Deployment
           You can view all changes that have not yet been deployed [here](https://github.com/${{ github.repository }}/compare/${{ steps.last_deployed.outputs.last_deployed }}...${{ github.sha }}).
@@ -217,7 +266,7 @@ runs:
 
           ${{ inputs.pr_extra_text }}
         add-paths: |
-          ${{ inputs.terraform_path }}/${{ inputs.plan_file_binary }}
+          ${{ inputs.terraform_path }}/${{ inputs.plan_file_pointer }}
           ${{ inputs.terraform_path }}/${{ inputs.plan_file_text }}
         base: ${{ inputs.pr_creation_branch }}
         reviewers: ${{ steps.reviewer.outputs.reviewer }}