Add CI/CD for staging deployment

[rudransh-shrivastava]

Proposed change

Resolves #2694

Required GitHub environment secrets:

• AWS_ACCESS_KEY_ID
• AWS_ACCOUNT_ID
• AWS_SECRET_ACCESS_KEY
• TF_STATE_BUCKET_NAME
• TF_STATE_DYNAMODB_TABLE_NAME
• ZAPPA_LAMBDA_FUNCTION_NAME

Required GitHub environment variables:

• AWS_AVAILABILITY_ZONES (e.g. ["ap-south-1a", "ap-south-1b", "ap-south-1c"])
• AWS_REGION (e.g. ap-south-1)
• DOMAIN_NAME (e.g. nest.rudransh.app)

Checklist

• Required: I followed the contributing workflow
• Required: I verified that my code works as intended and resolves the issue as described
• Required: I ran make check-test locally: all warnings addressed, tests passed
• I used AI for code, documentation, tests, or communication related to this PR

[coderabbitai]

Summary by CodeRabbit

• New Features

  • Terraform-driven staging infrastructure with improved deployment planning and visibility in CI/CD.
  • Enhanced ECS task orchestration including automated migrations and data indexing during deployments.
  • AWS ECR integration for containerized application deployments.

• Chores

  • Updated infrastructure configuration parameter paths and settings.
  • Replaced legacy deployment automation with modern Terraform-driven workflows.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Walkthrough

Replaces SSH/Ansible staging deployment with Terraform-driven CI/CD, adds AWS/ECR login and ECR tagging in workflows, surfaces Terraform plans/artifacts, orchestrates ECS task/migrations in staging, and renames project/SSM/S3 identifiers from "owasp-nest" to "nest".

Changes

Cohort / File(s)	Summary
CI/CD workflow .github/workflows/run-ci-cd.yaml	Adds AWS creds & ECR login, ECR image tagging, splits plan vs deploy jobs, installs Terraform, uploads/shows tfplan, runs Terraform init/plan/apply for staging, orchestrates ECS migrate/index-data tasks, increases timeouts.
Removed Ansible / Compose staging deploys .github/ansible/staging/nest.yaml, .github/ansible/staging/proxy.yaml, docker-compose/staging/compose.yaml	Deletes staging Ansible playbooks and staging docker-compose (SSH/Ansible-based deployment removed).
Backend build & Zappa backend/Makefile, backend/zappa_callback.py, backend/zappa_settings.template.json	Makefile: conditional SHELL via EXEC_MODE; zappa_callback: guard against missing alias (catch ResourceNotFoundException); zappa settings: SSM path/ARNs -> "nest", add /generated_videos/ exclusion, disable snap_start, change VPC SG variable.
Terraform — parameters & variables infrastructure/modules/parameters/main.tf, infrastructure/modules/parameters/variables.tf	Replace hardcoded SSM parameter values with variables; add server_csrf_url and server_graphql_url; change settings_module default to settings.staging.
Terraform — staging module & outputs infrastructure/staging/main.tf, infrastructure/staging/outputs.tf, infrastructure/staging/terraform.tfbackend.example, infrastructure/staging/terraform.tfvars.example	Add local bucket names and pass CSRF/GraphQL args to parameters module; rename/replace Zappa S3 outputs; add ECS/frontend outputs (cluster/service, ecs_security_group_id); update DynamoDB lock table and example project_name to "nest".
Terraform — modules: ecs & storage infrastructure/modules/ecs/outputs.tf, infrastructure/modules/storage/outputs.tf	Add ecr_repository_url and ecs_cluster_name outputs; rename zappa_s3_bucket → zappa_s3_bucket_name and change value to bucket id.
Infrastructure variable defaults infrastructure/staging/variables.tf, infrastructure/backend/variables.tf	Set fixtures_bucket_name and zappa_bucket_name defaults to null; change backend project_name default from "owasp-nest" to "nest".
Docs & tooling infrastructure/README.md, cspell/custom-dict.txt	README: reference zappa_settings.template.json; cspell: add envsubst to dictionary.
Inventory .github/ansible/inventory.yaml	Remove staging host entries previously used by deleted Ansible plays.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• Add Remote State Management #2699 — overlapping changes to Terraform remote backend/state and CI Terraform plan/apply flow.
• Add Frontend to Infrastructure #2832 — related modifications to frontend/ECS/ALB Terraform wiring and staging infrastructure.
• Restore unmerged changes from #2685 #2717 — touches ECS security-group wiring and outputs that overlap with ECS/SG changes here.

Suggested reviewers

• kasya
• arkid15r

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'Add CI/CD for staging deployment' accurately captures the main purpose of the changeset, which introduces CI/CD pipeline automation for staging environment deployment.
Description check	✅ Passed	The description is related to the changeset, referencing issue #2694 and specifying required AWS credentials and GitHub environment variables needed for the CI/CD pipeline implementation.
Linked Issues check	✅ Passed	The PR addresses issue #2694 by implementing a complete CI/CD pipeline for infrastructure deployment with Terraform automation, AWS ECR integration, and ECS orchestration for staging.
Out of Scope Changes check	✅ Passed	All changes are directly aligned with CI/CD infrastructure deployment automation: workflow updates, Terraform configurations, AWS parameter management, and removal of legacy SSH/Ansible deployment scripts.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

.github/workflows/run-ci-cd.yaml (1)

586-656: Guard against empty DOMAIN_NAME values in terraform.tfvars.

When DOMAIN_NAME is unset, the current code writes domain_name="", which Terraform treats as a non-null empty string. This causes var.domain_name != null checks to evaluate to true, enabling HTTPS with an empty domain and producing invalid URLs like https:// with no host. Emit domain_name=null instead when unset.

🐛 Suggested guard

       - name: Prepare terraform variables
         env:
           AVAILABILITY_ZONES: ${{ vars.AWS_AVAILABILITY_ZONES }}
           AWS_REGION: ${{ vars.AWS_REGION }}
           CREATE_RDS_PROXY: false
           DOMAIN_NAME: ${{ vars.DOMAIN_NAME }}
@@
         run: |
           umask 377
+          if [ -z "$DOMAIN_NAME" ]; then
+            DOMAIN_NAME_TF=null
+          else
+            DOMAIN_NAME_TF="\"$DOMAIN_NAME\""
+          fi
           cat > infrastructure/staging/terraform.tfvars <<-EOF
@@
-          domain_name="$DOMAIN_NAME"
+          domain_name=$DOMAIN_NAME_TF

🧹 Nitpick comments (2)

.github/workflows/run-ci-cd.yaml (2)

450-549: Prefer OIDC-based AWS auth over long‑lived access keys.

Static AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY increases credential exposure. Consider switching to OIDC with role-to-assume. Also verify the ECR repo URL secrets exist for staging so the new tags resolve. The timeout bump looks reasonable for multi-registry pushes.

♻️ Example OIDC switch

-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
-        with:
-          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
-          aws-region: ${{ vars.AWS_REGION }}
-          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+      - name: Configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
+        with:
+          aws-region: ${{ vars.AWS_REGION }}
+          role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
+          role-session-name: github-actions

---

687-851: Avoid drift by sourcing ECS network inputs from Terraform outputs.

The migrate/index tasks rely on subnet and SG IDs from static secrets. With Terraform now managing infra, those IDs can drift after terraform apply, causing task failures or cross‑VPC runs. Consider pulling these values from terraform output (or SSM) in the job; if index-data success is required, optionally wait for completion and check the exit code.

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

.github/workflows/run-ci-cd.yaml (1)

586-685: Validate AVAILABILITY_ZONES is a valid JSON array before writing tfvars.

If AWS_AVAILABILITY_ZONES is set in GitHub variables as a comma-separated string (e.g., us-east-2a,us-east-2b) instead of a JSON array, the line availability_zones=$AVAILABILITY_ZONES will write invalid HCL syntax and break terraform validate and terraform plan. Add validation to catch this early.

✅ Add a quick format check

       - name: Prepare terraform variables
         env:
           AVAILABILITY_ZONES: ${{ vars.AWS_AVAILABILITY_ZONES }}
@@
         run: |
           umask 377
+          echo "$AVAILABILITY_ZONES" | jq -e 'type=="array"' >/dev/null || {
+            echo "::error::AWS_AVAILABILITY_ZONES must be a JSON array, e.g. [\"us-east-2a\",\"us-east-2b\"]"
+            exit 1
+          }
           cat > infrastructure/staging/terraform.tfvars <<-EOF
           availability_zones=$AVAILABILITY_ZONES

🤖 Fix all issues with AI agents

In @.github/workflows/run-ci-cd.yaml:
- Around line 687-862: The Run ECS migrate task step can yield an empty tasks
array so capture the full run-task response and guard against missing task ARN:
after calling aws ecs run-task in the step with id migrate-task, check the
TASK_ARN (the variable written to GITHUB_OUTPUT and referenced as
steps.migrate-task.outputs.task_arn) and if it's empty or equals "None" fail
fast with a clear ::error:: message (include the run-task response or reason)
and exit 1 so the workflow doesn’t proceed to aws ecs wait/describe with an
invalid task ARN.

🧹 Nitpick comments (1)

.github/workflows/run-ci-cd.yaml (1)

450-459: Consider using GitHub OIDC instead of long‑lived AWS keys for ECR pushes.

Using role-to-assume with GitHub's OIDC token reduces secret exposure and eliminates rotation burden. This requires:

• Adding id-token: write to job permissions
• Creating an IAM role with an OIDC trust policy scoped to your repository
• Replacing aws-access-key-id and aws-secret-access-key with role-to-assume: ${{ secrets.AWS_ROLE_ARN }}

This pattern appears in multiple jobs throughout the workflow. The 10-minute timeout is appropriate for the current dual-registry push setup.

♻️ Example adjustment (OIDC)

 permissions:
   contents: read
+  id-token: write

 - name: Configure AWS credentials
   uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
   with:
-    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
-    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
     aws-region: ${{ vars.AWS_REGION }}
+    role-to-assume: ${{ secrets.AWS_ROLE_ARN }}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[rudransh-shrivastava]

currently has no effect.