Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: deploy redis and postgresql in localstack #620

Merged

Conversation

craigzour
Copy link
Contributor

@craigzour craigzour commented Mar 22, 2024

Summary | Résumé

Goes with cds-snc/platform-forms-client#3421

  • Moved both Redis and PostgreSQL from isolated Docker containers to Localstack Pro to align with what we have in production/staging
  • Added new optional environment variables to define how Localstack Pro persistence behaves
  • Reintroduce a clean option in the localstack_services.sh script to prevent Terraform state files from being deleted every time. This was causing deployment issues when running the script against an existing instance of Localstack where everything was already deployed.
  • Updated README.md to reflect all recent changes made to the infrastructure.

Testing

  • Deploy the infrastructure using the latest information contained in the README file.
  • Run and use the GC Forms web application after reviewing your .env file to make sure it is in sync with the latest .env.example.

@craigzour craigzour self-assigned this Mar 22, 2024
@craigzour craigzour force-pushed the feature/deploy-redis-and-postgresql-in-localstack branch 4 times, most recently from dc48889 to 3200f0c Compare March 26, 2024 15:02
@craigzour craigzour force-pushed the feature/deploy-redis-and-postgresql-in-localstack branch 2 times, most recently from 91cb2d4 to 01882ec Compare March 27, 2024 15:06
@craigzour craigzour force-pushed the feature/deploy-redis-and-postgresql-in-localstack branch from 01882ec to 3017ddd Compare March 27, 2024 15:15
@craigzour craigzour marked this pull request as ready for review March 27, 2024 15:16
@craigzour craigzour changed the title Feature/deploy redis and postgresql in localstack feature: deploy redis and postgresql in localstack Mar 27, 2024
@craigzour craigzour force-pushed the feature/deploy-redis-and-postgresql-in-localstack branch from 3017ddd to 06fcb35 Compare March 28, 2024 17:21
@craigzour craigzour force-pushed the feature/deploy-redis-and-postgresql-in-localstack branch from 06fcb35 to d13af55 Compare March 28, 2024 18:22
wmoussa-gc
wmoussa-gc previously approved these changes Mar 28, 2024
Copy link
Contributor

@wmoussa-gc wmoussa-gc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, so much cleanup! Well done!


printf "${color}...Purging stale localstack related files${reset}\n"
find $basedir/env/cloud -type d -name .terragrunt-cache -prune -print -exec rm -rf {} \;
if [[ "${ACTION}" == "clean" ]]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re added this parameter for now. The README file has been updated to explain when to use it.
Please note that a smarter version of this script is already being tested here. It will be able to detect whether it should run with or without the clean argument.

Copy link
Contributor

@bryan-robitaille bryan-robitaille left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nicely done!
I'll leave it up to you if you want to add that suggested comment in.

env/cloud/secrets/terragrunt.hcl Show resolved Hide resolved
Copy link

github-actions bot commented Apr 9, 2024

⚠ Terrform update available

Terraform: 1.7.5 (using 1.6.6)
Terragrunt: 0.56.2 (using 0.54.8)

Copy link
Contributor

@bryan-robitaille bryan-robitaille left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link

github-actions bot commented Apr 9, 2024

Staging: secrets

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

⚠️   Warning: resources will be destroyed by this change!

Plan: 1 to add, 0 to change, 1 to destroy
Show summary
CHANGE NAME
recreate aws_secretsmanager_secret_version.notify_callback_bearer_token
Show plan
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_secretsmanager_secret_version.notify_callback_bearer_token must be replaced
-/+ resource "aws_secretsmanager_secret_version" "notify_callback_bearer_token" {
      ~ arn            = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:notify_callback_bearer_token-aXJPLs" -> (known after apply)
      ~ id             = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:notify_callback_bearer_token-aXJPLs|4A43D97C-2937-4E91-A924-ADF60955E459" -> (known after apply)
      ~ secret_string  = (sensitive value) # forces replacement
      ~ version_id     = "4A43D97C-2937-4E91-A924-ADF60955E459" -> (known after apply)
      ~ version_stages = [
          - "AWSPREVIOUS",
        ] -> (known after apply)
        # (1 unchanged attribute hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"
Show Conftest results
WARN - plan.json - main - Missing Common Tags: ["aws_secretsmanager_secret.freshdesk_api_key"]
WARN - plan.json - main - Missing Common Tags: ["aws_secretsmanager_secret.notify_api_key"]
WARN - plan.json - main - Missing Common Tags: ["aws_secretsmanager_secret.notify_callback_bearer_token"]
WARN - plan.json - main - Missing Common Tags: ["aws_secretsmanager_secret.recaptcha_secret"]
WARN - plan.json - main - Missing Common Tags: ["aws_secretsmanager_secret.token_secret"]

24 tests, 19 passed, 5 warnings, 0 failures, 0 exceptions

Copy link

github-actions bot commented Apr 9, 2024

Staging: app

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

⚠️   Warning: resources will be destroyed by this change!

Plan: 1 to add, 0 to change, 1 to destroy
Show summary
CHANGE NAME
recreate aws_ecs_task_definition.form_viewer
Show plan
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_ecs_task_definition.form_viewer must be replaced
-/+ resource "aws_ecs_task_definition" "form_viewer" {
      ~ arn                      = "arn:aws:ecs:ca-central-1:687401027353:task-definition/form-viewer:1503" -> (known after apply)
      ~ arn_without_revision     = "arn:aws:ecs:ca-central-1:687401027353:task-definition/form-viewer" -> (known after apply)
      ~ container_definitions    = jsonencode(
          ~ [
              ~ {
                  - cpu              = 0
                  - essential        = true
                  ~ linuxParameters  = {
                      ~ capabilities = {
                          - add  = []
                            # (1 unchanged attribute hidden)
                        }
                    }
                  - mountPoints      = []
                    name             = "form_viewer"
                  ~ portMappings     = [
                      ~ {
                          - hostPort      = 3000
                          - protocol      = "tcp"
                            # (1 unchanged attribute hidden)
                        },
                    ]
                  - systemControls   = []
                  - volumesFrom      = []
                    # (4 unchanged attributes hidden)
                },
            ] # forces replacement
        )
      ~ id                       = "form-viewer" -> (known after apply)
      ~ revision                 = 1503 -> (known after apply)
      - tags                     = {} -> null
        # (9 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"
Show Conftest results
WARN - plan.json - main - Missing Common Tags: ["aws_appautoscaling_target.forms[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.forms"]
WARN - plan.json - main - Missing Common Tags: ["aws_codedeploy_app.app"]
WARN - plan.json - main - Missing Common Tags: ["aws_codedeploy_deployment_group.app"]
WARN - plan.json - main - Missing Common Tags: ["aws_ecs_cluster.forms"]
WARN - plan.json - main - Missing Common Tags: ["aws_ecs_service.form_viewer"]
WARN - plan.json - main - Missing Common Tags: ["aws_ecs_task_definition.form_viewer"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.cognito"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.forms_dynamodb"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.forms_kms"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.forms_s3"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.forms_secrets_manager"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.forms_sqs"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.codedeploy"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.forms"]

34 tests, 19 passed, 15 warnings, 0 failures, 0 exceptions

Copy link

github-actions bot commented Apr 9, 2024

Staging: lambdas

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

Plan: 0 to add, 7 to change, 0 to destroy
Show summary
CHANGE NAME
update aws_lambda_function.form_archiver
aws_lambda_function.nagware
aws_lambda_function.reliability
aws_lambda_function.vault_integrity
aws_s3_object.form_archiver_code
aws_s3_object.nagware_code
aws_s3_object.reliability_code
Show plan
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_lambda_function.form_archiver will be updated in-place
  ~ resource "aws_lambda_function" "form_archiver" {
        id                             = "Archive_Form_Templates"
      ~ last_modified                  = "2024-02-08T18:31:43.542+0000" -> (known after apply)
      ~ s3_object_version              = "79XIc6VR2aQuYzaduPsoOatLGFISakMg" -> (known after apply)
      ~ source_code_hash               = "Az0liG599yfRg7cqDAtpcwSYHtgtsQI2m7K2x/krsxY=" -> "tZom8NvYkLTCiFBHAx0p97qloJ1ZcMMumoKxxhc4SWc="
        tags                           = {}
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              - "ENVIRONMENT" = "staging" -> null
              - "LOCALSTACK"  = "false" -> null
                # (4 unchanged elements hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

  # aws_lambda_function.nagware will be updated in-place
  ~ resource "aws_lambda_function" "nagware" {
        id                             = "Nagware"
      ~ last_modified                  = "2024-02-08T19:33:25.528+0000" -> (known after apply)
      ~ s3_object_version              = "fuG43TeGDVNjxXr9SBeF7II5Kv7lVBoZ" -> (known after apply)
      ~ source_code_hash               = "2ZP5o8tCbep27P53xhm8tjX9DytomL5qhN8DZAlAVhQ=" -> "ewsERJHdgUqXPuAunXW9Djvue3VCJbRmdbJySbEUXMY="
        tags                           = {}
        # (21 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

  # aws_lambda_function.reliability will be updated in-place
  ~ resource "aws_lambda_function" "reliability" {
        id                             = "Reliability"
      ~ last_modified                  = "2024-03-26T18:42:41.000+0000" -> (known after apply)
      ~ s3_object_version              = "sGP63Ns5_df6wtyOh_.FvquFgx50U2A9" -> (known after apply)
      ~ source_code_hash               = "qdwZPGA+SOTsb3f31K4JE0IjD5jxYL3sUPXLUXdU1c8=" -> "VfnhFxgRiQhRaOOzhvqorDCLZ95Numn4J8eGPARS0uk="
        tags                           = {}
        # (21 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

  # aws_lambda_function.vault_integrity will be updated in-place
  ~ resource "aws_lambda_function" "vault_integrity" {
        id                             = "Vault_Data_Integrity_Check"
      ~ last_modified                  = "2024-03-26T18:42:41.000+0000" -> (known after apply)
      ~ source_code_hash               = "PD5RmSEVjAYMZc1lvfSSn6O3RadR+J09hCPKVbQS/+g=" -> "TowbMcppnki+0a5fq50Oral3CqleiwGw7U1igvFz0Ws="
        tags                           = {}
        # (24 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

  # aws_s3_object.form_archiver_code will be updated in-place
  ~ resource "aws_s3_object" "form_archiver_code" {
        id                     = "form_archiver_code"
      ~ source_hash            = "Az0liG599yfRg7cqDAtpcwSYHtgtsQI2m7K2x/krsxY=" -> "tZom8NvYkLTCiFBHAx0p97qloJ1ZcMMumoKxxhc4SWc="
        tags                   = {}
      ~ version_id             = "79XIc6VR2aQuYzaduPsoOatLGFISakMg" -> (known after apply)
        # (11 unchanged attributes hidden)
    }

  # aws_s3_object.nagware_code will be updated in-place
  ~ resource "aws_s3_object" "nagware_code" {
        id                     = "nagware_code"
      ~ source_hash            = "2ZP5o8tCbep27P53xhm8tjX9DytomL5qhN8DZAlAVhQ=" -> "ewsERJHdgUqXPuAunXW9Djvue3VCJbRmdbJySbEUXMY="
        tags                   = {}
      ~ version_id             = "fuG43TeGDVNjxXr9SBeF7II5Kv7lVBoZ" -> (known after apply)
        # (11 unchanged attributes hidden)
    }

  # aws_s3_object.reliability_code will be updated in-place
  ~ resource "aws_s3_object" "reliability_code" {
        id                     = "reliability_code"
      ~ source_hash            = "qdwZPGA+SOTsb3f31K4JE0IjD5jxYL3sUPXLUXdU1c8=" -> "VfnhFxgRiQhRaOOzhvqorDCLZ95Numn4J8eGPARS0uk="
        tags                   = {}
      ~ version_id             = "sGP63Ns5_df6wtyOh_.FvquFgx50U2A9" -> (known after apply)
        # (11 unchanged attributes hidden)
    }

Plan: 0 to add, 7 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"
Show Conftest results
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.audit_logs_archiver_lambda_trigger"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.form_archiver_lambda_trigger"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.nagware_lambda_trigger"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.reliability_dlq_lambda_trigger"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.response_archiver_lambda_trigger"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.archive_form_templates"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.audit_logs"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.audit_logs_archiver"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.dead_letter_queue_consumer"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.nagware"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.reliability"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.response_archiver"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.submission"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.vault_integrity"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_dynamodb"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_kms"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_logging"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_rds"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_s3"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_secrets"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_sns"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_sqs"]
WARN -...

Copy link

github-actions bot commented Apr 9, 2024

Staging: alarms

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

Plan: 0 to add, 2 to change, 0 to destroy
Show summary
CHANGE NAME
update aws_lambda_function.notify_slack
aws_s3_object.notify_slack_code
Show plan
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_lambda_function.notify_slack will be updated in-place
  ~ resource "aws_lambda_function" "notify_slack" {
        id                             = "NotifySlack"
      ~ last_modified                  = "2024-03-12T18:32:06.000+0000" -> (known after apply)
      ~ s3_object_version              = "PyMgtr32ws4WaGmnW8_J5v3xgSYhPXsD" -> (known after apply)
      ~ source_code_hash               = "OsJd8gBcFfPfRrCQT1iq3mOwD/HVbtvStHmskpExTvk=" -> "kbHJgDNOQodqCMI271CAcgHrZGTvsoUmV5UoyA6F7k8="
        tags                           = {}
        # (21 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "ENVIRONMENT"      = "Staging" -> "staging"
                # (2 unchanged elements hidden)
            }
        }

        # (2 unchanged blocks hidden)
    }

  # aws_s3_object.notify_slack_code will be updated in-place
  ~ resource "aws_s3_object" "notify_slack_code" {
        id                     = "notify_slack_code"
      ~ source_hash            = "OsJd8gBcFfPfRrCQT1iq3mOwD/HVbtvStHmskpExTvk=" -> "kbHJgDNOQodqCMI271CAcgHrZGTvsoUmV5UoyA6F7k8="
        tags                   = {}
      ~ version_id             = "PyMgtr32ws4WaGmnW8_J5v3xgSYhPXsD" -> (known after apply)
        # (11 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

Warning: Argument is deprecated

  with module.athena_bucket.aws_s3_bucket.this,
  on .terraform/modules/athena_bucket/S3/main.tf line 8, in resource "aws_s3_bucket" "this":
   8: resource "aws_s3_bucket" "this" {

Use the aws_s3_bucket_lifecycle_configuration resource instead

(and 3 more similar warnings elsewhere)

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"
Show Conftest results
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.codedeploy_sns"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.notify_slack"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.ELB_5xx_error_warn"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.UnHealthyHostCount-TargetGroup1"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.UnHealthyHostCount-TargetGroup2"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.alb_ddos"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.audit_log_dead_letter_queue_warn"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.cognito_login_outside_canada_warn"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.cognito_signin_exceeded"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.ddos_detected_forms_warn"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.ddos_detected_route53_warn[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.forms_cpu_utilization_high_warn"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.forms_memory_utilization_high_warn"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.reliability_dead_letter_queue_warn"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.response_time_warn"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.route53_ddos[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.twoFa_verification_exceeded"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.vault_data_integrity_check_lambda_iterator_age"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.notify_slack_lambda"]
WARN - plan.json -...

@craigzour craigzour merged commit 20e0fc1 into develop Apr 9, 2024
4 checks passed
@craigzour craigzour deleted the feature/deploy-redis-and-postgresql-in-localstack branch April 9, 2024 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants