TargetGroup for Port 443 uses HTTP Protocol instead of HTTPS

[johnrlowry]

What changes do I need to make to the manifest to create a targetgroup that listens on port 443 protocol HTTPS instead of on port 443 protocol HTTP?

I created my app using copilot app init --domain my.example.com

Here's my manifest:

# The manifest for the "php-apache" service.
# Read the full specification for the "Load Balanced Web Service" type at:
#  https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/

# Your service name will be used in naming your resources like log groups, ECS services, etc.
name: php-apache
# The "architecture" of the service you're running.
type: Load Balanced Web Service

image:
  # Docker build arguments. You can specify additional overrides here. Supported: dockerfile, context, args.
  build: apache/Dockerfile
  # Port exposed through your container to route traffic to it.
  port: 443

http:
  # Requests to this path will be forwarded to your service.
  # To match all requests you can use the "/" path.
  path: '/'
  # You can specify a custom health check path. The default is "/"
  healthcheck: '/'
  # You can enable sticky sessions.
  # stickiness: true

# Number of CPU units for the task.
cpu: 1024
# Amount of memory in MiB used by the task.
memory: 4096
# Number of tasks that should be running in your service.
count: 2

# Optional fields for more advanced use-cases.
#
#variables:                    # Pass environment variables as key value pairs.
#  LOG_LEVEL: info
#
#secrets:                      # Pass secrets from AWS Systems Manager (SSM) Parameter Store.
#  GITHUB_TOKEN: GITHUB_TOKEN  # The key is the name of the environment variable, the value is the name of the SSM parameter.

# You can override any of the values defined above by environment.
#environments:
#  test:
#    count: 2               # Number of tasks to run for the "test" environment.

[kohidave]

Hey there! I'm just curious - why do you want to change your target group port? When you provide a domain to copilot, we'll provision an ACM cert, assign it to your loadbalancer, and terminate TLS at the load balancer level. That way your service doesn't have to do any TLS termination.

[johnrlowry]

Technically, I'm not trying to change the port. I'm trying to change the protocol.

My application listens on port 443 however, the ALB is sending traffic from the load balancer to port 80 on the container. I opened a ticket with AWS support and they identified this as the problem. If I create a targetgroup that uses HTTPS as the protocol instead of HTTP and then change the forwarding rules to point to the new target group, I can access my service.

In the image below you can see the copilot created target group vs the one I created.

[iamhopaul123]

Hey @johnrlowry I just tested my own service which exposes port 443, and it works well. I wonder if it is because your service is expecting a tls connection?

[johnrlowry]

Yes the service is expecting tls from the load balancer. Is this not possible with copilot today?

[iamhopaul123]

Gotchu. Yeah it is not available yet in copilot. Could you elaborate your use case of using tls connection within? We have a plan of adding NLB as another service pattern. I am wondering if that could help.

[kohidave]

I'd also say that most Copilot services don't need to do TLS termination since we handle that at the load balancer level. You should be able to turn off TLS handling in your service and let the Application Loadbalancer Handle everything for you.

[jsutton]

Adding on to @kohidave's response, this was the response from an ELB engineer regarding TLS on the backside of an ALB:
https://kevin.burke.dev/kevin/aws-alb-validation-tls-reply/

One thing to note is that the ALB does not validate TLS certificates from internal services which reduces the utility of having the target groups support TLS in the first place.
https://kevin.burke.dev/kevin/amazons-albs-insecure-internal-traffic/

[mnitchie]

If it were up to me, those blog posts would be satisfying, but this is a bit problematic from a regulatory perspective. For instance, if you're managing healthcare/patient data, regulations strongly recommend network encryption end-to-end. If you don't have it, auditors ask questions and you have to explain yourself, and you are kind of up to their discretion and whim. It would just be easier to allow a config to forward HTTPS traffic to a sidecar nginx or something that could terminate ssl with a self-signed cert, similar to the suggestion here: #2981

Alternatively, if AWS has documentation more formal than an unrelated blog post on why encryption beyond the LB is not necessary, that may keep auditors at bay.

[johnrlowry]

I'm curious if any additional movement has been made on this topic. For situations where customers or regulatory recommendations indicate end-to-end encryption this is needed. I abandoned Copilot because of this issue, but am now considering it again. At the moment, the only thing keeping me from migrating our existing environments is that everything is automated by copilot except the creation of the target group that uses https protocol.

[efekarakus]

Hi @johnrlowry !

I wonder if nlb would solve your usecase? To allow ssl termination at the task level.

nlb:
  port: 443/tcp

[romulovmarques]

Hi everyone , have a great day.
I have a problem similar, but obtain success changing the healt check. In tg configure protocol https and port 443, in the healt chek protocol htttp, in traffic modified port to "override " in advanced health check settings, set port 80. In my case this is success

[craigjbass]

For compliance users, having SSL termination within tasks/containers is preferable, even if ALBs do not do any certificate validation.

[cd-billo]

Yes, the ability to allow termination WITHIN the ECS task (sidecar or just within the main task) would check a lot of boxes for a variety of compliance standards. The biggest being HIPAA.

[tmichalski]

I'm only echoing this need. We work with PCI and HIPAA apps and there is no compromise for not having end-to-end encryption between components, even within the VPC. Copilot has been great so far and I'm glad there is an alternative to use NLB to pass-thru the TLS termination to the container, but there is extra work to make each container capable of terminating TLS that we were hoping to avoid. I would strongly back exposing the CloudFormation TargetGroup settings within the manifest, or just allowing for the ability to override the CloudFormation like task overrides.

Btw, huge fan of copilot. Nice work overall.

[efekarakus]

Hi folks!

I just built a tiny prototype to get an understanding of how we could support this in Copilot and the changes are pretty tiny from our end! I just wanted to validate a few things:

1. Is it fair to assume that for end-to-end encryption the certificate attached to the Load Balancer will be imported? https://aws.github.io/copilot-cli/docs/manifest/environment/#http-public-certificates

2. I assume that you have a sidecar container such as Envoy ready that's going to terminate TLS is that accurate?
   For example, in my prototype I have a manifest that looks like this:

   http:
     alias: domain.com
     path: "/"
     target_container: "envoy"
   
   image:
     build: "./example/Dockerfile-app"
     port: 8080
   
   sidecars:
     envoy:
       port: 443
       image: "615288112214.dkr.ecr.us-west-2.amazonaws.com/demo/envoy-proxy:v2"

   Or are you looking for Copilot to provide a sidecar container to help with the TLS termination?

[cd-billo]

This implementation looks correct to me… I think there could be the assumption of a self-signed cert in whatever is used as the envoy image. On Sep 20, 2022, at 7:56 PM, Efe Karakus ***@***.******@***.***>> wrote: 1. Or are you looking for Copilot to provide a sidecar container to help with the TLS termination?

[coingraham]

Adding on to @kohidave's response, this was the response from an ELB engineer regarding TLS on the backside of an ALB: https://kevin.burke.dev/kevin/aws-alb-validation-tls-reply/

One thing to note is that the ALB does not validate TLS certificates from internal services which reduces the utility of having the target groups support TLS in the first place. https://kevin.burke.dev/kevin/amazons-albs-insecure-internal-traffic/

Let me chime in that this is actually the desirable effect. The certificate validation can happen at the LoadBalancer and the downstream (target group) systems can use self-signed certificates. This gets you numerous advantages:

1. End to end encryption.
2. Immutable infrastructure
3. Cost savings
4. Better security

Because you don't need to load an exported certificate on the internal systems, you don't need to purchase a certificate from a provider. AWS provides those for free. Additionally your process doesn't need to include managing and loading specific certificates per application. All build patterns will generate their own self signed certs and use those. Further since you're generating new certs on instance creation, the certs are rotated much more often.

So the use case for compliance workloads (HIPAA and PCI) is:

Client/User >> 443/https >> Secure ALB Listener (Cert validated here) >> 443/https >> Web Host (self-signed cert) or Envoy Sidecar (self-signed cert).

Today I'm accomplishing this by using the sidecar implementation of copilot and rolling my own LB/Listener/TargetGroup/Service in the addons. This requires two deployments to deploy because there isn't a way to get the TaskDefinition update from the Cloudformation Template (There's no !GetAtt for the Task Definition Arn).

[coingraham]

Hi folks!

I just built a tiny prototype to get an understanding of how we could support this in Copilot and the changes are pretty tiny from our end! I just wanted to validate a few things:

1. Is it fair to assume that for end-to-end encryption the certificate attached to the Load Balancer will be imported? https://aws.github.io/copilot-cli/docs/manifest/environment/#http-public-certificates

2. I assume that you have a sidecar container such as Envoy ready that's going to terminate TLS is that accurate?
   For example, in my prototype I have a manifest that looks like this:
   [snip]

   Or are you looking for Copilot to provide a sidecar container to help with the TLS termination?

1. It doesn't need to be imported into ACM. You could just request a certificate instead. Regardless, the implementation shouldn't care since the ALB Listener only cares that you have a Certificate ARN associated and doesn't know the difference of which type (request vs. import).
2. If you have the option for the target_container, you could choose the sidecar or go straight to your web server that's listening on a secured port.

I don't know what the level of effort is to adjust the configuration but it seems to me that the missing piece is the option for the target group connection and healthcheck be HTTPS instead of HTTP. I feel like that's the only thing missing.

[tmichalski]

I don't know what the level of effort is to adjust the configuration but it seems to me that the missing piece is the option for the target group connection and healthcheck be HTTPS instead of HTTP. I feel like that's the only thing missing.

I second this quote. If copilot would allow for a target group connection and healthcheck be HTTPS, then we're good to go.

Our containers manage TLS on their own using self-signed certs and expose an HTTPS port.

[efekarakus]

Thanks folks ❤️ , I think I'm fully ramped up on the task now. I've implemented this prototype and tested it: mainline...efekarakus:copilot-cli:prototype/e2e-encryption-alb and it seems to work as we want it ✅

I'll try to clean this up and merge it and hopefully get the feature in before next release 🤞

[efekarakus]

This feature is now released in v1.22 🚀 !

For the blog post: https://aws.github.io/copilot-cli/blogs/release-v122/
Release notes: https://github.com/aws/copilot-cli/releases/tag/v1.22.0