Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TargetGroup for Port 443 uses HTTP Protocol instead of HTTPS #1582

Closed
johnrlowry opened this issue Oct 28, 2020 · 21 comments · Fixed by #4021
Closed

TargetGroup for Port 443 uses HTTP Protocol instead of HTTPS #1582

johnrlowry opened this issue Oct 28, 2020 · 21 comments · Fixed by #4021

Comments

@johnrlowry
Copy link

johnrlowry commented Oct 28, 2020

What changes do I need to make to the manifest to create a targetgroup that listens on port 443 protocol HTTPS instead of on port 443 protocol HTTP?

I created my app using copilot app init --domain my.example.com

Here's my manifest:

# The manifest for the "php-apache" service.
# Read the full specification for the "Load Balanced Web Service" type at:
#  https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/

# Your service name will be used in naming your resources like log groups, ECS services, etc.
name: php-apache
# The "architecture" of the service you're running.
type: Load Balanced Web Service

image:
  # Docker build arguments. You can specify additional overrides here. Supported: dockerfile, context, args.
  build: apache/Dockerfile
  # Port exposed through your container to route traffic to it.
  port: 443

http:
  # Requests to this path will be forwarded to your service.
  # To match all requests you can use the "/" path.
  path: '/'
  # You can specify a custom health check path. The default is "/"
  healthcheck: '/'
  # You can enable sticky sessions.
  # stickiness: true

# Number of CPU units for the task.
cpu: 1024
# Amount of memory in MiB used by the task.
memory: 4096
# Number of tasks that should be running in your service.
count: 2

# Optional fields for more advanced use-cases.
#
#variables:                    # Pass environment variables as key value pairs.
#  LOG_LEVEL: info
#
#secrets:                      # Pass secrets from AWS Systems Manager (SSM) Parameter Store.
#  GITHUB_TOKEN: GITHUB_TOKEN  # The key is the name of the environment variable, the value is the name of the SSM parameter.

# You can override any of the values defined above by environment.
#environments:
#  test:
#    count: 2               # Number of tasks to run for the "test" environment.
@kohidave
Copy link
Contributor

Hey there! I'm just curious - why do you want to change your target group port? When you provide a domain to copilot, we'll provision an ACM cert, assign it to your loadbalancer, and terminate TLS at the load balancer level. That way your service doesn't have to do any TLS termination.

@johnrlowry
Copy link
Author

johnrlowry commented Oct 28, 2020

Technically, I'm not trying to change the port. I'm trying to change the protocol.

My application listens on port 443 however, the ALB is sending traffic from the load balancer to port 80 on the container. I opened a ticket with AWS support and they identified this as the problem. If I create a targetgroup that uses HTTPS as the protocol instead of HTTP and then change the forwarding rules to point to the new target group, I can access my service.

In the image below you can see the copilot created target group vs the one I created.
image

@iamhopaul123
Copy link
Contributor

Hey @johnrlowry I just tested my own service which exposes port 443, and it works well. I wonder if it is because your service is expecting a tls connection?

@johnrlowry
Copy link
Author

Yes the service is expecting tls from the load balancer. Is this not possible with copilot today?

@iamhopaul123
Copy link
Contributor

Gotchu. Yeah it is not available yet in copilot. Could you elaborate your use case of using tls connection within? We have a plan of adding NLB as another service pattern. I am wondering if that could help.

@kohidave
Copy link
Contributor

I'd also say that most Copilot services don't need to do TLS termination since we handle that at the load balancer level. You should be able to turn off TLS handling in your service and let the Application Loadbalancer Handle everything for you.

@jsutton
Copy link

jsutton commented Nov 16, 2021

Adding on to @kohidave's response, this was the response from an ELB engineer regarding TLS on the backside of an ALB:
https://kevin.burke.dev/kevin/aws-alb-validation-tls-reply/

One thing to note is that the ALB does not validate TLS certificates from internal services which reduces the utility of having the target groups support TLS in the first place.
https://kevin.burke.dev/kevin/amazons-albs-insecure-internal-traffic/

@mnitchie
Copy link

mnitchie commented Apr 5, 2022

If it were up to me, those blog posts would be satisfying, but this is a bit problematic from a regulatory perspective. For instance, if you're managing healthcare/patient data, regulations strongly recommend network encryption end-to-end. If you don't have it, auditors ask questions and you have to explain yourself, and you are kind of up to their discretion and whim. It would just be easier to allow a config to forward HTTPS traffic to a sidecar nginx or something that could terminate ssl with a self-signed cert, similar to the suggestion here: #2981

Alternatively, if AWS has documentation more formal than an unrelated blog post on why encryption beyond the LB is not necessary, that may keep auditors at bay.

@johnrlowry
Copy link
Author

I'm curious if any additional movement has been made on this topic. For situations where customers or regulatory recommendations indicate end-to-end encryption this is needed. I abandoned Copilot because of this issue, but am now considering it again. At the moment, the only thing keeping me from migrating our existing environments is that everything is automated by copilot except the creation of the target group that uses https protocol.

@efekarakus
Copy link
Contributor

Hi @johnrlowry !

I wonder if nlb would solve your usecase? To allow ssl termination at the task level.

nlb:
  port: 443/tcp

@romulovmarques
Copy link

Hi everyone , have a great day.
I have a problem similar, but obtain success changing the healt check. In tg configure protocol https and port 443, in the healt chek protocol htttp, in traffic modified port to "override " in advanced health check settings, set port 80. In my case this is success

@craigjbass
Copy link
Contributor

craigjbass commented Aug 24, 2022

For compliance users, having SSL termination within tasks/containers is preferable, even if ALBs do not do any certificate validation.

@cd-billo
Copy link

Yes, the ability to allow termination WITHIN the ECS task (sidecar or just within the main task) would check a lot of boxes for a variety of compliance standards. The biggest being HIPAA.

@tmichalski
Copy link

I'm only echoing this need. We work with PCI and HIPAA apps and there is no compromise for not having end-to-end encryption between components, even within the VPC. Copilot has been great so far and I'm glad there is an alternative to use NLB to pass-thru the TLS termination to the container, but there is extra work to make each container capable of terminating TLS that we were hoping to avoid. I would strongly back exposing the CloudFormation TargetGroup settings within the manifest, or just allowing for the ability to override the CloudFormation like task overrides.

Btw, huge fan of copilot. Nice work overall.

@efekarakus
Copy link
Contributor

Hi folks!

I just built a tiny prototype to get an understanding of how we could support this in Copilot and the changes are pretty tiny from our end! I just wanted to validate a few things:

  1. Is it fair to assume that for end-to-end encryption the certificate attached to the Load Balancer will be imported? https://aws.github.io/copilot-cli/docs/manifest/environment/#http-public-certificates

  2. I assume that you have a sidecar container such as Envoy ready that's going to terminate TLS is that accurate?
    For example, in my prototype I have a manifest that looks like this:

    http:
      alias: domain.com
      path: "/"
      target_container: "envoy"
    
    image:
      build: "./example/Dockerfile-app"
      port: 8080
    
    sidecars:
      envoy:
        port: 443
        image: "615288112214.dkr.ecr.us-west-2.amazonaws.com/demo/envoy-proxy:v2"

    Or are you looking for Copilot to provide a sidecar container to help with the TLS termination?

@cd-billo
Copy link

cd-billo commented Sep 21, 2022 via email

@coingraham
Copy link

Adding on to @kohidave's response, this was the response from an ELB engineer regarding TLS on the backside of an ALB: https://kevin.burke.dev/kevin/aws-alb-validation-tls-reply/

One thing to note is that the ALB does not validate TLS certificates from internal services which reduces the utility of having the target groups support TLS in the first place. https://kevin.burke.dev/kevin/amazons-albs-insecure-internal-traffic/

Let me chime in that this is actually the desirable effect. The certificate validation can happen at the LoadBalancer and the downstream (target group) systems can use self-signed certificates. This gets you numerous advantages:

  1. End to end encryption.
  2. Immutable infrastructure
  3. Cost savings
  4. Better security

Because you don't need to load an exported certificate on the internal systems, you don't need to purchase a certificate from a provider. AWS provides those for free. Additionally your process doesn't need to include managing and loading specific certificates per application. All build patterns will generate their own self signed certs and use those. Further since you're generating new certs on instance creation, the certs are rotated much more often.

So the use case for compliance workloads (HIPAA and PCI) is:

Client/User >> 443/https >> Secure ALB Listener (Cert validated here) >> 443/https >> Web Host (self-signed cert) or Envoy Sidecar (self-signed cert).

Today I'm accomplishing this by using the sidecar implementation of copilot and rolling my own LB/Listener/TargetGroup/Service in the addons. This requires two deployments to deploy because there isn't a way to get the TaskDefinition update from the Cloudformation Template (There's no !GetAtt for the Task Definition Arn).

@coingraham
Copy link

Hi folks!

I just built a tiny prototype to get an understanding of how we could support this in Copilot and the changes are pretty tiny from our end! I just wanted to validate a few things:

  1. Is it fair to assume that for end-to-end encryption the certificate attached to the Load Balancer will be imported? https://aws.github.io/copilot-cli/docs/manifest/environment/#http-public-certificates

  2. I assume that you have a sidecar container such as Envoy ready that's going to terminate TLS is that accurate?
    For example, in my prototype I have a manifest that looks like this:
    [snip]

    Or are you looking for Copilot to provide a sidecar container to help with the TLS termination?

  1. It doesn't need to be imported into ACM. You could just request a certificate instead. Regardless, the implementation shouldn't care since the ALB Listener only cares that you have a Certificate ARN associated and doesn't know the difference of which type (request vs. import).
  2. If you have the option for the target_container, you could choose the sidecar or go straight to your web server that's listening on a secured port.

I don't know what the level of effort is to adjust the configuration but it seems to me that the missing piece is the option for the target group connection and healthcheck be HTTPS instead of HTTP. I feel like that's the only thing missing.

@tmichalski
Copy link

I don't know what the level of effort is to adjust the configuration but it seems to me that the missing piece is the option for the target group connection and healthcheck be HTTPS instead of HTTP. I feel like that's the only thing missing.

I second this quote. If copilot would allow for a target group connection and healthcheck be HTTPS, then we're good to go.

Our containers manage TLS on their own using self-signed certs and expose an HTTPS port.

@efekarakus
Copy link
Contributor

Thanks folks ❤️ , I think I'm fully ramped up on the task now. I've implemented this prototype and tested it: mainline...efekarakus:copilot-cli:prototype/e2e-encryption-alb and it seems to work as we want it ✅

I'll try to clean this up and merge it and hopefully get the feature in before next release 🤞

@mergify mergify bot closed this as completed in #4021 Sep 21, 2022
mergify bot pushed a commit that referenced this issue Sep 21, 2022
…4021)

When the target container's port is set to 443 for Load Balanced Web Services or Backend Services with internal ALB then we set the TargetGroup's protocols to HTTPS allowing for e2e encryption even within the VPC.

Resolves #1582, resolves #2981

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.
@efekarakus
Copy link
Contributor

This feature is now released in v1.22 🚀 !

For the blog post: https://aws.github.io/copilot-cli/blogs/release-v122/
Release notes: https://github.com/aws/copilot-cli/releases/tag/v1.22.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.