Make sidecar usable on ECS Fargate

[v3rm0n]

Sidecar is mostly meant to be used on Kubernetes and it is possible to get it working on ECS with Fargate, but it's a bit of a hassle. I'll describe some of the issues so maybe you can take it into consideration for the future.

Biggest issue is how the configuration works: in Kubernetes it is very easy to add files to a pod using a configmap so this file based configuration works fine, but in ECS the only way to add files is to build your own image or by mounting some persistent storage like EFS and then going in and manually making the changes to the files.

For more advanced configuration this is probably even acceptable, but it would be nice if the sidecar would start on ECS Fargate into a usable state without all this config file management.

Why it currently does not is because EE images use ports 443 and 80 by default which are privileged ports, but the proxy runs as non privileged xroad user. Fargate doesn't support capabilities (except CAP_SYS_PTRACE).

So currently to get the sidecar working, I have to start it up with an EFS persistent mount on /etc/xroad launch an EC2 machine (Since amazon-efs-utils is not supported on MacOS), mount the EFS to the EC2 machine and add:

[proxy]
client-https-port = 8443

to the config file so that the proxy server starts up.

If I want it to stay on port 443 I also need to add an nginx container which proxies the traffic from port 443->8443 (Fargate doesn't allow remapping ports).

events {}
stream {
  server {
    listen     443;
    proxy_pass localhost:8443;
  }
}

Getting the privileged ports working seems to be a bit harder but adding an environment variable override for the proxy port(s) would already give a huge impact.

[raits]

Hello @v3rm0n!

Just to clarify so that I understand correctly. The biggest issue with the EE metapackages in ECS Fargate currently are the ports used (80 and 443) and the main change that would make this easier would be to allow overriding the client proxy ports via environmental variables?

[v3rm0n]

Yes, that would remove the need to change any configuration files.

[TaaviMeinberg]

Hey @v3rm0n
We at RIA (Riigi Infosüsteemi Amet) have also done a bit of testing regarding setting up a X-Road Security Server cluster on Kubernetes.

You can find an overview of our experience and some of the complexities that we discovered here: https://koodivaramu.eesti.ee/x-tee/x-road-helm . We have also forwarded them to NIIS as feedback and points of improvement.

I hope that documentation is of help to you or that it's just interesting to read. Any and all feedback is also welcomed.

[v3rm0n]

Thanks! Definitely an interesting read.