NET-5186 Allow dataplane container to bind to privileged ports

[nathancoleman]

Describe the issue

Consul-dataplane is currently unable to bind to privileged ports (< 1024).

This is important for ingress-gateway use cases where customers have historically been able to bind to ports such as 443 and are encountering runtime failures when attempting to upgrade to Consul 1.15+ and the corresponding Helm chart versions. In these newer versions, consul-dataplane has taken the place of the envoyproxy/envoy containers that were used previously.

Example of failure:

[warning] envoy.config(13) delta config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) http:0.0.0.0:443: cannot bind '0.0.0.0:443': Permission denied

Describe the fix

It appears that Envoy containers, which consul-dataplane has replaced for ingress-gateway use cases, run as root and then use su-exec to run as a different user. I'm thinking that we can set the NET_BIND_SERVICE capability directly on the Envoy and dataplane binaries and avoid starting up as root, but I'm depending on my own testing and reviewers here to validate this.

• The consul-dataplane container still needs to function correctly when deployed into OpenShift
• The consul-dataplane container still needs to function correctly with Set privileged to false unless on OpenShift without CNI consul-k8s#2755

This PR adds a new stage to set the net_bind_service capability on the Envoy and dataplane binaries that are copied into the release image. The final images then copy their Envoy and dataplane binaries from this new stage instead of their previous source.

How to test

• Create an ingress-gateway that binds to a privileged port, such as 443 (see example values below)
  • Before this change, the consul-dataplane container will appear healthy but spew logs containing the error below
  • After this change, the consul-dataplane container will appear healthy, and the logs will indicate successful xDS config
• Do the same testing on OpenShift to verify no negative impact

Example `values.yaml`

global:
  name: consul
  logLevel: debug

connectInject:
  replicas: 1
  enabled: true

server:
  replicas: 3

ingressGateways:
  enabled: true
  gateways:
    - name: my-ingress
      service:
        type: LoadBalancer
        ports:
          - port: 443

[nathancoleman]

Defaulting these for building locally like we do in consul-k8s

[curtbushko]

Thanks Nathan!

For other reviewers: we had hours upon hours of back and forth discussions on how to do this and this 'setcap' method was the best of all horrible ways to solve this problem.

We couldn't setcap at runtime since we are running a distroless image (no shell to run the setcap command...).