Enable app developers to perform privileged actions if the operator allows it

[maxmoehl]

Proposed Change

As a developer,
I want this ability to capture tcpdumps or perform other privileged actions,
So that I can perform in-depth troubleshooting of my application.

The original request started with the ask to find a way to enable app developers to perform tcpdumps. This is a topic that has been going on in the background for quite some time but so far we never came to a solution for CF (for BOSH we have the introduced the bosh-pcap command).

When troubleshooting applications it is common to require elevated privileges, for example to dump the core on a process or start a packet capture on a network interface. Today there is no possibility for an app developer to perform such actions when they deploy their application via buildpacks.

If the operator allows docker apps to be pushed, the app owner can push such an app in a configuration that makes the executing user root and gain all the necessary privileges.

Option 1) sudo (discarded)

The proposal is to add another switch (like the docker one) that allows operators to make the vcap user a sudoer. This has security implications but if docker apps are already allowed I don't see how this is any different than building the app locally and pushing as a docker image with the root user configured.

Option 2) setcap on tcpdump

This would be a change in the stack to assign the tcpdump binary the necessary capabilities to perform packet captures (namely CAP_NET_RAW and CAP_NET_ADMIN). The scope here is much narrower as this only enables tcpdump but a change in the stack is hard to hide behind a feature flag and would probably be rolled out unconditionally to everyone. With the capabilities set on the tcpdump binary any regular user can use it and via that binary (and only that binary) gain the necessary capabilities to perform packet captures.

Option 2b) setcap on custom packet capturing tool

A reduced tcpdump written in go. With gopacket it's trivial to start a capture and output the stream to stdout. We'd only support specifying an interface, filter and snaplen to reduce the attack surface - there even should be code in the archived pcap-release that pretty much does that. We can then setcap that binary to allow regular users to capture traffic. If the binary is injected via diego (say similar to diego-ssh) this can even become a per-app feature flag (although I'd prefer just making it a platform flag for simplicity).

Acceptance criteria

Application developers can perform tcpdumps in app containers.

Related links

• Slack discussion in #security
• Cross-post to #garden
• pcap-release (archived)
• RFC0019 PCAP BOSH

[maxmoehl]

Once we can run tcpdump inside the application container we can easily replicate the logic that we build into the bosh-pcap command either as a CF CLI plugin or built-in command. The main reason for doing so is that the data can be streamed back live instead of storing it locally in the app container. Storing it locally has the risk of loosing all data when the app crashes or is evacuated and the storage space is usually very limited.

[vlast3k]

i think that restricting buildpack based apps is not bad, as it adds a layer of security and people do not need to care about it.
when they deploy Docker apps and explicitly allow root user, this opens potential for the app to be exploited externally if not properly protected (no risk for the platform, but same as using phone with unlocked bootloader)
therefore i would propose to make it in a similar way and allow the app developer and not to the platform operator to allow this with a configuration in the manifest or some other means

[maxmoehl]

I've rephrased the issue description to also explain the setcap approach along the sudo one.

If we can agree on an approach I can write or more detailed proposal on how this could work out.

[tcdowney]

Kubernetes supports ephemeral containers that can be created in a Pod that share namespaces with the other containers in a Pod. This is mostly useful for debugging, but means you can also bring along other debug utilities that you might not want to be present in the main app container.

https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/

This would be significantly more work to implement I'm sure, but just throwing it out there as food for thought.

[ameowlia]

Out of the options I am leaning towards (2). I like the smaller scope and I like not opening the can of worms (🪱) about giving more capabilities to apps and who would have the power to give those capabilities.

but a change in the stack is hard to hide behind a feature flag and would probably be rolled out unconditionally to everyone.

I've never worked in the stack before. You mentioned that they were other binaries that work in a similar way, like "ping".

❓ Can you provide a link to where ping is setup in the stack and the area of code that would be altered if we did option 2?

[tcdowney]

I also want to call out that there has been recent work in CAPI around adding the ability for developers to specify a different run user for their LRPs/Tasks along with an operator-configured allowlist of permitted users.

cloudfoundry/cloud_controller_ng#4372

There is some discussion there around the fact that allowing Docker apps to run with 'root' may not always be a good thing and I think @Gerg is going to add some opt-in properties to restrict that. Since Docker lifecycle apps are currently disabled by default I don't think it's 100% true to say that all devs can already do this.

[maxmoehl]

I've never worked in the stack before. You mentioned that they were other binaries that work in a similar way, like "ping".

❓ Can you provide a link to where ping is setup in the stack and the area of code that would be altered if we did option 2?

I had to learn that ping no longer uses this mechanism. But fping still does:

[testing] $ sudo getcap /usr/bin/fping
/usr/bin/fping cap_net_raw=ep

[thomasthal]

In general, I do not have security concerns to offer CAT_NET_RAW within CF application containers and with the support of docker images the feature is already available. It might still be a good idea to offer this as a feature flag, because to my knowledge docker image support is optional and there might be different use cases or opinions about the criticality of having CAT_NET_RAW enabled. So, I favour option 1 at the moment.

Option 2 tries to limit the additional permissions to the tcpdump binary. I think this approach does not work, because tcpdump offers functionality to run arbitrary code in the security context of tcpdump. At least I found the following (yet untested) code snippet:

COMMAND='id'
TF=$(mktemp)
echo "$COMMAND" > $TF
chmod +x $TF
tcpdump -ln -i lo -w /dev/null -W 1 -G 1 -z $TF

[maxmoehl]

I want to take a deeper look into the PoC for code execution pointed out by @thomasthal but I was somewhat expecting that tcpdump isn't really made for this kind of setup.

So let me propose a different option which is slightly more maintenance but puts us in control: a reduced tcpdump written in go. With gopacket it's trivial to start a capture and output the stream to stdout. We'd only support specifying an interface, filter and snaplen to reduce the attack surface - there even should be code in the archived pcap-release that pretty much does that. We can then setcap that binary to allow regular users to capture traffic. If the binary is injected via diego (say similar to diego-ssh) this can even become a per-app feature flag (although I'd prefer just making it a platform flag for simplicity).

Thoughts?

_{Realistically I don't care too much which route we go down but I really want this.}

[maxmoehl]

Since Docker lifecycle apps are currently disabled by default I don't think it's 100% true to say that all devs can already do this.

@tcdowney yes, this is true. In any case it's probably best to let the operator decide whether to allow this on their platform - just like with docker apps. We could probably also make that work for setcap somehow without modifying the stack.

[ameowlia]

> tcpdump offers functionality to run arbitrary code in the security context of tcpdump. At least I found the following (yet untested) code snippet:

If this is true, it is unfortunate. It if is true, then I am in favor of option 3 with a foundation wide config flag.