feat(blog): New post on running containers with full host isolation from Jenkins via DinD

[Pexers]

Hey there!

I've been running this solution for a couple of months now, and I believe others could benefit from it as well. So I figured, why not write a blog post about it?

In this post, I discuss how to effectively use Docker-in-Docker as a sidecar container to keep Jenkins execution environments fully isolated from the host system. This approach also eliminates the need for custom builds to include Docker binaries.

Regardless of whether or not it gets accepted, I'd be grateful for any feedback that I get!😄

Thank you.

Cheers,
Pedro R.

The preview can be seen at: https://deploy-preview-7654--jenkins-io-site-pr.netlify.app/blog/2024/11/02/containerized-pipelines-with-full-host-isolation-via-dind/ .

[dduportal]

Hello, many thanks for this proposal! Wether it is published or not, it is really commendable that you share your experience and worked on this proposal!

I believe it is worth a discussion as many of us would be against recommending using DinD at all for security reason, while this posts proposes solution to problem which raise many questions on how Jenkins community could help.

My (personal) main concern is the DinD container. It enables the privileged option AND runs as root user. Both are huge security holes which allows transversal attacks.

In the example in this proposal, the fact that privileged is used defeats the whole purpose of using a side container: if you can execute a container with the Docker engine then you can access the agent container as root.

It also allows any pipeline command to attack the host (as it runs as root AND is privileged: it is the same as giving sudo permissions to pipeline).

@Pexers It could be interesting to extend the scope of your blog post to add the following (and open wider discussion without endangering users):

• Running DinD in rootless mode (Docker provides resources on that matter). It's closes one of the 2 security holes and would open awareness
• Check how to run this setup, out of the box, inside a confined system such as firecracker

My personal recommendation is to use VMs to have Docker Engine withing (either as ephemeral agents with ec2, azure-vms , etc. plugin) or with a static agent (with clean up).

Ping @MarkEWaite @kmartens27 @gounthar @darinpope for awareness: this could start valuable discussion about:

• Getting rid of DinD on all of our documentation pages to avoid endangering users
• Discuss the pertiencne to keep the "docker agent" like plugins or at least shows the limits of it
• Start addressing the need of Jenkins users to run Docker containers for builds (documentation, examples, etc.) which includes the safety concerns

=> I'm not blocking the PR as I'm only one voice, and this requires wider community discussion

[kmartens27]

Thank you very much @Pexers for all the work done to put this together and share your insights.

I will agree with @dduportal that the security risk of DinD is the main concern with sharing/encouraging this sort of information and workflow. We have been working to update the tutorials so that they rely on docker compose -up instead, to enhance the user experience and remove the security risk that DinD opens.

Alternatively, from a documentation standpoint, the end goal of this work is to update the rest of the documentation so that docker compose is the standard usage. We are looking to incorporate the docker compose workflow into the installation docs, among other things, and this could create confusion amongst readers. If there are alternative ways to provide a similar configuration without DinD, that would be very welcome.

[Pexers]

Hey @dduportal and @kmartens27! First and foremost, thank you so much for taking a look at it, and for the detailed feedback!

I understand you guys' position on this topic, and I was somewhat expecting it, but I wanted to give it a try anyway haha.

I still believe a solution involving DinD has the potential to attract a significant number of readers, and in my opinion, adapting the existing post to exclude DinD entirely would be too big of a change (it would essentially defeat its original purpose).

So I will follow @dduportal's advice and further investigate the possibility of running DinD in rootless mode.

In case I do make any new discoveries, I’ll update the blog post accordingly. 🙂

Thank you once again!

[meiswjn]

As @dduportal stated, it doesn't make a difference from security perspective to mount the socket or have it as a sidecar container, sadly.

Interestingly enough, I demonstrated a container escape from docker:dind literally today, as we are also investigating secure methods of using DIND. --privileged allows you to mount your host's file system, for example.
Don't use --privileged / privileged: true, it is dangerous regardless of using TCP or mounting the socket directly. If the original socket can be accessed, your system can be accessed. Rootless Docker prevents you from having root permissions but it still allows you to edit other containers running in parallel.

I recommend having a look at Sysbox: https://github.com/nestybox/sysbox
That basically "behaves" like a VM. Doesn't require the socket of the parent container anymore and is actually isolated.