Provide docker registry credentials to custom host container source

[springroll12]

What I'd like:
A way to SSH into the "control" container. Current approach is to build a custom host container and store it in a private (non-ECR) registry. Is there a secure way to pass registry credentials to the bottlerocket userdata? Is there any documentation of the format of the "SOURCE" parameter for custom host containers?

See #1326 for details.

Any alternatives you've considered:

• Re-use admin container image in superpowered = false mode?
• Publish custom host image publicly to docker hub or elsewhere
• Ignore control container and always enable admin container. Running in private subnet this seems about equivalent risk to running a normal ec2 instance in a private subnet. This the simplest and least secure option.

[etungsten]

Hi @springroll12, thanks for the request!

I’d like to understand your use case a bit better. Do you want to make Bottlerocket API calls, and so the particular host container doesn’t matter much, or are you trying to use other tools on the system? Are there particular tools you want from the control container that aren't in the admin container, or do you want SSH access without there being a superpowered container on the other end? I'm guessing the latter, because the control container is fairly minimal.

Since you're in a private network (which I'm assuming is trusted) I think your third option, keeping the admin container enabled, is about the same in terms of security, and should give you any tools you need. Regarding your first option: Since host containers have access to the API, and you can enable additional host containers through the API, the superpowered flag wouldn't (isn't meant to) stop intentional abuse, so running the admin container with superpowered=false wouldn't be much of a benefit.

If you would still like to use a custom control container, you should be able to store the image in a private ECR registry and use it in Bottlerocket without needing to explicitly specify credentials to pull the image if you’re running in EC2. If you’re running Bottlerocket outside of AWS or can’t use private ECR registries, unfortunately it’s currently not possible to specify image registry credentials in Bottlerocket. #1227 is the issue for tracking support for configuring image registry credentials.

Please let us know what works best for you.

[springroll12]

@etungsten Thanks for clarifying!

Essentially I'm trying to enable SSH access to Bottlerocket nodes through a bastion host in a public network. The recommended EKS deployment pattern is to put all worker nodes in a private subnet bridged to a public subnet via NAT gateway, so a bastion host would have access to the nodes, but needs an SSH daemon to connect to. This can be either from the admin or control container, but I was trying to follow the recommended pattern:

1. Enable control container by default
2. Disable admin container by default
3. Login to control container to enable admin container only when necessary
4. SSH into admin container

This seems difficult to achieve without SSM. I would like to achieve step 3 with SSH.

I don't have any particular need for extra tooling in the control container (other than SSH). The use cases I am trying to satisfy are emergency debugging ("escape hatch") and possibly Bottlerocket upgrades via API.

All that said, if it is indeed "just as secure" to run the admin container with SSH daemon at all times that is certainly much simpler for a variety of reasons. I am not sure what extra privileges the admin container has over a standard AL2 instance ec2-user account for example, so it is unclear to me if this is really true. However it would be much easier to sell a "two-hop" SSH flow (bastion->admin container) than a "three-hop" SSH flow (bastion->control container -> admin container) to my team.

#1227 looks like a duplicate of the question I asked in the first post, so maybe it makes sense to close or rename this issue to avoid confusion.

[bcressey]

If we'd had apiclient exec at the outset, we likely would've removed the conflation between "SSH access and superpowers" that exists in the admin container as it's currently formulated.

Instead we might have had "sshd" and "ssm" containers, each of which provided an entry point via the preferred remote management protocol, and the "admin" container wouldn't be remotely reachable. Under that scheme, after logging in via method of choice, the user would run:

enable-admin-container
apiclient exec admin bash

The appeal of that approach is just consistency in the overall break-glass workflow. You could create your own "sshd" and "admin" containers based on that scheme, but as you note there's not a big difference between the powers granted to ec2-user logging into the admin container on a Bottlerocket host, vs. ec2-user logging into ssh on a standard AL2 instance. Both can use sudo without a password and run in something like an unconfined SELinux domain.

The overall design goal is that gaining "superpowers" should require an intentional step, because you can use those powers to break the host in a way that's not otherwise possible. For example, directly modifying files in the host's /etc or configuration snippets in /var/lib/bottlerocket.

It's not a security boundary; processes in a "regular" host container have access to the API and can create "superpowered" host containers as they like. It's more like a support boundary, though not a hard and fast one. It's similar in my mind to "unsafe" code in Rust - if you modify the host via superpowered container, you have to uphold the same invariants that the host programs expect, or the host might fail at runtime or on upgrade.

We're about to ship support for registry credentials and you'd asked about passing registry credentials securely. We can write up a discussion Q&A about this if it's helpful, but the quick version would be to use a bootstrap container marked as essential and set to run once, which could retrieve the credentials from an SSM secure string parameter and then push them into the API.

[stmcginnis]

I think the mechanisms exist for credentials, and I think the open questions have been answered. Not sure if there are any foundational changes needed in Bottlerocket, but if so, please file a new issue that details the missing requirements.