AWS-independent ssh mechanism or documentation

[springroll12]

What I'd like:

With the understanding that connecting directly to a shell on the node is against the design philosophy of Bottlrocket, it would still be nice to have a way to ssh into Bottlerocket control or admin containers that doesn't involve SSM or custom setup. I realize this is a "feature", but I feel there is some DX that could be improved. It is very hard to sell Bottlerocket to a team if the "escape hatch" of sshing into an instance involves many steps and AWS-specific tools.

This may be only a documentation issue, but the only way I've gathered to do this is to create your own control container with an SSH server in it, however it is unclear if this will work to connect from there to the admin container. Related to this issue which seems to imply this is not simply a documentation problem #367 .

Any alternatives you've considered:

• Building a custom control container and adding a bastion host
• Moving to standard AmazonLinux2 nodes

[tjkirch]

Hi @springroll12, thanks for your feedback!

You’re right, there are still some AWS dependencies at the moment. The default admin (SSH) container is stored in AWS’s container registry, ECR. However, if you’re running an aws-* variant of Bottlerocket, you shouldn’t need SSM or a custom setup to connect to it, just an SSH client and the SSH key you gave when launching the Bottlerocket instance. (I’m assuming you enabled the admin container and have network access to the instance, as described in the quickstart guide.)

Let us know if you’re having trouble connecting. We’d like to hear more about your use case, too. It’s always an option to customize the containers used, but we want to make common use cases easy.

That said, we’re currently working on removing AWS dependencies and supporting other platforms. We have open PRs to allow specifying the SSH key in user data (here) and then generate default user data when in AWS (here). The goal is to allow easier integration with other platforms by separating those concerns. We also recently added a vmware-dev variant as the first step toward a supported non-AWS build, and further work on that project will involve other improvements related to the host containers.

[springroll12]

Thanks. After seeing the ssh example you linked it is more clear to me how to provide the userdata I need to enable the admin container by default (with eksctl). With nodes provisioned in a private subnet (recommended eks setup), I imagine I'll need to connect through a bastion host of some kind.

If I wanted to not have SSH enabled on my nodes by default, would the correct approach be to enable the control container on my nodes, then enable the admin container from there? It seems like this still requires SSM, but I imagine a custom control container with an SSH daemon could solve this. Maybe the best approach for my use case (ad-hoc "escape-hatch") is to have a control/admin containers with SSH deployed with a manifest, rather than through eksctl or Bottlerocket API.

It's great to see progress being made on genericizing(?) Bottlerocket! I am very glad to see that the community is interested in bringing this project into the mainstream. 👍

[tjkirch]

If I wanted to not have SSH enabled on my nodes by default, would the correct approach be to enable the control container on my nodes, then enable the admin container from there?

Yeah, we have an enable-admin-container helper script in the control container to try to make that easy. (The control container is enabled by default.) We also have an issue to figure out easier access to the admin container from the control container, without having to jump outside of the host.

It seems like this still requires SSM, but I imagine a custom control container with an SSH daemon could solve this.

Yeah, given that you're in a private network and would be accessing it through a bastion host anyway, it wouldn't increase your risks too much. You could use it to enable the admin container, if your goal is occasional troubleshooting.

Maybe the best approach for my use case (ad-hoc "escape-hatch") is to have a control/admin containers with SSH deployed with a manifest, rather than through eksctl or Bottlerocket API.

One difference with that approach is that it would no longer be running as a host container and so it wouldn't have easy access to the "superpowered" flag that gives extra privileges and access to the host filesystem. (The exact changes are here.) You could request privileged access in the manifest, along with appropriate mounts and the super_t label, to get some of that back. The other difference is that the container would be running on the orchestrator's instance of containerd, whereas host containers have their own instance of containerd for reliability and separation purposes. It all depends on your goals for these containers.

[stmcginnis]

I believe this had been addressed and the docs have been improved, so going to close this one out. Please feel free to reopen or file a new issue if there are still concerns.