Is a data disk size of 20gb required?

[andreialecu]

I'm using Bottlerocket with AWS ECS to orchestrate EC2 Spot instances.

I noticed each instance has two EBS volumes, one 2GB and the other 20GB.

My question is if the data volume is not unnecessarily large at 20GB, as this contributes to some costs for our AWS bill.

As a test, I reduced it to 10GB in the block devices configuration of our launch template and everything works just fine.

What am I risking? Can someone provide some guidance?

Thanks!

[larvacea]

As you may have guessed already, this is the persistent, read-write storage for your instance. The 2GB volume is read only. We mount /var, /local, /opt, and /mnt on that larger read-write EBS volume as xfs file systems. Of those, I would expect /var/lib/containerd (EKS) or /var/lib/docker (ECS) to be the largest and most variable consumer (but my experience is limited and non-production). The most serious risk would be that the instance would crash or become nonresponsive, as with any disk-completely-full scenario, and the most likely cause would be something in your world causing dockerd to fill its data-root. I'm afraid I don't have any specific, actionable guidance, though.

[yeazelm]

My question is if the data volume is not unnecessarily large at 20GB, as this contributes to some costs for our AWS bill.
As a test, I reduced it to 10GB in the block devices configuration of our launch template and everything works just fine.

To summarize @larvacea's comments: The root volume must be the 2GB in size, but the secondary volume only defaults to 20GB. If you can run your workload with smaller disks, you can override the default 20GB using block device mappings. In fact, many users of Bottlerocket set the secondary disk to be larger but you can also set it to smaller as you discovered. The primary concern is that this secondary disk holds nearly all variable data: so your logs, container images, etc all default to that volume. The size of this volume really is workload dependent so there isn't a general one-size-fits all recommendation but if you found it works with 10GB, you should be fine as long as you watch how much you are writing to that volume.

[larvacea]

The read-only root volume must be at least 2GB in size. I have seen instances where a customer has chosen a larger size, but I'd like to state here that this has no benefit: the instance will not use any extra capacity over the 2GB default. Resizing the data volume up or down does matter. It behaves somewhat like the root volume in a general-purpose Linux distro, and the same considerations apply. It is only somewhat-like, I should say. For instance, in most distros, /etc is part of the root volume and persists between boots, but in Bottlerocket /etc is ephemeral (a tmpfs in-memory filesystem), and is rebuilt during each boot. This surprises me more often than I'd like, even though I've seen how systemd builds it and I should know better.