Change default inotify settings

[backjo]

What I'd like:

In #371, there was discussion about adopting the learning from From 30 to 230 docker containers per host . As part of this discussion, it looks like the inotify limits were discussed, but the default settings seemed to have been interpreted as what the settings should have been changed to ( see discrepancy in the settings here compared to the referenced article). Tuning these settings to the recommendations in the article:

fs.inotify.max_user_instances = 4096
fs.inotify.max_user_watches = 32768

will allow for more containers to run per node, assuming they're hitting the inotify limits already.
Any alternatives you've considered:
Building a custom image with these settings tweaked. This will always be something that heavy users of inotify will have to do, but tweaking the defaults will probably provide a better developer experience for your average operator.

[zmrow]

Hi @backjo ! Thanks for bringing this to our attention; I barely remember discussing those things in the very early days of Bottlerocket. :)

Are you running into these limits or having to tweak them on your end? Can you provide some additional details about your use case?

[backjo]

Our use case is more or less described by this post -> http://blog.travisgosselin.com/configured-user-limit-inotify-instances/.

Basically, we've got a bunch of .NET Core pods running across various K8S namespaces on our EKS cluster. Each of those pods shares a base container image and leverages the same user ('app'). If enough of these pods end up running on the same node, we start seeing failures due to hitting the inotify limit.

We can (and do) workaround this case in a few ways - per-app UIDs, do less file watches, etc - but I think a higher default value might make sense.

Note: max_user_watches was increased significantly in the AL2 EKS AMI a few months back - awslabs/amazon-eks-ami#589

[ehedei206]

Hi, is there an update on this? Is there any way we can change fs.inotify.max_user_instances through the bottlerocket api in the control container? or userData toml?

Thanks!

[duboisf]

I've encountered an issue where the default value of fs.inotify.max_user_instances was too low (128 in my case).

To reproduce this in kubernetes, I created a troubleshoot karpenter provisioner that would only provision largish Bottlerocket nodes, something like:

# ...
labels:
  dedicated: troubleshoot
# ...
- key: karpenter.k8s.aws/instance-size
  operator: In
  values:
  - 24xlarge

Then I deployed a simple app in an istio-enabled namespace, with 100 replicas and a node selector dedicated: troubleshoot. Not all the istio proxy sidecars become ready, roughly a third crash with a "too many open files" error. Increasing fs.inotify.max_user_instances to 1024 solved the issue here.

---

@ehedei206 to answer your question about if there's a way to fix this using user data, yes there is. I used the following user data for my Bottlerocket nodes in kubernetes:

[settings.kernel.sysctl]
"fs.inotify.max_user_instances" = "1024"
"fs.inotify.max_user_watches" = "32768"

EDIT: typos

[ehedei206]

Thanks @duboisf!