Is the complexity of this module getting too high?

[max-rocket-internet]

A general question for users and contributors of this module 🙂

My feeling is that complexity getting too high and quality is suffering somewhat. We are squeezing a lot of features in a single module. This is by far the most complex module in the Terraform AWS modules org. But perhaps I just feel the pressure since I'm a maintainer?

Some recent examples:

• Adding global create_eks variable: Add destroy time flag #580
• Adding Managed Node Groups: Support for EKS Managed Node Groups #602
• Use kubernetes provider: Use kubernetes provider to manage aws auth #355
• Add Windows support: Add Windows support #555

And now:

• Adding IRSA: Add IRSA support #632
• Adding Fargate profiles: Add Fargate Profiles Support #634

The amount of new resources being added to this module is just getting higher. And I don't see this slowing down as AWS is doubling down on EKS, as is everyone on k8s in general.

Don't get me wrong, these are all awesome contributions 💙 but I feel there's a price here. On every new release there's always bunch of new issues related to the recent changes.

Let me know your thoughts 😃

[RothAndrew]

Yep. I've felt this way for quite a while now. And it's already too late. Too many new "features" have been added without letting things settle down and stabilize.

And, it's not fair to you as the maintainer, or to me as the user trying to run this thing in production. I'm cringing right now thinking about upgrading my prod cluster to v7.x. There's just no way I can reasonably safely do it.

IMO: deprecate this module and make several smaller, focused ones. Think long and hard about each new addition. Solicit feedback from multiple sources before leaping into a new feature. Even if somebody shows up with a PR with it all done if it ups the complexity of the module that should be considered carefully.

Another idea I have is to stop trying to support windows. There are unfortunately instances where a local_exec needs to run, and lots of complexity has been added trying to support windows alongside *nix. Instead, we can say "this is supported by linux (and probably mac). If you are on Windows, here is instructions on how to run this thing in a docker container.

[RothAndrew]

Examples of smaller, focused modules:

1. Worker group module, separate from the main module(s)
2. Main EKS Cluster module
3. Fargate k8s module

[RothAndrew]

At a bare minimum, documentation should be added for each new feature. On top of just adding the new params in the table. But I don't think that's going far enough to solve the predicament this module is already in.

[alaa]

I agree with ditching the support for windows as local_exec is required sometimes. and every new feature should have its own documentation, params, examples, and switch flag.

[kamirendawkins]

First, thank you @max-rocket-internet for maintaining this module for everyone.

As far as complexity I would agree this module is certainly complex but I would not expect much less when trying to maintain both halves of a managed Kubernetes Cluster. It is also very reasonable to assume it will only get more complex based on the AWS roadmap.

As for paths forward I agree with @RothAndrew and @alaa in regards to documentation and dropping windows support for now.

As for the module itself, I would hate to see this module depreciated after seeing it come this far. It would be better in my opinion to determine the direction it should go and instead work towards that as a community. It certainly does make sense to shard the module out in a way but unfortunately no matter how you look at it they will always be co-dependent on one another. I struggle to visualize a world where I would use an eks-fargate module without using eks-cluster along side it. That being said separating cluster configuration from worker configuration by means of a sub module may be a more manageable way to handle feature creep.

To add to this, if there is any way I as an individual can lend a hand I would certainly love to be more involved.

[dpiddockcmp]

I'm going to voice the other side of the no-Windows argument: local-exec should be banned from within the module now that's its gone:

• Its use makes supporting Windows difficult. Both for maintainers blindly guessing at support and for users
• It requires installing extra tools. We heavily use terraform inside short lived CI containers. Maintaining extra CI images is a pain. Or we waste time every run installing extra commands that might not be called. Ideally we should only need pure terraform.
• They have long been a source of issue and bugs.
• local-exec also makes the module difficult to use on terraform cloud where extra tools are also not available. You need more local-exec hacks to get them installed first. Support for aws-auth in terraform enterprise #226
• I don't think there's anything on master right now that's blocking Windows users?

Maybe splitting up in to sub modules would isolate some of the complexity rather than running as a single flat layer? Similar to how the RDS module is designed. Upgrade paths will be painful to get to that state, of course. And the aws-auth map will always add complexity for submodules that create workers.

Submodules could either be directly used by the top level as required (RDS style) or as version linked partner modules that users include as necessary, similar to how security-group works. Would need much documenting and examples in the second use case though due to aws-auth.

I think trying to coordinate stand-alone intertwined modules will be a maintenance and user nightmare. But it's not something I've tried.

Burning the module and starting again will present its own problems. There will still need to be some upgrade path for users who used this module. aws_eks_cluster is not a flexible resource. The master IAM role name is an example of something people wanted to change but existing users are locked in to.

Could something be done to increase release velocity? But without swamping Max with administrative tasks?

• master only receives bug fixes for stable features. And regular point releases. The VPC and RDS modules receive a point release after every merge of bug fixes or trivial changes.
• exciting new or breaking features, like kubernetes provider, are kept in a branch for testing and fixing. Before finally merging to master once settled enough and a major release.

[max-rocket-internet]

Burning the module and starting again will present its own problems

I would hate to see this module depreciated after seeing it come this far

To be clear, I'm not thinking we should do either of these, I was just thinking perhaps we could restructure or split parts of it 🙂

[RothAndrew]

Another suggestion: start rejecting PRs that want to introduce breaking changes that aren't providing great documentation on how to upgrade and preferably automated migration.

[deanrock]

Splitting it up into submodules like RDS does seems like a really good idea. However, unless Terraform supports easy migration of resources that I don't know of, it seems like this would require a significant effort to migrate existing projects to the new version of the module. 😞

Even if it gets split, some PRs, such as #555 and #580, would require similar changes as they did now, since they affect multiple modules.

At the moment, I'm really interested in IRSA and Fargate support, since we currently resort to specifying those resources outside of the module. Maybe we can start with adding those two as a separate submodules? They don't depend on much apart from eks instance.

If I can help with getting these features merged (e.g. help with splitting, writing examples), please let me know.

[peterloron]

While refactoring projects because the eks module changes (or goes away) will be a pain for those who depend on it, the module should not be held prisoner by it. A possible path: deprecate the current module, and fix only security and critical functionality problems. Create new module(s) which break up the parts (eks, node groups, etc). This will avoid breaking existing users, but will make a cleaner path forward. Existing users can pick when they spend the time to switch to the new hotness.

[nauxliu]

Deprecate this module will be a nightmare for existing users, I can't imagine i need to delete current clusters then recreate them using new modules or import all resources to new modules.I'll feel less pain if we could continue to maintain this module and deprecate and move features to smaller modules gradually.

[max-rocket-internet]

OK here's my thoughts...

IMO: deprecate this module and make several smaller, focused ones.

This was my initial thought also.

Solicit feedback from multiple sources before leaping into a new feature. Even if somebody shows up with a PR with it all done if it ups the complexity of the module that should be considered carefully.

I like this idea but there's no one to do it apart from maintainers. People want their PRs merged but they rarely want to review other PRs (apart from @dpiddockcmp)

I agree with ditching the support for windows

Cool. I think so too. But this only solves local_exec issues AFAIK.

Maybe splitting up in to sub modules would isolate some of the complexity rather than running as a single flat layer?

Yes. I was thinking perhaps modules for:

• The cluster including aws-auth configmap
• Fargate
• Managed Nodes Groups
• Normal worker groups

I think trying to coordinate stand-alone intertwined modules will be a maintenance and user nightmare

I think so too. I don't have enough time for this module let alone a bunch of others.

Could something be done to increase release velocity? But without swamping Max with administrative tasks?

We need more maintainers. It's as simple as that. I have less time these days.

Another suggestion: start rejecting PRs that want to introduce breaking changes that aren't providing great documentation

That's a great idea.

At the moment, I'm really interested in IRSA and Fargate support, since we currently resort to specifying those resources outside of the module.

Sure but AFAIK IRSA is only a single resource? https://github.com/terraform-aws-modules/terraform-aws-eks/pull/632/files

Maybe we can start with adding those two as a separate submodules?

@antonbabenko please chime in. I think we should consider creating some new modules to split out some of the functionality of this module. Personally I think separate github repos but not really up to me.

[antonbabenko]

Here are my 5 cents, though I have not used or followed the development of this module closely for some time - I feel that everyone who commented on this page is already on the same page (lol).

The complexity of this repository is large but still controllable because of the good patterns used here. Good job, everyone!

I don't think that splitting this module into submodules or separate repos for Fargate, Managed Nodes Group, Normal Worker Groups, etc will give any benefits to users and maintainers who will have to manage similar EKS-related-things in multiple places.

I also don't think we need to add more maintainers, because most of the time is usually spent on answering issues and triaging bugs, and this can be done by anyone. Adding more people will not increase velocity, but releasing more often (after every PR is merged) will.

If there is not an obvious feature, consider adding examples show-casing it. Same for bugs if they keep reoccurring.

[barryib]

To me this module is quite complex, but it's still maintainable and readable.

I'm not against splitting this module into submodules, but I'm against any kind of code duplication and the really annoying migration path. Furthermore, I'm not really confident this will help.

Thinking out loud, I would say we have a lack of automated tests which can tell us easily if a new feature introduce bugs or regressions. If a change breaks something. Features will came quickly as kubernetes and EKS are evolving really fast, and people will always want to get those feature as quick as possible. There is no way to do that without automation. I know it's a long way to go, and I don't really know how we can achieve that goal with free resources, but I think it's worth digging a little deeper to get confident on changes we make on this module.