Execution environment support for community.general

[felixfontein]

This is something we would like to have (see for example #2968), but that's not so easy, since community.general has a huge amount of modules and plugins, all of them having potentially different requirements.

First, we have to classify modules and plugins into two categories:

1. Stuff that will not run in the execution environment (but on a remote machine). Examples are the timezone module.
2. Stuff that can or wil run in the execution environment (this includes all non-module plugins).

Only requirements for 2. have to be included in the execution environment dependency files. But just compiling a list of these plugins/modules and figuring out the required dependencies is already a huge task.

Proposition 1: Let's start with a subset, and try to increase it until everything is covered. This will be a community task resp. something for module maintainers to do, I'll do some initial work and cover some plugins / modules / module groups I'm interested in / know something about, but I won't finish this.

The second problem is that there isn't adequate tooling for such a large and diverse collection such as community.general. We do not want to have one bindep.txt and one Python requirements.txt file for the whole collections: first these cannot have comments, and second it will be a huge mess and nobody will remember which dependencies belong to which module. Sooner or later we will have some dependencies in there which are for modules/plugins that have long been removed from this collection (for whatever reason), and nobody will notice or be sure. To avoid this:

Proposition 2: Let's store these dependencies locally next to the modules and plugins (filenames *-bindep.txt or bindep.txt for system dependencies, and *-requirements.txt or requirements.txt for Python dependencies) in plugins/<plugin_type>/ or plugins/modules/, where the filenames starting with * are for individual modules/plugins or smaller groups of such plugins which do not have their own directory (for example: plugins/modules/files/xml-requirements.txt), and the standalone form of the filenames are for module groups in their own subdirectory (for example: plugins/modules/source_control/gitlab/requirements.txt). Then we add a tool (I'll happily create that one; it should be part of this repository) which creates the combined EE files from these individual fragments. This tool needs to be integrated in the release process and also into CI.

What do you think?

[felixfontein]

bot_skip

[russoz]

The one thing I dislike about both propositions is the fact that we have one place where requirements are meant to be listed and now we are creating another place where it also needs tobe listed. Keeping both in sync will be yet another task for the (module and collection) maintainers.

So, Proposition 3 from the Ideas-from-the-ideal-world dept, the requirements item in the documentation blob could look like:

requirements:
  controller:
    python: ">=3.8"
    python_requirements:
      - requests
      - some_pkg: ">3.2"
    bindeps:
      - libcurl
      - mysql-client
  managed:
    python: ">=2.6"
    python_requirements: ...
    bindeps: ...

(or bindeps could be the textual blob of the bindeps.txt file).

This bumps into the problem that https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_documenting.html#documentation-block describes requirements as:

• List of requirements (if applicable).
• Include minimum versions.

That could, perhaps, be solved by ansible-doc (and whatever other piece of code reading these blobs) checking whether the parsed value of requirements is a list (to keep compatibility) or a dict (to move forward to this better(!?) model). If list, do whatever we do today. If dict, then we can generate the HTML docs and EE files and whatever else we need.

Anyway, all propositions rely on the community maintainers adding the exact information to their plugins. Also, the resulting, effective, bindep for this collection, given its diversity, is bound to be quite bloated. The ideal solution, IMO, would be to have something that would run through the playbook and assemble the right dependencies for that playbook - or playbooks. That would require something like Proposition 2 or 3, a tool to list all the modules matching a collection prefix (e.g. community.general) out of a set of playbooks, and the tool from Prop. 2 to be able to read one such list and generate the EE specs for that list.

In summary, I think this is a problem for the Ansible team, not just for collection maintainers.

PS: I am not very familiar with how the EE model would work, so part of this might already exist, and/or I might be missing something important.

[felixfontein]

The one thing I dislike about both propositions is the fact that we have one place where requirements are meant to be listed and now we are creating another place where it also needs tobe listed. Keeping both in sync will be yet another task for the (module and collection) maintainers.

So, Proposition 3 from the Ideas-from-the-ideal-world dept, the requirements item in the documentation blob could look like:

Changing the format of that entry will be next to impossible (and will need a couple of years), since it affects multiple RedHat products and thus will take a looong time of preparations. (Believe me, already adding new semantic markup is taking forever because of that...) So while this is a nice idea, it's nothing that will happen anytime soon.

What we could do is create a new place for this documenation, and have a linter which ensures that the requirements in the plugins/modules is up-to-date with what's specified elsewhere.

It will likely be more complex than you proposed though, since we have some cases where requirements are a lot more complex (optional requirements, mutually exclusive requirements, requirements with strange conditions, ...).

In any case, I don't think it's worth to try to solve this in its full generality if we ever want to be able to ship something in say a few months. (It still would be nice to have something more general though...)

[russoz]

Fair enough. OTOH, whatever "quicker" solution we come up with will have to be maintained and will require backwards compatibility when "something more general" comes around, making it even harder to implement. I think we are feeding the beast.

Personally I think that selling long term support for a rapidly-evolving product is a good way to kill the product's rapid evolution. On the other other hand, I know there is a lot more complexity to that, as you said before. Sigh. I don't have any good solution to that.

---

That being said, I think proposition 2 is the less bitter pill to achieve the goal, but although we will be able to tick the box "c.g. has support for EE", it won't actually support it for a very long while (500+ modules, plus other plugin types).

It just does not feel like a good use of our already thinly stretched time.

[felixfontein]

The last week I got two more data points on EEs which let me doubt this whole thing is a really great idea:

1. Make EE a proper EE with dependencies installed ansible-community/images#22 (comment) - trying to build an EE out of the collections contained in Ansible 5.8 is already failing because some of the few collections which declare their dependencies have incompatible ones! This basically makes it impossible to build EEs with these collections, even if you just need content of these collections that do not actually need the libraries that trigger the error.

One could now argue that if we'd sick to more simple dependencies, with preferably no lower bounds, would solve this. But:

2. community.docker has two very simple Python dependencies: docker and docker-compose. Unfortunately the latter is incompatible with a dependency of ansible-lint, whence community.docker is now no longer part of the creator EE: Remove community.docker and docker from creator-ee ansible/creator-ee#68

So ... we'd have to restrict to really basic and simple dependencies, and nothing something which is remotely complex. Or, maybe better, not include any dependencies? Except maybe something very, very basic? But what would be safe?!

[felixfontein]

After thinking about this some more time, I think my personal conclusion is that community.general should not install anything into an EE except possibly very generic dependencies that cover a larger set of modules / plugins. To me, this would be someting like PyYAML or Jinja2, which both are already present in EEs since they contain ansible-core themselves. The only potential requirement I can think of that would make sense are things like lxml and requests. lxml has only four users in tihs collection though, and both lxml and requests can be installed in multiple ways, which can easily collide with how other collections prefer to install them, so it might be better to keep the EE creator specify such dependencies manually.

[russoz]

Short of analysing the actual playbook that will use the collection, we have no way of telling which dependencies will be really used. Trying to solve the runtime dependency puzzle during collection's development time is bound to be a nightmare.

Don't get me wrong, I love the idea of containerising the runtimes and isolate them, but I reckon that would work out better with a tool that would go through the playbooks we want to run, and compile the list of plugins used, and from that derive the depen... hey, wait! we do not have a standard/parseable way to declare plugins' dependencies, so it will depend on the developer anyways.

So at the end of the day, I'd say it is likely devs will build their own containers rather than relying on the collection's EE.