Draft: Implement faster xml_template mode using NetBox inventory

[ypid]

I implemented a different way how to put together the XML that you might be interested in. The main reason is scalability, especially when you want to configure many VLANs, interfaces and so on, running hundreds or thousands of Ansible tasks was slow, at least when I wrote the xml_template mode in 2021-05.

With this mode, I am able to generate my fw config in 30 seconds.

Another interesting feature this offers: The generated config can include parts of the device config. It works like this:

    {{ lookup("xmlfile", local_config_path_fetch, xpath="//opnsense/filter/rule[not(contains(descr, 'Ansible managed.'))]") }}

It is currently a draft. If you are interested, I can get it polished, update the template based on my private template which is updated to the latest OPNsense release (24.7.x). What do you say?

I had this PR open against https://github.com/naturalis/ansible-opnsense but it was deleted sadly.

[zerwes]

Hello @ypid

Thank you for your interest in improving this role.

I can vaguely remember your interesting PR in the original naturalis repo of this playbook.
Unfortunately the people there abandoned the repo after a time of inactivity (#2 and #35).

Well, your idea sounds quite interesting.
I know the issues using the playbook with many interfaces/vlans/aliases/rules.
For example I have here a 3-stage fw setup with nearly 100 VLANs/interfaces and accordingly many aliases and rules. A ansible run, even with selective tasks, takes time, not to say it is a pain ...

So I am interested in speeding it up ...

I was thinking about some changes too, using jinja templates or the opnsense api ...

Well, the templates have their own order of elements and opnsense has its own, so I have refrained from going this way, as I considered this a far to complex and error prone process. I think with jinja templates the config will be often have changes in the structure without real changes to the content.

Using the opnsense api ... a long and winding road to that goal ...
Would be nice to just use the opnsense internal mechanism and not to reinvent the wheel ..
But the API is AFAIK still not yet covering all features.
And regarding the speed: a invocation of the API for each element over a sometimes unreliable connection (we have several offices spread over the world) will definitely be even slower then the current implementation.
Maybe the mixture of paring the current cfg using xml tools and just calling the API for required changes would be a viable way ...

Well ... much speak about it ... just put it in a nutshell: I see the need for a different implementation / improvements, have some ideas but see some issues ...
Yes, I am interested in this.

I have only taken a rough look at, do not yet see through all the changes, but I am willing to try it out ... but I have to admit our/my time resources are very limited, so I can not promise when I can have a first test run.

I can see you have addressed already some issue regarding #12
And it looks like you cover some parts of https://github.com/Rosa-Luxemburgstiftung-Berlin/ansible-opnsense-plugpack ... yes, could be a good idea to merge the roles ...

Until then, I would be very happy if you can document the changes, so we can test the new features easily. Especially: will there be some way to convert existing VARs driving the current playbook to the netbox inventory?
So please be so kind and provide us with some hints regarding the way we can test this.

Using FQCN notation for the ansible modules would be great too (you can do this very fast using https://github.com/zerwes/ansible-fqcn-converter) and usning ansible-lint for the changes too.

Greetings
Klaus

[zerwes]

I am currently on vacation and will take care of it when I get back ...

[ypid]

Enjoy! I still have quite some work to do on this pull request (update the template, see if I can make NetBox optional, write basic docs). Thanks already for your long replay and that you like the idea in general.

[ypid]

I have played with this more and there is also another route I will evaluate. How to Generate Device Configurations with NetBox. So the heavy lifting of generating the config.xml would happen in NetBox where I have easy access to all interfaces, IP addresses, VLANs, Virtual IPs, CARP and so on. I came to this conclusion while attempting to switching to FHRP Group for my firewall cluster. Accessing this related info within Configuration Templates is trivial. But with Ansible, it is a pain.
This should accelerate config generation quite a bit because it avoids lookups/roundtrips to NetBox that currently Ansible does on each template rendering bringing it down below 1 second.

hostname {{ virtualmachine.name }}

{% for interface in virtualmachine.interfaces.all() %}
{{ interface.fhrp_group_assignments.first() }}
{% endfor %}

PS: This repetitive objects in NetBox like IP addresses, VLANs, CARP should itself managed by something like Terraform modules to lifecycle management.