Configure HTTP-based ARP information fetching from Palo Alto PIO-OS firewalls using management profiles

NOTES.rst

@@ -8,6 +8,55 @@ existing bug reports, go to https://github.com/uninett/nav/issues .
 To see an overview of upcoming release milestones and the issues they resolve,
 please go to https://github.com/uninett/nav/milestones .
 
+NAV 5.12
+========
+Deprecation warnings
+--------------------
+.. warning:: The ``[paloaltoarp]`` section of :file:`ipdevpoll.conf`, used for
+             configuring HTTP-based ARP fetching from Palo Alto firewalls, is
+             deprecated and will be ignored in NAV 5.12 and future versions.
+             HTTP-based ARP fetching from Palo Alto
+             firewalls *must* now be configured using management profiles,
+             analogous to configuration of SNMP-based fetching.  :ref:`See below
+             for more details<5.12-new-http-rest-api-management-profile-type>`.
+
+.. _5.12-new-http-rest-api-management-profile-type:
+New way to configure fetching of Palo Alto firewall ARP cache data
+------------------------------------------------------------------
+.. NOTE:: Stated in the paragraphs below is the rationale for this
+          change. You can safely skip this reading and head
+          straight to the
+          :ref:`management profile reference documentation<http-rest-api-management-profile>`
+          for instructions on how to reconfigure your Palo Alto firewall
+          devices in NAV 5.12 to enable support for fetching of their
+          ARP information.
+
+NAV is often able to fetch ARP tables from devices (and do many other management
+tasks) using SNMP. However, some devices serve parts of their management-related
+interfaces such as ARP-table querying over alternative protocols that often
+demand a different set of configuration parameters than SNMP.
+
+Usually the act of configuring netboxes to enable NAV using some protocol, be it
+SNMP or another protocol, is done by creating a protocol-specific management
+profile with protocol-specific communication parameters, and assigning this
+profile to the relevant netboxes.
+
+An outlier to this configuration pattern in NAV 5.10 and NAV 5.11 was the
+configuration of parameters for Palo Alto firewall netboxes running PAN-OS, who
+serves ARP information over HTTP, because there was no management profile type
+for HTTP protocols.  Instead, the HTTP-protocol specific parameters
+for such netboxes needed to be supplied in the ``[paloaltoarp]`` section of
+:file:`ipdevpoll.conf` instead of the standard way of assigning the netbox
+a parameter-bearing management profile through the netbox's "edit netbox" SeedDB page.
+
+Starting with NAV 5.12, a new HTTP REST API management profile type has been
+added to NAV for configuring HTTP-specific parameters used in fetching of ARP
+information from Palo Alto firewalls running PAN-OS. Currently, this management
+profile type is only used to configure Palo Alto firewall devices. If support
+for other devices that similarly can be managed using HTTP is added to NAV in future
+releases, you can expect to be able to configure HTTP parameters for these
+devices the standard way by using management profiles as well.
+
 NAV 5.11
 ========
 

changelog.d/use-management-profiles-to-configure-paloaltoarp.md

@@ -0,0 +1,3 @@
+The ipdevpoll plugin to fetch ARP cache data from a netbox's Palo Alto firewall
+API is now configured through a new management profile type assigned to that
+netbox.

doc/reference/ipdevpoll.rst

@@ -106,35 +106,6 @@ Section [linkstate]
   The value ``any`` will generate alerts for all link state changes, but
   **this is not recommended** for performance reasons.
 
-Section [paloaltoarp]
----------------------
-
-This section configures the Palo Alto ARP plugin.  Palo Alto firewalls do
-support SNMP.  They do not, however, support fetching ARP cache data using
-SNMP.  This plugin enables fetching ARP records from Palo Alto firewalls using
-their built-in REST API.
-
-Currently, there is no management profile type for this type of REST APIs, so
-credentials to access a Palo Alto firewall's API must be configured in this
-section.
-
-If you have a Palo Alto firewall named ``example-fw.example.org``, with an IP
-address of ``10.0.42.42`` and a secret API token of
-``762e87e0ec051a1c5211a08dd48e7a93720eee63``, you can configure this in this
-section by adding::
-
-  example-fw.example.org = 762e87e0ec051a1c5211a08dd48e7a93720eee63
-
-Or, alternatively::
-
-  10.0.42.42 = 762e87e0ec051a1c5211a08dd48e7a93720eee63
-
-
-.. warning:: The Palo Alto ARP plugin does not currently verify TLS
-             certificates when accessing a Palo Alto API.  This will be changed
-             at a later date, but if it worries you, you should not use the
-             plugin yet.
-
 
 Job sections
 ------------

doc/reference/management-profiles.rst

@@ -86,7 +86,38 @@ Use keys
 Alternate port
   If access to the switch is not on the default port (22, in the case of the
   JunOS driver), put the alternate port here.
-  
+
 
 .. _`NAPALM`: https://napalm.readthedocs.io/en/latest/
 .. _`NETCONF`: https://en.wikipedia.org/wiki/NETCONF
+
+.. _http-rest-api-management-profile:
+HTTP REST APIs
+--------------
+As of NAV 5.12, HTTP REST API profiles are used to configure access to
+services of the following devices.
+
+`Palo Alto PAN-OS firewalls`_
+  A HTTP REST API profile is needed for NAV to access the firewall's ARP information.
+
+.. warning:: The Palo Alto ARP implementation in NAV does not currently verify TLS
+             certificates when accessing a Palo Alto API.  This will be changed
+             at a later date, but if it worries you, you should not configure
+             any netboxes to use the Palo Alto Arp service yet.
+
+.. image:: http-rest-api-profile-example.png
+
+If you have a Palo Alto firewall running on a netbox managed by NAV,
+with a secret API key of ``762e87e0ec051a1c5211a08dd48e7a93720eee63``,
+you can configure NAV to fetch ARP information from this firewall by
+creating a new management profile with
+
+* protocol set to ``HTTP REST API``,
+
+* API key set to ``762e87e0ec051a1c5211a08dd48e7a93720eee63``,
+
+* service set to ``Palo Alto ARP``,
+
+and then add this management profile to the netbox.
+
+.. _`Palo Alto PAN-OS firewalls`: https://docs.paloaltonetworks.com/pan-os/11-0/pan-os-panorama-api/pan-os-xml-api-request-types/configuration-api/get-active-configuration/use-xpath-to-get-arp-information

python/nav/ipdevpoll/plugins/paloaltoarp.py

@@ -28,6 +28,7 @@
 import xml.etree.ElementTree as ET
 
 from IPy import IP
+from nav.models.manage import Netbox, ManagementProfile, NetboxProfile
 from twisted.internet import defer, reactor, ssl
 from twisted.internet.defer import returnValue
 from twisted.web import client
@@ -39,65 +40,67 @@
 
 
 class PaloaltoArp(Arp):
-    configured_devices: dict[str, str] = {}
-
-    @classmethod
-    def on_plugin_load(cls):
-        """Loads the list of PaloAlto access keys from ipdevpoll.conf into the plugin
-        class instance, so that `can_handle` will be able to answer which devices
-        this plugin can run for.
-        """
-        from nav.ipdevpoll.config import ipdevpoll_conf
-
-        cls._logger.debug("loading paloaltoarp configuration")
-        if 'paloaltoarp' not in ipdevpoll_conf:
-            cls._logger.debug("PaloaltoArp config section NOT found")
-            return
-        cls._logger.debug("PaloaltoArp config section found")
-        cls.configured_devices = dict(ipdevpoll_conf['paloaltoarp'])
-
     @classmethod
+    @defer.inlineCallbacks
     def can_handle(cls, netbox):
         """Return True if this plugin can handle the given netbox."""
-        return (
-            netbox.sysname in cls.configured_devices
-            or str(netbox.ip) in cls.configured_devices
-        )
+        has_configurations = yield run_in_thread(cls._has_paloalto_configurations, netbox)
+        defer.returnValue(has_configurations)
 
     @defer.inlineCallbacks
     def handle(self):
         """Handle plugin business, return a deferred."""
-
-        api_key = self.configured_devices.get(
-            str(self.netbox.ip), self.configured_devices.get(self.netbox.sysname, "")
-        )
         self._logger.debug("Collecting IP/MAC mappings for Paloalto device")
 
-        mappings = yield self._get_paloalto_arp_mappings(self.netbox.ip, api_key)
-        if mappings is None:
-            self._logger.info("No mappings found for Paloalto device")
-            returnValue(None)
-
-        yield self._process_data(mappings)
-
-        returnValue(None)
+        configurations = yield run_in_thread(self._get_paloalto_configurations, self.netbox)
+#        api_keys = [config["api_key"] for config in configurations]
+        mappings = yield self._get_paloalto_arp_mappings(self.netbox.ip, api_keys)
+        if mappings is not None:
+            yield self._process_data(mappings)
 
     @defer.inlineCallbacks
-    def _get_paloalto_arp_mappings(self, address: str, key: str):
-        """Get mappings from Paloalto device"""
-
-        arptable = yield self._do_request(address, key)
-        if arptable is None:
-            returnValue(None)
+    def _get_paloalto_arp_mappings(self, ip: IP, api_keys: list[str]):
+        """
+        Get ARP mappings from Paloalto device
 
-        # process arpdata into an array of mappings
-        mappings = parse_arp(arptable.decode('utf-8'))
+        The Paloalto device is expected to give the same result for two correct but different keys in api_keys.
+        Hence, a request to the Paloalto device is made for each api key only until a successful response from the device.
+        """
+        mappings = None
+        arptable = yield self._do_request(ip, api_key)
+        if arptable is not None:
+            mappings = parse_arp(arptable.decode('utf-8'))
         returnValue(mappings)
 
-    @defer.inlineCallbacks
-    def _do_request(self, address: str, key: str):
-        """Make request to Paloalto device"""
+    @classmethod
+    def _has_paloalto_configurations(cls, netbox: Netbox) -> bool:
+        """
+        Make a blocking database request to check if the netbox has any
+        management profile that configures access to Palo Alto ARP data via HTTP
+        """
+        return NetboxProfile.objects.filter(
+            netbox_id=netbox.id,
+            profile__protocol=ManagementProfile.PROTOCOL_HTTP_REST,
+            profile__configuration__contains={"service": "Palo Alto ARP"},
+        ).exists()
+
+    @classmethod
+    def _get_paloalto_configurations(cls, netbox: Netbox) -> list[dict]:
+        """
+        Make a blocking database request that fetches all management profiles of
+        the netbox that configures access to Palo Alto ARP data via HTTP
+        """
+        return NetboxProfile.objects.filter(
+            netbox_id=netbox.id,
+            profile__protocol=ManagementProfile.PROTOCOL_HTTP_REST,
+            profile__configuration__contains={"service": "Palo Alto ARP"},
+        ).values_list("profile__configuration", flat=True)
 
+    @defer.inlineCallbacks
+    def _do_request(self, address: IP, key: str):
+        """
+        Make request to Paloalto device
+        """
         class SslPolicy(client.BrowserLikePolicyForHTTPS):
             def creatorForNetloc(self, hostname, port):
                 return ssl.CertificateOptions(verify=False)

python/nav/models/manage.py

@@ -131,10 +131,12 @@ class ManagementProfile(models.Model):
     PROTOCOL_SNMP = 1
     PROTOCOL_NAPALM = 2
     PROTOCOL_SNMPV3 = 3
+    PROTOCOL_HTTP_REST = 4
     PROTOCOL_CHOICES = [
         (PROTOCOL_SNMP, "SNMP"),
         (PROTOCOL_NAPALM, "NAPALM"),
         (PROTOCOL_SNMPV3, "SNMPv3"),
+        (PROTOCOL_HTTP_REST, "HTTP REST API"),
     ]
     if settings.DEBUG:
         PROTOCOL_CHOICES.insert(0, (PROTOCOL_DEBUG, 'debug'))
@@ -340,6 +342,12 @@ def get_preferred_snmp_management_profile(
         if profiles:
             return profiles[0]
 
+    def get_http_rest_management_profiles(self, service: str) -> models.QuerySet:
+        protocol = ManagementProfile.PROTOCOL_HTTP_REST
+        return self.profiles.filter(
+            protocol=protocol, configuration__contains={"service": service}
+        )
+
     def is_up(self):
         """Returns True if the Netbox isn't known to be down or in shadow"""
         return self.up == self.UP_UP

python/nav/web/seeddb/page/management_profile/forms.py

@@ -62,6 +62,28 @@ def _post_clean(self):
                 cfg[field] = self.cleaned_data.get(field)
 
 
+class HttpRestForm(ProtocolSpecificMixIn, forms.ModelForm):
+    PROTOCOL = ManagementProfile.PROTOCOL_HTTP_REST
+    PROTOCOL_NAME = PROTOCOL_CHOICES.get(PROTOCOL)
+
+    class Meta(object):
+        model = ManagementProfile
+        configuration_fields = ['api_key', 'service']
+        fields = []
+
+    api_key = forms.CharField(
+        label="API key",
+        help_text="Key/token to authenticate to the service",
+        required=True,
+    )
+
+    service = forms.ChoiceField(
+        choices=(("Palo Alto ARP", "Palo Alto ARP"),),
+        help_text="",
+        required=True,
+    )
+
+
 class DebugForm(ProtocolSpecificMixIn, forms.ModelForm):
     PROTOCOL = ManagementProfile.PROTOCOL_DEBUG
     PROTOCOL_NAME = PROTOCOL_CHOICES.get(PROTOCOL)