community.general.proxmox inventory plugin fails for a cluster setup in 9.3.0

[kennethso168]

Summary

When I tried to run a community.general.proxmox inventory plugin with 9.3.0, it fails with an exception. Rolling back to 9.2.0 works

Issue Type

Bug Report

Component Name

plugins/inventory/proxmox.py

Ansible Version

$ ansible --version
ansible [core 2.17.3]
  config file = /home/kenneth/.ansible.cfg
  configured module search path = ['/home/kenneth/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /var/home/kenneth/miniforge3/envs/ansible/lib/python3.12/site-packages/ansible
  ansible collection location = /home/kenneth/.ansible/collections:/usr/share/ansible/collections
  executable location = /var/home/kenneth/miniforge3/envs/ansible/bin/ansible
  python version = 3.12.5 | packaged by conda-forge | (main, Aug  8 2024, 18:36:51) [GCC 12.4.0] (/var/home/kenneth/miniforge3/envs/ansible/bin/python3.12)
  jinja version = 3.1.4
  libyaml = True

Community.general Version

$ ansible-galaxy collection list community.general

# /home/kenneth/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.general 9.3.0  

# /var/home/kenneth/miniforge3/envs/ansible/lib/python3.12/site-packages/ansible_collections
Collection        Version
----------------- -------
community.general 9.2.0

Configuration

$ ansible-config dump --only-changed
CONFIG_FILE() = /home/kenneth/.ansible.cfg
DEFAULT_VAULT_PASSWORD_FILE(/home/kenneth/.ansible.cfg) = /home/kenneth/.local/bin/vault-keyring-client.py
EDITOR(env: EDITOR) = /usr/bin/nano

OS / Environment

Bazzite (Fedora based) with ansible installed using conda

Steps to Reproduce

plugin: community.general.proxmox
url: https://pve.pitwinken.com
validate_certs: true
user: ansible@pve
token_id: ansible-homeserv-v2
token_secret: redacted
want_facts: true
want_proxmox_nodes_ansible_host: false
groups:
  pve_cts_prod_server: >-
    ('prod' in (proxmox_tags_parsed|list))
    and (proxmox_vmtype == 'lxc') and (proxmox_status == 'running')
  pve_vms_prod_server: >-
    ('prod' in (proxmox_tags_parsed|list) or 'pbs' in (proxmox_tags_parsed))
    and ('windows' not in (proxmox_tags_parsed|list))
    and (proxmox_vmtype == 'qemu') and (proxmox_status == 'running')

Run the following command

$ ansible-inventory -i inventory --graph -vvvv

Expected Results

Graph of inventory returned with no errors/exceptions

Actual Results

ansible-inventory [core 2.17.3]
  config file = /home/kenneth/.ansible.cfg
  configured module search path = ['/home/kenneth/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /var/home/kenneth/miniforge3/envs/ansible/lib/python3.12/site-packages/ansible
  ansible collection location = /home/kenneth/.ansible/collections:/usr/share/ansible/collections
  executable location = /var/home/kenneth/miniforge3/envs/ansible/bin/ansible-inventory
  python version = 3.12.5 | packaged by conda-forge | (main, Aug  8 2024, 18:36:51) [GCC 12.4.0] (/var/home/kenneth/miniforge3/envs/ansible/bin/python3.12)
  jinja version = 3.1.4
  libyaml = True
Using /home/kenneth/.ansible.cfg as config file
The vault password file /home/kenneth/.local/bin/vault-keyring-client.py is a client script.
Executing vault password client script: /home/kenneth/.local/bin/vault-keyring-client.py --vault-id default
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /home/kenneth/ansible-homeserv-v2/inventory/30-pve2.proxmox.yml as it did not pass its verify_file() method
script declined parsing /home/kenneth/ansible-homeserv-v2/inventory/30-pve2.proxmox.yml as it did not pass its verify_file() method
Loading collection community.general from /home/kenneth/.ansible/collections/ansible_collections/community/general
Using inventory plugin 'ansible_collections.community.general.plugins.inventory.proxmox' to process inventory source '/home/kenneth/ansible-homeserv-v2/inventory/30-pve2.proxmox.yml'
Trying secret ClientScriptVaultSecret(filename='/home/kenneth/.local/bin/vault-keyring-client.py', vault_id='default') for vault_id=default
toml declined parsing /home/kenneth/ansible-homeserv-v2/inventory/30-pve2.proxmox.yml as it did not pass its verify_file() method
[WARNING]:  * Failed to parse /home/kenneth/ansible-homeserv-v2/inventory/30-pve2.proxmox.yml with auto plugin: can only concatenate list (not "NoneType") to list
  File "/var/home/kenneth/miniforge3/envs/ansible/lib/python3.12/site-packages/ansible/inventory/manager.py", line 292, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/var/home/kenneth/miniforge3/envs/ansible/lib/python3.12/site-packages/ansible/plugins/inventory/auto.py", line 58, in parse
    plugin.parse(inventory, loader, path, cache=cache)
  File "/home/kenneth/.ansible/collections/ansible_collections/community/general/plugins/inventory/proxmox.py", line 685, in parse
    self._populate()
  File "/home/kenneth/.ansible/collections/ansible_collections/community/general/plugins/inventory/proxmox.py", line 643, in _populate
    name = self._handle_item(node['node'], ittype, item)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kenneth/.ansible/collections/ansible_collections/community/general/plugins/inventory/proxmox.py", line 558, in _handle_item
    self._get_lxc_interfaces(properties, node, vmid)
  File "/home/kenneth/.ansible/collections/ansible_collections/community/general/plugins/inventory/proxmox.py", line 371, in _get_lxc_interfaces
    ret = self._get_json("%s/api2/json/nodes/%s/lxc/%s/interfaces" % (self.proxmox_url, node, vmid), ignore_errors=[501])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kenneth/.ansible/collections/ansible_collections/community/general/plugins/inventory/proxmox.py", line 333, in _get_json
    data = data + json['data']
           ~~~~~^~~~~~~~~~~~~~
[WARNING]:  * Failed to parse /home/kenneth/ansible-homeserv-v2/inventory/30-pve2.proxmox.yml with yaml plugin: Plugin configuration YAML file, not YAML inventory
  File "/var/home/kenneth/miniforge3/envs/ansible/lib/python3.12/site-packages/ansible/inventory/manager.py", line 292, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/var/home/kenneth/miniforge3/envs/ansible/lib/python3.12/site-packages/ansible/plugins/inventory/yaml.py", line 113, in parse
    raise AnsibleParserError('Plugin configuration YAML file, not YAML inventory')
[WARNING]:  * Failed to parse /home/kenneth/ansible-homeserv-v2/inventory/30-pve2.proxmox.yml with ini plugin: Invalid host pattern 'plugin:' supplied, ending in ':' is not allowed, this character is reserved to provide a port.
  File "/var/home/kenneth/miniforge3/envs/ansible/lib/python3.12/site-packages/ansible/inventory/manager.py", line 292, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/var/home/kenneth/miniforge3/envs/ansible/lib/python3.12/site-packages/ansible/plugins/inventory/ini.py", line 137, in parse
    raise AnsibleParserError(e)
[WARNING]: Unable to parse /home/kenneth/ansible-homeserv-v2/inventory/30-pve2.proxmox.yml as an inventory source
setting up inventory plugins
host_list declined parsing /home/kenneth/ansible-homeserv-v2/inventory/70-static.yml as it did not pass its verify_file() method
script declined parsing /home/kenneth/ansible-homeserv-v2/inventory/70-static.yml as it did not pass its verify_file() method
Skipping empty key (vars) in group (openwrt)
Parsed /home/kenneth/ansible-homeserv-v2/inventory/70-static.yml inventory source with yaml plugin
@all:
...

I have tried some prelim debugging by adding print calls in the _get_json function:

    def _get_json(self, url, ignore_errors=None):

        if not self.use_cache or url not in self._cache.get(self.cache_key, {}):

            if self.cache_key not in self._cache:
                self._cache[self.cache_key] = {'url': ''}

            data = []
            s = self._get_session()
            while True:
                ret = s.get(url, headers=self.headers)
                if ignore_errors and ret.status_code in ignore_errors:
                    break
                ret.raise_for_status()
                json = ret.json()

                # process results
                # FIXME: This assumes 'return type' matches a specific query,
                #        it will break if we expand the queries and they dont have different types
                if 'interfaces' in url and 'lxc' in url: ## ADDED FOR DEBUGGING
                    print(url) ## ADDED FOR DEBUGGING
                    print(data) ## ADDED FOR DEBUGGING
                    print(json) ## ADDED FOR DEBUGGING
                if 'data' not in json:
                    # /hosts/:id does not have a 'data' key
                    data = json
                    break
                elif isinstance(json['data'], MutableMapping):
                    # /facts are returned as dict in 'data'
                    data = json['data']
                    break
                else:
                    # /hosts 's 'results' is a list of all hosts, returned is paginated
                    data = data + json['data']
                    break

            self._cache[self.cache_key][url] = data

        return make_unsafe(self._cache[self.cache_key][url])

Running over many times, I found that json['data'] is not None when the lxc container is running on pve2 (but not the other node pve1 in the same cluster. pve.pitwinken.com is a reverse proxy with load balancing pointing to one of the nodes. Currently it is reverse proxying from pve2.

...
data=[]
url='https://pve.pitwinken.com/api2/json/nodes/pve2/lxc/142/interfaces'
json={'data': [{'inet6': '::1/128', 'hwaddr': '00:00:00:00:00:00', 'inet': '127.0.0.1/8', 'name': 'lo'}, {'inet6': 'fe80::83:88ff:fede:bd5f/64', 'hwaddr': '02:83:88:de:bd:5f', 'inet': '10.151.40.42/24', 'name': 'eth0'}]}
data=[]
url='https://pve.pitwinken.com/api2/json/nodes/pve1/lxc/220/interfaces'
json={'data': None}
...

Further investigation leads to the discovery of bug in Proxmox API that the return data is none if the lxc container is running on other nodes of the same cluster. I have opened a bug report on Proxmox's bug tracker. But it'd be great if the plugin can handle json['data'] being None gracefully anyway.

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• plugins/modules/proxmox

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @Ajpantuso @Thulium-Drake @UnderGreen @joshainglis @karmab @krauthosting
click here for bot help

[felixfontein]

!component =plugins/inventory/proxmox.py

[felixfontein]

Yesterday @a17t created #8794 which might fix this problem (at least to me it looks related :) ).

[ansibullbot]

Files identified in the description:

• plugins/inventory/proxmox.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @ilijamt
click here for bot help

[kennethso168]

Yesterday @a17t created #8794 which might fix this problem (at least to me it looks related :) ).

I have tried his fix and that seems to workaround the issue.

I'll close this issue once that PR gets merged.