AttributeError: 'NoneType' object has no attribute '__version__'

[ojc97]

SUMMARY

When using the community.postgresql.postgresql_user or community.postgresql.postgresql_schema ansible modules, I am facing a python stacktrace which has the error "AttributeError: 'NoneType' object has no attribute 'version'".

I have looked online and seen that this could be due to 'psycopg2' being missing but I have added an ansible step to ensure this is present.

Any help greatly appreciated.

ISSUE TYPE

• Bug Report

COMPONENT NAME

community.postgresql.postgresql_user
community.postgresql.postgresql_schema

ANSIBLE VERSION

ansible [core 2.12.4]
  config file = None
  configured module search path = ['/Users/olliecole/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/homebrew/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/olliecole/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/homebrew/bin/ansible
  python version = 3.9.12 (main, Mar 26 2022, 15:44:31) [Clang 13.1.6 (clang-1316.0.21.2)]
  jinja version = 3.1.1
  libyaml = True

COLLECTION VERSION

# /Users/olliecole/.ansible/collections/ansible_collections
Collection           Version
-------------------- -------
community.postgresql 2.1.3  

# /opt/homebrew/lib/python3.9/site-packages/ansible_collections
Collection           Version
-------------------- -------
community.postgresql 1.7.1  

CONFIGURATION

~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
(END)

OS / ENVIRONMENT

MacOS 12.3.1
Database is on AWS RDS Postgres.

STEPS TO REPRODUCE

---
- name: Create ansible_tooling user, set password, grant privs
  community.postgresql.postgresql_user:
    login_host: "{{ cloud_database_postgresql_fqdn }}"
    port: "{{ tooling_ansible_database_postgresql_port }}"
    login_user: "{{ cloud_database_postgresql_master_username }}"
    login_password: "{{ cloud_database_postgresql_master_password }}"
    name: "{{ tooling_ansible_database_postgresql_username }}"
    password: "{{ tf_tooling_ansible.outputs.tooling_ansible_database_postgresql_password }}"
    role_attr_flags: "{{ tooling_ansible_database_postgresql_role_attr_flags }}"

EXPECTED RESULTS

Playbook should run and configure users.

ACTUAL RESULTS

The full traceback is:
Traceback (most recent call last):
  File "/Users/olliecole/.ansible/tmp/ansible-tmp-1651226309.226697-22461-187613661793656/AnsiballZ_postgresql_user.py", line 107, in <module>
    _ansiballz_main()
  File "/Users/olliecole/.ansible/tmp/ansible-tmp-1651226309.226697-22461-187613661793656/AnsiballZ_postgresql_user.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/Users/olliecole/.ansible/tmp/ansible-tmp-1651226309.226697-22461-187613661793656/AnsiballZ_postgresql_user.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.community.postgresql.plugins.modules.postgresql_user', init_globals=dict(_module_fqn='ansible_collections.community.postgresql.plugins.modules.postgresql_user', _modlib_path=modlib_path),
  File "/opt/homebrew/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/opt/homebrew/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/opt/homebrew/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/var/folders/dr/2fw5_ncd7hn_x18fyfcnvq700000gn/T/ansible_community.postgresql.postgresql_user_payload_hsbp_1tn/ansible_community.postgresql.postgresql_user_payload.zip/ansible_collections/community/postgresql/plugins/modules/postgresql_user.py", line 1038, in <module>
  File "/var/folders/dr/2fw5_ncd7hn_x18fyfcnvq700000gn/T/ansible_community.postgresql.postgresql_user_payload_hsbp_1tn/ansible_community.postgresql.postgresql_user_payload.zip/ansible_collections/community/postgresql/plugins/modules/postgresql_user.py", line 954, in main
  File "/var/folders/dr/2fw5_ncd7hn_x18fyfcnvq700000gn/T/ansible_community.postgresql.postgresql_user_payload_hsbp_1tn/ansible_community.postgresql.postgresql_user_payload.zip/ansible_collections/community/postgresql/plugins/module_utils/postgres.py", line 192, in get_conn_params
AttributeError: 'NoneType' object has no attribute '__version__'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/Users/olliecole/.ansible/tmp/ansible-tmp-1651226309.226697-22461-187613661793656/AnsiballZ_postgresql_user.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/Users/olliecole/.ansible/tmp/ansible-tmp-1651226309.226697-22461-187613661793656/AnsiballZ_postgresql_user.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/Users/olliecole/.ansible/tmp/ansible-tmp-1651226309.226697-22461-187613661793656/AnsiballZ_postgresql_user.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.community.postgresql.plugins.modules.postgresql_user', init_globals=dict(_module_fqn='ansible_collections.community.postgresql.plugins.modules.postgresql_user', _modlib_path=modlib_path),\n  File \"/opt/homebrew/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py\", line 210, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/opt/homebrew/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/opt/homebrew/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/var/folders/dr/2fw5_ncd7hn_x18fyfcnvq700000gn/T/ansible_community.postgresql.postgresql_user_payload_hsbp_1tn/ansible_community.postgresql.postgresql_user_payload.zip/ansible_collections/community/postgresql/plugins/modules/postgresql_user.py\", line 1038, in <module>\n  File \"/var/folders/dr/2fw5_ncd7hn_x18fyfcnvq700000gn/T/ansible_community.postgresql.postgresql_user_payload_hsbp_1tn/ansible_community.postgresql.postgresql_user_payload.zip/ansible_collections/community/postgresql/plugins/modules/postgresql_user.py\", line 954, in main\n  File \"/var/folders/dr/2fw5_ncd7hn_x18fyfcnvq700000gn/T/ansible_community.postgresql.postgresql_user_payload_hsbp_1tn/ansible_community.postgresql.postgresql_user_payload.zip/ansible_collections/community/postgresql/plugins/module_utils/postgres.py\", line 192, in get_conn_params\nAttributeError: 'NoneType' object has no attribute '__version__'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

[Andersson007]

@ojc97 hello, welcome to the project and thank you for reporting the issue!

I think it relates to psycopg2 (missed or i don't know why the interpreter doesn't see it), could you please recheck? Open the interpreter Ansible use on the target machine and try to run import psycopg2.

The issue helps spot another logical bug @hunleyd @jchancojr:

• In plugins/module_utils/postgres.py, we have the ensure_required_libs function which is called in the connect_to_db function.
• In the reported modules, psycopg2 is invoked in the get_ conn_params function which also lives in plugins/module_utils/postgres.py
• The problem is that get_conn_params is invoked in the modules earlier than connect_to_db and ensure_required_libs in it resp. In sort, now we check required libs after we call it the first time...

That's why it reports that object is None @ojc97. It should report that the lib is missed.

So the solution can be, for example:

1. Import ensure_required_libs in modules
2. Invoke it there before get_conn_params in all of the modules
3. Remove ensure_required_libs from connect_to_db
   Feel free to suggest your solution.

Does anyone want to pick it up? Should be an easy fix. I'd be happy to help if there are any issues.
What do you folks think?

[ojc97]

@Andersson007 First of all thanks for your swift reply!!

I have already installed psycopg2. I will try and run import psycopg2 now and report back shortly.

[ojc97]

@Andersson007 I assume this is what you meant.

Doesn't work as expected though.

➜  ~ python3                    
Python 3.9.12 (main, Mar 26 2022, 15:44:31) 
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/lib/python3.9/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: dlopen(/opt/homebrew/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_PQbackendPID'
>>> 

[Andersson007]

@Andersson007 I assume this is what you meant.

@ojc97 yep
If i understand correctly your controller is on MacOS and you want your playbook to run on your controller and connect to the remote postgres via psycopg2, correct?
If yes, i don't see in your example the task was delegated to your controller, e.g. via delegate_to: localhost.
Also in this case you need to install psycopg2 on your controller as it would be like an Ansible target host.
Did you do that?

[Andersson007]

in case described above, that task that installs psycopg2 should be also with delegate_to: localhost so it will install in on your controller.
If you did that, there's homebrew mentioned, i have no experience in working with it. So there's probably issues in the local configuration. Maybe you could try to install psycopg2 manually.

[Andersson007]

https://stackoverflow.com/questions/67013694/django-unable-to-load-psycopg2-module-python-is-also-unable-to-load-it-even-th
people recommend to run pip install psycopg2-binary

[jchancojr]

Does anyone want to pick it up? Should be an easy fix. I'd be happy to help if there are any issues. What do you folks think?

I can pick this up @Andersson007! Should be able to get to it ~today or this weekend.

[Andersson007]

@jchancojr thanks! This week or next one would be imo fine:) No rush with this

[Andersson007]

@ojc97 when you investigate what's the reason and how to fix it, please let us know.
It can help other users with a similar case find a solution.
If you have any questions, feel free to ask.

[ojc97]

@Andersson007 I have been looking into the reasons regarding this today and i am yet to have an answer...

It appears that lots of people have problems with psycopg2 on mac os. Even more with M1 ARM Macs.

If you have any recommendations, id love to hear them.

So far i have reinstalled, used psycopg2-binary instead, added exports as suggested here and some others...

[Andersson007]

@ojc97 as a workaround you could try to:

1. take a look at pyenv or
2. install a virtual machine with a linux distro, say, fedora or ubuntu, and run your controller in there

[mschwrdtnr]

@Andersson007 @ojc97 I am facing the same issue on a current CentOS machine. I think I have everything necessary installed such as:

• python3-devel
• libpq-devel
• psycopg2

python3 -c "import psycopg2; print(psycopg2.__version__)"
2.9.3 (dt dec pq3 ext lo64)

pg_config --version
PostgreSQL 13.5

but it still don't work...

[ojc97]

Hi all - @mschwrdtnr @Andersson007

I believe i have fixed the issues on my end...

• brew install openssl
• pip uninstall psycopg2
• pip install psycopg2-binary --no-cache-dir - the "no-cache-dir" was paramount.

Hope this helps.

[ojc97]

This appeared to be an M1 Mac specific issue.

[Andersson007]

@ojc97 thanks for the solution!
@mschwrdtnr could you please try 2 last steps of #252 (comment) ?
If it helps, maybe we should update the doc.

[blindrood]

I face the same issue on custom awx-ee. I can create container out of it and run

python
import psycopg2

with no issues but ansible job that calls postgresql_query is failing with same error

"module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 107, in <module>
 File \"<stdin>\", line 99, in _ansiballz_main
 File \"<stdin>\", line 48, in invoke_module
 File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code
    exec(code, run_globals)\n  File \"/tmp/ansible_postgresql_query_payload_mz1vgebp/ansible_postgresql_query_payload.zip/ansible_collections/community/postgresql/plugins/modules/postgresql_query.py\", line 510, in <module>
  File \"/tmp/ansible_postgresql_query_payload_mz1vgebp/ansible_postgresql_query_payload.zip/ansible_collections/community/postgresql/plugins/modules/postgresql_query.py\", line 404, in main
  File \"/tmp/ansible_postgresql_query_payload_mz1vgebp/ansible_postgresql_query_payload.zip/ansible_collections/community/postgresql/plugins/module_utils/postgres.py\", line 192, in get_conn_params\nAttributeError: 'NoneType' object has no attribute '__version__'\n",

[Andersson007]

@blindrood thanks for the feedback!
The module is just a python file which runs import psycopg2. The output says that the module can't do it.
The None object is there because we had a logical bug in the code (psycopg2 was invoked before checking it present).
After we release the collection containing #253, it will report "missing required library psycopg2".
Maybe #252 (comment) will help.

[blindrood]

@Andersson007 I understand but psycopg2 (or rather psycopg2-binary) is part of awx-ee image, it's being installed during its creation and is working as I can access it from container. You can find image here:
custom awx-ee image

[Andersson007]

@blindrood i don't know how it works, could you please explain if this image is used as a target host?
The library has to be installed on the target hosts. If this image is for controller (where Ansible itself lives), it won't work.

[blindrood]

This is ansible-runner custom image (https://github.com/ansible/ansible-runner)
build using ansible-builder (https://github.com/ansible/ansible-builder)

AWX runs ansible templates (playbooks) from inside such container to ensure reliability and reapitability.

Dockerfile is generated using builder and it basically installs OS packages, install pip packages and galaxy roles.

It was working for now but I had to recreate image due to issue with Cryptography and Openssl package issues after AWX upgrade.
After downgrading Cryptography by adding this to requirements file

cryptography>=35.0,<37.0.0
pyopenssl

I started facing issue with postgres templates that run postgresql_query module with error above

[Andersson007]

@blindrood thank you for the explanation!
I don't think postgres modules use those libraries, only psycopg2.
If you can edit your collection files in the image, the following can help debug the issue:

1. Find module_utils/postgres.py
2. Instead of the following at the top of the file

try:
    import psycopg2
    HAS_PSYCOPG2 = True
except ImportError:
    HAS_PSYCOPG2 = False

put

 import psycopg2
 HAS_PSYCOPG2 = True

3. Run again. As there will be not try/except, you should see the actual error

If it doesn't help, i would recommend to create an issue in ee repo

[Andersson007]

cryptography>=35.0,<37.0.0
pyopenssl

Maybe psycopg2 uses one/both of them as a dependency? I.e. the newer psycopg can't be installed. Try to find the dependency and downgrade psycopg2 to a correspond versions of the libs

[blindrood]

Hi, I followed Your suggestion and indeed this revealed that

ModuleNotFoundError: No module named 'psycopg2'

How I can overcome this during Docker image creation if part of it already is pip install psycopg2-binary ?

[Andersson007]

Hi, maybe something from https://stackoverflow.com/questions/64690093/why-is-installing-python-psycopg2-on-my-docker-image-not-preventing-the-subseque will help.
Anyway, if you discover a solution, please share it here. Will be helpful for other folks.

[blindrood]

Looks like I resolved issue on my side. Suspicious for me was that errors refer to python3.6 while I'm explicitly installing and using python3.8. dnf is not reporting python3.6 to be installed but files are there. Enabling debug in AWX I noticed that Ansible is refering to python at /usr/libexec/platform-python which is symbolic link to python3.6 binary. I changed this symlink to point to current pytho

ln -sf /usr/bin/python /usr/libexec/platform-python

and all started working. I think this is a bug in ansible-runner base image and I'll report this there.
Thanks for the support !

[Andersson007]

@blindrood thanks for the feedback!

I think this is a bug in ansible-runner base image and I'll report this there.

Does make sense.

You're welcome!

[Andersson007]

closed via #253