Handle openstacksdk < 0.10.0: fix AttributeError #50285

pilou- · 2018-12-24T01:06:22Z

SUMMARY

Handle openstacksdk < 0.10.0: fix AttributeError

Error was:

The full traceback is:
Traceback (most recent call last):
  File "$HOME/.ansible/tmp/ansible-tmp-1545612308.8-46792777824159/AnsiballZ_os_security_group.py", line 113, in <module>
    _ansiballz_main()
  File "$HOME/.ansible/tmp/ansible-tmp-1545612308.8-46792777824159/AnsiballZ_os_security_group.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "$HOME/.ansible/tmp/ansible-tmp-1545612308.8-46792777824159/AnsiballZ_os_security_group.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_os_security_group_payload_keFTIJ/__main__.py", line 163, in <module>
  File "/tmp/ansible_os_security_group_payload_keFTIJ/__main__.py", line 115, in main
  File "/tmp/ansible_os_security_group_payload_keFTIJ/ansible_os_security_group_payload.zip/ansible/module_utils/openstack.py", line 121, in openstack_cloud_from_module
AttributeError: 'module' object has no attribute 'version'

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

lib/ansible/module_utils/openstack.py

ADDITIONAL INFORMATION

ansibot · 2018-12-24T01:09:28Z

cc @Shrews @cloudnull @dagnello @emonty @evrardjp @juliakreger @mnaser @odyssey4me @rcarrillocruz
click here for bot help

opendev-zuul · 2018-12-24T01:39:11Z

Build succeeded (third-party-check pipeline).

openstacksdk-ansible-devel-functional-devstack : SUCCESS in 30m 23s

ansibot · 2019-04-07T03:30:07Z

cc @gtema
click here for bot help

dominikholler · 2019-04-18T16:37:03Z

lib/ansible/module_utils/openstack.py

+        module.fail_json(
+            msg="To utilize this module, the installed version of "
+                "the openstacksdk library MUST be >=0.10.0")
+


Why this code change is required?
Isn't this already checked in the lines above?

min_version also defaults to 0.12.0

Another version could be used.

When openstacksdk 0.9.19 is used (assuming caller is using a lower version than the default one), without the additional check, the following exception occurs:

Traceback (most recent call last): File "~/.ansible/tmp/ansible-tmp-1555747400.49-76880204403181/AnsiballZ_os_security_group.py", line 113, in <module> _ansiballz_main() File "~/.ansible/tmp/ansible-tmp-1555747400.49-76880204403181/AnsiballZ_os_security_group.py", line 105, in _ansiballz_main invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS) File "~/.ansible/tmp/ansible-tmp-1555747400.49-76880204403181/AnsiballZ_os_security_group.py", line 48, in invoke_module imp.load_module('__main__', mod, module, MOD_DESC) File "/tmp/ansible_os_security_group_payload_0EWiaz/__main__.py", line 163, in <module> File "/tmp/ansible_os_security_group_payload_0EWiaz/__main__.py", line 115, in main File "/tmp/ansible_os_security_group_payload_0EWiaz/ansible_os_security_group_payload.zip/ansible/module_utils/openstack.py", line 162, in openstack_cloud_from_module AttributeError: 'module' object has no attribute 'exceptions'

The current code isn't compatible with openstacksdk < 0.10.0.
Because this is a module_utils, other module not belonging to the ansible repository might use this method with another value than the default one.

@marsavela @dominikholler: please, could you take a look at this ?

dominikholler · 2019-04-30T08:36:14Z

lib/ansible/module_utils/openstack.py

@@ -114,16 +114,22 @@ def openstack_cloud_from_module(module, min_version='0.12.0'):
        # Due to the name shadowing we should import other way
        import importlib
        sdk = importlib.import_module('openstack')
+        sdk_version = importlib.import_module('openstack.version')


Why do you think this additional line is helpful?

Did you read the stacktrace mentioned in the description of this PR and in the first commit ?

sdk.version.__version__ expression raises an AttributeError exception when openstacksdk < 0.10.0 is used.

Did you read the stacktrace mentioned in the description of this PR and in the first commit ?

Sure, but I did not understand how this is related, but with your additional exlaination, I understand now.
The commit message could be improved by writing something like
"openstack.version.__version__ expression raises an AttributeError exception when openstacksdk < 0.10.0 is used. openstack.version is now imported as a module, which works for all openstacksdk versions.

PR updated:

commit message updated

rebased on devel

dominikholler · 2019-04-30T08:41:16Z

lib/ansible/module_utils/openstack.py

@@ -114,16 +114,22 @@ def openstack_cloud_from_module(module, min_version='0.12.0'):
        # Due to the name shadowing we should import other way
        import importlib
        sdk = importlib.import_module('openstack')
+        sdk_version = importlib.import_module('openstack.version')
    except ImportError:
        module.fail_json(msg='openstacksdk is required for this module')



The current code isn't compatible with openstacksdk < 0.10.0.
Because this is a module_utils, other module not belonging to the ansible repository might use this method with another value than the default one.

Sounds like min_version should be set to max(StrictVersion('0.10.0'), StrictVersion(min_version)) here.
This would avoid the redundancy below, too.

Some modules rely on this default value and don't work with 0.10.0.

For example using os_security_group:

with devel branch

the following diff applied
-def openstack_cloud_from_module(module, min_version='0.12.0'): +def openstack_cloud_from_module(module, min_version='0.10.0'):

openstacksdk==0.10.0 installed (same error occurs with 0.11.0)

this task:
- os_security_group: cloud: mycloud state: present name: foo

this exception occurs:

TASK [os_security_group] *** An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'Connection' object has no attribute 'current_project_id' fatal: [localhost]: FAILED! => { "changed": false, "rc": 1 } MSG: MODULE FAILURE See stdout/stderr for the exact error MODULE_STDERR: Traceback (most recent call last): File "$HOME/.ansible/tmp/ansible-tmp-1556617879.09-260636442237090/AnsiballZ_os_security_group.py", line 125, in <module> _ansiballz_main() File "$HOME/.ansible/tmp/ansible-tmp-1556617879.09-260636442237090/AnsiballZ_os_security_group.py", line 117, in _ansiballz_main invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS) File "$HOME/.ansible/tmp/ansible-tmp-1556617879.09-260636442237090/AnsiballZ_os_security_group.py", line 54, in invoke_module imp.load_module('__main__', mod, module, MOD_DESC) File "/tmp/ansible_os_security_group_payload_uLUg4r/__main__.py", line 163, in <module> File "/tmp/ansible_os_security_group_payload_uLUg4r/__main__.py", line 123, in main AttributeError: 'Connection' object has no attribute 'current_project_id'

Then if 0.10.0 is used by default, callers must be modified in order to pass 0.12.0 when required. Is that OK with you ?

Some modules rely on this default value and don't work with 0.10.0.

No doubts about this.

Then if 0.10.0 is used by default, callers must be modified in order to pass 0.12.0 when required. Is that OK with you ?

Sorry, I did not mean to use 0.10.0 as the default, but process the bigger value, either 0.10.0 or the parmeter value, which is 0.12.0 by default.
This way we would use 0.10.0 if < 0.10.0 is requested or 0.12.0 if the default parmater value is used. If the parameter is set explicitly to > 0.10.0, we would use this.
Is this explained understandable?

Yes, thanks!

PR updated.

opendev-zuul · 2019-04-30T10:41:55Z

Build succeeded (third-party-check pipeline).

openstacksdk-ansible-devel-functional-devstack : SUCCESS in 28m 51s

dominikholler

Thanks for the updates!
Did can you please write in a comment, for which versions of openstack sdk you verified this code?

dominikholler · 2019-04-30T10:56:49Z

lib/ansible/module_utils/openstack.py

@@ -114,15 +114,20 @@ def openstack_cloud_from_module(module, min_version='0.12.0'):
        # Due to the name shadowing we should import other way
        import importlib
        sdk = importlib.import_module('openstack')
+        sdk_version = importlib.import_module('openstack.version')
    except ImportError:
        module.fail_json(msg='openstacksdk is required for this module')

    if min_version:


Maybe we can remove the if min_version:? min_version has a useful default, and if min_version=None is passed, it is OK to explode, from my point of view.

It's not OK (see).

opendev-zuul · 2019-04-30T11:19:55Z

Build succeeded (third-party-check pipeline).

openstacksdk-ansible-devel-functional-devstack : SUCCESS in 31m 41s

pilou- · 2019-04-30T11:46:46Z

Did can you please write in a comment, for which versions of openstack sdk you verified this code?

0.9.0

$ pip list |grep openstacksdk
openstacksdk       0.9.0

TASK [os_security_group] 
fatal: [localhost]: FAILED! => {
    "changed": false
}

MSG:

To utilize this module, the installed version of the openstacksdk library MUST be >=0.12.

0.10.0

$ pip list |grep openstacksdk
openstacksdk       0.10.0

TASK [os_security_group] 
fatal: [localhost]: FAILED! => {
    "changed": false
}

MSG:

To utilize this module, the installed version of the openstacksdk library MUST be >=0.12.

0.11.0

$ pip list |grep openstacksdk
openstacksdk       0.11.0

TASK [os_security_group] 
fatal: [localhost]: FAILED! => {
    "changed": false
}

MSG:

To utilize this module, the installed version of the openstacksdk library MUST be >=0.12.

0.12.0

$ pip list |grep openstacksdk
openstacksdk       0.12.0

No error.

Manual unit test with openstacksdk 0.9.0 installed (these exceptions doesn't occur when using 0.12.0)

>>> from units.module_utils.urls.test_fetch_url import FakeAnsibleModule
>>> from ansible.module_utils.openstack import openstack_cloud_from_module
>>> try:
...     openstack_cloud_from_module(FakeAnsibleModule(), min_version=None)
... except Exception as e:
...   print(e.kwargs)
{'msg': 'To utilize this module, the installed version of the openstacksdk library MUST be >=0.10.'}
>>> try:
...     openstack_cloud_from_module(FakeAnsibleModule(), min_version='0.9.0')
... except Exception as e:
...   print(e.kwargs)
{'msg': 'To utilize this module, the installed version of the openstacksdk library MUST be >=0.10.'}

pilou- · 2019-05-03T16:11:22Z

bot_status

ansibot · 2019-05-03T16:20:44Z

Components

lib/ansible/module_utils/openstack.py
support: community
maintainers: Shrews cloudnull dagnello emonty evrardjp gtema juliakreger mnaser odyssey4me rcarrillocruz

Metadata

waiting_on: maintainer
changes_requested_by: null
needs_info: False
needs_revision: False
needs_rebase: False
merge_commits: []
too many files or commits: False
mergeable_state: clean
shippable_status: success
maintainer_shipits (module maintainers): 0
community_shipits (namespace maintainers): 0
ansible_shipits (core team members): 0
shipit_actors (maintainer or core team member): []
shipit_actors_other: []
automerge: automerge shipit test failed

click here for bot help

emonty · 2019-05-09T14:44:57Z

We should not attempt to handle openstacksdk < 0.10.0. The Ansible OpenStack modules have never supported <0.10.0 and should not attempt to start doing so now. People who want to use older versions of libraries with Ansible for out-of-tree drivers should be using the shade library, which is what was needed for the Ansible OpenStack modules before sdk 0.10.0. Even still, I'm not sure we even want to attempt to support 0.10.0 when the default min is 0.12.0. Let's just keep the default min as 0.12.

That said - I actually like most of this PR - since it prevents people from having modules that call this with a lower min_version than our actual min and gives them a better error message. Let's update it to use 0.12 instead of 0.12 and then I'm good with it.

emonty · 2019-05-09T14:45:47Z

lib/ansible/module_utils/openstack.py

-                msg="To utilize this module, the installed version of"
-                    "the openstacksdk library MUST be >={min_version}".format(
-                        min_version=min_version))
+        min_version = max(StrictVersion('0.10.0'), StrictVersion(min_version))


Let's change this to 0.12 to match our min.

emonty · 2019-05-09T14:45:56Z

lib/ansible/module_utils/openstack.py

-                        min_version=min_version))
+        min_version = max(StrictVersion('0.10.0'), StrictVersion(min_version))
+    else:
+        min_version = StrictVersion('0.10.0')


`openstack.version.__version__` expression raises an `AttributeError` exception when openstacksdk < 0.10.0 is used. `openstack.version` is now imported as a module, which works for all openstacksdk versions. Error was: The full traceback is: Traceback (most recent call last): File "$HOME/.ansible/tmp/ansible-tmp-1545612308.8-46792777824159/AnsiballZ_os_security_group.py", line 113, in <module> _ansiballz_main() File "$HOME/.ansible/tmp/ansible-tmp-1545612308.8-46792777824159/AnsiballZ_os_security_group.py", line 105, in _ansiballz_main invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS) File "$HOME/.ansible/tmp/ansible-tmp-1545612308.8-46792777824159/AnsiballZ_os_security_group.py", line 48, in invoke_module imp.load_module('__main__', mod, module, MOD_DESC) File "/tmp/ansible_os_security_group_payload_keFTIJ/__main__.py", line 163, in <module> File "/tmp/ansible_os_security_group_payload_keFTIJ/__main__.py", line 115, in main File "/tmp/ansible_os_security_group_payload_keFTIJ/ansible_os_security_group_payload.zip/ansible/module_utils/openstack.py", line 121, in openstack_cloud_from_module AttributeError: 'module' object has no attribute 'version'

opendev-zuul · 2019-05-09T16:24:34Z

Build succeeded (third-party-check pipeline).

openstacksdk-ansible-devel-functional-devstack : SUCCESS in 41m 37s

pilou- · 2019-05-09T17:23:36Z

@emonty thanks for the review !

PR rebased and updated following your feedback (openstack documentation fragment has been updated).

ready_for_review

emonty

shipit

gtema

shipit

pilou- · 2019-05-13T08:26:25Z

bot_status

ansibot · 2019-05-13T08:33:26Z

Components

lib/ansible/module_utils/openstack.py
support: community
maintainers: Shrews cloudnull dagnello emonty evrardjp gtema juliakreger mnaser odyssey4me rcarrillocruz

lib/ansible/plugins/doc_fragments/openstack.py
support: community
maintainers:

Metadata

waiting_on: maintainer
changes_requested_by: null
needs_info: False
needs_revision: False
needs_rebase: False
merge_commits: []
too many files or commits: False
mergeable_state: clean
shippable_status: success
maintainer_shipits (module maintainers): 1
community_shipits (namespace maintainers): 0
ansible_shipits (core team members): 1
shipit_actors (maintainers or core team members): emonty gtema
shipit_actors_other: []
automerge: automerge is_module test failed

click here for bot help

pilou- · 2019-05-14T16:49:49Z

@emonty please, could you manually merge this one ? automerge isn't available when module_utils paths are involved.

ansibot added the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Jan 1, 2019

pilou- mentioned this pull request Apr 18, 2019

OpenStack community pinboard ansible/community#447

Closed

ssbarnea approved these changes Apr 18, 2019

View reviewed changes

dominikholler reviewed Apr 18, 2019

View reviewed changes

ansibot removed the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Apr 29, 2019

dominikholler reviewed Apr 30, 2019

View reviewed changes

pilou- force-pushed the AttributeError_module_utils_openstack_module_version branch from 6d2f527 to f3e0a72 Compare April 30, 2019 10:11

pilou- force-pushed the AttributeError_module_utils_openstack_module_version branch from f3e0a72 to 970d508 Compare April 30, 2019 10:46

dominikholler suggested changes Apr 30, 2019

View reviewed changes

ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed community_review In order to be merged, this PR must follow the community review workflow. labels Apr 30, 2019

dominikholler approved these changes Apr 30, 2019

View reviewed changes

ansibot added community_review In order to be merged, this PR must follow the community review workflow. and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels Apr 30, 2019

emonty suggested changes May 9, 2019

View reviewed changes

pilou- added 3 commits May 9, 2019 17:35

Add missing space & dot in error message

a04031c

Always require openstacksdk >= 0.12

6e9b003

pilou- force-pushed the AttributeError_module_utils_openstack_module_version branch from 970d508 to 6e9b003 Compare May 9, 2019 15:36

ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed community_review In order to be merged, this PR must follow the community review workflow. labels May 9, 2019

ansibot added community_review In order to be merged, this PR must follow the community review workflow. and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels May 9, 2019

emonty approved these changes May 10, 2019

View reviewed changes

gtema approved these changes May 10, 2019

View reviewed changes

ansibot added shipit This PR is ready to be merged by Core and removed community_review In order to be merged, this PR must follow the community review workflow. labels May 10, 2019

emonty merged commit 4f25435 into ansible:devel May 24, 2019

ansible locked and limited conversation to collaborators Aug 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle openstacksdk < 0.10.0: fix AttributeError #50285

Handle openstacksdk < 0.10.0: fix AttributeError #50285

pilou- commented Dec 24, 2018

ansibot commented Dec 24, 2018

opendev-zuul bot commented Dec 24, 2018

ansibot commented Apr 7, 2019

dominikholler Apr 18, 2019

marsavela Apr 18, 2019

pilou- Apr 18, 2019

pilou- Apr 20, 2019

pilou- Apr 29, 2019

dominikholler Apr 30, 2019

pilou- Apr 30, 2019

dominikholler Apr 30, 2019

pilou- Apr 30, 2019

dominikholler Apr 30, 2019

pilou- Apr 30, 2019

dominikholler Apr 30, 2019

pilou- Apr 30, 2019

opendev-zuul bot commented Apr 30, 2019

dominikholler left a comment

dominikholler Apr 30, 2019

pilou- Apr 30, 2019

opendev-zuul bot commented Apr 30, 2019

pilou- commented Apr 30, 2019

pilou- commented May 3, 2019

ansibot commented May 3, 2019

emonty commented May 9, 2019

emonty May 9, 2019

emonty May 9, 2019

opendev-zuul bot commented May 9, 2019

pilou- commented May 9, 2019 •

edited

Loading

emonty left a comment

gtema left a comment

pilou- commented May 13, 2019

ansibot commented May 13, 2019

pilou- commented May 14, 2019

Handle openstacksdk < 0.10.0: fix AttributeError #50285

Handle openstacksdk < 0.10.0: fix AttributeError #50285

Conversation

pilou- commented Dec 24, 2018

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

ansibot commented Dec 24, 2018

opendev-zuul bot commented Dec 24, 2018

ansibot commented Apr 7, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opendev-zuul bot commented Apr 30, 2019

dominikholler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opendev-zuul bot commented Apr 30, 2019

pilou- commented Apr 30, 2019

pilou- commented May 3, 2019

ansibot commented May 3, 2019

emonty commented May 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opendev-zuul bot commented May 9, 2019

pilou- commented May 9, 2019 • edited Loading

emonty left a comment

Choose a reason for hiding this comment

gtema left a comment

Choose a reason for hiding this comment

pilou- commented May 13, 2019

ansibot commented May 13, 2019

pilou- commented May 14, 2019

pilou- commented May 9, 2019 •

edited

Loading