setup.py test fails with Python 3.6: AttributeError: module 'S3.Progress' has no attribute 'ProgressCR'

[ghost]

python3.6 -B setup.py test fails. I haven't checked other Python 3 versions. The error is as follows.

AttributeError: module 'S3.Progress' has no attribute 'ProgressCR'

I understand that no tests are run by setup.py. We are building s3cmd and executing tests in an automated way. We are currently working around this by disabling the tests. It would be nice if the problem can still be fixed.

The tests don't fail with Python 2.7.

As far as I can see the problem is due to a circular dependency: S3/Progress.py -> S3/Utils.py -> S3/Config.py -> S3/Progress.py.

Here is the full output.

$ tar xf s3cmd-2.0.2.tar.gz
$ cd s3cmd-2.0.2/
$ python3.6 -B setup.py config
Using xml.etree.ElementTree for XML processing
running config
$ python3.6 -B setup.py build
Using xml.etree.ElementTree for XML processing
running build
running build_py
creating build
creating build/lib
creating build/lib/S3
copying S3/Progress.py -> build/lib/S3
copying S3/ConnMan.py -> build/lib/S3
copying S3/Custom_httplib3x.py -> build/lib/S3
copying S3/SortedDict.py -> build/lib/S3
copying S3/FileDict.py -> build/lib/S3
copying S3/CloudFront.py -> build/lib/S3
copying S3/PkgInfo.py -> build/lib/S3
copying S3/HashCache.py -> build/lib/S3
copying S3/MultiPart.py -> build/lib/S3
copying S3/Exceptions.py -> build/lib/S3
copying S3/Config.py -> build/lib/S3
copying S3/S3.py -> build/lib/S3
copying S3/Utils.py -> build/lib/S3
copying S3/BidirMap.py -> build/lib/S3
copying S3/FileLists.py -> build/lib/S3
copying S3/ACL.py -> build/lib/S3
copying S3/Custom_httplib27.py -> build/lib/S3
copying S3/S3Uri.py -> build/lib/S3
copying S3/AccessLog.py -> build/lib/S3
copying S3/ExitCodes.py -> build/lib/S3
copying S3/__init__.py -> build/lib/S3
copying S3/Crypto.py -> build/lib/S3
warning: build_py: byte-compiling is disabled, skipping.

running build_scripts
creating build/scripts-3.6
copying and adjusting s3cmd -> build/scripts-3.6
changing mode of build/scripts-3.6/s3cmd from 644 to 755
$ python3.6 -B setup.py test
Using xml.etree.ElementTree for XML processing
running test
running egg_info
writing s3cmd.egg-info/PKG-INFO
writing dependency_links to s3cmd.egg-info/dependency_links.txt
writing requirements to s3cmd.egg-info/requires.txt
writing top-level names to s3cmd.egg-info/top_level.txt
reading manifest file 's3cmd.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 's3cmd.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
  File "setup.py", line 112, in <module>
    install_requires = ["python-dateutil", "python-magic"]
  File "/usr/host/lib/python3.6/site-packages/setuptools/__init__.py", line 131, in setup
    return distutils.core.setup(**attrs)
  File "/usr/host/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/host/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/host/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/host/lib/python3.6/site-packages/setuptools/command/test.py", line 228, in run
    self.run_tests()
  File "/usr/host/lib/python3.6/site-packages/setuptools/command/test.py", line 250, in run_tests
    exit=False,
  File "/usr/host/lib/python3.6/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/host/lib/python3.6/unittest/main.py", line 118, in parseArgs
    self._do_discovery(argv[2:])
  File "/usr/host/lib/python3.6/unittest/main.py", line 229, in _do_discovery
    self.test = loader.discover(self.start, self.pattern, self.top)
  File "/usr/host/lib/python3.6/unittest/loader.py", line 341, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/usr/host/lib/python3.6/unittest/loader.py", line 398, in _find_tests
    full_path, pattern, namespace)
  File "/usr/host/lib/python3.6/unittest/loader.py", line 475, in _find_test_path
    tests = self.loadTestsFromModule(package, pattern=pattern)
  File "/usr/host/lib/python3.6/site-packages/setuptools/command/test.py", line 54, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/host/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
  File "/home/markus/tmp/s3cmd-2.0.2/S3/Progress.py", line 14, in <module>
    import S3.Utils
  File "/home/markus/tmp/s3cmd-2.0.2/S3/Utils.py", line 51, in <module>
    import S3.Config
  File "/home/markus/tmp/s3cmd-2.0.2/S3/Config.py", line 85, in <module>
    class Config(object):
  File "/home/markus/tmp/s3cmd-2.0.2/S3/Config.py", line 101, in Config
    progress_class = Progress.ProgressCR
AttributeError: module 'S3.Progress' has no attribute 'ProgressCR'

Is I said, the steps don't fail with Python 2.7.

$ tar xf s3cmd-2.0.2.tar.gz
$ cd s3cmd-2.0.2/
$ python2.7 -B setup.py config
[...]
$ python2.7 -B setup.py build
[...]
$ python2.7 -B setup.py test
Using xml.etree.ElementTree for XML processing
running test
running egg_info
writing requirements to s3cmd.egg-info/requires.txt
writing s3cmd.egg-info/PKG-INFO
writing top-level names to s3cmd.egg-info/top_level.txt
writing dependency_links to s3cmd.egg-info/dependency_links.txt
reading manifest file 's3cmd.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 's3cmd.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

[ghost]

The reason for the failure seems to be that Python's unittest module imports each and every module it can find in order to find tests by name (see loadTestsFromName function in /usr/lib/python3.6/unittest/loader.py). S3.Progress simply can't be loaded as such.

$ pwd
/home/markus/tmp/s3cmd-2.0.2
$ python3.6
Python 3.6.5 (default, Jun 14 2018, 18:26:01) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import S3.Progress
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/markus/tmp/s3cmd-2.0.2/S3/Progress.py", line 14, in <module>
    import S3.Utils
  File "/home/markus/tmp/s3cmd-2.0.2/S3/Utils.py", line 51, in <module>
    import S3.Config
  File "/home/markus/tmp/s3cmd-2.0.2/S3/Config.py", line 85, in <module>
    class Config(object):
  File "/home/markus/tmp/s3cmd-2.0.2/S3/Config.py", line 101, in Config
    progress_class = Progress.ProgressCR
AttributeError: module 'S3.Progress' has no attribute 'ProgressCR'
>>>

If on the other hand, S3.Config is imported before, then S3.Progress can be imported.

$ pwd
/home/markus/tmp/s3cmd-2.0.2
$ python3.6
Python 3.6.5 (default, Jun 14 2018, 18:26:01) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import S3.Config
>>> import S3.Progress
>>>

[ghost]

Adding from . import Config to S3/__init__.py solves the problem from above.

Unfortunately, python3.6 -B setup.py test still doesn't pass.

python3.6 setup.py test
Using xml.etree.ElementTree for XML processing
running test
running egg_info
creating s3cmd.egg-info
writing s3cmd.egg-info/PKG-INFO
writing dependency_links to s3cmd.egg-info/dependency_links.txt
writing requirements to s3cmd.egg-info/requires.txt
writing top-level names to s3cmd.egg-info/top_level.txt
writing manifest file 's3cmd.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 's3cmd.egg-info/SOURCES.txt'
running build_ext
Custom_httplib27 (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: Custom_httplib27 (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: Custom_httplib27
Traceback (most recent call last):
  File "/usr/host/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
  File "/home/rothe/repos/s3cmd/S3/Custom_httplib27.py", line 4, in <module>
    import httplib
ModuleNotFoundError: No module named 'httplib'


----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)
Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0>

[fviard]

Having imports right for python3 is very tricky when we have multiple way to use a command/lib.

But I don't understand what you are trying to achieve, just don't use "setup.py test",
it doesn't make any sense at all because we are not using unittest.

If you want to perform some self test on s3cmd, there are 2 run_tests.py scripts dedicated to do that.
You can have a look at the travis recept to see how to perform this automatically on a fresh environement :
https://github.com/s3tools/s3cmd/blob/master/.travis.yml

It involves running a self contained server binary "minio" to act as a server.

[ghost]

Yes, I understand that not running the non-existing tests is a solution. Nevertheless, it wasn't easy for me to decide if setup.py test failed due to some real error or due to a problem with the unit tests.

In my opinion setup.py test shouldn't fail even though there are no tests.

When I get the output from Python 2.7 (Ran 0 tests in 0.000s [..] OK), I can easily determine that no tests were run and everything is as expected. When I use Python 3.6 I cannot be certain about the source of the error without reading the source code.

Each and every (Linux) distribution has to double check this, when packaging s3cmd.

Feel free to close this issue.

Also, thank you for writing and maintaining s3cmd. 👍

[fviard]

Fixed in MASTER now.