Switch to Python 3 (golemfactory#1128)

* wqRemove OpenEXR

Signed-off-by: Adam Banasiak <[email protected]>

* Update CI to Python 3.6

Signed-off-by: Adam Banasiak <[email protected]>

* 2to3 applied

Signed-off-by: Lukasz Foniok <[email protected]>

* Adding dependencies from devp2p.crypto

Signed-off-by: Lukasz Foniok <[email protected]>

* 2to3 on latest commits

Signed-off-by: Lukasz Foniok <[email protected]>

* Removing class advice, adds decorator

Signed-off-by: Lukasz Foniok <[email protected]>

* Changin keys.privtoaddr to privttoaddr

Signed-off-by: Lukasz Foniok <[email protected]>

* Forking pystun, fix keys encoding

Signed-off-by: Lukasz Foniok <[email protected]>

* Fixes for encode/decode strings

Signed-off-by: Lukasz Foniok <[email protected]>

* Fix ui generation and pystun imports

Signed-off-by: Lukasz Foniok <[email protected]>

* Fix for DataBuffer

Signed-off-by: Lukasz Foniok <[email protected]>

* DataBuffer, use bytes instead of bytes strings

Signed-off-by: Lukasz Foniok <[email protected]>

* Fix window_resize method

Signed-off-by: Lukasz Foniok <[email protected]>

* Fix docker scripts are python2 compatible

Signed-off-by: Lukasz Foniok <[email protected]>

* Flooring bucket midpoint

Signed-off-by: Lukasz Foniok <[email protected]>

* Fix etherum address format

Signed-off-by: Lukasz Foniok <[email protected]>

* Runnable tests

* Runnable tests, continued

* circle.yml

* circle.yml, continued

* UI files generation

* Test imports

* Requirements

* Fix test_startapp

Signed-off-by: Lukasz Foniok <[email protected]>

* Correct SHA3 function

* Correct SHA3 function (old behaviour)

* Minor fixes and tests

* keysauth

* Batch of micro fixes + tests

* Fix tests imports

Signed-off-by: Lukasz Foniok <[email protected]>

* Ethereum

* CircleCI: Python 3.5

* Ethereum, continued

* Fix for blender task exr/sgi read/write

Signed-off-by: Lukasz Foniok <[email protected]>

* CLI

* Monitor

* Docker manager

* Core + misc

* Client

* Network

* Lux docker task fix

* Resources

* RPC

* Ethereum

* Task

* Resources

* GUI

* Apps

* PaymentProcessor quickfix

* Circle CI

* Convert MSE to int in calculate_mse tests

* Dummy task

* Change seed option is not crashing

Signed-off-by: Lukasz Foniok <[email protected]>

* Unified encode_hex and decode_hex; explicit '0x' prefixing

* Travis CI

* Appveyor CI

* Enable remote debugging (Appveyor)

* Taskcollector
Temp dir fixture fix (Windows specific)
An attempt on fixing dummy task tests (Windows specific)

* Configure txaio in config_logging (first/only use of txaio in tests)

* Docker tests (Windows specific)

* Core tests (Windows specific)

* Remove datadir on node shutdown in DummyTask test

* Remove datadir on node shutdown in DummyTask test, continued

* Cleanup 2to3 print statements

* Cleanup 2to3 'for' loops

* Docker image build fix

* macOS tests

* python3 syntax in spec file

Signed-off-by: Lukasz Foniok <[email protected]>

* Fix --peer option

Signed-off-by: Lukasz Foniok <[email protected]>

* enum34 dependency removed

* Handle HTTPS redirects in stats.golem.network

* Python 3: extra python package repository (golemfactory#1237)

* Python 3 Windows requirements (golemfactory#1239)

Author: banasiakadam60

.travis.yml

@@ -1,6 +1,5 @@
 language: generic
 
-
 os: osx
 osx_image: xcode8.3  # OS X 10.12
 
@@ -18,7 +17,7 @@ cache:
     - $HOME/Library/Caches/pip
 
 before_install:
-  - brew install [email protected] openexr freeimage
+  - brew install python3 qt5 openexr freeimage
 
   - wget https://gethstore.blob.core.windows.net/builds/${GETH_VER}.tar.gz
   - tar xvf ${GETH_VER}.tar.gz
@@ -28,15 +27,13 @@ before_install:
   - cd golem-hyperdrive && npm install --save && cd ..
   - node golem-hyperdrive/src/index.js &
 
-  - sudo -HE pip install --upgrade pip setuptools six virtualenv pytest-cov codecov
-  - pip install https://github.com/golemfactory/golem/wiki/wheels/sip-4.19-cp27-cp27m-macosx_10_12_x86_64.whl
-  - pip install https://github.com/golemfactory/golem/wiki/wheels/PyQt5-5.7.1-cp27-cp27m-macosx_10_12_x86_64.whl
-  - sudo -HE pip install -r requirements.txt
-  - sudo -HE pip install -r requirements-test.txt
-  - sudo -HE python setup.py develop
+  - sudo -HE pip3 install --upgrade pip setuptools six virtualenv pytest-cov codecov
+  - sudo -HE pip3 install -r requirements.txt
+  - sudo -HE pip3 install -r requirements-test.txt
+  - sudo -HE python3 setup.py develop
 
 script:
-  - pytest --cov=golem
+  - python3 -m pytest --cov=golem
 
 after_success:
-  - codecov
+  - python3 -m codecov

apps/appsmanager.py

@@ -1,7 +1,5 @@
-from __future__ import absolute_import
-
 import os
-from ConfigParser import ConfigParser
+from configparser import ConfigParser
 from collections import OrderedDict
 from importlib import import_module
 

apps/blender/benchmark/benchmark.py

@@ -25,7 +25,7 @@ def __init__(self):
         task_def.output_format = "png"
         task_def.options = BlenderRendererOptions()
         task_def.options.frames = "1"
-        task_def.task_id = u"blender_benchmark"
+        task_def.task_id = "blender_benchmark"
         main_scene_file = pathlib.Path(self.blender_task_path)
         main_scene_file /= "bmw27_cpu.blend"
         task_def.main_scene_file = str(main_scene_file)

apps/blender/resources/blenderloganalyser.py

@@ -8,20 +8,20 @@ def make_log_analyses(log_content, return_data):
     _get_warnings(log_content, return_data)
     rendering_time = find_rendering_time(log_content)
     if rendering_time:
-        return_data[u"rendering_time"] = to_unicode(rendering_time)
+        return_data["rendering_time"] = to_unicode(rendering_time)
     output_path = find_filepath(log_content)
     if output_path:
-        return_data[u"output_path"] = to_unicode(output_path)
+        return_data["output_path"] = to_unicode(output_path)
     frames = find_frames(log_content)
     # FIXME Convert to nice string and to unicode
     if frames:
-        return_data[u"frames"] = frames
+        return_data["frames"] = frames
     resolution = find_resolution(log_content)
     if resolution:
-        return_data[u"resolution"] = resolution
+        return_data["resolution"] = resolution
     file_format = find_file_format(log_content)
     if file_format:
-        return_data[u"file_format"] = to_unicode(file_format)
+        return_data["file_format"] = to_unicode(file_format)
 
 
 def _get_warnings(log_content, return_data):
@@ -33,13 +33,13 @@ def _get_warnings(log_content, return_data):
 
     wrong_engine = find_wrong_renderer_warning(log_content)
     if wrong_engine:
-        warnings.append(u"\n{}\n".format(wrong_engine))
+        warnings.append("\n{}\n".format(wrong_engine))
 
     if warnings:
-        if return_data.get(u"warnings"):
-            return_data[u"warnings"] += u"".join(warnings)
+        if return_data.get("warnings"):
+            return_data["warnings"] += "".join(warnings)
         else:
-            return_data[u"warnings"] = u"".join(warnings)
+            return_data["warnings"] = "".join(warnings)
 
 
 def find_wrong_renderer_warning(log_content):
@@ -62,10 +62,10 @@ def find_missing_files(log_content):
 
 
 def _format_missing_files_warning(missing_files):
-    missing_files = [u"    {}\n".format(missing_file)
+    missing_files = ["    {}\n".format(missing_file)
                      for missing_file in missing_files]
-    ret = u"Additional data is missing:\n" + u"".join(missing_files)
-    ret += u"\nTry to add missing files to resources before " \
+    ret = "Additional data is missing:\n" + "".join(missing_files)
+    ret += "\nTry to add missing files to resources before " \
            "you start rendering."
     return ret
 
@@ -99,7 +99,7 @@ def find_frames(log_content):
         start_frame = int(frames.group(1))
         end_frame = int(frames.group(2))
         frame_step = int(frames.group(3))
-        return range(start_frame, end_frame + 1, frame_step)
+        return list(range(start_frame, end_frame + 1, frame_step))
 
 
 def find_file_format(log_content):

apps/blender/task/blenderrendertask.py

@@ -1,10 +1,10 @@
-from __future__ import division
+
 import logging
 import math
 import os
 import random
 from collections import OrderedDict
-from itertools import ifilter
+
 
 import time
 from PIL import Image, ImageChops
@@ -229,10 +229,10 @@ def __get_border(cls, start, end, parts, res_x, res_y):
 
         upper = offsets[start]
         lower = offsets[end + 1]
-        for i in xrange(upper, lower):
+        for i in range(upper, lower):
             border.append((0, i))
             border.append((x, i))
-        for i in xrange(0, x):
+        for i in range(0, x):
             border.append((i, upper))
             border.append((i, lower))
         return border
@@ -544,7 +544,7 @@ def _update_frame_preview(self, new_chunk_file_path, frame_num, part=1,
             self._update_frame_task_preview()
 
     def _put_image_together(self):
-        output_file_name = u"{}".format(self.output_file, self.output_format)
+        output_file_name = "{}".format(self.output_file, self.output_format)
         logger.debug('_put_image_together() out: %r', output_file_name)
         self.collected_file_names = OrderedDict(sorted(self.collected_file_names.items()))
         if not self._use_outer_task_collector():
@@ -554,7 +554,7 @@ def _put_image_together(self):
             collector.finalize().save(output_file_name, self.output_format)
         else:
             self._put_collected_files_together(os.path.join(self.tmp_dir, output_file_name),
-                                               self.collected_file_names.values(), "paste")
+                                               list(self.collected_file_names.values()), "paste")
 
     def mark_part_on_preview(self, part, img_task, color, preview_updater, frame_index=0):
         lower = preview_updater.get_offset(part)
@@ -580,7 +580,7 @@ def _mark_task_area(self, subtask, img_task, color, frame_index=0):
     def _put_frame_together(self, frame_num, num_start):
         directory = os.path.dirname(self.output_file)
         output_file_name = os.path.join(directory, self._get_output_name(frame_num))
-        frame_key = unicode(frame_num)
+        frame_key = str(frame_num)
         collected = self.frames_given[frame_key]
         collected = OrderedDict(sorted(collected.items()))
         if not self._use_outer_task_collector():
@@ -589,7 +589,7 @@ def _put_frame_together(self, frame_num, num_start):
                 collector.add_img_file(file)
             collector.finalize().save(output_file_name, self.output_format)
         else:
-            self._put_collected_files_together(output_file_name, collected.values(), "paste")
+            self._put_collected_files_together(output_file_name, list(collected.values()), "paste")
         self.collected_file_names[frame_num] = output_file_name
         self._update_frame_preview(output_file_name, frame_num, final=True)
         self._update_frame_task_preview()
@@ -607,13 +607,13 @@ def build_dictionary(cls, definition):
         parent = super(BlenderRenderTaskBuilder, cls)
 
         dictionary = parent.build_dictionary(definition)
-        dictionary[u'options'][u'compositing'] = definition.options.compositing
+        dictionary['options']['compositing'] = definition.options.compositing
         return dictionary
 
     @classmethod
     def build_full_definition(cls, task_type, dictionary):
         parent = super(BlenderRenderTaskBuilder, cls)
-        options = dictionary[u'options']
+        options = dictionary['options']
 
         definition = parent.build_full_definition(task_type, dictionary)
         definition.options.compositing = options.get('compositing', False)

apps/blender/task/verificator.py

@@ -1,4 +1,4 @@
-from __future__ import division
+
 import math
 import random
 

apps/core/benchmark/minilight/src/camera.py

@@ -5,8 +5,8 @@
 
 
 from math import pi, tan
-from raytracer import RayTracer
-from vector3f import Vector3f
+from .raytracer import RayTracer
+from .vector3f import Vector3f
 
 import re
 SEARCH = re.compile('(\(.+\))\s*(\(.+\))\s*(\S+)').search
@@ -18,12 +18,12 @@ class Camera(object):
 
     def __init__(self, in_stream):
         for l in in_stream:
-            if type(l) == type(u""):
+            if type(l) == type(""):
                 line = l.encode('ascii','ignore')
             else:
                 line = l
             if not line.isspace():
-                p, d, a = SEARCH(line).groups()
+                p, d, a = SEARCH(line.decode('utf-8')).groups()
                 self.view_position = Vector3f(p)
                 self.view_direction = Vector3f(d).unitize()
                 if self.view_direction.is_zero():

apps/core/benchmark/minilight/src/image.py

@@ -5,7 +5,7 @@
 
 
 from math import log10
-from vector3f import Vector3f
+from .vector3f import Vector3f
 
 IMAGE_DIM_MAX = 4000
 PPM_ID = 'P6'
@@ -19,8 +19,7 @@ class Image(object):
     def __init__(self, in_stream):
         for line in in_stream:
             if not line.isspace():
-                self.width, self.height = map(lambda dimension:
-                    min(max(1, int(dimension)), IMAGE_DIM_MAX), line.split())
+                self.width, self.height = [min(max(1, int(dimension)), IMAGE_DIM_MAX) for dimension in line.split()]
                 self.pixels = [0.0] * self.width * self.height * 3
                 break
 

apps/core/benchmark/minilight/src/img.py

@@ -5,7 +5,7 @@
 
 
 from math import log10
-from vector3f import Vector3f
+from .vector3f import Vector3f
 
 IMAGE_DIM_MAX = 4000
 PPM_ID = 'P6'

apps/core/benchmark/minilight/src/maxilight.py

@@ -1,6 +1,6 @@
-from camera import Camera
-from image import Image
-from scene import Scene
+from .camera import Camera
+from .image import Image
+from .scene import Scene
 
 from mtrenderer import RenderWorker
 
@@ -59,9 +59,9 @@ def render_taskable(image, image_file_pathname, camera, scene, num_samples, num_
 
     def main():
         if len(argv) < 2 or argv[1] == '-?' or argv[1] == '--help':
-            print HELP
+            print(HELP)
         else:
-            print BANNER
+            print(BANNER)
             model_file_pathname = argv[1]
             image_file_pathname = model_file_pathname + '.ppm'
             model_file = open(model_file_pathname, 'r')
@@ -81,9 +81,12 @@ def main():
             avgSpeed = float(numSamples) / duration
             expectedTime = totalSamples / avgSpeed
 
-            print "\nSummary:"
-            print "    Rendering scene with {} rays took {} seconds".format(numSamples, duration)
-            print "    giving an average speed of {} rays/s".format(avgSpeed)
-            print "    estimated time for the whole scene is {} seconds".format(expectedTime)
+            print("\nSummary:")
+            print("    Rendering scene with {} rays took {} seconds"
+                  .format(numSamples, duration))
+            print("    giving an average speed of {} rays/s"
+                  .format(avgSpeed))
+            print("    estimated time for the whole scene is {} seconds"
+                  .format(expectedTime))
 
     main()

apps/core/benchmark/minilight/src/minilight.py

@@ -11,10 +11,10 @@
 import sys
 
 sys.path.append("src")
-from camera import Camera
-from image import Image
-from scene import Scene
-from randommini import Random
+from .camera import Camera
+from .image import Image
+from .scene import Scene
+from .randommini import Random
 
 
 BANNER = '''
@@ -69,7 +69,7 @@ def makePerfTest(filename, cfg_filename, num_cores):
     model_file_pathname = filename
     image_file_pathname = model_file_pathname + '.ppm'
     model_file = open(model_file_pathname, 'r')
-    if model_file.next().strip() != MODEL_FORMAT_ID:
+    if model_file.readline().strip() != MODEL_FORMAT_ID:
         raise 'invalid model file'
     for line in model_file:
         if not line.isspace():
@@ -84,9 +84,11 @@ def makePerfTest(filename, cfg_filename, num_cores):
     duration = render_taskable(image, image_file_pathname, camera, scene, iterations)
 
     numSamples = image.width * image.height * iterations
-    print "\nSummary:"
-    print "    Rendering scene with {} rays took {} seconds".format(numSamples, duration)
-    print "    giving an average speed of {} rays/s".format(float(numSamples) / duration)
+    print("\nSummary:")
+    print("    Rendering scene with {} rays took {} seconds"
+          .format(numSamples, duration))
+    print("    giving an average speed of {} rays/s"
+          .format(float(numSamples) / duration))
     cfg_file = open(cfg_filename, 'w')
     average = float(numSamples) / duration
     average = average * num_cores
@@ -117,9 +119,9 @@ def render_orig(image, image_file_pathname, camera, scene, iterations):
                 image_file = open(image_file_pathname, 'wb')
                 image.get_formatted(image_file, frame_no)
                 image_file.close()
-        print '\nfinished'
+        print('\nfinished')
     except KeyboardInterrupt:
-        print '\ninterrupted'
+        print('\ninterrupted')
 
 @timedafunc
 def render_taskable(image, image_file_pathname, camera, scene, num_samples):
@@ -152,17 +154,17 @@ def render_taskable(image, image_file_pathname, camera, scene, num_samples):
         # image.get_formatted(image_file, num_samples)
         # image_file.close()
 
-        print '\nfinished'
+        print('\nfinished')
     except KeyboardInterrupt:
-        print '\ninterrupted'
+        print('\ninterrupted')
 
 
 
 def main():
     if len(argv) < 2 or argv[1] == '-?' or argv[1] == '--help':
-        print HELP
+        print(HELP)
     else:
-        print BANNER
+        print(BANNER)
         model_file_pathname = argv[1]
         image_file_pathname = model_file_pathname + '.ppm'
         model_file = open(model_file_pathname, 'r')
@@ -181,9 +183,11 @@ def main():
         duration = render_taskable(image, image_file_pathname, camera, scene, iterations)
 
         numSamples = image.width * image.height * iterations
-        print "\nSummary:"
-        print "    Rendering scene with {} rays took {} seconds".format(numSamples, duration)
-        print "    giving an average speed of {} rays/s".format(float(numSamples) / duration)
+        print("\nSummary:")
+        print("    Rendering scene with {} rays took {} seconds"
+              .format(numSamples, duration))
+        print("    giving an average speed of {} rays/s"
+              .format(float(numSamples) / duration))
         cfg_file = open('minilight.ini', 'w')
         average = float(numSamples) / duration
         average = average * multiprocessing.cpu_count()
@@ -192,4 +196,4 @@ def main():
 
 
 if __name__ == "__main__":
-    main()
+    main()