modernize images and dependencies

[ehrenb]

• Updated Alpine base
• Updated Ubuntu base
• Update various pypi deps
• Update OrientDB + Python driver
  • using git+https://github.com/orientechnologies/pyorient.git@ca7690a95f57262b6b91f3545ac98def052d4193#egg=pyorient for now (https://github.com/orientechnologies/orientdb/issues/8852)
• COPY module src per analysis module Dockerfile
• rmq

[ehrenb]

python3 run.py 
/src/pyorient/pyorient/utils.py:82: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if _cluster_id[0] is '#':
Traceback (most recent call last):
  File "/machina/run.py", line 3, in <module>
    from jar import JarAnalyzer
  File "/machina/jar.py", line 5, in <module>
    from machina.core.worker import Worker
  File "/machina/machina/core/worker.py", line 11, in <module>
    from machina.core.models import init_orientdb
  File "/machina/machina/core/models/__init__.py", line 4, in <module>
    from pyorient.ogm.declarative import declarative_node, declarative_relationship
  File "/src/pyorient/pyorient/ogm/__init__.py", line 2, in <module>
    from .graph import Graph
  File "/src/pyorient/pyorient/ogm/graph.py", line 7, in <module>
    from .query import Query
  File "/src/pyorient/pyorient/ogm/query.py", line 6, in <module>
    from .mapping import CacheMixin
  File "/src/pyorient/pyorient/ogm/mapping.py", line 1, in <module>
    from collections import Iterable
ImportError: cannot import name 'Iterable' from 'collections' (/usr/lib/python3.10/collections/__init__.py)

Fixed by upgrading Python and pyorient driver

[ehrenb]

Error upon submission:

[*] resolved to: {'type': 'zip', 'reason': 'detailed_types', 'value': 'Zip archive data'}
Exception in thread Thread-2 (callback):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/machina/src/identifier.py", line 117, in callback
    node = c.objects.create(md5=body['hashes']['md5'],
  File "/usr/lib/python3.10/site-packages/pyorient/ogm/broker.py", line 56, in create
    return self.g.create_vertex(self.element_cls, **kwargs)
  File "/usr/lib/python3.10/site-packages/pyorient/ogm/graph.py", line 532, in create_vertex
    result = self.client.command(self.create_vertex_command(vertex_cls, **kwargs))[0]
  File "/usr/lib/python3.10/site-packages/pyorient/orient.py", line 481, in command
    return self.get_message("CommandMessage").prepare((QUERY_CMD,) + args).send().fetch_response()
  File "/usr/lib/python3.10/site-packages/pyorient/utils.py", line 48, in wrap_function
    return wrap(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/pyorient/utils.py", line 61, in wrap_function
    return wrap(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/pyorient/messages/commands.py", line 127, in prepare
    payload = b''.join(
  File "/usr/lib/python3.10/site-packages/pyorient/messages/commands.py", line 128, in <genexpr>
    self._encode_field(x) for x in _payload_definition
  File "/usr/lib/python3.10/site-packages/pyorient/messages/database.py", line 379, in _encode_field
    _content = struct.pack("!i", len(v)) + v
TypeError: object of type 'VertexCommand' has no len()

Fixed by switching to this fork: baesystemsFASTLabs/pyorient@7e46e16

see report w PR: orientechnologies/pyorient#44

[ehrenb]

When testing with AndroguardAnalysis:

[*] resolved path: /data/binaries/20221214010929825926/e6e66bbee9b574db612fa540e6bd27f9
Exception in thread Thread-2 (callback):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/machina/src/androguardanalysis.py", line 24, in callback
    a, d, dx = AnalyzeAPK(target)
  File "/usr/lib/python3.10/site-packages/androguard/misc.py", line 71, in AnalyzeAPK
    dx.add(df)
  File "/usr/lib/python3.10/site-packages/androguard/core/analysis/analysis.py", line 1004, in add
    self.methods[method] = MethodAnalysis(vm, method)
  File "/usr/lib/python3.10/site-packages/androguard/core/analysis/analysis.py", line 286, in __init__
    self._create_basic_block()
  File "/usr/lib/python3.10/site-packages/androguard/core/analysis/analysis.py", line 301, in _create_basic_block
    v = dvm.determineNext(i, idx, self.method)
  File "/usr/lib/python3.10/site-packages/androguard/core/bytecodes/dvm.py", line 354, in determineNext
    for target in data.get_targets():
AttributeError: 'Instruction10x' object has no attribute 'get_targets'

This seems to trigger when analyzing a complicated APK like Andr.PegasusB.apk, but succeeds on other APKs. Revisit later.