[MXNET-793] ★ Virtualized testing in CI with QEMU ★ (apache#12094)

* virtual testing with qemu

* Add install procedure

* update installation

* Refine test run

* use direct ssh

* update readme

* Fix uneccesary cp

* Minor refinements

* Refine error conditions in startup

* requirements installed inside QEMU

* Update base image

* Fix license

* Dockerfile rename fallout

* license fixes

* refine documentation

* license fix

* update readme

* Update qemu base image and refine documentation

* Address CR comments wrt shebangs.

* Address CR comments wrt comments.

* adjust vda2 -> vda1

* Disable SMP, bug with newer kernel

* Remove commented out code

* Fix licenses

* CR comments addressed

* increase ram to 4096mb

* Revert dockerfile renaming

* Fix undo rename of dockerfiles

* Address CR comments

* CR

ci/README.md

@@ -90,3 +90,13 @@ For all builds a directory from the host system is mapped where ccache will stor
 compiled object files (defaults to /tmp/ci_ccache). This will speed up rebuilds 
 significantly. You can set this directory explicitly by setting CCACHE_DIR environment 
 variable. All ccache instances are currently set to be 10 Gigabytes max in size.
+
+
+## Testing with QEMU
+To run the unit tests under qemu:
+```
+./build.py -p armv7 && ./build.py -p test.arm_qemu ./runtime_functions.py run_ut_py3_qemu
+```
+
+To get a shell on the container and debug issues with the emulator itself:
+Run the output of `./build.py -p test.arm_qemu --print-docker-run`

ci/build.py

@@ -394,7 +394,7 @@ def main() -> int:
                         help="platform",
                         type=str)
 
-    parser.add_argument("--build-only",
+    parser.add_argument("-b", "--build-only",
                         help="Only build the container, don't build the project",
                         action='store_true')
 

ci/docker/Dockerfile.build.test.arm_qemu

@@ -0,0 +1,44 @@
+# -*- mode: dockerfile -*-
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Dockerfile to build and run MXNet on Ubuntu 16.04 for CPU
+
+FROM ubuntu:16.04
+
+WORKDIR /work
+
+RUN apt-get update
+COPY install/ubuntu_python.sh /work/
+RUN /work/ubuntu_python.sh
+
+COPY install/ubuntu_arm_qemu.sh /work
+RUN /work/ubuntu_arm_qemu.sh
+
+COPY install/ubuntu_arm_qemu_bin.sh /work
+RUN /work/ubuntu_arm_qemu_bin.sh
+
+ARG USER_ID=0
+ARG GROUP_ID=0
+COPY install/ubuntu_adduser.sh /work/
+RUN /work/ubuntu_adduser.sh
+
+COPY runtime_functions.sh /work/
+COPY qemu/* /work/
+COPY qemu/ansible.cfg /etc/ansible/ansible.cfg 
+
+CMD ["./runtime_functions.py","run_ut_py3_qemu"]

ci/docker/install/ubuntu_arm_qemu.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# build and install are separated so changes to build don't invalidate
+# the whole docker cache for the image
+
+set -exuo pipefail
+
+apt-get install -y \
+    cmake \
+    curl \
+    wget \
+    git \
+    qemu \
+    qemu-system-arm \
+    unzip \
+    bzip2 \
+    vim-nox
+
+pip3 install ansible ipython

ci/docker/install/ubuntu_arm_qemu_bin.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# build and install are separated so changes to build don't invalidate
+# the whole docker cache for the image
+
+set -exuo pipefail
+
+#
+# This disk image and kernels for virtual testing with QEMU  is generated with some manual OS
+# installation steps with the scripts and documentation found in the ci/qemu/ folder.
+#
+# The image has a base Debian OS and MXNet runtime dependencies installed.
+# The root password is empty and there's a "qemu" user without password. SSH access is enabled as
+# well.
+#
+# See also: ci/qemu/README.md
+#
+
+REMOTE="https://s3-us-west-2.amazonaws.com/mxnet-ci-prod-slave-data"
+curl -f ${REMOTE}/vda_debian_stretch.qcow2.bz2 | bunzip2 > vda.qcow2
+curl -f ${REMOTE}/vmlinuz -o vmlinuz
+curl -f ${REMOTE}/initrd.img -o initrd.img
+

ci/docker/install/ubuntu_python.sh

@@ -22,7 +22,7 @@
 
 set -ex
 # install libraries for mxnet's python package on ubuntu
-apt-get install -y python-dev python3-dev virtualenv
+apt-get install -y python-dev python3-dev virtualenv wget
 
 # the version of the pip shipped with ubuntu may be too lower, install a recent version here
 wget -nv https://bootstrap.pypa.io/get-pip.py

ci/docker/qemu/README.md

@@ -0,0 +1 @@
+These are files used in the docker container that runs QEMU

ci/docker/qemu/ansible.cfg

@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[defaults]
+host_key_checking = False
+ansible_python_interpreter = /usr/bin/python3

ci/docker/qemu/playbook.yml

@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+---
+- name: provision QEMU VM
+  hosts: all
+  gather_facts: no
+  become: true
+  become_user: root
+  tasks:
+    - name: Wait until ssh is available
+      wait_for_connection:
+        delay: 0
+        sleep: 3
+        timeout: 400
+    - command: hostname
+      register: h
+    - debug: msg="{{ h.stdout }}"
+
+    - name: copy mxnet artifacts
+      copy:
+        src: "{{ item }}"
+        dest: mxnet_dist/
+      with_fileglob: "/work/mxnet/build/*.whl"
+
+    - name: copy runtime_functions.py
+      copy:
+        src: "/work/runtime_functions.py"
+        dest: .
+    - file:
+        path: runtime_functions.py
+        mode: 0755
+
+

ci/docker/qemu/qemu_run.sh

@@ -0,0 +1,31 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -exuo pipefail
+
+qemu-system-arm -M virt -m 1024 \
+  -kernel vmlinuz \
+  -initrd initrd.img \
+  -append 'root=/dev/vda1' \
+  -drive if=none,file=vda.qcow2,format=qcow2,id=hd \
+  -device virtio-blk-device,drive=hd \
+  -netdev user,id=mynet,hostfwd=tcp::2222-:22 \
+  -device virtio-net-device,netdev=mynet \
+  -nographic \
+  -display none

ci/docker/qemu/runtime_functions.py

@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# -*- coding: utf-8 -*-
+"""Runtime functions to use in docker / testing"""
+
+__author__ = 'Pedro Larroy'
+__version__ = '0.1'
+
+import os
+import sys
+import subprocess
+import argparse
+import logging
+from subprocess import call, check_call, Popen, DEVNULL, PIPE
+import time
+import sys
+import types
+import glob
+
+def activate_this(base):
+    import site
+    import os
+    import sys
+    if sys.platform == 'win32':
+        site_packages = os.path.join(base, 'Lib', 'site-packages')
+    else:
+        site_packages = os.path.join(base, 'lib', 'python%s' % sys.version[:3], 'site-packages')
+    prev_sys_path = list(sys.path)
+    sys.real_prefix = sys.prefix
+    sys.prefix = base
+    # Move the added items to the front of the path:
+    new_sys_path = []
+    for item in list(sys.path):
+        if item not in prev_sys_path:
+            new_sys_path.append(item)
+            sys.path.remove(item)
+    sys.path[:0] = new_sys_path
+
+def run_ut_py3_qemu():
+    from vmcontrol import VM
+    with VM() as vm:
+        logging.info("VM provisioning with ansible")
+        check_call(["ansible-playbook", "-v", "-u", "qemu", "-i", "localhost:{},".format(vm.ssh_port), "playbook.yml"])
+        logging.info("VM provisioned successfully.")
+        logging.info("sync tests")
+        check_call(['rsync', '-e', 'ssh -p{}'.format(vm.ssh_port), '-a', 'mxnet/tests', 'qemu@localhost:mxnet'])
+        logging.info("execute tests")
+        check_call(["ssh", "-o", "ServerAliveInterval=5", "-p{}".format(vm.ssh_port), "qemu@localhost", "./runtime_functions.py", "run_ut_python3_qemu_internal"])
+        logging.info("tests finished, vm shutdown.")
+        vm.shutdown()
+
+def run_ut_python3_qemu_internal():
+    """this runs inside the vm, it's run by the playbook above by ansible"""
+    pkg = glob.glob('mxnet_dist/*.whl')[0]
+    logging.info("=== NOW Running inside QEMU ===")
+    logging.info("PIP Installing %s", pkg)
+    check_call(['sudo', 'pip3', 'install', pkg])
+    logging.info("PIP Installing mxnet/tests/requirements.txt")
+    check_call(['sudo', 'pip3', 'install', '-r', 'mxnet/tests/requirements.txt'])
+    logging.info("Running tests in mxnet/tests/python/unittest/")
+    check_call(['nosetests', '--with-timer', '--with-xunit', '--xunit-file', 'nosetests_unittest.xml', '--verbose', 'mxnet/tests/python/unittest/test_ndarray.py:test_ndarray_fluent'])
+
+
+def parsed_args():
+    parser = argparse.ArgumentParser(description="""python runtime functions""", epilog="")
+    parser.add_argument('command',nargs='*',
+        help="Name of the function to run with arguments")
+    args = parser.parse_args()
+    return (args, parser)
+
+def script_name() -> str:
+    return os.path.split(sys.argv[0])[1]
+
+def chdir_to_script_directory():
+    # We need to be in the same directory than the script so the commands in the dockerfiles work as
+    # expected. But the script can be invoked from a different path
+    base = os.path.split(os.path.realpath(__file__))[0]
+    os.chdir(base)
+
+def main():
+    logging.getLogger().setLevel(logging.DEBUG)
+    logging.basicConfig(format='{}: %(asctime)-15s %(message)s'.format(script_name()))
+    chdir_to_script_directory()
+
+    # Run function with name passed as argument
+    (args, parser) = parsed_args()
+    logging.info("%s", args.command)
+    if args.command:
+        fargs = args.command[1:]
+        globals()[args.command[0]](*fargs)
+        return 0
+    else:
+        parser.print_help()
+        fnames = [x for x in globals() if type(globals()[x]) is types.FunctionType]
+        print('\nAvailable functions: {}'.format(' '.join(fnames)))
+        return 1
+
+if __name__ == '__main__':
+    sys.exit(main())
+