-
Notifications
You must be signed in to change notification settings - Fork 448
VboxApps
Running apps in VirtualBox virtual machines
BOINC supports "VM apps" - applications that run in VirtualBox virtual machines. This provides several benefits:
- You don't need to build app versions for different platforms. You can develop your app in your environment of choice (say, Debian Linux), and then bundle the resulting executable with a virtual machine image containing an appropriate runtime environment. The application can then be run on all platforms (Windows, Mac OS X, all versions of Linux) with no additional work on your part.
- Virtual machines provide the strongest available security sandbox; a VM app cannot access or modify the host system. This makes it feasible to deploy untrusted applications.
- VM apps don't need to have their own checkpoint/restart mechanism; BOINC provides one.
VM apps have the following limitations:
- Only a fraction of volunteer computers have VirtualBox installed.
- VirtualBox runs only on Intel-compatible processors. If you want to support other processors (such as ARM, SPARC, etc.), you'll need to use non-VM app versions.
- Currently you can't run GPU applications in VirtualBox VMs.
VM apps use a program called "vboxwrapper" that connects the BOINC client to VirtualBox.
There are two ways to package VM apps. NOTE: in the following, application and application version have BOINC-specific meanings; executable refers to the program that runs within the VM.
- Single-purpose app: Include the executable with the application version. Create a separate application for each executable you want to run.
- Multi-purpose app: Include the executable in each workunit. This allows you to use a single application for as many executables as you like. In this case, consider making the executable file sticky; that way, clients will download it only once.
You must create app versions for each platform you want to support; the app versions differ in which vboxwrapper executable they use.
You must associate a plan class with each app version, such as "vbox64" (for 64-bit machines). To enable multiple cores use "vbox64_mt". By default this will assign 2 threads (virtual cores) per VM task.
For single-purpose apps, an app version includes the following files:
- The VM image, in VirtualBox format. See "Multiattach mode" below. If you don't use this mode, the image:
- Must have logical name "vm_image.vdi".
- Must have the copy_file attribute.
- Should have the gzip attribute for faster download to 7.0+ clients.
- The application executable to be run in the VM image.
- This may be a shell script or a binary program.
- The logical name must be shared/boinc_app.
- Other files needed by the application, all with logical names starting with shared/.
- An XML Vbox job description file with logical name vbox_job.xml (see below)
- vboxwrapper, compiled for the platform (executables are available below).
- All scripts and executables must have the execute permission set.
For multi-purpose apps, any of these files except vboxwrapper may be included in the workunit instead of the app version.
Include <dont_throttle/> in the version.xml file; VirtualBox does its own CPU throttling.
Typically you can use the same VM image for multiple applications. This reduces network traffic and client disk usage.
As of version 26204, vboxwrapper can deal with multiattach-mode/differencing disk images as described in the VirtualBox manual. This avoids the need to copy a large "vm_image.vdi" file to the worker slot each time a task starts. Instead the vdi image remains in the project directory, is opened read-only and used for all tasks referring to it. VirtualBox transparently ensures that all disk writes go to a separate vdi file within the slot directory of a task. That "differencing image" is usually much smaller than the parent vdi image.
To enable this mode add <multiattach_vdi_file>filename_version.vdi</multiattach_vdi_file> to vbox_job.xml with filename_version.vdi being the name of the parent vdi image in the project directory. Compared to the setup above the app version needs to be configured slightly different:
- The VM image, in VirtualBox format.
- The logical name "vm_image.vdi" should not be set.
- The copy_file attribute should not be set to avoid copying the vdi file to the slot directory.
- Should have the sticky attribute set.
- During contextualization/creation of the vdi image, multi-attach mode must not be used.
The VBox job description file has logical name vbox_job.xml (its physical name should include a version number and other info). It has following structure:
<vbox_job>
<memory_size_mb>N</memory_size_mb>
<os_name>name</os_name>
[ <completion_trigger_file>filename</completion_trigger_file> ]
[ <copy_to_shared>filename</copy_to_shared> ]
[ <copy_cmdline_to_shared>0|1</copy_cmdline_to_shared> ]
[ <enable_cache_disk>0|1</enable_cache_disk> ]
[ <enable_cern_dataformat>0|1</enable_cern_dataformat> ]
[ <enable_floppyio>0|1</enable_floppyio> ]
[ <enable_isocontextualization>0|1</enable_isocontextualization> ]
[ <enable_network/> ]
[ <enable_remotedesktop>0|1</enable_remotedesktop> ]
[ <enable_shared_directory/> ]
[ <enable_graphics_support/> ]
[ <boot_iso/> ]
[ <fraction_done_filename>filename</fraction_done_filename> ]
[ <job_duration>X</job_duration> ]
[ <minimum_checkpoint_interval>N</minimum_checkpoint_interval> ]
[ <multiattach_vdi_file>filename_version.vdi</multiattach_vdi_file> ]
[ <network_bridged_mode/> ]
[ <pf_guest_port>N</pf_guest_port> ]
[ <port_forward>
<host_port>H</host_port>
<guest_port>G</guest_port>
[<is_remote>0|1</is_remote>]
[<nports>N</nports]
</port_forward> ]
[ <trickle_trigger_file>filename</trickle_trigger_file> ]
[ <vm_disk_controller_model>LSILogic|LSILogicSAS|BusLogic|IntelAHCI|PIIX3|PIIX4|ICH6|I82078</vm_disk_controller_model> ]
[ <vm_disk_controller_type>ide|sata|scsi|floppy|sas</vm_disk_controller_type> ]
[ <vm_graphics_controller_type>VBoxSVGA|VBoxVGA|VMSVGA</vm_graphics_controller_type> ]
[ <vram_size_mb>N</vram_size_mb> ]
</vbox_job>
Required elements:
The amount of physical memory allocated to the VM, in megabytes.
The name of the guest OS as defined by VirtualBox, e.g. "Linux26", "Linux26_64", "Linux24", etc. To see a list of all available OS names, install VirtualBox on a system, and type "vboxmanage list ostypes".
Optional elements:
This provides a more bulletproof way for VM apps to exit; sometimes VMs fail to shut down, and the task hangs indefinitely. When the VM app is done, it writes a file of this name in the shared directory; the file can optionally contain an integer exit code (first line) a bool value for whether it should bubble up to the volunteer (second line) and stderr text (subsequent lines). If vboxwrapper finds this file, it cleans up the VM and exits with the given code (default 0).
Copy the given file from the slot directory to the shared directory before launch. For example, you can use to copy init_data.xml into the VM. This directive can be used more than once.
Write vboxwrapper's command line to a file shared/cmdline. This lets you pass information into the VM without input files. The max size of the command line is on the order of 60KB.
Mount a virtual disk in the VM. The virtual disk is described by a VDI file in the app version with logical name vm_cache.vdi and the copy_file attribute.
If enable_floppyio is used (see below) initialize the floppy disk image to contain user and host ID and credit as name=value pairs.
The VM image is an ISO file named vm_isocontext.iso, rather than a VDI file. Also, vboxwrapper will mount the host's guest additions ISO (VBoxGuestAdditions.iso) as a DVD in the VM.
Create a floppy disk image in the VM, containing the contents of init_data.xml.
If present, allow the VM to do network access.
If the Oracle VirtualBox Extension are installed, it'll enable the use of a remote desktop client to view the console of the VM.
If present, create a directory that is shared between the host OS and the guest OS. Must be set if your application has input or output files.
If present, creates and updates a graphics status file. This is used by HTMLGfx. (v26155+)
if both a disk and an ISO are specified to be attached to the VM, by default the VM will first attempt to boot from the disk. with boot_iso given, it will instead first boot from the ISO.
The name of a file to which the app will periodically write its fraction done (0 to 1). This is used by the wrapper to report overall fraction done.
Specifies the name of an output file that, when present, should be uploaded as soon as possible.
Specifies the maximum elapsed time of the job, after which vboxwrapper will kill the VM and exit normally.
Minimum number of seconds before a checkpoint/snapshot can be created. Defaults to 10 minutes. (v26086+)
Enables multiattach-mode/differencing vdi images to be used. The filename given here becomes the parent image. It is expected to be in the projects directory. The filename should have a version number to avoid conflicts when an app version gets updated.
If enable_network is set, use bridged mode; default is NAT mode.
Enable port forwarding to port N within the VM. This is assumed to be a web server providing application graphics.
Defines a port forwarding between the given host and guest ports. If nports is specified, N ports will be forwarded: H to G, H+1 to G+1, ... H+N-1 to G+N-1. If is_remote is set, the host ports can be accessed from other computers; otherwise they can be accessed only from processes on this computer. There may be more than one of these elements.
Specifies the name of a file that, if present, causes the wrapper to temporarily exit.
Provides a mechanism for the VM to send trickle-up messages. If a file of the given name appears in the shared directory, vboxwrapper sends a trickle-up message whose variety is the filename and whose contents is the contents of the file, then deletes the file.
Which disk controller model to emulate. As of version 26204 vboxwrapper uses <vm_disk_controller_model>IntelAHCI</vm_disk_controller_model> together with <vm_disk_controller_type>sata</vm_disk_controller_type> as default setting.
Which disk controller type to emulate. As of version 26204 vboxwrapper uses <vm_disk_controller_model>IntelAHCI</vm_disk_controller_model> together with <vm_disk_controller_type>sata</vm_disk_controller_type> as default setting.
Which graphics controller type to emulate. For details see VirtualBox manual. Vboxwrapper as of version 26204 uses VBoxVGA as default.
The amount of video memory allocated by the virtual graphics controller, in megabytes. VirtualBox allows it to be between 8 and 128 MB. For Linux VMs using VBoxVGA 16 MB is the recommended minimum. This value is used by default.
Send a trickle message reporting elapsed time every X seconds. Use might this for incremental credit granting, or as a "heartbeat" mechanism.
Create a virtual machine that will use N cores.
Use vm_image_N.vdi as the VM image, rather than vm_image.vdi. This lets you create an app version with several images, and the app_plan function can decide which one to use for the particular host.
Register the VM but don't run it. For debugging; see below.
The input and output files of a VM app must
- Have logical names starting with shared/.
- Have the copy_file attribute.
This causes the BOINC client to copy them to and from the slots/x/shared/ directory.
See the vboxwrapper release notes.
Windows:
x64: vboxwrapper_26206_windows_x86_64.exe.zip
Mac OS X:
x64: vboxwrapper_26206_x86_64-apple-darwin.zip>
Linux:
x64: vboxwrapper_26206_x86_64-pc-linux-gnu
These VM images were built using the above instructions for creating VM images. They contain Debian 4.0, without GCC or any build tools installed. They contain the example startup script.
x64: vmimage_x64.zip
In most cases, you can use these VM images with no modifications. If your application uses libraries not on the VM images, you can add them as follows:
- Run VirtualBox, and open the VM image
- Hit CTRL-C when see "BOINC VM starting" in the console window
- Install whatever you want (can use apt-get install to install Debian packages).
- when you're done, type
shutdown -hP 0
The VM image now has the additional libraries. Rename it to avoid confusion with the original version.
To debug a VM within the BOINC/VboxWrapper framework:
- Launch BOINC with --exit_before_start
- When BOINC exits, launch vboxwrapper with the --register_only option.
- Set the VBOX_USER_HOME environment variable to the vbox directory under the slot directory. This changes where the VirtualBox applications look for the root VirtualBox configuration files. It may or may not apply to your installation of VirtualBox. It depends on where your copy of VirtualBox came from and what type of system it is installed on.
- Now Launch the VM using the VirtualBox UI. You should now be able to interact with your VM.
To debug what is going on within the guest VM you can use vboxmonitor which writes whatever is received in stdin to the VM guest log (VBox.log). This in turn is read and rewritten to stderr.txt in the slot directory by vboxwrapper.
To use you must either use the premade vboxmonitor or build your own. Once on the guest VM you must execute setuid on it so that it runs with root permissions.
Usage:
[root@localhost vboxmonitor]# echo this is a test | ./vboxmonitor
this is a test
Log Output:
01:08:57.938896 Guest Log: this is a test
Usage:
[root@localhost vboxmonitor]# ls -la ../vboxwrapper | ./vboxmonitor
total 37052
drwxrwxr-x 5 boincadm boincadm 4096 Jun 2 12:36 .
drwxrwxr-x 18 boincadm boincadm 4096 Oct 3 2013 ..
-rw-rw-r-- 1 boincadm boincadm 3831 Apr 24 2013 BuildMacVboxWrapper.sh
drwxrwxr-x 2 boincadm boincadm 4096 Apr 24 2013 cernvm
drwxrwxr-x 3 boincadm boincadm 4096 Apr 24 2013 deprecated
-rw-rw-r-- 1 boincadm boincadm 11585 Apr 24 2013 floppyio.cpp
-rw-rw-r-- 1 boincadm boincadm 5910 Apr 24 2013 floppyio.h
-rw-rw-r-- 1 boincadm boincadm 71444 May 21 20:01 floppyio.o
lrwxrwxrwx 1 boincadm boincadm 48 May 21 20:01 libstdc++.a -> /usr/lib/gcc/i386-redhat-linux/4.1.2/libstdc++.a
-rw-rw-r-- 1 boincadm boincadm 927 May 29 2013 Makefile
-rw-rw-r-- 1 boincadm boincadm 1135 Apr 24 2013 Makefile_mac
-rw-rw-r-- 1 boincadm boincadm 90494 Jun 2 12:33 vbox.cpp
-rw-rw-r-- 1 boincadm boincadm 7573 Jun 2 12:33 vbox.h
-rw-rw-r-- 1 boincadm boincadm 249892 Jun 2 12:35 vbox.o
-rw-rw-r-- 1 boincadm boincadm 43848 Jun 2 12:33 vboxwrapper.cpp
-rw-rw-r-- 1 boincadm boincadm 1458 Dec 9 19:21 vboxwrapper.h
-rw-rw-r-- 1 boincadm boincadm 178548 Jun 2 12:35 vboxwrapper.o
-rw-rw-r-- 1 boincadm boincadm 438 May 21 19:56 vboxwrapper_win.h
-rw-rw-r-- 1 boincadm boincadm 2127 May 21 19:56 vboxwrapper_win.rc
drwxrwxr-x 2 boincadm boincadm 4096 Apr 24 2013 vboxwrapper.xcodeproj
Log Output:
01:09:25.148427 Guest Log: total 37052
01:09:25.148762 Guest Log: drwxrwxr-x 5 boincadm boincadm 4096 Jun 2 12:36 .
01:09:25.149071 Guest Log: drwxrwxr-x 18 boincadm boincadm 4096 Oct 3 2013 ..
01:09:25.149445 Guest Log: -rw-rw-r-- 1 boincadm boincadm 3831 Apr 24 2013 BuildMacVboxWrapper.sh
01:09:25.149729 Guest Log: drwxrwxr-x 2 boincadm boincadm 4096 Apr 24 2013 cernvm
01:09:25.150070 Guest Log: drwxrwxr-x 3 boincadm boincadm 4096 Apr 24 2013 deprecated
01:09:25.150374 Guest Log: -rw-rw-r-- 1 boincadm boincadm 11585 Apr 24 2013 floppyio.cpp
01:09:25.150700 Guest Log: -rw-rw-r-- 1 boincadm boincadm 5910 Apr 24 2013 floppyio.h
01:09:25.151029 Guest Log: -rw-rw-r-- 1 boincadm boincadm 71444 May 21 20:01 floppyio.o
01:09:25.151573 Guest Log: lrwxrwxrwx 1 boincadm boincadm 48 May 21 20:01 libstdc++.a -> /usr/lib/gcc/i386-redhat-linux/4.1.2/libstdc++.a
01:09:25.151864 Guest Log: -rw-rw-r-- 1 boincadm boincadm 927 May 29 2013 Makefile
01:09:25.152190 Guest Log: -rw-rw-r-- 1 boincadm boincadm 1135 Apr 24 2013 Makefile_mac
01:09:25.152476 Guest Log: -rw-rw-r-- 1 boincadm boincadm 90494 Jun 2 12:33 vbox.cpp
01:09:25.152757 Guest Log: -rw-rw-r-- 1 boincadm boincadm 7573 Jun 2 12:33 vbox.h
01:09:25.153045 Guest Log: -rw-rw-r-- 1 boincadm boincadm 249892 Jun 2 12:35 vbox.o
01:09:25.199092 Guest Log: -rw-rw-r-- 1 boincadm boincadm 43848 Jun 2 12:33 vboxwrapper.cpp
01:09:25.199479 Guest Log: -rw-rw-r-- 1 boincadm boincadm 1458 Dec 9 19:21 vboxwrapper.h
01:09:25.199806 Guest Log: -rw-rw-r-- 1 boincadm boincadm 178548 Jun 2 12:35 vboxwrapper.o
01:09:25.200147 Guest Log: -rw-rw-r-- 1 boincadm boincadm 438 May 21 19:56 vboxwrapper_win.h
01:09:25.200490 Guest Log: -rw-rw-r-- 1 boincadm boincadm 2127 May 21 19:56 vboxwrapper_win.rc
01:09:25.201024 Guest Log: drwxrwxr-x 2 boincadm boincadm 4096 Apr 24 2013 vboxwrapper.xcodeproj
The VM, when booted, must do the following:
- If the applications has input or output files, mount the shared directory using
mount -t vboxsf shared /root/shared
where /root/shared is the path where the shared directory is to be mounted.
In this case the VM must contain the VirtualBox "guest additions".
Guest additions are required for shared folders to work.
-
Run the application.
-
When the application is finished, shut down the VM (e.g., by running shutdown on Linux).
These steps are typically done by a startup script in the VM image. An example startup script is given below. This script runs the application by doing the following:
- cd into the shared directory
- execute boinc_app, and wait for it to exit.
Using this script, your application executable must have logical name share/boinc_app.
Doing things this way, the VM image is independent of the application. You can use the a single VM image for many applications.
Attention: If your boinc_app is a bash or perl script you may get problems when the VM is restored from a snapshot. To circumvent this you have to change your startup script to copy the contents of the shared/ directory to another directory 'inside' the VM and execute it there. For example:
echo -- Launching boinc_app
if [ -f /root/shared/boinc_app ]; then
mkdir /root/worker
cp -r /root/shared/* /root/worker/
cd /root/worker/
./boinc_app
cd /root/
rm -rf /root/worker
shutdown -hP 0
else
This way you can still reuse the VM for other applications but have to make sure that your boinc_app control script is copying the output files of the application to ´/root/shared/´ before exiting. This may take some time, so you should do something like:
cp outfile1.zip /root/shared/out1.zip.tmp
cp outfile2.zip /root/shared/out2.zip.tmp
{...}
mv /root/shared/out1.zip.tmp /root/shared/out1.zip
mv /root/shared/out2.zip.tmp /root/shared/out2.zip
The example startup script follows. You can deploy it by appending to /root/.bashrc in the VM image.
echo --- BOINC VM starting
sleep 5
The "sleep 5" gives you time to break into a console session via CTRL-C if you need to make changes to the VM in the future.
echo --- Mounting shared directory
mount -t vboxsf shared /root/shared
if [ $? -ne 0 ]; then
echo --- Failed to mount shared directory
sleep 5
shutdown -hP 0
fi
echo -- Launching boinc_app
if [ -f /root/shared/boinc_app ]; then
cd /root/shared
./boinc_app
shutdown -hP 0
else
echo --- Failed to launch script
sleep 5
fi
shutdown -hP 0
Using the example startup script, the steps in running a vboxwrapper app are:
- BOINC client
- Create slot directory, say
slots/0 - Create slots/0/shared, and copy input files there
- Execute vboxwrapper in the slot directory
- vboxwrapper
- Create and run virtual machine
- Virtual machine
- Startup script
- mounts shared directory
- cd into shared directory
- execute boinc_app
- when boinc_app exits, shut down virtual machine
- vboxwrapper
- delete virtual machine
- call boinc_finish()
- BOINC client
- copy output files from
slots/0/sharedto project directory
On nearly all modern Non-Windows computer systems the CMOS Real Time Clock (RTC) is set to UTC and the OS ensures the correct local time is presented to users and processes. Windows traditionally expects the RTC to be set to local time but can be configured to also accept UTC. VirtualBox VMs can either be set to use UTC or local time for their virtual RTC. As of version 26204 vboxwrapper automatically forwards the host setting to the VMs.
The VM image that you distribute need contain only the runtime environment for your applications. In particular, it need not contain:
- Development tools such as gcc
- GUI software such as X11, gtk etc.
Reducing the VM image size reduces the network load on your server and on volunteer hosts, and the disk usage on volunteer hosts.
The easiest way to make a "small" Linux VM is to install the network install of Debian within the VM. You can find the netinst images here. Such VMs have VirtualBox and guest additions installed by default. They have the runtime libraries needed to run C and C++ applications.
During install you'll be asked what role should this Linux machine be configured for. Make sure all roles are unselected before continuing.
First step is to remove non-critical packages that are essential (source). For Debian Wheezy 7.6 that are:
acpi acpid busybox debconf-i18n eject groff-base iamerican ibritish info ispell laptop-detect logrotate installation-report manpages man-db net-tools os-prober rsyslog tasksel tasksel-data traceroute usbutils wamerican
Use aptitude to also remove linux-headers-* packages but don't remove dkms or virtualbox-guest-dkms! Run apt-get autoremove after this to get rid of now unused packages. apt-get autoclean and apt-get clean should remove downloaded archives to free up space.
In order to really purge all files from previously removed packages this command is helpful as it purges all configuration files from uninstalled packages:
dpkg --purge `dpkg --get-selections | grep deinstall | cut -f1`
Now, we can remove the contents of the following folders:
- /usr/share/locale/ - since we don't need all locales.
- /usr/share/doc/ - since we don't need the documentation.
- /usr/share/man/ - since we don't need the manuals and we've removed the man program.
- /var/log/ - since we won't be needing all that logging.
- /var/cache/debconf/ - since that cache is disposable.
- /var/lib/apt/lists/ - since those lists are huge and can quickly be recreated with apt-get update.
If you want to speed up the boot process,
change the default timeout for grub by modifying /etc/default/grub:
GRUB_TIMEOUT = 0
After saving the update run:
update-grub
To configure Linux for automatic login you'll need to install a different terminal handler. mingetty works well for our purposes.
To install mingetty:
root@boinc-vm-image:/etc/default# apt-get install mingetty
Next you'll need to change the terminal handler assigned to the first virtual terminal by modifying /etc/inittab.
Change line:
1:2345:respawn:/sbin/getty 38400 tty1
To:
1:2345:respawn:/sbin/mingetty --autologin root --noclear tty1
This will disable the periodic fsck run that occurs every 30 times the VM is turned on: tune2fs -c -1 /dev/sda1.
To get a better compression ratio you may install and run the zerofree tool to overwrite all empty space on the VDI with zeros.
apt-get install zerofree
telinit 1
{enter root password}
mount -o remount,ro /dev/sda1
zerofree -v /dev/sda1
shutdown -hP 0
Now open a terminal on the host System and compact the VDI file:
vboxmanage modifyhd FILENAME.vdi --compact
TODO - EXPLAIN THE FOLLOWING
The state file has the name of vbox_webapi.xml.
It has following structure:
<webapi>
<host_port>X</host_port>
</webapi>
Required elements:
The port, if configured for it, vboxwrapper has assigned to the task for WebAPI requests. (HTTP, XML-RPC, JSON)
The state file has the name of vbox_remote_desktop.xml.
It has following structure:
<remote_desktop>
<host_port>X</host_port>
</remote_desktop>
Required elements:
The port, if configured for it, vboxwrapper has assigned to the task for Remote Desktop requests. (RDP)