Skip to content

Setting up ob2 (the hard way)

Jump to bottom
Roger Chen edited this page Jan 25, 2016 · 4 revisions

The CS162 staff runs ob2 on a Ubuntu 14.04LTS AMD64 virtual machine, which is provided by EECS instructional support. These instructions have been designed for a similar host machine, but they can be adapted to another system.

Reminder: Turn off debug mode in production

Don’t forget to disable the debug_mode setting in production.

Docker

Docker is the most complex part of ob2’s requirements, so we take time to explain it in detail. You should make sure that each of these requirements is satisfied by your server.

Docker PPA

You should NOT use the docker package that comes with Ubuntu 14.04LTS, because it is very out of date. Instead, we use the official Docker Ubuntu repository, which is hosted at https://get.docker.io/ubuntu with the signing key 36A1D7869245C8950F966E92D8576A8BA88D21E9 (available at keyserver.ubuntu.com). Once you’ve added this repository, install the lxc-docker package (NOT docker or docker.io).

AUFS or other layer-based file system

This is not a strict requirement, but you will suffer severe performance issues if you do not use a layered file system for Docker. A layered file system uses copy-on-write semantics to provide very fast provisioning for new VM's. We use AUFS (ships with Docker by default) in CS162.

You will need to make sure you have the linux-image-extra-* Ubuntu package, which adds kernel support for AUFS. This package may have come pre-installed on your system, but you should make sure it is installed. (This is automatically installed by the Vagrant provisioner, if you’re using Vagrant.)

sudo apt-get install linux-image-extra-$(uname -r)

A reboot is required after installing this package. You can use docker info | grep Storage\ Driver to check if AUFS is enabled.

Docker access permissions

You do not need to run ob2 as root, and you probably shouldn’t. In CS162, we run ob2 as the cs162ta user. You need to make sure that ob2’s user account is in the docker group, which gives it access to /var/run/docker.sock (0660 root:docker, by default). If you're using Vagrant, the vagrant user is added to the docker group automatically.

Docker image

In order to best use Docker, you probably want to prepare a Docker image that contains all of the build dependencies and tools required to run your autograding scripts. This may include programming language runtimes, compilers, shared libraries, and debugging tools. You can create this image however you want to, but make sure you tag your image after you’re done. When ob2 starts, it stops all containers and deletes all untagged images. The assumption is that leftover containers were probably the result of a previous ob2 instance that exited or crashed.

For example, CS162 takes the following steps to build its Docker image:

# Get our Puppet-based VM provisioner from GitHub
git clone https://github.com/Berkeley-CS162/vagrant.git

# Go into the project directory
cd vagrant/

# Build the Docker image using the Dockerfile
docker build .

# Get the image ID and copy it
docker images

# Tag the image (replace the X's with the actual image ID)
docker tag XXXXXXXX cs162:latest

Docker AppArmor Profiles

By default, Docker runs with a very restrictive set of permissions to provide security and isolation. On Ubuntu, AppArmor is used to enforce these restrictions. CS162 wanted to enable a container to ptrace its own processes, so we created our own AppArmor profile, which is provided below. You can set a custom AppArmor profile by changing the dockergrader_apparmor_profile configuration value.

#include <tunables/global>


profile ob2docker flags=(attach_disconnected,mediate_deleted) {

  #include <abstractions/base>


  network,
  capability,
  file,
  umount,
  ptrace peer=@{profile_name},

  deny @{PROC}/sys/fs/** wklx,
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/mem rwklx,
  deny @{PROC}/kmem rwklx,
  deny @{PROC}/sys/kernel/[^s][^h][^m]* wklx,
  deny @{PROC}/sys/kernel/*/** wklx,

  deny mount,

  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/efi/efivars/** rwklx,
  deny /sys/kernel/security/** rwklx,
}

This AppArmor profile is also installed by the Vagrant provisioner, if you’re using Vagrant.

Python and SQLite

There are many Python packages that ob2 depends upon. Some of these packages also depend on system libraries. On Ubuntu, you can install the required system libraries with:

sudo apt-get install python-dev libffi-dev libssl-dev

This is done automatically by the Vagrant provisioner if you’re using Vagrant.

You should use a virtualenv to contain the Python packages needed for ob2. Use pip to install the packages in requirements.txt.

APSW

There’s one more Python package that you need to install, but is not available via pip: APSW (Another Python SQLite Wrapper). This package is the interface between ob2 and the database, and it was chosen specifically for its superior architecture to competing Python/SQLite libraries. The included ./build_apsw.sh will download APSW from GitHub and install it in the local Python environment (you can/should run this script in a virtualenv).

GitHub Integration

You need to set up GitHub OAuth in order to log in to the autograder. There are other GitHub integration points that should be configured as well. See the GitHub integration guide article for more information.

Autograder Jobs

You must configure assignments and autograder jobs before ob2 will do anything useful. See the Autograder jobs article for more information.

NGINX (or another reverse proxy)

You should use NGINX (or similar) to proxy connections to the autograder. This is useful for multiplexing multiple websites (you might want to run a course website in addition to the autograder) and HTTPS termination (required if you want your autograder to be secure). Here is an abbreviated version of the CS162 virtual host configuration:

server {
    listen 80;
    server_name cs162.eecs.berkeley.edu;
    return 302 https://$server_name$request_uri;
}

server {
    listen 443 ssl;
    server_name cs162.eecs.berkeley.edu;

    # (SSL settings are omitted)
    # You should consult your system administrator for more
    # information about your server's SSL certificate

    # For the course website (runs on localhost port 5000)
    location / {
        proxy_pass         http://127.0.0.1:5000/;

        proxy_redirect     off;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $server_name;
        proxy_set_header   X-Forwarded-Proto $scheme;
    }

    # For the autograder (runs on localhost port 5002)
    location /autograder {
        proxy_pass         http://127.0.0.1:5002/autograder;

        proxy_redirect     off;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $server_name;
        proxy_set_header   X-Forwarded-Proto $scheme;
    }

}

You don't need to serve static files via NGINX. The autograder web server will take care of that for you, and the overhead is negligible because of the low rate of traffic.

Clone this wiki locally