Skip to content

Autograder jobs

Jump to bottom
Roger Chen edited this page Jan 8, 2016 · 1 revision

Once you have created your course’s ob2 config directory, you can start configuring your assignments and your autograder jobs.

Creating an autograded assignment

Add the following to your config.yaml inside your course’s config directory:

assignments:

# My first assignment (hw1)
- name:                hw1
  full_score:          1.0
  min_score:           0.0
  max_score:           1.0
  weight:              1.00000
  category:            Homework
  is_group:            false
  manual_grading:      false
  not_visible_before:  "2016-01-01 00:00:00 -0800"
  start_auto_building: "2016-01-01 00:00:00 -0800"
  end_auto_building:   "2020-12-31 00:00:00 -0800"
  due_date:            "2020-12-31 23:59:59 -0800"
  cannot_build_after:  "2020-12-31 23:59:59 -0800"

Then, add the following to your functions.py file:

from ob2.util.hooks import register_job

@register_job("hw1")
def hw1_job_handler(repo, commit_hash):
    print repo, commit_hash
    return "I think this deserves 100%", 1.0

Now, start ob2 and try running your new assignment's autograder.

Manually-graded assignments

Here’s an example of a manually graded assignment. It doesn’t require as many fields as the autograded ones do.

- name:                midterm
  full_score:          100.0
  min_score:           0.0
  max_score:           100.0
  weight:              1.00000
  category:            Exams
  is_group:            false
  manual_grading:      true
  not_visible_before:  "2015-01-01 00:00:00 -0800"
  due_date:            "2015-01-01 00:00:00 -0800"

Improving the organization of your config directory

If you put all of your autograder scripts in functions.py, it will get very cluttered very quickly. You can improve the organization of your config directory by creating a Python module to store your autograder scripts.

Create a new folder inside your config directory called cs162, and then create an empty file named __init__.py inside of it.

cs162/init.py

# This file is empty
# (except for this comment)

Now, your config directory should look like:

config_dir/
config_dir/config.yaml
config_dir/functions.py
config_dir/cs162/
config_dir/cs162/__init__.py

Add the following code to your functions.py file:

functions.py (append to the end of the file)

from os.path import abspath, dirname

import ob2.config as config

# Insert the current directory into sys.path
#
# In order to avoid naming conflicts, we created a new "cs162" module that contains all of our
# custom code. This module has many self-referencing imports, so we need all of Python's module
# infrastructure in order to support it.
current_directory = dirname(abspath(__file__))
sys.path.insert(0, current_directory)

# .. and then make sure all the submodules get initialized.
for assignment in config.assignments:
    if not assignment.manual_grading:
        import_module("cs162.%s" % assignment.name)

# Then, get rid of it, so we don't pollute sys.path any more
sys.path.remove(current_directory)

This code will add the cs162 directory as a Python module. It attempts to import all of your autograding scripts, under the assumption that they will be inside cs162/hw1.py (for an assignment named hw1).

Finally, let's create a new job definition for our hw1 job.

cs162/hw1.py

from ob2.util.hooks import register_job

@register_job("hw1")
def hw1_job_handler(repo, commit_hash):
    return "I live inside a module!", 1.0

You can also create a file containing code that is shared between all the autograder jobs:

cs162/common.py

from os.path import dirname, join, realpath

RESOURCES_DIR = join(dirname(realpath(__file__)), "resources")

And then use this code inside your autograder script:

cs162/hw1.py

import cs162.common as common

from ob2.util.hooks import register_job

@register_job("hw1")
def hw1_job_handler(repo, commit_hash):
    print common.RESOURCES_DIR
    return "I live inside a module!", 1.0

Integrating Docker into your autograder scripts

Octobear 2 comes with a rich library of helper functions to interface with Docker. Let’s start with an example, and then I’ll show you what each of the helper functions does.

from os.path import join
from ob2.dockergrader.helpers import (
    copy,
    copytree,
    download_repository,
    ensure_files_exist,
    ensure_no_binaries,
    extract_repository,
    get_working_directory,
    safe_get_results,
)
from ob2.dockergrader.job import JobFailedError
from ob2.dockergrader.rpc import DockerClient, TimeoutError
from ob2.util.hooks import register_job

from cs162.common import RESOURCES_DIR

# This is the docker image we'll use to run our autograder
docker_image = "cs162:latest"


@register_job("hw1")
def build(source, commit):
    # We're using 2 Python context managers here:
    #   - get_working_directory() of them creates a temporary directory on the host
    #   - DockerClient().start() creates a new Docker container for this autograder job
    #     We mount our temporary directory to "/host" inside the Docker container, so we can
    #     communicate with it.
    with get_working_directory() as wd, \
            DockerClient().start(docker_image, volumes={wd: "/host"}) as container:

        # These helper functions download and extract code from GitHub.
        try:
            download_repository(source, commit, join(wd, "hw1.tar.gz"))
            extract_repository(container, join("/host", "hw1.tar.gz"), "/home/vagrant/ag",
                               user="vagrant")
        except TimeoutError:
            raise JobFailedError("I was downloading and extracting your code from GitHub, but I "
                                 "took too long to finish and timed out. Try again?")

        # For testing purposes, we can use solution code instead of GitHub code to test our
        # autograder. You're free to leave in commented code in order to support this.
        # container.bash("mkdir -p /home/vagrant/ag/hw1", user="vagrant")
        # copytree(join(RESOURCES_DIR, "hw-exec", "solutions"), join(wd, "solutions"))
        # container.bash("cp -R /host/solutions/. /home/vagrant/ag/hw1/", user="vagrant")

        # These functions will raise JobFailedError if they find problems with the student code
        ensure_no_binaries(container, "/home/vagrant/ag")
        ensure_files_exist(container, "/home/vagrant/ag", ["./hw1/Makefile", "./hw1/main.c",
                                                           "./hw1/map.c", "./hw1/wc.c"])

        # Our autograder consists of 2 Python scripts. You are free to use whatever language you
        # want in your autograders, as long as you make sure your Docker container can run them.
        copy(join(RESOURCES_DIR, "hw-exec", "check.py"), join(wd, "check.py"))
        copy(join(RESOURCES_DIR, "common.py"), join(wd, "common.py"))

        # We run our autograder, which produces 2 outputs: a build log and a score.
        try:
            container.bash("""cd /host
                              python2.7 check.py /home/vagrant/ag/hw0 &>build_log 162>score
                           """, user="vagrant", timeout=60)
        except TimeoutError:
            raise JobFailedError("I was grading your code, but the autograder took too long to " +
                                 "finish and timed out. Try again?")

        # This function will safely retrieve the build log and the score value, without throwing
        # unexpected exceptions.
        return safe_get_results(join(wd, "build_log"), join(wd, "score"))

All of the CS162 autograder scripts look something like this.

Some more tips

  • Assignments and autograder jobs are separate entities. Assignments are defined by the 'assignments' section inside config.yaml. Autograder jobs are registered with the @register_job decorator.
  • However, assignments and autograder jobs usually have the same name. An autograder job named hw0 will assign a score to the assignment named hw0.
  • Not all assignments need an autograder job. If an assignment is marked as manual_grading: True, then ob2 does not expect to find an autograder job for it.
Clone this wiki locally