Clockwork Raven is a web application that allows users to easily interact with Mechanical Turk. It is actively used at Twitter to gather thousands of judgments every day.
If you've ever wanted a friendlier interface to Mechanical Turk, then Clockwork Raven is for you.
Clockwork Raven is designed for individuals, or organizations that share a single Mechanical Turk account. Users can upload data, design evaluations to send to Mechanical Turk through a simple, drag-and-drop interface, and review results all in a single place. It also provides a Thrift API for users who wish to programmatically run human computation tasks or gather real-time judgments.
To control the quality of responses, Clockwork Raven allows you to restrict tasks to Mechanical Turk's "Categorization Masters", or to only those users whom you have marked as Trusted in Clockwork Raven.
- Check out the code from https://github.com/twitter/clockworkraven
- Requirements:
- Make sure the machine that you're using has Ruby 1.9.3 installed. The easiest way to install and manage Ruby is with RVM.
- You'll need the RubyGem "bundler" installed, and then just run
bundle install
from the Clockwork Raven directory to install all of the libraries needed by Clockwork Raven. - Clockwork Raven uses Resque to run
tasks in the background. Resque requires a Redis server -- see
Resque's instructions for installing Redis.
By default Clockwork Raven assumes your Redis server is running on
localhost:6379
. If this isn't the case, editconfig/resque.yml
. - In a production environment (e.g. any environment where Clockwork Raven will be accessible to users), it should be run over SSL to protect users' credentials when they log in. If you don't use SSL, these credentials will be sent over the network in the clear!
- Configure:
-
Generate a secret key. Copy
config/secret.example.yml
toconfig/secret.yml
. Then, runrake secret
and copy the output toconfig/secret.yml
. -
Copy
config/database.example.yml
toconfig/database.yml
and modify it to point to your MySQL database. Currently, Clockwork Raven only supports MySQL. -
Copy
config/mturk.example.yml
toconfig/mturk.yml
. Follow the instructions in that file to connect Clockwork Raven to your Mechanical Turk account. -
Configure authentication:
LDAP Authentication
LDAP authentication is the recommended way to manage account in Clockwork Raven. If your LDAP server supports SSL/TLS, copy
config/auth.example_ldap_encrypted.yml
toconfig/auth.yml
. If your LDAP server does not, copyconfig/auth.example_ldap_unencrypted.yml
toconfig/auth.yml
. Follow the instructions in that file to connect Clockwork Raven to your LDAP server and grant access to specific LDAP groups and users.Password Authentication
If you can't use an LDAP server, you can configure Clockwork Raven to use "password authentication," which will allow you to manually create accounts. Copy
config/auth.example_password.yml
toconfig/auth.yml
. Then, you can create accounts by running "rake users:add" and change passwords withrake users:change_password
. Note that you will need to set up your database (explained below) before using these rake tasks.
-
- Set up the database. If the databases you configured Clockwork Raven to use in
config/database.yml
do not exist, runrake db:create
to create them. Then, runrake db:structure:load
to load the database structure into your database. - Start up the background workers. Just
run
rake raven:resque
to start up 4 background workers. You can start up more background workers by passing an argument to the rake task:rake raven:resque[16]
will start up 16 background workers. - Run the server. To run the server, run
rails server
.
Administrators can separate users into "privileged" and "unprivileged" sets manually or based on LDAP groups. Unprivileged uses are not allowed to spend money, but can submit evaluations to the Mechanical Turk sandbox to test out the system.
Clockwork Raven is designed for situations where everyone who has access to the system is relatively trusted. Because the form designer allows users to use arbitrary HTML, anyone with access to the system could execute an XSS attack and compromise the system.
Documentation is available on the wiki.
Follow @clockworkraven for updates and notifications. Submit bug report and feature requests to the issue tracker.
Join the mailing list, [email protected], on Google Groups to ask questions and discuss development.
We would love any help adding ideas or implementing them!
- JSON/REST API.
- Provide the option to have multiple Mechanical Turk users complete each task.
- Provide in-depth analytics about workers and automate the process of choosing trusted workers.
To contribute to Clockwork Raven, fork the repo, make your changes, and
submit a pull request. All pull requests should be against *-wip
branches.
Nothing gets committed/merged directly to master. To merge your pull request,
you'll need to include appropriate documentation and tests. Get in touch if you
have any questions about what you need to do to get your contributions accepted.
- Ben Weissmann, @benweissmann
- Edwin Chen, @echen
- Dave Buchfuhrer, @daveFNbuck
The current version is in the VERSION file and accessible in the code as ClockworkRaven::VERSION. Releases will be tagged with their release number in Git.
Clockwork Raven uses semantic versioning. Basically, this means that versions will be of the form X.Y.Z, where X is the major version (incremented when backwards-incompatible changes are introduced), Y is the minor version (incremented when backwards-compatible features are introduced), and X is the patch number (incremented when backwards-compatible bug fixes are introduced). Note, however, that these are only hard rules once Clockwork Raven reaches 1.x. Until then, we will do our best to adhere to these policies (particularly with regards to not introducing backwards-incompatible changes in patch releases), but we may make backwards-incompatible changes while only incrementing the minor version number.
Copyright 2012 Twitter, Inc.
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0