Merge pull request #36 from bcoe/manual-tasks

Manual Tasks Refactor

README.md

@@ -19,18 +19,15 @@ Thumbd requires the following environment variables to be set:
 
 * **AWS_KEY** the key for your AWS account (the IAM user must have access to the appropriate SQS and S3 resources).
 * **AWS_SECRET** the AWS secret key.
-* **AWS_REGION** the AWS Region of the bucket. Defaults to: `us-east-1`.
 * **BUCKET** the bucket to download the original images from. The thumbnails will also be placed in this bucket.
+* **AWS_REGION** the AWS Region of the bucket. Defaults to: `us-east-1`.
 * **CONVERT_COMMAND** the ImageMagick convert command. Defaults to `convert`.
 * **REQUEST_TIMEOUT** how long to wait in milliseconds before aborting a remote request. Defaults to `15000`.
 * **S3_ACL** the acl to set on the uploaded images. Must be one of `private`, or `public-read`. Defaults to `private`.
 * **S3_STORAGE_CLASS** the storage class for the uploaded images. Must be either `STANDARD` or `REDUCED_REDUNDANCY`. Defaults to `STANDARD`.
 * **SQS_QUEUE** the queue name to listen for image thumbnailing.
-	* As of version 2.0.0, the integer identifier from the queue URL is no longer required.
 
-You can export these variables to your environment, or specify them when running the thumbd CLI.
-
-Personally, I set these environment variables in a .env file and execute thumbd using Foreman.
+When running locally, I set these environment variables in a .env file and execute thumbd using Foreman.
 
 Server
 ------
@@ -39,10 +36,10 @@ The thumbd server:
 
 * listens for thumbnailing jobs on the queue specified.
 * downloads the original image from our thumbnailng S3 bucket, or from an HTTP(s) resource.
-	* HTTP resources are prefixed with __http://__ or __https://__.
-	* S3 resources are a path to the image in the S3 bucket indicated by the __BUCKET__ environment variable.
+	* HTTP resources are prefixed with `http://` or `https://`.
+	* S3 resources are a path to the image in the S3 bucket indicated by the `BUCKET` environment variable.
 * Uses ImageMagick to perform a set of transformations on the image.
-* uploads the thumbnails created back to S3, with the following naming convention: [original filename excluding extension]\_[thumbnail suffix].jpg
+* uploads the thumbnails created back to S3, with the following naming convention: `[original filename excluding extension]\_[thumbnail suffix].[thumbnail format]`
 
 Assume that the following thumbnail job was received over SQS:
 
@@ -81,7 +78,7 @@ Once thumbd processes the job, the files stored in S3 will look something like t
 Client
 ------
 
-Submit thumbnailing jobs from your application by creating an instance of a thumbd client (clients will soon be offered for other languages).
+Submit thumbnailing jobs from your application by creating an instance of a thumbd client (contribute by submitting clients in other languages).
 
 ```javascript
 var Client = require('./thumbd').Client,
@@ -108,17 +105,19 @@ The descriptions received in the thumbnail job describe the way in which thumbna
 
 _description_ accepts the following keys:
 
-* **suffix** a suffix describing the thumbnail.
-* **width** the width of the thumbnail.
-* **height** the height of the thumbnail.
-* **background** background color for matte.
-* **format** what should the output format of the image be, e.g., `jpg`, `gif`, defaults to `jpg`.
-* **strategy** indicate an approach for creating the thumbnail.
-	* **matted** maintain aspect ratio, places image on _width x height_ matte.
-	* **bounded (default)** maintain aspect ratio, don't place image on matte.
-	* **fill** both resizes and zooms into an image, filling the specified dimensions.
-    * **strict** resizes the image, filling the specified dimensions changing the aspect ratio
-* **quality** the quality of the thumbnail, in percent. e.g. `90`.
+* **suffix:** a suffix describing the thumbnail.
+* **width:** the width of the thumbnail.
+* **height:** the height of the thumbnail.
+* **background:** background color for matte.
+* **format:** what should the output format of the image be, e.g., `jpg`, `gif`, defaults to `jpg`.
+* **strategy:** indicate an approach for creating the thumbnail.
+	* **bounded (default):** maintain aspect ratio, don't place image on matte.
+	* **matted:** maintain aspect ratio, places image on _width x height_ matte.
+	* **fill:** both resizes and zooms into an image, filling the specified dimensions.
+  * **strict:** resizes the image, filling the specified dimensions changing the aspect ratio
+	* **manual:** allows for a custom convert command to be passed in:
+	  * `%(command)s -border 0 %(localPaths[0])s %(convertedPath)s`
+* **quality:** the quality of the thumbnail, in percent. e.g. `90`.
 
 CLI
 ---
@@ -138,18 +137,39 @@ thumbd thumbnail --remote_image=<path to image s3 or http> --descriptions=<path
 * **remote_image** indicates the S3 object to perform the thumbnailing operations on.
 * **thumbnail_descriptions** the path to a JSON file describing the dimensions of the thumbnails that should be created (see _example.json_ in the _data_ directory).
 
+Advanced Options
+----------------
+
+* **Creating a Mosaic:** Rather than performing an operation on a single S3 resource, you can perform an operation on a set
+of S3 resources. A great example of this would be converting a set of images into a mosaic:
+
+```json
+{
+	"resources": [
+		"images/image1.png",
+		"images/image2.png"
+	],
+	"descriptions": [{
+		"strategy": "%(command)s -border 0 -tile 2x1 -geometry 160x106 '%(localPaths[0])s' '%(localPaths[1])s' %(convertedPath)s",
+		"command": "montage",
+		"suffix": "stitch"
+	}]
+}
+```
+
+The custom strategy can be used for a variety of purposes, _experiment with it :tm:_
+
 Production Notes
 ----------------
 
-At Attachments.me, thumbd thumbnails tens of thousands of images a day. There are a few things you should know about our production deployment:
+At Attachments.me, thumbd thumbnailed tens of thousands of images a day. There are a few things you should know about our production deployment:
 
 ![Thumbd in Production](https://dl.dropboxusercontent.com/s/r2sce6tekfsvolt/thumbnailer.png?token_hash=AAHI0ARNhPdra24jqmDFpoC7nNiNTL8ELwOtaQB_YqVwpg "Thumbd in Production")
 
 * thumbd was not designed to be bullet-proof:
 	* it is run with an Upstart script, which keeps the thumbnailing process on its feet.
 * Node.js is a single process, this does not take advantage of multi-processor environments.
 	* we run an instance of thumbd per-CPU on our servers.
-* we use Foreman's export functionality to simplify the process of creating Upstart scripts.
 * be midful of the version of ImageMagick you are running:
 	* make sure that you build it with the appropriate extensions for images you would like to support.
 	* we've had issues with some versions of ImageMagick, we run 6.6.2-6 in production.
@@ -161,7 +181,10 @@ At Attachments.me, thumbd thumbnails tens of thousands of images a day. There ar
 The Future
 ----------
 
-thumbd is a rough first pass at creating an efficient, easy to deploy, thumbnailing pipeline. Please be liberal with your feature-requests, patches, and feedback.
+thumbd is a rough first pass at creating an efficient, easy to deploy, thumbnailing pipeline. Please be liberal with your feature-requests, patches, and feedback:
+
+* **If you create a client in a language other than JavaScript, let me know.**
+* **If you build something cool using thumbd, I will list it here.
 
 Copyright
 ---------

lib/client.js

@@ -1,22 +1,29 @@
 var aws = require('aws-sdk'),
 	config = require('./config').Config,
-	Saver = require('./saver').Saver;
+	Saver = require('./saver').Saver,
+	_ = require('underscore');
 
 /**
  * Initialize the Client
  *
  * @param object opts The Client options
  */
 function Client(opts) {
-	config.extend(opts);
 
-	// Create an SQS client for
-	// submitting thumbnailing work.
-	this.sqs = new aws.SQS({
-		accessKeyId: config.get('awsKey'),
-		secretAccessKey: config.get('awsSecret'),
-		region: config.get('awsRegion')
-	});
+	// allow sqs to be overridden
+	// in tests.
+	if (opts && opts.sqs) {
+		this.sqs = opts.sqs;
+		delete opts.sqs;
+	} else {
+		this.sqs = new aws.SQS({
+			accessKeyId: config.get('awsKey'),
+			secretAccessKey: config.get('awsSecret'),
+			region: config.get('awsRegion')
+		});
+	}
+
+	config.extend(opts);
 
 	config.set('sqsQueueUrl', this.sqs.endpoint.protocol + '//' + this.sqs.endpoint.hostname + '/' + config.get('sqsQueue'));
 }
@@ -36,23 +43,45 @@ Client.prototype.upload = function(source, destination, callback) {
 /**
  * Submit a thumbnailing job over SQS.
  *
- * @param string originalImagePath Path to the image in S3 that thumbnailing should be performed on.
+ * @param string originalImagePaths Path to the image in S3 that thumbnailing should be performed on,
+ *    can optionally be an array of resources.
  * @param array thumbnailDescriptions Thumbnailing meta information, see README.md.
+ * @param object opts additional options
+ *   @opt prefix alternative prefix for saving thumbnail.
  * @param function callback The callback function. Optional.
  */
-Client.prototype.thumbnail = function(originalImagePath, thumbnailDescriptions, callback) {
+Client.prototype.thumbnail = function(originalImagePaths, thumbnailDescriptions, opts, callback) {
 	/**
 		job = {
-			"original": "/foo/awesome.jpg",
+			"resources": [
+				"/foo/awesome.jpg"
+			],
+			"prefix": "/foo/awesome",
 			"descriptions": [{
 				"suffix": "small",
 				"width": 64,
 				"height": 64
 			}],
 		}
 	*/
+
+	// additional options can be provided.
+	if (typeof opts === 'function') {
+		callback = opts;
+		opts = {};
+	}
+
+	// allow for either a single S3 resource, or an array.
+	if (!_.isArray(originalImagePaths)) originalImagePaths = [originalImagePaths];
+
+	// override defaults with opts.
+	opts = _.extend({
+		prefix: originalImagePaths[0].split('.').slice(0, -1).join('.')
+	}, opts);
+
 	this.sqs.sendMessage({QueueUrl: config.get('sqsQueueUrl'), MessageBody: JSON.stringify({
-		original: originalImagePath,
+		resources: originalImagePaths,
+		prefix: opts.prefix,
 		descriptions: thumbnailDescriptions
 	})}, function (err, result) {
 		if (callback) callback(err, result);

lib/thumbnailer.js

@@ -1,4 +1,5 @@
 var exec = require('child_process').exec,
+	sprintf = require('sprintf-js').sprintf,
 	_ = require('underscore'),
 	fs = require('fs'),
 	config = require('./config').Config;
@@ -21,19 +22,25 @@ function Thumbnailer(opts) {
  * @param localPath The local path to the image
  * @param function onComplete The callback function
  */
-Thumbnailer.prototype.execute = function(description, localPath, onComplete) {
+Thumbnailer.prototype.execute = function(description, localPaths, onComplete) {
 	var _this = this;
 
+	// Convert single path to array
+	if (!_.isArray(localPaths)) {
+		localPaths = [localPaths];
+	}
+
 	// parameters for a single execution
 	// of the thumbnailer.
 	_.extend(this, {
-		localPath: localPath,
+		localPaths: localPaths,
 		width: description.width,
 		height: description.height,
 		format: (description.format || 'jpg'),
 		strategy: (description.strategy || 'bounded'),
 		background: (description.background || 'black'),
 		quality: (description.quality || null),
+		command: (description.command || config.get('convertCommand')),
 		onComplete: onComplete,
 		thumbnailTimeout: 20000
 	});
@@ -45,15 +52,31 @@ Thumbnailer.prototype.execute = function(description, localPath, onComplete) {
 			return;
 		}
 
-		// apply the thumbnail creation strategy.
-		if (!_this[_this.strategy]) {
-			_this.onComplete('could not find strategy ' + _this.strategy);
-		} else {
-			_this[_this.strategy]();
-		}
+		_this[_this._guessStrategy()]();
 	});
 };
 
+/**
+ * Choose an appropriate image manipulation
+ * strategy, based on 'strategy' key in job.
+ * If the strategy contains, %(command)s, assume
+ * manual strategy:
+ *
+ * "%(command)s -border 0 -tile 3x1 -geometry 160x106 "%(localPaths[0])s" "%(localPaths[1])s" "%(localPaths[2])s" -quality 90 %(convertedPath)s"
+ *
+ * @return string strategy to execute.
+ * @throw strategy not found.
+ */
+Thumbnailer.prototype._guessStrategy = function() {
+	if (this.strategy.match(/%\(.*\)s/)) {
+		return 'manual'
+	} else if (!this[this.strategy]) {
+		this.onComplete(Error('could not find strategy ' + this.strategy));
+	} else {
+		return this.strategy;
+	}
+}
+
 /**
  * Create a temp file for the converted image
  *
@@ -99,12 +122,34 @@ Thumbnailer.prototype.execCommand = function(command) {
 	});
 };
 
+/**
+ * Convert the image using the manual strategy.
+ * looks for a strategy of the form:
+ *
+ * "%(command)s -border 0 -tile 3x1 -geometry 160x106 '%(localPath[0])s' '%(localPath[1])s' '%(localPath[2])s' -quality 90 %(convertedPath)s
+ *
+ * The custom strategy has access to all variables set on
+ * the thumbnailer instance:
+ *   * command: the conversion command to run.
+ *   * localPaths: the local temp images to apply operation to.
+ *   * convertedPath: path to store final thumbnail to on S3.
+ */
+Thumbnailer.prototype.manual = function() {
+	try {
+		var thumbnailCommand = sprintf(this.strategy, this);
+	} catch (err) {
+		this.onComplete(err);
+	}
+
+	this.execCommand(thumbnailCommand);
+};
+
 /**
  * Convert the image using the matted strategy
  */
 Thumbnailer.prototype.matted = function() {
 	var qualityString = (this.quality ? '-quality ' + this.quality : ''),
-		thumbnailCommand = config.get('convertCommand') + ' "' + this.localPath + '[0]" -thumbnail ' + (this.width * this.height) + '@ -gravity center -background ' + this.background + ' -extent ' + this.width + 'X' + this.height + ' ' + qualityString + ' ' + this.convertedPath;
+		thumbnailCommand = config.get('convertCommand') + ' "' + this.localPaths[0] + '[0]" -thumbnail ' + (this.width * this.height) + '@ -gravity center -background ' + this.background + ' -extent ' + this.width + 'X' + this.height + ' ' + qualityString + ' ' + this.convertedPath;
 
 	this.execCommand(thumbnailCommand);
 };
@@ -115,7 +160,7 @@ Thumbnailer.prototype.matted = function() {
 Thumbnailer.prototype.bounded = function() {
 	var dimensionsString = this.width + 'X' + this.height,
 		qualityString = (this.quality ? '-quality ' + this.quality + ' ' : ''),
-		thumbnailCommand = config.get('convertCommand') + ' "' + this.localPath + '[0]" -thumbnail ' + dimensionsString + ' ' + qualityString + this.convertedPath;
+		thumbnailCommand = config.get('convertCommand') + ' "' + this.localPaths[0] + '[0]" -thumbnail ' + dimensionsString + ' ' + qualityString + this.convertedPath;
 
 	this.execCommand(thumbnailCommand);
 };
@@ -126,7 +171,7 @@ Thumbnailer.prototype.bounded = function() {
 Thumbnailer.prototype.fill = function() {
 	var dimensionsString = this.width + 'X' + this.height,
 		qualityString = (this.quality ? '-quality ' + this.quality : ''),
-		thumbnailCommand = config.get('convertCommand') + ' "' + this.localPath + '[0]" -resize ' + dimensionsString + '^ -gravity center -extent ' + dimensionsString + ' ' + qualityString + ' ' + this.convertedPath;
+		thumbnailCommand = config.get('convertCommand') + ' "' + this.localPaths[0] + '[0]" -resize ' + dimensionsString + '^ -gravity center -extent ' + dimensionsString + ' ' + qualityString + ' ' + this.convertedPath;
 
 	this.execCommand(thumbnailCommand);
 };