0-planout-intro.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:da46e81e5a6940ad3b5405735709fdfe2531b930aa029d0cd0312a42fc2b369c"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Introduction to PlanOut for computational social science"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "A simple A/B test (single factor experiment)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This imports basic operators for doing random assignment and SimpleExperiment, the base class for logging"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from planout.ops.random import *\n",
      "from planout.experiment import SimpleExperiment\n",
      "import pandas as pd"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can define new experiments by subclassing `SimpleExperiment`, and implementing an `assign()` method."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class VotingExperiment(SimpleExperiment):\n",
      "  def assign(self, params, userid):\n",
      "    params.button_text = UniformChoice(\n",
      "      choices=[\"I'm a voter\", \"I'm voting\"],\n",
      "      unit=userid)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can get randomized assignments for your input units by creating instances of the class. The code below gets the parameter values, `button_text` for `userid` = 4."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "e = VotingExperiment(userid=212)\n",
      "print e.get('button_text')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "I'm voting\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here are the assignments for 10 userids."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for i in xrange(10):\n",
      "    e = VotingExperiment(userid=i)\n",
      "    print \"cookie = %s: %s\" % (i, e.get('button_text'))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "cookie = 0: I'm a voter\n",
        "cookie = 1: I'm a voter\n",
        "cookie = 2: I'm voting\n",
        "cookie = 3: I'm voting\n",
        "cookie = 4: I'm voting\n",
        "cookie = 5: I'm voting\n",
        "cookie = 6: I'm a voter\n",
        "cookie = 7: I'm a voter\n",
        "cookie = 8: I'm voting\n",
        "cookie = 9: I'm a voter\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To check to see that the experiment is doing what we expect it to, we can simulate assignments for many userids and construct a dataframe with all of the assignments:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "e.get_params()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 5,
       "text": [
        "{'button_text': \"I'm a voter\"}"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sim_users = [VotingExperiment(userid=i).get_params() for i in xrange(10000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)\n",
      "print assignments.groupby(['button_text']).size()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "button_text\n",
        "I'm a voter    4959\n",
        "I'm voting     5041\n",
        "dtype: int64\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Factorial designs"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Parameter assigns (by default) are performed independently, so a 2x2 design can be constructed simply by setting another parameter."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class VotingExperiment(SimpleExperiment):\n",
      "  def assign(self, params, userid):\n",
      "    params.button_text = UniformChoice(\n",
      "      choices=[\"I'm a voter\", \"I'm voting\"], unit=userid)\n",
      "    params.has_social_cues = UniformChoice(\n",
      "      choices=[1, 0], unit=userid)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "e = VotingExperiment(userid=4)\n",
      "print e.get('button_text'), e.get('has_social_cues')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "I'm voting 0\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A few examples of assignments produced by the data..."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sim_users = [VotingExperiment(userid=i).get_params() for i in xrange(10000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)\n",
      "print assignments[:5]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "   button_text  has_social_cues\n",
        "0  I'm a voter                0\n",
        "1  I'm a voter                0\n",
        "2   I'm voting                0\n",
        "3   I'm voting                1\n",
        "4   I'm voting                0\n",
        "\n",
        "[5 rows x 2 columns]\n"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Crosstabs: "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print assignments.groupby(['button_text', 'has_social_cues']).size()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "button_text  has_social_cues\n",
        "I'm a voter  0                  2488\n",
        "             1                  2471\n",
        "I'm voting   0                  2581\n",
        "             1                  2460\n",
        "dtype: int64\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Unequal probability assignment with `WeightedChoice`"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The `WeightedChoice` operator lets you choose among multiple choices with different frequencies. The `weights` parameter is any set of weights (integer or floating point) to select among `choices`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class VotingExperiment(SimpleExperiment):\n",
      "  def assign(self, params, userid):\n",
      "    params.button_text = UniformChoice(\n",
      "      choices=[\"I'm a voter\", \"I'm voting\"], unit=userid)\n",
      "    params.has_social_cues = WeightedChoice(\n",
      "      choices=[1, 0], weights=[0.8, 0.2], unit=userid)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The button text frequencies reflect these weights, while the button color continues to be split in equal proportions."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sim_users = [VotingExperiment(userid=i).get_params() for i in xrange(2000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)\n",
      "print assignments.groupby(['button_text', 'has_social_cues']).size()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "button_text  has_social_cues\n",
        "I'm a voter  0                  191\n",
        "             1                  793\n",
        "I'm voting   0                  191\n",
        "             1                  825\n",
        "dtype: int64\n"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Arbitrary cell probabilites in a factorial design"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Consider the case where there are organizational factors that limit the size of your experiment. For example, if in a hypothetical get out the vote experiment at Facebook only 5% of the users could be used as part of an experiment used to evaluate the effects of various encouragements to vote.\n",
      "\n",
      "Because of this, the experimenters decided they wanted to assign users with a matrix of probabilities like:\n",
      "\n",
      "$$ \\left[\n",
      "  \\begin{array}{ l c c }\n",
      "     & \\text{has megaphone} & \\text{no megaphone} \\\\\n",
      "     \\text{has feed} & 0.95 & 0.015 \\\\\n",
      "     \\text{no feed} & 0.02 & 0.015\n",
      "  \\end{array} \\right]$$\n",
      "  \n",
      "There are multiple ways to do this."
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "Method 1: Conditional probabilities"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Method 1 first assigns users to see or not see the megaphone with probability 0.97. Half of those 3% are then assigned to see feed stories about voting, while 98% of those who see megaphones are assigned to the see the stories. This produces the assignment probabilities given about."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class VotingExperiment(SimpleExperiment):\n",
      "  def assign(self, params, userid):\n",
      "    params.has_megaphone = BernoulliTrial(\n",
      "      p=0.97, unit=userid)\n",
      "    cond_probs = [0.5, 0.98]\n",
      "    params.has_feed = BernoulliTrial(\n",
      "      p=cond_probs[params.has_megaphone], unit=userid)\n",
      "\n",
      "sim_users = [VotingExperiment(userid=i).get_params() for i in xrange(10000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)\n",
      "print assignments.groupby(['has_feed', 'has_megaphone']).size()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "has_feed  has_megaphone\n",
        "0         0                 159\n",
        "          1                 201\n",
        "1         0                 150\n",
        "          1                9490\n",
        "dtype: int64\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "x = 2\n",
      "print x"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "2\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "Method 2: Binary encoding of cell positions"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Method 2 selects the four conditions with their respective probabilities from a flat list. It then picks out elements of an array corresponding to the cell positions. For a very small number of subjects, this method will produce slightly more imbalance, but for larger N it will acheive similar levels of balance compared to method 1."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class VotingExperiment(SimpleExperiment):\n",
      "  def assign(self, params, userid):\n",
      "    params.idx = WeightedChoice(\n",
      "      choices=[0,1,2,3], weights=[0.95, 0.02, 0.015, 0.015], unit=userid)\n",
      "    params.has_megaphone = [1,1,0,0][params.idx]\n",
      "    params.has_feed = [1,0,1,0][params.idx]\n",
      "    \n",
      "sim_users = [VotingExperiment(userid=i).get_params() for i in xrange(10000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)\n",
      "print assignments.groupby(['has_feed', 'has_megaphone']).size()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "has_feed  has_megaphone\n",
        "0         0                 166\n",
        "          1                 212\n",
        "1         0                 133\n",
        "          1                9489\n",
        "dtype: int64\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "More than two factor levels"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Small changes in appearance can have significant effects on individual behavior. Google is infamous for testing [41 different shades of blue](http://www.nytimes.com/2009/03/01/business/01marissa.html) for their link colors. Let's implement that experiment."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class ColorExperiment(SimpleExperiment):\n",
      "  def assign(self, params, userid):\n",
      "    params.blue_value = RandomInteger(min=215, max=255, unit=userid)\n",
      "    params.link_color = '#0000%s' % format(params.blue_value, '02x')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ColorExperiment(userid=10).get_params()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 17,
       "text": [
        "{'blue_value': 238, 'link_color': '#0000ee'}"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sim_users = [ColorExperiment(userid=i).get_params() for i in xrange(20000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "assignments[:5]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>blue_value</th>\n",
        "      <th>link_color</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 231</td>\n",
        "      <td> #0000e7</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 223</td>\n",
        "      <td> #0000df</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 226</td>\n",
        "      <td> #0000e2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 242</td>\n",
        "      <td> #0000f2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 247</td>\n",
        "      <td> #0000f7</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>5 rows \u00d7 2 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 19,
       "text": [
        "   blue_value link_color\n",
        "0         231    #0000e7\n",
        "1         223    #0000df\n",
        "2         226    #0000e2\n",
        "3         242    #0000f2\n",
        "4         247    #0000f7\n",
        "\n",
        "[5 rows x 2 columns]"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "assignments['blue_value'].hist(bins=41);"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEACAYAAABS29YJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X9s3VX9x/FXR2vkxyYzslvshVzWn7sbtAXcjIlJl9qO\nEJnjRyoVSAfDP1xIRJOlA2LEKLkXDfplmP2jwzXBwKaEdSFQK2bvBEmwCFtMVnRza7W/dgVKx1h0\nZb3v7x9bL+vWe+9pez/3fe7u65E02+353Huf59P2tD29ty1RVQUREV3UFlkHEBFR8LjYExEVAS72\nRERFgIs9EVER4GJPRFQEuNgTERUBp8V+YmICd911F1asWIFoNIq//OUvGB8fR0tLC2pqatDa2oqJ\niYnU8bFYDNXV1airq0Nvb29g8URE5MZpsf/ud7+LW2+9Fe+++y7+9re/oa6uDvF4HC0tLTh06BCa\nm5sRj8cBAP39/di1axf6+/vR09ODzZs3I5lMBjoJIiLKLOtif/z4cbz++ut44IEHAAClpaX43Oc+\nh71796KjowMA0NHRgT179gAAuru70d7ejrKyMkQiEVRVVaGvry/AKRARUTZZF/uBgQFcddVVuP/+\n+3HjjTfi29/+Nk6ePIlEIoFQKAQACIVCSCQSAIDR0VGEw+HU9cPhMEZGRgLKJyIiF1kX+9OnT+Od\nd97B5s2b8c477+Dyyy9PbdlMKykpQUlJSdrbyDRGRETBK812QDgcRjgcxpe+9CUAwF133YVYLIby\n8nIcO3YM5eXlGBsbw7JlywAAFRUVGBoaSl1/eHgYFRUVM26zoqICo6OjuZwHEdFFr7KyEv/85z/n\ndd2sX9mXl5fjmmuuwaFDhwAAr732GlauXInbbrsNXV1dAICuri5s2LABALB+/Xq88MILmJycxMDA\nAA4fPozVq1fPuM3R0VGoqvcvP/zhD80b2MnOQm1kZ+5fjhw5Mq+FHnD4yh4AnnnmGdxzzz2YnJxE\nZWUlfvOb32BqagptbW3YsWMHIpEIdu/eDQCIRqNoa2tDNBpFaWkptm/fXrDbOIODg9YJTtiZW4XQ\nWQiNADt94rTY19fX46233rrg9a+99tqsxz/66KN49NFHF1ZGREQ5w2fQZrBx40brBCfszK1C6CyE\nRoCdPilR1bz/8ZKSkhIY3C0RUUFbyNrJr+wzEBHrBCfszK1C6CyERoCdPuFiT0RUBLiNQ0RUILiN\nQ0REGXGxz6BQ9vHYmVuF0FkIjQA7fcLFnoioCHDPnoioQHDPnoiIMuJin0Gh7OOxM7cKobMQGgF2\n+oSLPRFREeCePRFRgeCePRERZcTFPoNC2cdjZ24VQmchNALs9AkXe48tWfL51N/3ne1lyZLPWycS\nUYHgnr3HzvyFr0znieeRqJhwz56IiDLiYp9BoezjsTO3CqGzEBoBdvqEiz0RURHgnr3HfN+zX7Lk\n8zhx4sOMxyxevBQffTSepyKii9tC1k4u9h7zfbHP3gdYNxJdTPgD2oAUyj4eO3OrEDoLoRFgp0+4\n2BMRFQFu43iM2zhEdC5u4xARUUZc7DMolH08duZWvjvn82sxeC5zq1A6F6LUOoCo2J15+Gr6b81P\nnCjJXwxdtJz27CORCJYsWYJLLrkEZWVl6Ovrw/j4OL75zW/iX//6FyKRCHbv3o0rr7wSABCLxfDs\ns8/ikksuwbZt29Da2jrzTrln74R79sXB97cznZHteSX5eE5J4Hv2JSUlEBHs378ffX19AIB4PI6W\nlhYcOnQIzc3NiMfjAID+/n7s2rUL/f396OnpwebNm5FMJucV5zP+RkqiTxXDx8On34HN/pLtCYbW\nnPfsz/9ssnfvXnR0dAAAOjo6sGfPHgBAd3c32tvbUVZWhkgkgqqqqtQniEKTaR/Ppzd8oew3zqfT\nYhEphPM518Zs57Gk5DMLOs/pPx72oRAWwny9zS0/KTp/Zf+1r30NN998M371q18BABKJBEKhEAAg\nFAohkUgAAEZHRxEOh1PXDYfDGBkZyXU3FQmfPqkWsmznEfgk4zjPc25Yvj87/YD2jTfewNVXX433\n3nsPLS0tqKurmzE+/VkpnUxj85Gv38nS1NS0oOvnSyF3urwt860QzmchNJ7RZB3gpHDO5/w5LfZX\nX301AOCqq67C7bffjr6+PoRCIRw7dgzl5eUYGxvDsmXLAAAVFRUYGhpKXXd4eBgVFRUX3ObGjRsR\niUQAAFdeeSUaGhpSJ/yyyxbjv//9OEvVPnz6jiRn//308okTa1NHTn+LNn37ubr8qQvvf8aop7cf\n/Pyz396ZhX5fhuuXLOj2C+Hyp3OafX7T18nX++t8398Wen3fL+dqfnN5fxYR7Ny5EwBS6+W8aRYn\nT57Ujz76SFVVP/74Y/3KV76if/jDH3TLli0aj8dVVTUWi2lnZ6eqqh48eFDr6+v11KlTevToUV2+\nfLkmk8kZt5ntbgEooBleso1nv4/Fi5dm+p5WFy9eqvv27VtQ40K53kemziDN9e0wW+fC39YLfzuf\nL9/ncz5zPLcx2xw/fQnuPKe//r6cfTwEyeVtnouP+Vyc5/nK+pV9IpHA7bffDgA4ffo07rnnHrS2\ntuLmm29GW1sbduzYkXroJQBEo1G0tbUhGo2itLQU27dvz/k2jptSh/vVtCN8bPPFoRgew55tjmcU\n/jyz8eGhkT7z8nfjuDzu2O2deyG3sfDGhZ5a3x9/nYvH2S/8bW3/dsrG7ecSQc7xzG1c7Oc56IZc\n3P5Cb4O/G4dm5cdjn0szNhSD7I+EIQoeF/sMCuHx1kD6Tj8etnj6nPvcN0uHfwrh7V4IjWeIdYCT\nwjmf88fFnojMZX/SVyE8Czfzd7HW38lyzz7DuPUeZS7292z3MM/cx0LfDj6fAxf2P5dwuw/L8+w2\nhzKc+U4xE9s9+1y8HYLaszf7rZenTp2yumtHLo/mIaL8md4STCfbx2vmj+mL/dE6Zts4l1++ZNaX\nyy67wirpPKcx+x5zbvaaXb5tdVU4+41iHTCL7N96+7h9kP+3+Xx/0C75CnRw7s+Pzn/Z58kzuYN7\nQIPZYj81dWrWl2TyL1ZJeZX9ERp+/vDy4jPbAjDzk7wfi4C1TAsl31dzJ7jzbLZnnz78HQA3ZRgH\n8rVnv7Dxhe4vujVwzz4/P/+x3ev1Y8++ED4eCuF9KcjGTPiXqgKz0P3FfMj2c4kynPltiJSJj7/M\nzT+F8PFwceNDLzMS64AsFrq/l+1bxsy/9nbuZB7XsSBzOtriSVP8OU2uiXVA4PiVfUGbXqwFs/8q\nWX61dHHgI8No4bhn7+24Dw2FMQfrPftc7MlfDG+Hi2EO3LMn8ha/6iVywT37jMQ6wJFYBziSAG4z\niIeqSU7KgiXWAY7EOsCRWAcEjos9EVER4J69t+M+NHAOZyz8d7LYz8G6oTDmwD17oqLGx4hT4eM2\nTkZiHeBIrAMciXWAI7EOcCDWAY7EOsCR4GL/Qzv8yp6ICMDF/h0c9+y9HfehgXPwo4Fz8KPBjznw\nb9ASEVFaXOwzEusAR2Id4EisAxyJdYADsQ5wJNYBjsQ6IHBc7ImIigD37L0d96GBc/CjgXPwo8GP\nOXDPnoiI0uJin5FYBzgS6wBHYh3gSKwDHIh1gCOxDnAk1gGB42JPRFQEuGfv7bgPDZyDHw2cgx8N\nfsyBe/ZERJSW02I/NTWFxsZG3HbbbQCA8fFxtLS0oKamBq2trZiYmEgdG4vFUF1djbq6OvT29gZT\nnTdiHeBIrAMciXWAI7EOcCDWAY7EOsCRWAcEzmmxf/rppxGNRlO/DCgej6OlpQWHDh1Cc3Mz4vE4\nAKC/vx+7du1Cf38/enp6sHnzZiSTyeDqiYjISdbFfnh4GK+88goefPDB1F7R3r170dHRAQDo6OjA\nnj17AADd3d1ob29HWVkZIpEIqqqq0NfXF2B+0JqsAxw1WQc4arIOcNRkHeCgyTrAUZN1gKMm64DA\nZV3sv/e97+FnP/sZFi369NBEIoFQKAQACIVCSCQSAIDR0VGEw+HUceFwGCMjI7luJiKiOcr4K45f\nfvllLFu2DI2NjRCRWY/J9rue049tBBA5+/8rATRg5mdXOefy9H27jp/fOt/x6fsI6vZzNf5/mHn+\nfOmb7fimORxvdfn88zl9TLrjLcYPAHjY8fqCmfI5fv5Yvu/fdfxAHu5/PpcFwM6zlyNYEM3gkUce\n0XA4rJFIRMvLy/Wyyy7Te++9V2tra3VsbExVVUdHR7W2tlZVVWOxmMZisdT1161bp2+++eYFtwtA\nAU3z8naW8WzXdz3GZXxfwLefqzmk6wy6ca5zmK3TunG28fM7fW/M18fDfMb35ej2g57Dvjyco9zM\nYb6cryki+vWvf11VVbds2aLxeDy1wHd2dqqq6sGDB7W+vl5PnTqlR48e1eXLl2symSzgxd5y3IcG\nzsGPBs7BjwY/5jBfc/pLVdNbMlu3bkVbWxt27NiBSCSC3bt3AwCi0Sja2toQjUZRWlqK7du3XxR/\nzouIqNDxGbQZx/ch/U/pfXq2nWD2Tt+eMSi4sNO6cbZxwcxO3xt9fvap4EynH88+TT8uANYGePu5\nug0+g5aIiDLgV/bejvvQwDn40cA5+NHgxxz4lT0REaXFxT4jsQ5wJNYBjsQ6wJFYBzgQ6wBHYh3g\nSKwDAsfFnoioCHDP3ttxHxo4Bz8aOAc/GvyYA/fsiYgoLS72GYl1gCOxDnAk1gGOxDrAgVgHOBLr\nAEdiHRA4LvZEREWAe/bejvvQwDn40cA5+NHgxxy4Z09ERGlxsc9IrAMciXWAI7EOcCTWAQ7EOsCR\nWAc4EuuAwHGxJyIqAtyz93bchwbOwY8GzsGPBj/mwD17IiJKi4t9RmId4EisAxyJdYAjsQ5wINYB\njsQ6wJFYBwSOiz0RURHgnr234z40cA5+NHAOfjT4MQfu2RMRUVpc7DMS6wBHYh3gSKwDHIl1gAOx\nDnAk1gGOxDogcFzsiYiKAPfsvR33oYFz8KOBc/CjwY85cM+eiIjS4mKfkVgHOBLrAEdiHeBIrAMc\niHWAI7EOcCTWAYHjYk9EVAS4Z+/tuA8NnIMfDZyDHw1+zIF79kRElBYX+4zEOsCRWAc4EusAR2Id\n4ECsAxyJdYAjsQ4IXMbF/n//+x/WrFmDhoYGRKNRPPLIIwCA8fFxtLS0oKamBq2trZiYmEhdJxaL\nobq6GnV1dejt7Q22noiI3GgWJ0+eVFXVTz75RNesWaOvv/66btmyRZ988klVVY3H49rZ2amqqgcP\nHtT6+nqdnJzUgYEBrays1KmpqQtuE4ACmubl7Szj2a7veozv4z40cA5+NHAOfjT4MYf5yrqNc9ll\nlwEAJicnMTU1haVLl2Lv3r3o6OgAAHR0dGDPnj0AgO7ubrS3t6OsrAyRSARVVVXo6+ub8ycgIiLK\nrayLfTKZRENDA0KhENauXYuVK1cikUggFAoBAEKhEBKJBABgdHQU4XA4dd1wOIyRkZGA0vNBrAMc\niXWAI7EOcCTWAQ7EOsCRWAc4EuuAwJVmO2DRokU4cOAAjh8/jnXr1mHfvn0zxktKSs4+lHJ26cc2\nAoic/f+VABoANJ0zLudclrP/uo4LZrrYxw8Y33+68YUeb3X5/PM5fUy64y3GD8zh+oKZOH7h+IEs\n47m4//lcFgA7z16OYCHm9Dj7H//4x7j00kvx61//GiKC8vJyjI2NYe3atfj73/+OeDwOANi6dSsA\n4JZbbsGPfvQjrFmzZuad8nH2DuM+NHAOfjRwDn40+DGHQB5n//7776ceafPf//4Xf/zjH9HY2Ij1\n69ejq6sLANDV1YUNGzYAANavX48XXngBk5OTGBgYwOHDh7F69ep5hRERUe5k3MYZGxtDR0cHkskk\nkskk7rvvPjQ3N6OxsRFtbW3YsWMHIpEIdu/eDQCIRqNoa2tDNBpFaWkptm/fnnGLx3+Cmd8q+0rA\nzlwS+N8p8L8RKKzOixt/XULG8X1I/47q07d8gtk7ffu2VXBhp3XjbOOCmZ2+N/q8BSI40+nHFkj6\ncQGwNsDbz9VtzH8bh4u9t+M+NHAOfjRwDn40+DEH/m4cIiJKi4t9RmId4EisAxyJdYAjsQ5wINYB\njsQ6wJFYBwSOiz0RURHgnr234z40cA5+NHAOfjT4MQfu2RMRUVpc7DMS6wBHYh3gSKwDHIl1gAOx\nDnAk1gGOxDogcFzsiYiKAPfsvR33oYFz8KOBc/CjwY85cM+eiIjS4mKfkVgHOBLrAEdiHeBIrAMc\niHWAI7EOcCTWAYHjYk9EVAS4Z+/tuA8NnIMfDZyDHw1+zIF79kRElBYX+4zEOsCRWAc4EusAR2Id\n4ECsAxyJdYAjsQ4IHBd7IqIiwD17b8d9aOAc/GjgHPxo8GMO3LMnIqK0uNhnJNYBjsQ6wJFYBzgS\n6wAHYh3gSKwDHIl1QOC42BMRFQHu2Xs77kMD5+BHA+fgR4Mfc+CePRERpcXFPiOxDnAk1gGOxDrA\nkVgHOBDrAEdiHeBIrAMCx8WeiKgIcM/e23EfGjgHPxo4Bz8a/JgD9+yJiCgtLvYZiXWAI7EOcCTW\nAY7EOsCBWAc4EusAR2IdEDgu9kRERSDrYj80NIS1a9di5cqVWLVqFbZt2wYAGB8fR0tLC2pqatDa\n2oqJiYnUdWKxGKqrq1FXV4fe3t7g6gPXZB3gqMk6wFGTdYCjJusAB03WAY6arAMcNVkHBE+zGBsb\n0/3796uq6okTJ7Smpkb7+/t1y5Yt+uSTT6qqajwe187OTlVVPXjwoNbX1+vk5KQODAxoZWWlTk1N\nzbhNAApompe3s4xnu77rMb6P+9DAOfjRwDn40eDHHOYr61f25eXlaGhoAABcccUVWLFiBUZGRrB3\n7150dHQAADo6OrBnzx4AQHd3N9rb21FWVoZIJIKqqir09fXN+ZOQH8Q6wJFYBzgS6wBHYh3gQKwD\nHIl1gCOxDgjcnPbsBwcHsX//fqxZswaJRAKhUAgAEAqFkEgkAACjo6MIh8Op64TDYYyMjOQwmYiI\n5qrU9cCPP/4Yd955J55++mksXrx4xlhJScnZx87PbvaxjQAiZ/9/JYAGzNw3k3Muy9l/XccFM813\nPOjbz9X49Ous7j/deKFenn5dusvZjs/XOLKMpzs+n+NNxvfv0/h8LguAnWcvR7AgLns9k5OT2tra\nqr/4xS9Sr6utrdWxsTFVVR0dHdXa2lpVVY3FYhqLxVLHrVu3Tt98803u2XMOBTruQwPn4EeDH3MI\nbM9eVbFp0yZEo1E8/PDDqdevX78eXV1dAICuri5s2LAh9foXXngBk5OTGBgYwOHDh7F69eo5fQLy\nh1gHOBLrAEdiHeBIrAMciHWAI7EOcCTWAYHLuo3zxhtv4LnnnsMNN9yAxsZGAGceWrl161a0tbVh\nx44diEQi2L17NwAgGo2ira0N0WgUpaWl2L59e8YtHiIiCh5/N4634z40cA5+NHAOfjT4MYf5Ltl8\nBi0RURHgYp+RWAc4EusAR2Id4EisAxyIdYAjsQ5wJNYBgeNiT0RUBLhn7+24Dw2cgx8NnIMfDX7M\ngXv2RESUFhf7jMQ6wJFYBzgS6wBHYh3gQKwDHIl1gCOxDggcF3sioiLAPXtvx31o4Bz8aOAc/Gjw\nYw7csyciorS42Gck1gGOxDrAkVgHOBLrAAdiHeBIrAMciXVA4LjYExEVAe7ZezvuQwPn4EcD5+BH\ngx9z4J49ERGlxcU+I7EOcCTWAY7EOsCRWAc4EOsAR2Id4EisAwLHxZ6IqAhwz97bcR8aOAc/GjgH\nPxr8mAP37ImIKC0u9hmJdYAjsQ5wJNYBjsQ6wIFYBzgS6wBHYh0QOC72RERFgHv23o770MA5+NHA\nOfjR4MccuGdPRERpcbHPSKwDHIl1gCOxDnAk1gEOxDrAkVgHOBLrgMBxsSciKgLcs/d23IcGzsGP\nBs7BjwY/5sA9eyIiSouLfUZiHeBIrAMciXWAI7EOcCDWAY7EOsCRWAcEjos9EVER4J69t+M+NHAO\nfjRwDn40+DGHwPbsH3jgAYRCIVx//fWp142Pj6OlpQU1NTVobW3FxMREaiwWi6G6uhp1dXXo7e2d\nVxQREeVW1sX+/vvvR09Pz4zXxeNxtLS04NChQ2hubkY8HgcA9Pf3Y9euXejv70dPTw82b96MZDIZ\nTHleiHWAI7EOcCTWAY7EOsCBWAc4EusAR2IdELisi/1Xv/pVLF26dMbr9u7di46ODgBAR0cH9uzZ\nAwDo7u5Ge3s7ysrKEIlEUFVVhb6+vgCyiYhoLub1A9pEIoFQKAQACIVCSCQSAIDR0VGEw+HUceFw\nGCMjIznItNJkHeCoyTrAUZN1gKMm6wAHTdYBjpqsAxw1WQcEbsGPxikpKTn7A9f040REZKt0PlcK\nhUI4duwYysvLMTY2hmXLlgEAKioqMDQ0lDpueHgYFRUVaW5lI4DI2f9fCaABMz+7yjmX5ey/ruOC\nmeY7Pn0fQd1+rsb/DzPPny99sx3fNIfjrS6ffz6nj0l3vMX4AQAPO15fMFM+x88fy/f9u44fyMP9\nz+eyANh59nIEC6IOBgYGdNWqVanLW7Zs0Xg8rqqqsVhMOzs7VVX14MGDWl9fr6dOndKjR4/q8uXL\nNZlMXnB7ABTQNC9vZxnPdn3XY1zG9wV8+7maQ7rOoBvnOofZOq0bZxs/v9P3xnx9PMxnfF+Obj/o\nOezLwznKzRzmK+s17777br366qu1rKxMw+GwPvvss/rBBx9oc3OzVldXa0tLi3744Yep45944gmt\nrKzU2tpa7enpmf1OM07Ip8XectyHBs7BjwbOwY8GP+YwX3xSlbfjPjRwDn40cA5+NPgxh/ku2fx1\nCRmJdYAjsQ5wJNYBjsQ6wIFYBzgS6wBHYh0QOC72RERFgNs43o770MA5+NHAOfjR4MccuI1DRERp\ncbHPSKwDHIl1gCOxDnAk1gEOxDrAkVgHOBLrgMBxsSciKgLcs/d23IcGzsGPBs7BjwY/5sA9eyIi\nSouLfUZiHeBIrAMciXWAI7EOcCDWAY7EOsCRWAcEjos9EVER4J69t+M+NHAOfjRwDn40+DEH7tkT\nEVFaXOwzEusAR2Id4EisAxyJdYADsQ5wJNYBjsQ6IHBc7ImIigD37L0d96GBc/CjgXPwo8GPOXDP\nnoiI0uJin5FYBzgS6wBHYh3gSKwDHIh1gCOxDnAk1gGB42JPRFQEuGfv7bgPDZyDHw2cgx8NfsyB\ne/ZERJQWF/uMxDrAkVgHOBLrAEdiHeBArAMciXWAI7EOCBwXeyKiIsA9e2/HfWjgHPxo4Bz8aPBj\nDtyzJyKitLjYZyTWAY7EOsCRWAc4EusAB2Id4EisAxyJdUDguNgTERUB7tl7O+5DA+fgRwPn4EeD\nH3Pgnj0REaUVyGLf09ODuro6VFdX48knnwziLvJErAMciXWAI7EOcCTWAQ7EOsCRWAc4EuuAwOV8\nsZ+amsJDDz2Enp4e9Pf34/nnn8e7776b67vJkwPWAY7YmVuF0FkIjQA7/ZHzxb6vrw9VVVWIRCIo\nKyvD3Xffje7u7lzfTZ5MWAc4YmduFUJnITQC7PRHzhf7kZERXHPNNanL4XAYIyMjub4bIiKag9Jc\n3+CZR9pkt2TJbbO+fmpqAidP5rJoIQatAxwNWgc4GrQOcDRoHeBg0DrA0aB1gKNB64DA5Xyxr6io\nwNDQUOry0NAQwuHwjGMqKytx5MjLWW4p2ycNl08qC72NrrMvQd1+LueQrjPoxrnOYbZO68bZxs/v\n9L0xHx8P8x3vyjLuevu5uA3r8YXdRmVlpcPtp7nVXD/O/vTp06itrcWf/vQnfPGLX8Tq1avx/PPP\nY8WKFbm8GyIimoOcf2VfWlqKX/7yl1i3bh2mpqawadMmLvRERMZMnkFLRET5lfNH4wwNDWHt2rVY\nuXIlVq1ahW3btgEAfve732HlypW45JJL8M4776SOHxwcxKWXXorGxkY0NjZi8+bNuU6aU+eWLVuw\nYsUK1NfX44477sDx48dT14nFYqiurkZdXR16e3u97PTtfP7gBz9AfX09Ghoa0NzcPOPnOT6dz3Sd\nvp3PaU899RQWLVqE8fHx1OvyfT7n2ujbuXz88ccRDodTPa+++mrqOj69b57f2dPTA2Ae51NzbGxs\nTPfv36+qqidOnNCamhrt7+/Xd999V//xj39oU1OTvv3226njBwYGdNWqVbnOmHdnb2+vTk1Nqapq\nZ2endnZ2qqrqwYMHtb6+XicnJ3VgYEArKytTx/nU6dv5/Oijj1LHbNu2TTdt2qSq/p3PdJ2+nU9V\n1X//+9+6bt06jUQi+sEHH6iqzfmca6Nv5/Lxxx/Xp5566oLjfXvfTNc51/OZ86/sy8vL0dDQAAC4\n4oorsGLFCoyOjqKurg41NTW5vrt5S9fZ0tKCRYvOnJY1a9ZgeHgYANDd3Y329naUlZUhEomgqqoK\nfX193nVaSde5ePHi1DEff/wxvvCFLwDw73ym67SSrhMAvv/97+OnP/3pjOMtzudcG63M1jn93B+d\nZRfbp/fNTJ1zFegvQhscHMT+/fuxZs2ajMcNDAygsbERTU1N+POf/xxk0qzSdT777LO49dZbAQCj\no6MzHkJq8WQxl07Av/P52GOP4dprr8XOnTvxyCOPAPDzfE53dnV1YevWranjfDqf3d3dCIfDuOGG\nG2YcY30+XRoBf87ll7/8ZQDAM888g/r6emzatAkTE2eeRWt9Ll07gTmezxx9B3KBEydO6E033aQv\nvfTSjNefv41z6tQpHR8fV1XVt99+W6+55poZ31IHLV3nT37yE73jjjtSlx966CF97rnnUpc3bdqk\nL774onedvp5PVdVYLKYbN25UVX/P5/mdPp3PkydP6urVq/X48eOqqhqJRPT9999XVdvz6dro07lU\nVU0kEppMJjWZTOpjjz2mDzzwgKr6976ZrnOu5zOQr+w/+eQT3Hnnnbj33nuxYcOGjMd+5jOfwdKl\nSwEAN954IyorK3H48OEgsi6QrnPnzp145ZVX8Nvf/jb1uvOfLDY8PIyKigrvOn08n9O+9a1v4a23\n3gLg5/m6m8YkAAABz0lEQVScrdOn83nkyBEMDg6ivr4e1113HYaHh3HTTTchkUiYnU/Xxv/85z9e\nnUsAWLZsGUpKSlBSUoIHH3wwtVXj2/tmus45n89cf1ZKJpN633336cMPPzzreFNTk/71r39NXX7v\nvff09OnTqqp65MgRraio0A8//DDXWc6dr776qkajUX3vvfdmvH76hzanTp3So0eP6vLlyzWZTHrX\n6dv5PHToUOr/27Zt03vvvVdV/Tuf6Tp9O5/nmu0HtPk8n3Nt9O1cjo6Opv7/85//XNvb21XVv/fN\ndJ1zPZ85X+xff/11LSkp0fr6em1oaNCGhgZ95ZVX9KWXXtJwOKyf/exnNRQK6S233KKqqr///e91\n5cqV2tDQoDfeeKO+/PLLuU6aU2dVVZVee+21qdd95zvfSV3niSee0MrKSq2trdWenh4vO307n3fe\neaeuWrVK6+vr9Y477tBEIpG6jk/nM13niy++6NX5PNd1112XWkhV838+59ro27m877779Prrr9cb\nbrhBv/GNb+ixY8dS1/HpfTNd51zPJ59URURUBPhnCYmIigAXeyKiIsDFnoioCHCxJyIqAlzsiYiK\nABd7IqIiwMWeiKgIcLEnIioC/w/Oc3tST4gW1QAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x105a29c10>"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Within-subjects designs (simple random assignment)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In some cases you might want to assign user-item pairs or user-session pairs to parameters. You can do this by simply passing more units into `assign()` and applying multiple units."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "class SelectiveExposureExperiment(SimpleExperiment):\n",
      "  def assign(self, params, subjectid, url):\n",
      "    params.has_source_cues = BernoulliTrial(p=0.5, unit=subjectid)\n",
      "    params.has_social_cues = BernoulliTrial(p=0.5, unit=subjectid)\n",
      "    \n",
      "    if params.has_source_cues:\n",
      "      sources = ['Fox News','USA Today', 'CNN', 'MSNBC']\n",
      "      params.source = UniformChoice(choices=sources, unit=[subjectid, url])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 21
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Within-subjects designs (balanced random assignment)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "If there are a small number of URLs in the case above, it is sometimes possible to not select a particular source, or disproportionately select one source. If a complete list of URLs is known in advance, it is possible to acheive complete balance by permuting a pre-constructed list of sources containing an exactly balanced list. The following code cycles through all sources, so that e.g., if there are 6 URLs, each source appears exactly twice. It also shuffles the URLs (in a deterministic fashion)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from itertools import cycle, islice\n",
      "\n",
      "class SelectiveExposureExperiment(SimpleExperiment):\n",
      "  def assign(self, params, subjectid, urls): \n",
      "    params.shuffled_urls = Sample(choices=urls, unit=subjectid)\n",
      "    \n",
      "    params.has_source_cues = BernoulliTrial(p=0.5, unit=subjectid)\n",
      "    params.has_social_cues = BernoulliTrial(p=0.5, unit=subjectid)\n",
      "    \n",
      "    if params.has_source_cues:\n",
      "      balanced_sources = list(islice(cycle(['Fox News','USA Today', 'CNN', 'MSNBC']), len(urls)))\n",
      "      params.shuffled_sources = Sample(choices=balanced_sources, unit=subjectid)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 26
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ex_urls = ['http://reuters.com/article%s.html' % i for i in xrange(8)]\n",
      "\n",
      "subject = 18\n",
      "e = SelectiveExposureExperiment(subjectid=subject, urls=ex_urls)\n",
      "pd.DataFrame.from_dict(e.get_params())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>has_social_cues</th>\n",
        "      <th>has_source_cues</th>\n",
        "      <th>shuffled_sources</th>\n",
        "      <th>shuffled_urls</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> USA Today</td>\n",
        "      <td> http://reuters.com/article5.html</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td>     MSNBC</td>\n",
        "      <td> http://reuters.com/article3.html</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td>       CNN</td>\n",
        "      <td> http://reuters.com/article2.html</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td>       CNN</td>\n",
        "      <td> http://reuters.com/article7.html</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td> USA Today</td>\n",
        "      <td> http://reuters.com/article4.html</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5</th>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td>  Fox News</td>\n",
        "      <td> http://reuters.com/article6.html</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6</th>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td>     MSNBC</td>\n",
        "      <td> http://reuters.com/article0.html</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7</th>\n",
        "      <td> 0</td>\n",
        "      <td> 1</td>\n",
        "      <td>  Fox News</td>\n",
        "      <td> http://reuters.com/article1.html</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>8 rows \u00d7 4 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 33,
       "text": [
        "   has_social_cues  has_source_cues shuffled_sources  \\\n",
        "0                0                1        USA Today   \n",
        "1                0                1            MSNBC   \n",
        "2                0                1              CNN   \n",
        "3                0                1              CNN   \n",
        "4                0                1        USA Today   \n",
        "5                0                1         Fox News   \n",
        "6                0                1            MSNBC   \n",
        "7                0                1         Fox News   \n",
        "\n",
        "                      shuffled_urls  \n",
        "0  http://reuters.com/article5.html  \n",
        "1  http://reuters.com/article3.html  \n",
        "2  http://reuters.com/article2.html  \n",
        "3  http://reuters.com/article7.html  \n",
        "4  http://reuters.com/article4.html  \n",
        "5  http://reuters.com/article6.html  \n",
        "6  http://reuters.com/article0.html  \n",
        "7  http://reuters.com/article1.html  \n",
        "\n",
        "[8 rows x 4 columns]"
       ]
      }
     ],
     "prompt_number": 33
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Clustered random assignment"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Random assignment can also occur at the cluster level. For example, a course on MOOCs might randomize classes into treatment conditions.  Here, course ID, rather than the student ID, is used for random assignment. Note that while the student ID is not actually used for random assignment, it is still useful to pass it in to the experiment object for logging purposes."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class BadgeExperiment(SimpleExperiment):\n",
      "  def assign(self, params, studentid, courseid):\n",
      "    params.has_faces = BernoulliTrial(p=0.8, unit=courseid)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 34
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Stepped-wedge design"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Stepped wedge designs expose subjects or clusters of subjects to treatments at a certain random point in time. This can improve precision when there are time-correlated exogenous events that may affect subjects' behaviors, or when there are constraints on how rapidly you can roll out treatments."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class BadgeExperiment(SimpleExperiment):\n",
      "  def assign(self, params, studentid, courseid, week):\n",
      "    params.rollout_week = RandomInteger(min=0, max=8, unit=courseid)\n",
      "    if week >= params.rollout_week:\n",
      "      params.has_faces = 1\n",
      "    else:\n",
      "      params.has_faces = 0"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 37
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "example_course = 17\n",
      "e = BadgeExperiment(studentid=2, courseid=example_course, week=0)\n",
      "rollout_week = e.get('rollout_week')\n",
      "print 'The badge rollout for course %s is %s...' % (example_course, rollout_week)\n",
      "\n",
      "for week in xrange(9):\n",
      "  has_faces = BadgeExperiment(studentid=2, courseid=example_course, week=week).get('has_faces')\n",
      "  print 'week %s: has_faces = %s ' % (week, has_faces)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "The badge rollout for course 17 is 2...\n",
        "week 0: has_faces = 0 \n",
        "week 1: has_faces = 0 \n",
        "week 2: has_faces = 1 \n",
        "week 3: has_faces = 1 \n",
        "week 4: has_faces = 1 \n",
        "week 5: has_faces = 1 \n",
        "week 6: has_faces = 1 \n",
        "week 7: has_faces = 1 \n",
        "week 8: has_faces = 1 \n"
       ]
      }
     ],
     "prompt_number": 38
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Social cues experiment"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let's consider a social cues experiment where an individual has a certain number of friends that we might show in association with a page, and we want to deterministically randomly sample from some subset of those friends. This is the setup from [Social Influence in Social Advertising: Evidence from Field Experiments](http://arxiv.org/pdf/1206.4327.pdf)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class CueExperiment(SimpleExperiment):\n",
      "  def assign(self, params, userid, page, friends):\n",
      "    params.num_friends_shown = RandomInteger(\n",
      "      min=0, max=min(3, len(friends)), unit=[userid, page])\n",
      "    params.friends_shown = Sample(\n",
      "      choices=friends, draws=params.num_friends_shown, unit=[userid, page])\n",
      "\n",
      "for p in xrange(12):\n",
      "    some_friends = [\"Sean\", \"Cat\", \"Solomon\", \"Dean\", \"Anne\", \"Moira\", \"Michael\", \"Mike\"]\n",
      "    print CueExperiment(userid=2, page=p, friends=some_friends).get('friends_shown')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "['Dean', 'Solomon']\n",
        "['Solomon', 'Cat']\n",
        "[]\n",
        "['Sean', 'Anne', 'Dean']\n",
        "['Sean']\n",
        "['Cat', 'Michael']\n",
        "['Michael', 'Solomon']\n",
        "['Anne', 'Solomon']\n",
        "['Sean', 'Mike']\n",
        "['Mike', 'Cat']\n",
        "[]\n",
        "['Sean', 'Dean', 'Solomon']\n"
       ]
      }
     ],
     "prompt_number": 39
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "More complex between subjects experiment"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Sometimes we are interested in how manipulations may in aggregate affect a subjects' outcomes. For example, **to what extent does feedback on a user's content future content production?** To answer this question, we can modulate the amount of feedback focal subjects receive by varying encouragements to peers providing feedback."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class ContentExperiment(SimpleExperiment):\n",
      "  def assign(self, params, producerid, storyid, viewerid):\n",
      "    params.p = RandomFloat(min=0, max=1, unit=producerid)\n",
      "    params.collapse = BernoulliTrial(p=params.p, unit=[storyid, viewerid])\n",
      "\n",
      "\n",
      "def sim_impressions(producerid, storyid):\n",
      "    viewers = range(5)\n",
      "    sim_users = [ContentExperiment(producerid=producerid, storyid=storyid, viewerid=viewerid).get_params() for viewerid in viewers]\n",
      "    df = pd.DataFrame.from_dict(sim_users)\n",
      "    df['producerid'] = producerid\n",
      "    df['storyid'] = storyid\n",
      "    df['viewerid'] = viewers\n",
      "    return df"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 39
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Each user has some overall probability that viewers of their story will see an expanded or collapsed version of their post. Expanded versions receive more feedback. Subjects are assigned some proportion of impressions of their stories to be collapsed. The `collapse` parameter is assigned using both the post and the viewer, so that different posts and different viewers may see the story collapsed some times but not others.\n",
      "\n",
      "_ Tutorial participants: think about the effects of experiments which do not include post as a unit in randomization _"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now let's try with a few different producers."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sim_impressions(producerid=10, storyid=2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>collapse</th>\n",
        "      <th>p</th>\n",
        "      <th>producerid</th>\n",
        "      <th>storyid</th>\n",
        "      <th>viewerid</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.127167</td>\n",
        "      <td> 10</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.127167</td>\n",
        "      <td> 10</td>\n",
        "      <td> 2</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.127167</td>\n",
        "      <td> 10</td>\n",
        "      <td> 2</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.127167</td>\n",
        "      <td> 10</td>\n",
        "      <td> 2</td>\n",
        "      <td> 3</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.127167</td>\n",
        "      <td> 10</td>\n",
        "      <td> 2</td>\n",
        "      <td> 4</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>5 rows \u00d7 5 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 43,
       "text": [
        "   collapse         p  producerid  storyid  viewerid\n",
        "0         0  0.127167          10        2         0\n",
        "1         0  0.127167          10        2         1\n",
        "2         0  0.127167          10        2         2\n",
        "3         0  0.127167          10        2         3\n",
        "4         1  0.127167          10        2         4\n",
        "\n",
        "[5 rows x 5 columns]"
       ]
      }
     ],
     "prompt_number": 43
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The same proportion of viewers see collapsed stories for a given viewer, but exactly which friends see the collapsed version depends on which post they are looking at."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sim_impressions(producerid=10, storyid=4)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>collapse</th>\n",
        "      <th>p</th>\n",
        "      <th>producerid</th>\n",
        "      <th>storyid</th>\n",
        "      <th>viewerid</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.127167</td>\n",
        "      <td> 10</td>\n",
        "      <td> 4</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.127167</td>\n",
        "      <td> 10</td>\n",
        "      <td> 4</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.127167</td>\n",
        "      <td> 10</td>\n",
        "      <td> 4</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.127167</td>\n",
        "      <td> 10</td>\n",
        "      <td> 4</td>\n",
        "      <td> 3</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.127167</td>\n",
        "      <td> 10</td>\n",
        "      <td> 4</td>\n",
        "      <td> 4</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>5 rows \u00d7 5 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 41,
       "text": [
        "   collapse         p  producerid  storyid  viewerid\n",
        "0         0  0.127167          10        4         0\n",
        "1         0  0.127167          10        4         1\n",
        "2         0  0.127167          10        4         2\n",
        "3         1  0.127167          10        4         3\n",
        "4         0  0.127167          10        4         4\n",
        "\n",
        "[5 rows x 5 columns]"
       ]
      }
     ],
     "prompt_number": 41
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Different producers have different proportions of their story impressions collapsed."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sim_impressions(producerid=12, storyid=4)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>collapse</th>\n",
        "      <th>p</th>\n",
        "      <th>producerid</th>\n",
        "      <th>storyid</th>\n",
        "      <th>viewerid</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.814347</td>\n",
        "      <td> 12</td>\n",
        "      <td> 4</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.814347</td>\n",
        "      <td> 12</td>\n",
        "      <td> 4</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 0</td>\n",
        "      <td> 0.814347</td>\n",
        "      <td> 12</td>\n",
        "      <td> 4</td>\n",
        "      <td> 2</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.814347</td>\n",
        "      <td> 12</td>\n",
        "      <td> 4</td>\n",
        "      <td> 3</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 1</td>\n",
        "      <td> 0.814347</td>\n",
        "      <td> 12</td>\n",
        "      <td> 4</td>\n",
        "      <td> 4</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>5 rows \u00d7 5 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 42,
       "text": [
        "   collapse         p  producerid  storyid  viewerid\n",
        "0         1  0.814347          12        4         0\n",
        "1         0  0.814347          12        4         1\n",
        "2         0  0.814347          12        4         2\n",
        "3         1  0.814347          12        4         3\n",
        "4         1  0.814347          12        4         4\n",
        "\n",
        "[5 rows x 5 columns]"
       ]
      }
     ],
     "prompt_number": 42
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Different modifications identify different effects"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Doing assignment based on the the post-id level tells you something about like-hearding. e.g.,"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class ContentExperiment(SimpleExperiment):\n",
      "  def assign(self, params, producerid, storyid, viewerid):\n",
      "    params.p = RandomFloat(min=0, max=1, unit=storyid)\n",
      "    params.collapse = BernoulliTrial(p=params.p, unit=[storyid, viewerid])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 45
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "How random assignment works in PlanOut"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "PlanOut hashes input data to provide developers with randomized parameter assignments. Unless otherwise specified, all random assignments are independent. It combines experiment-level and parameter-level salts with the given input units in a way that ensures that:\n",
      "  * The same units (e.g., user ids) get mapped to different values for different experiments or parameters.\n",
      "  * Assignments are as good as random.\n",
      "\n",
      "Underneath the hood, PlanOut computes a hash that looks like\n",
      "\n",
      "    f(SHA1(experiment_name.parameter_name.unit_id))\n",
      "\n",
      "So for example, in the experiment below, PlanOut computes something that looks like:\n",
      "\n",
      "    SHA1(RandomExample1.x.4) % 2\n",
      "  \n",
      "to select the value for `x` when the given `userid` is 4."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Parameter-level salts"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class RandomExample1(SimpleExperiment):\n",
      "  def assign(self, params, userid):\n",
      "    params.x = UniformChoice(choices=[0, 1], unit=userid)\n",
      "    params.y = UniformChoice(choices=['a','b'], unit=userid)\n",
      "    \n",
      "sim_users = [RandomExample1(userid=i).get_params() for i in xrange(2000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)\n",
      "print assignments.groupby(['x', 'y']).size()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "PlanOut automatically \"salts\" each random assignment operator with the name of the parameter you are assigning. By writing `params.foo = Bar(...)`, you are implicitly passing the salt, \"foo\", into `Bar()`. The following experiment is equivalent to the code above."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class RandomExample1(SimpleExperiment):\n",
      "  def assign(self, params, userid):\n",
      "    params.x = UniformChoice(choices=[0, 1], unit=userid, salt='x')\n",
      "    params.y = UniformChoice(choices=['a','b'], unit=userid, salt='y')\n",
      "    \n",
      "sim_users = [RandomExample1(userid=i).get_params() for i in xrange(2000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)\n",
      "print assignments.groupby(['x', 'y']).size()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Comparing the cross tabs from the first and second experiments, it's clear that the two experiments produce identical assignments."
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "Changing the salts change the assignments:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class RandomExample1(SimpleExperiment):\n",
      "  def assign(self, params, userid):\n",
      "    params.x = UniformChoice(choices=[0, 1], unit=userid, salt='x2')\n",
      "    params.y = UniformChoice(choices=['a','b'], unit=userid, salt='y2')\n",
      "    \n",
      "sim_users = [RandomExample1(userid=i).get_params() for i in xrange(2000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)\n",
      "print assignments.groupby(['x', 'y']).size()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "Parameters with the same salt will have correlated assignments. If you use the same salt for the exact same kind of random operation, your assignments will be perfectly correlated."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class RandomExample1(SimpleExperiment):\n",
      "  def assign(self, params, userid):\n",
      "    params.x = UniformChoice(choices=[0, 1], unit=userid, salt='x')\n",
      "    params.y = UniformChoice(choices=['a','b'], unit=userid, salt='x')\n",
      "    \n",
      "sim_users = [RandomExample1(userid=i).get_params() for i in xrange(2000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)\n",
      "print assignments.groupby(['x', 'y']).size()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Experiment-level salts"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Each experiment also has its own salt. This makes it so that parameters with the same name will have independent random assignments, and also allows you to synchronize assignments across experiments in special situations."
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "By default, experiment class names are used as experiment-level salts"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This keeps parameter assignments for parameters with the same name independent of one another."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class RandomExample1(SimpleExperiment):\n",
      "  def assign(self, params, userid):\n",
      "    params.x = UniformChoice(choices=[0, 1], unit=userid)\n",
      "    params.y = UniformChoice(choices=['a','b'], unit=userid)\n",
      "    \n",
      "class RandomExample2(SimpleExperiment):\n",
      "  def assign(self, params, userid):\n",
      "    params.x = UniformChoice(choices=[4, 8], unit=userid)\n",
      "    params.y = UniformChoice(choices=['m','n'], unit=userid)\n",
      "\n",
      "sim_users = [RandomExample1(userid=i).get_params() for i in xrange(4000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)\n",
      "print assignments.groupby(['x', 'y']).size()\n",
      "\n",
      "sim_users = [RandomExample2(userid=i).get_params() for i in xrange(4000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)\n",
      "print assignments.groupby(['x', 'y']).size()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "Experiment-level salts can be specified by setting `self.salt`"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The `self.salt` attribute of an experiment object specifies the experiment-level salt. You can set this attribute in the `setup()` method, which gets called before any assignments take place."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class RandomExample1(SimpleExperiment):\n",
      "  def setup(self):\n",
      "        self.salt = 'RandomExample2'\n",
      "\n",
      "  def assign(self, params, userid):\n",
      "    params.x = UniformChoice(choices=[0, 1], unit=userid)\n",
      "    params.y = UniformChoice(choices=['a','b'], unit=userid)\n",
      "\n",
      "sim_users = [RandomExample2(userid=i).get_params() for i in xrange(4000)]\n",
      "assignments = pd.DataFrame.from_dict(sim_users)\n",
      "print assignments.groupby(['x', 'y']).agg(len)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Additional notes on random assignment"
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "Random assignment with multiple units"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When multiple units are used (e.g., in the case when user-item pairs are assigned to parameters in a within-subjects design), units are concatinated, so that if the input units are `userid=4` and `url='http://news.ycombinator.com'`, the hash operation would look like:\n",
      "\n",
      "    f(SHA1('RandomExperiment1.show_thumbnail.6.http://news.ycombinator.com'))"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class RandomExample1(SimpleExperiment):\n",
      "  def assign(self, params, userid, url):\n",
      "    params.show_thumbnail = BernoulliTrial(p=0.15, unit=[userid, url])\n",
      "\n",
      "RandomExample1(userid=6, url='http://news.ycombinator.com').get('show_thumbnail')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Note that since the names of units are not used, assignment does depend on the order of the units:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class RandomExample1(SimpleExperiment):\n",
      "  def assign(self, params, userid, url):\n",
      "    params.show_thumbnail = BernoulliTrial(p=0.15, unit=[url, userid])\n",
      "\n",
      "RandomExample1(userid=6, url='http://news.ycombinator.com').get('show_thumbnail')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "Namespaces"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When an experiment is running under a namespace, the namespace name is concatenated with the experiment-level salt. See the PlanOut documentation for more information on namespaces."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}