DSCI6006-4.1-Lecture AB Testing .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Design of A/B Testing  \n",
    "\n",
    "**Reference**: [A/B Testing - Udacity](https://www.udacity.com/course/ab-testing--ud257)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is  A/B Testing?  \n",
    "\n",
    "> [A/B testing](https://en.wikipedia.org/wiki/A/B_testing) is a term for a randomized experiment with two variants, A and B, which are the control and variation in the controlled experiment. A/B testing is a form of statistical hypothesis testing with two variants leading to the technical term, two-sample hypothesis testing, used in the field of statistics. Other terms used for this method include bucket tests and split-run testing.  \n",
    "\n",
    "* In the online world, the goal of A/B testing is to determine whether or not the users will like a particular new product / feature.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experimental vs Observational  \n",
    "\n",
    "* Experimental\n",
    "    * Apply treatments to experimental units (people, animals, land, etc) and observe effect of treatment\n",
    "    * Can be used to establish causality\n",
    "    * Example: Randomly assign students to two groups, only give homework to one group, and measure the performance of the two groups  \n",
    "    \n",
    "* Observational\n",
    "    * Observe subjects and measure variables of interest without assigning treatments to subjects\n",
    "    * Can’t be used to establish causality\n",
    "    * Example: Observe and record whether or not the studets do homework and their grades  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confounding Factor  \n",
    "\n",
    "* An extraneous attribute that correlates with the dependent variable (performance) and the independent variable (homework or not)  \n",
    "* Example:  \n",
    "    * How hard-working the student is  \n",
    "    * The students that are more hard-working are more likely to complete their homework and perform better"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experimental Design   \n",
    "\n",
    "* Randomization into groups of equal size  \n",
    "    * Example: Randomly generate number from 0 - 1, if < 0.5, assign homework (**treatment/experiment group**); otherwise, no homework (**control group**) \n",
    " \n",
    "* Assume independent observations  \n",
    "    * Assume the students don’t know if the other students have homework or not  \n",
    "    * Otherwise that knowledge might affect performance  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q**: What can we test with A/B testing? Can you think of any example when we can't use A/B testing?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Techniques  \n",
    "\n",
    "* Retrospective / observational study  \n",
    "* User experience research  \n",
    "* Focus groups\n",
    "* Surveys\n",
    "* Human evaluation    \n",
    "\n",
    "**Note**: These techniques can also be used for generating or validating the metrics used in A/B testing.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choosing Metrics  \n",
    "\n",
    "**Example**: An online education company (like Udacity) is trying to test features that increase student engagement. A typical user flow through their website might look like:  \n",
    "* Visit the homepage  \n",
    "* Explore the site  \n",
    "* Create an account  \n",
    "* Complete a class / make a purchase\n",
    "\n",
    "We'll consider an experimental change to the \"Start Now\" button on the company's homepage. If users click this button, they will see a list of the online courses. We want to test the hypothesis for that changing the \"Start Now\" button from orange to pink will increase how many students explore the online courses.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q**: What metrics should we use for testing the hypothesis?  \n",
    "\n",
    "* Total number of courses completed?  \n",
    "* The number of users click on the \"Start Now\" button?  \n",
    "* Click-through rate (CTR): the number of clicks divided by the number of pageviews?  \n",
    "* Click-through probability: the number of unique visitors who click at least once divided by the number\n",
    "of unique visitors who view the page?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q**: How would you measure the metric(s) of your choice?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q**: Say if we choose click-through probability as the metric. We observe that 1000 visitors visited the page, and there are 100 unique clicks, can we construct a 95% confidence interval for the click-through probability? How? What does this interval tell us?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 95% confidence interval for one sample proportion $p$ is given by  \n",
    "\n",
    "$$ (\\hat{p} - 1.96 \\sqrt{\\frac{\\hat{p}(1 - \\hat{p})}{n}} \\text{, } \\hat{p} + 1.96 \\sqrt{\\frac{\\hat{p}(1 - \\hat{p})}{n}}) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "\t1-sample proportions test with continuity correction\n",
       "\n",
       "data:  100 out of 1000, null probability 0.5\n",
       "X-squared = 638.4, df = 1, p-value < 2.2e-16\n",
       "alternative hypothesis: true p is not equal to 0.5\n",
       "95 percent confidence interval:\n",
       " 0.08245237 0.12069092\n",
       "sample estimates:\n",
       "  p \n",
       "0.1 \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# One sample proportion test in R\n",
    "\n",
    "prop.test(x = 100, n = 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>0.0824523714093577</li>\n",
       "\t<li>0.120690923047258</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 0.0824523714093577\n",
       "\\item 0.120690923047258\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 0.0824523714093577\n",
       "2. 0.120690923047258\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 0.08245237 0.12069092\n",
       "attr(,\"conf.level\")\n",
       "[1] 0.95"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# To get the confidence interval\n",
    "res = prop.test(x = 100, n = 1000)\n",
    "res$conf.int"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q**: Can you construct a 99% confidence interval for the click-through probability?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing the Two Versions  \n",
    "\n",
    "**Example**: Say we randomly show half of the visitors the homepage with orange \"Start Now\" button, and the other half the pink \"Start Now\" button. 1000 visitors see the orange button and there are 100 unique clicks on it, while 1000 visitors see the pink button and there are 130 unique clicks on it. \n",
    "\n",
    "**Q**: Which group is the control group? Which is the treatment/experiment group?  \n",
    "\n",
    "**Q**: How should we compare the click-through probability between the two groups?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say if we are just interested in testing if there is a statistically significant difference between the two groups,  \n",
    "\n",
    "$H_0: p_0 = p_1$ or $p_0 - p_1 = 0$  \n",
    "\n",
    "$H_a: p_0 \\neq p_1$ or $p_0 - p_1 \\neq 0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The overall (pooled) probability is:  \n",
    "\n",
    "$$ \\hat{p}_{pool} = \\frac{X_0 + X_1}{n_0 + n_1} $$  \n",
    "\n",
    "And the 95% confidence interval for $p_0 - p_1$ is give by  \n",
    "\n",
    "$$ (\\hat{p}_0 - \\hat{p}_1) \\pm 1.96 \\sqrt{\\hat{p}_{pool}(1 - \\hat{p}_{pool}) \\left(\\frac{1}{n_0} + \\frac{1}{n_1} \\right)} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "\t2-sample test for equality of proportions with continuity correction\n",
       "\n",
       "data:  c(100, 130) out of c(1000, 1000)\n",
       "X-squared = 4.1317, df = 1, p-value = 0.04209\n",
       "alternative hypothesis: two.sided\n",
       "95 percent confidence interval:\n",
       " -0.058932066 -0.001067934\n",
       "sample estimates:\n",
       "prop 1 prop 2 \n",
       "  0.10   0.13 \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "prop.test(x = c(100, 130), n = c(1000, 1000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Practical Significance vs Statistical Significance  \n",
    "\n",
    "When we conclude from the previous hypothsis test that there is a significance difference between the two groups, we are talking about statistical significance. But usually we don't just want to test if the difference is 0, and we would be more interested in something like the click-through probability increases 1% due to the new feature - this is talking about the practical significance.   \n",
    "\n",
    "**Q**: Is the experiment result practically significant?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Determining the Sample Size   \n",
    "\n",
    "Now that we have chosen a metric, how can we decide how large of a sample do we need for the experiment?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statistical Power  \n",
    "\n",
    "The statistical power of a test is defined as the probability that the test correctly rejects the null hypothesis ($H_0$) when the alternative hypothesis ($H_a$) is true. I.e.,  \n",
    "\n",
    "$$power = P(\\text{reject } H_0 \\ | \\ H_a \\text{ is true}) = 1 - \\beta$$  \n",
    "\n",
    "**Q**: What is the power of the test if the true difference in click-through probability is 0.02? The sample size for each group is 1000, and the click-through probability for the control group is 10%.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "     Two-sample comparison of proportions power calculation \n",
       "\n",
       "              n = 1000\n",
       "             p1 = 0.1\n",
       "             p2 = 0.12\n",
       "      sig.level = 0.05\n",
       "          power = 0.2977321\n",
       "    alternative = two.sided\n",
       "\n",
       "NOTE: n is number in *each* group\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "power.prop.test(n = 1000, p1 = 0.1, p2 = 0.1 + 0.02)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**: Now say if we know that our baseline click-through probability is 10% (the click-through probability before the new feature is introduced), and to be practically significant, we need an absolute difference of 2% in the click-through probability between the control and the experiment groups. In order to have a statistical power of 80%, what is the required sample size for each group?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "     Two-sample comparison of proportions power calculation \n",
       "\n",
       "              n = 3840.847\n",
       "             p1 = 0.1\n",
       "             p2 = 0.12\n",
       "      sig.level = 0.05\n",
       "          power = 0.8\n",
       "    alternative = two.sided\n",
       "\n",
       "NOTE: n is number in *each* group\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "power.prop.test(p1 = 0.1, p2 = 0.1 + 0.02, power = 0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q**: How does the required sample size change as you:  \n",
    "\n",
    "* Increase the practical significance level?  \n",
    "* Increase the confidence level?  \n",
    "* Increase the desired power of the test?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "     Two-sample comparison of proportions power calculation \n",
       "\n",
       "              n = 1773.976\n",
       "             p1 = 0.1\n",
       "             p2 = 0.13\n",
       "      sig.level = 0.05\n",
       "          power = 0.8\n",
       "    alternative = two.sided\n",
       "\n",
       "NOTE: n is number in *each* group\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "power.prop.test(p1 = 0.1, p2 = 0.1 + 0.03, power = 0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "     Two-sample comparison of proportions power calculation \n",
       "\n",
       "              n = 5715.417\n",
       "             p1 = 0.1\n",
       "             p2 = 0.12\n",
       "      sig.level = 0.01\n",
       "          power = 0.8\n",
       "    alternative = two.sided\n",
       "\n",
       "NOTE: n is number in *each* group\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "power.prop.test(p1 = 0.1, p2 = 0.1 + 0.02, sig.level = 0.01, power = 0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "     Two-sample comparison of proportions power calculation \n",
       "\n",
       "              n = 5141.306\n",
       "             p1 = 0.1\n",
       "             p2 = 0.12\n",
       "      sig.level = 0.05\n",
       "          power = 0.9\n",
       "    alternative = two.sided\n",
       "\n",
       "NOTE: n is number in *each* group\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "power.prop.test(p1 = 0.1, p2 = 0.1 + 0.02, power = 0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Two Uses of Metrics  \n",
    "\n",
    "* **Invariant checking**: the metrics that shouldn't change across the experiment and control  \n",
    "    * Do you have the same number of users across the two?  \n",
    "    * Do you have comparable numbers of users across countries, or by language?  \n",
    "    \n",
    "* **Evaluation**: \n",
    "    * High level business metrics, e.g., how much revenue you make, how many users you have  \n",
    "    * More detailed metrics, e.g., how long do the users stay on your page  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q**: For the following new features to the online education site, think about what metrics you would use to test them:  \n",
    "\n",
    "* Adding course descriptions on the course list page  \n",
    "* Increase the size of \"Start Now\" button  \n",
    "* Explain the benefits of the paid services  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some metrics can be difficult to obtain / measure:  \n",
    "\n",
    "* You don't have access to that data\n",
    "* It takes too long to collect  \n",
    "\n",
    "**Q**: Why might the following metrics difficult to obtain?  \n",
    "\n",
    "* The rate of students return to take a 2nd course after taking the 1st course  \n",
    "* The percentage of the students who get jobs after taking the online courses  \n",
    "* The average happiness level of Amazon shoppers  \n",
    "* The probability that the users find the information they look for on Google  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the Metrics  \n",
    "\n",
    "**Q**: For the click-through probability we chose in the previous example, how exactly are we going to capture this probability from observed data? (How do we define \"Unique\"?)  \n",
    "\n",
    "* For each time interval (1 minute, 1 hour, 1 day, etc.),  \n",
    "\n",
    "$$\\frac{\\text{number of clicks from unique cookies}}{\\text{number of unique cookies}} $$  \n",
    "\n",
    "* For each time interval (1 minute, 1 hour, 1 day, etc.),  \n",
    "\n",
    "$$\\frac{\\text{number of pageviews with clicks}}{\\text{number of pageviews}} $$ \n",
    "\n",
    "* Click-through rate (CTR)    \n",
    "\n",
    "$$ \\frac{\\text{number of clicks}}{\\text{number of pageviews}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q**: In certain cases, the metric(s) we defined might not be measuring what we think they are measuring, can you think of any cases?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q**: how would you check to see if there's any problem with the data for the defined metrics?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary Metrics  \n",
    "\n",
    "* Sum and count  \n",
    "* Mean, median, percentiles\n",
    "* Probability and rates\n",
    "* Ratios\n",
    "\n",
    "**Q**: Say if you want to measure the loading time of videos on your website, how would you choose a metric?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sensitivity and Robustness\n",
    "\n",
    "* Sensitivity: a metric that picks up changes you care about\n",
    "* Robustness: is robust against changes that you don't care about  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variability of Metrics  \n",
    "\n",
    "* Analytically\n",
    "    * If the metric follows a known distribution, we can find the variance of the matrics analytically by finding the variance of its distribution\n",
    "    * E.g., the mean would follow a Normal distribution by CLT\n",
    "\n",
    "* Empirically  \n",
    "    * A/A tests\n",
    "    * Bootstrap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q**: Say if we measured the number of daily visits to our website, and we chose a metric of mean number of site visits per day. How would you estimate the variance of the metric?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "visits = c(452, 593, 932, 854, 362, 481, 459, 512, 835, 480, 783, 291, 452, 843, 673, 841, 733, 910, 486, 928)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Design the Experiment  \n",
    "\n",
    "### Unit of Diversion  \n",
    "\n",
    "* The subject of the experiment\n",
    "    * How do we define what an individual subject is in the experiment?  \n",
    "* Commonly used:  \n",
    "    * User ID\n",
    "    * Anonymous ID (cookies)\n",
    "    * Event\n",
    "* Less commonly used:  \n",
    "    * Device ID\n",
    "    * IP address"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q**: Which unit of diversion would you use for the following experiments on the online education site:  \n",
    "\n",
    "* Reduce video loading time  \n",
    "* Change the color and size of a button  \n",
    "* Change the order of search results\n",
    "* Add instructor notes before quizzes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unit of Analysis  \n",
    "\n",
    "* Basically whatever the denominator of your metric is\n",
    "* If the unit of analysis and unit of diversion are the same, the analytical variability of the metric is likely to be close to the empirical one\n",
    "* In general, the unit of diversion needs to be at least as \"big\" as your unit of analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q**: If the metric of the test is the click-through rate (# clicks / # pageviews), and the unit of diversion is cookie, would you expect the analytical variance to match the empirical variance?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Target Population  \n",
    "\n",
    "* You have to decide who you're targeting in your users  \n",
    "    * E.g. by browswer, country, language, demographic information, etc.  \n",
    "* You might want to restrict how many of your users can see the feature  \n",
    "* You might not want to overlap with other experiments running at the same time  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sizing and Duration  \n",
    "\n",
    "* Based on the unit of diversion we choose, we may have to estimate the varibility of the metric empirically, then choose the sample size based on the variance  \n",
    "\n",
    "**Q**: How can you esimate the variance of the metric if the unit of diversion is different from the unit of analysis?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Then you need to decide \n",
    "    * What percentage of your users that you want to expose the new feature to\n",
    "    * How long you have to run the experiment  \n",
    "    \n",
    "**Q**: Why wouldn't you want to expose all your users to the experiment?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis of A/B Testing Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Now that we've decided on the metric(s) to evaluate, chosen the sample size and run the experiment, we want to see what we can conclude / recommend from the data collected.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Invariant Checking  \n",
    "\n",
    "* Before we dive into comparing the click-through probability, we need to do some sanity checks to make sure the experiment is actually run properly\n",
    "    * Something might have gone wrong in the experiment diversion, are your control and experiment groups still comparable?  \n",
    "    * Did the data capture the events you were looking for?  \n",
    "\n",
    "* We need to check if the experiment population and the control populations are actually comparable  \n",
    "* The invariants shouldn't change when you run your experiment  \n",
    "* \n",
    "**Q**: Say the metric we want to evaluate is the click-through rate, what metrics can we use for invariant checking?   \n",
    "\n",
    "**Q**: If we observed a total of 8294 pageviews in the control group and 8095 pageviews in the experiment group, how do we use it for invariant checking?   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We can perform a Binomial/proportion test on the invariant\n",
    "prop.test(8294, 8294 + 8095)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing the Results for Single Metric   \n",
    "\n",
    "* We have looked at how to analyze the results for a single metric during the morning lecture  \n",
    "* Another test we might be interested in performing is the 'Sign Test'  \n",
    "\n",
    "### Sign Test\n",
    "\n",
    "* The sign test can be used if we want to furthur confirm the results, or if the metric doesn't follow any known distribution  \n",
    "\n",
    "**Example**: We run a test for one week, and obtained the daily CTR for the control and the experiment groups below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ctr_cont = c(0.33, 0.45, 0.39, 0.40, 0.57, 0.63, 0.61)\n",
    "ctr_exp = c(0.53, 0.55, 0.61, 0.58, 0.68, 0.72, 0.70)\n",
    "\n",
    "# If we take the difference of the daily CTR's\n",
    "ctr_exp - ctr_cont"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* If the true CTR were the same between the two groups, what would we expect to see in terms of the signs of the differeces?   \n",
    "\n",
    "* What is the probability of observing 7 positive differences?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The one sample proportion test won't work too well\n",
    "# Since the sample size is too small for the Normal approximation\n",
    "prop.test(7, 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We can calculate the p-value \n",
    "# by using the PMF/CDF of a Binomial distribution\n",
    "\n",
    "dbinom(7, 7, 0.5) * 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysing the Results for Multiple Metrics   \n",
    "\n",
    "**Q**: If we are testing 20 different metrics, and use the 95% confidence interval to make the decision on each test, what is the probability that you get at least one significant result by chance?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonferroni Correction  \n",
    "\n",
    "* The more tests we run, the more likely that we will have significant results just by chance  \n",
    "* One way to adjust for this, is to reduce the significance level of each individual test\n",
    "* The Bonferroni correction controls the familywise error rate (FWER) by rejecting the null hypothesis for all $p_i \\leq \\frac{\\alpha}{m}$, where $p_i$ is the p-value for the $i$th test, $m$ is the number of tests\n",
    "    * The familywise error rate (FWER) is the probability of rejecting at least one true $H_0$, i.e. the probability of making at least one type I error  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### False Discovery Rate  (FDR)    \n",
    "\n",
    "* Instead of controlling the FWER, we can also control the expected proportion of false discoveries - the FDR  \n",
    "\n",
    "$$ FDR = E \\left(\\frac{\\text{number of false positives}}{\\text{number of rejections of the } H_0} \\right)$$  \n",
    "\n",
    "* Benjamini–Hochberg procedure:\n",
    "    * Order the p-values of the $m$ tests: $p_{(1)}, p_{(2)}, \\dots, p_{(m)}$\n",
    "    * For a given $\\alpha$ , find the largest $k$ such that $p_{(k)} \\leq \\frac{k}{m} \\alpha$\n",
    "    * Reject the null hypothesis for tests, $1, 2, \\dots, k$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make Recommendations  \n",
    "\n",
    "* Do we have statistically and practically significant results?  \n",
    "* Do we understand the results? Do they make sense?\n",
    "* Do we want to launch the change? Is it worth it?  \n",
    "* Do we want to launch the change for a slice of the users?  \n",
    "* Do we need to run further tests?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}