05_advanced_testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced testing\n",
    "\n",
    "Up till now we have been testing functions where the output is entirely predictable. In these cases, a handful of tests is usually enough to provide confidence that the software is working as expected. In the real world, however, you might be developing a complex piece of sofware to implement an entirely new algorithm, or model. In certain cases it might not even be clear what the expected outcome is meant to be. Things can be particularly challenging when the software is involves a stochastic element.\n",
    "\n",
    "Let us consider a class to simulate the behaviour of a dice. One is provided in the `dice` package. Let's import it and see how it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dice import Dice\n",
    "help(Dice)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How could we test that the dice is fair?\n",
    "\n",
    "Well, first of all we could check that the value of a dice roll is in range.\n",
    "\n",
    "```python\n",
    "# dice/test/test_dice.py\n",
    "def test_valid_roll():\n",
    "    \"\"\" Test that a dice roll is valid. \"\"\"\n",
    "\n",
    "    # Intialise a standard, six-sided dice.\n",
    "    dice = Dice()\n",
    "\n",
    "    # Roll the dice.\n",
    "    roll = dice.roll()\n",
    "\n",
    "    # Check that the value is valid.\n",
    "    assert roll > 0 and roll < 7\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pytest dice/test/test_dice.py::test_valid_roll"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, that worked. Although, it could just be a fluke...\n",
    "\n",
    "In practice, we need to check that the assertions hold repeatedly.\n",
    "\n",
    "```python\n",
    "# dice/test/test_dice.py\n",
    "def test_always_valid_roll():\n",
    "    \"\"\" Test that a dice roll is \"always\" valid. \"\"\"\n",
    "\n",
    "    # Intialise a standard, six-sided dice.\n",
    "    dice = Dice()\n",
    "\n",
    "    # Roll the dice lots of times.\n",
    "    for i in range(0, 10000):\n",
    "        roll = dice.roll()\n",
    "\n",
    "        # Check that the value is valid.\n",
    "        assert roll > 0 and roll < 7\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pytest dice/test/test_dice.py::test_always_valid_roll"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, that's better. Or is it...\n",
    "\n",
    "![xkcd: random](https://imgs.xkcd.com/comics/random_number.png)\n",
    "\n",
    "Not again!\n",
    "\n",
    "Perhaps we should test the average value. We know that this should equal the sum of the faces of the dice, divided by the number of sides, i.e. 3.5 for a six-sided dice.\n",
    "\n",
    "```python\n",
    "# dice/test/test_dice.py\n",
    "def test_average():\n",
    "    \"\"\" Test that the average dice roll is correct. \"\"\"\n",
    "\n",
    "    # Intialise a standard, six-sided dice.\n",
    "    dice = Dice()\n",
    "\n",
    "    # Work out the expected average roll.\n",
    "    exp = sum(range(1, 7)) / 6\n",
    "\n",
    "    # Calculate the sum of the dice rolls.\n",
    "    total = 0\n",
    "    for i in range(0, 100000):\n",
    "        total += dice.roll()\n",
    "\n",
    "    # Check that the average matches the expected value.\n",
    "    average = total / rolls\n",
    "    assert average == pytest.approx(3.5, rel=1e-2)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pytest dice/test/test_dice.py::test_average"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good... Hang on, hold your horses!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(1 + 3 + 4 + 6) / 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dang! We need to test that the _distrubtion_ of outcomes is correct, i.e. that each of the six possible outcomes is equally likely.\n",
    "\n",
    "```python\n",
    "# dice/test/test_dice.py\n",
    "def test_fair():\n",
    "    \"\"\" Test that a dice is fair. \"\"\"\n",
    "\n",
    "    # Intialise a standard, six-sided dice.\n",
    "    dice = Dice()\n",
    "\n",
    "    # Set the number of rolls.\n",
    "    rolls = 1000000\n",
    "\n",
    "    # Create a dictionary to hold the tally for each outcome.\n",
    "    tally = {}\n",
    "    for i in range(1, 7):\n",
    "        tally[i] = 0\n",
    "\n",
    "    # Roll the dice 'rolls' times.\n",
    "    for i in range(0, rolls):\n",
    "        tally[dice.roll()] += 1\n",
    "\n",
    "    # Assert that the probability is correct.\n",
    "    for i in range(1, 7):\n",
    "        assert tally[i] / rolls == pytest.approx(1 / 6, 1e-2)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pytest dice/test/test_dice.py::test_fair"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Phew, thanks goodness! Testing is hard.\n",
    "\n",
    "# Exercise\n",
    "\n",
    "#### Exercise 1\n",
    "\n",
    "The file `dice/test/test_dice.py` contains an empty function, `test_double_roll`, for checking that the distribution for the sum of two six-sided dice rolls is correct. Fill in the body of this function and run `pytest` to verify that your test passes.\n",
    "\n",
    "_Hints_:\n",
    "\n",
    "For any two n-sided dice, the probability of the sum of two rolls being a value of `x` is given by:\n",
    "\n",
    "$$p(x) = \\frac{n - |x - (n+1)|}{n^2},\\quad\\mathrm{for}\\ x=2\\ \\mathrm{to}\\ 2n$$\n",
    "\n",
    "We've provided a helper function called `prob_double_roll(x, n)` that will calculate this probability for you, i.e.\n",
    "\n",
    "```python\n",
    "prob = prob_double_roll(4, 6)\n",
    "```\n",
    "\n",
    "will return the probability of rolling a sum of 4 with two six-sided dice.\n",
    "\n",
    "#### Exercise 2\n",
    "\n",
    "Parametrize your test so that it works for any pair of n-sided dice. Test it using pairs of five- and seven-sided dice."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}