docs/GANDA_TOUR.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Tour of `ganda`\n",
    "\n",
    "This user guide is built in an interactive `bash` Jupyter Notebook.  If you've got `ganda` [installed](../README.md#installation) and in your `PATH` you can run the same commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ganda found in PATH\n"
     ]
    }
   ],
   "source": [
    "# ensure that the ganda executable is in your PATH\n",
    "which ganda >/dev/null && echo \"ganda found in PATH\" || echo \"ganda not found in PATH\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "# `ganda` Usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:\n",
      "   ganda - make http requests in parallel\n",
      "\n",
      "USAGE:\n",
      "   <urls/requests on stdout> | ganda [options]\n",
      "\n",
      "VERSION:\n",
      "   dev none unknown\n",
      "\n",
      "DESCRIPTION:\n",
      "   Pipe urls to ganda over stdout to make http requests to each url in parallel.\n",
      "\n",
      "AUTHOR:\n",
      "   Ted Naleid <contact@naleid.com>\n",
      "\n",
      "COMMANDS:\n",
      "   echoserver  Starts an echo server, --port <port> to override the default port of 8080\n",
      "   help, h     Shows a list of commands or help for one command\n",
      "\n",
      "GLOBAL OPTIONS:\n",
      "   --base-retry-millis value                              the base number of milliseconds to wait before retrying a request, exponential backoff is used for retries (default: 1000)\n",
      "   --response-body value, -B value                        transforms the body of the response. Values: 'raw' (unchanged), 'base64', 'discard' (don't emit body), 'escaped' (JSON escaped string), 'sha256' (default: raw)\n",
      "   --connect-timeout-millis value                         number of milliseconds to wait for a connection to be established before timeout (default: 10000)\n",
      "   --header value, -H value [ --header value, -H value ]  headers to send with every request, can be used multiple times (gzip and keep-alive are already there)\n",
      "   --insecure, -k                                         if flag is present, skip verification of https certificates (default: false)\n",
      "   --json-envelope, -J                                    emit result with JSON envelope with url, status, length, and body fields, assumes result is valid json (default: false)\n",
      "   --color                                                if flag is present, add color to success/warn messages (default: false)\n",
      "   --output-directory value                               if flag is present, save response bodies to files in the specified directory\n",
      "   --request value, -X value                              HTTP request method to use (default: \"GET\")\n",
      "   --retry value                                          max number of retries on transient errors (5XX status codes/timeouts) to attempt (default: 0)\n",
      "   --silent, -s                                           if flag is present, omit showing response code for each url only output response bodies (default: false)\n",
      "   --subdir-length value                                  length of hashed subdirectory name to put saved files when using --output-directory; use 2 for > 5k urls, 4 for > 5M urls (default: 0)\n",
      "   --throttle-per-second value                            max number of requests to process per second, default is unlimited (default: -1)\n",
      "   --workers value, -W value                              number of concurrent workers that will be making requests, increase this for more requests in parallel (default: 1)\n",
      "   --help, -h                                             show help (default: false)\n",
      "   --version, -v                                          print the version (default: false)\n"
     ]
    }
   ],
   "source": [
    "ganda help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "# `ganda` Basics\n",
    "\n",
    "\n",
    "`ganda` makes HTTP requests, similar to `curl`, just pipe it an URL on stdin and it will make a `GET` request and echo the body of the response on stdout.  The status code of the URL will be sent to stderr.\n",
    "\n",
    "We'll use [httpbin.org](http://httpbin.org) for the first few requests.  It returns a JSON representation of the request in the body of the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"args\": {}, \n",
      "  \"data\": \"\", \n",
      "  \"files\": {}, \n",
      "  \"form\": {}, \n",
      "  \"headers\": {\n",
      "    \"Accept-Encoding\": \"gzip\", \n",
      "    \"Host\": \"httpbin.org\", \n",
      "    \"User-Agent\": \"Go-http-client/1.1\", \n",
      "    \"X-Amzn-Trace-Id\": \"Root=1-66a91ed1-104785e5182ad5d848a0096c\"\n",
      "  }, \n",
      "  \"json\": null, \n",
      "  \"method\": \"GET\", \n",
      "  \"origin\": \"173.16.32.166\", \n",
      "  \"url\": \"http://httpbin.org/anything/1\"\n",
      "}\n",
      "Response: 200 http://httpbin.org/anything/1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "echo \"http://httpbin.org/anything/1\" | ganda "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "You can pipe multiple URLs to `ganda`.  It happily lives in the middle of shell pipes for making requests.\n",
    "\n",
    "Here we make 3 requests to `/anything/1`, `/anything/2`, and `/anything/3` and pipe them to `jq` where we grab just the `method` and `url` properties from the response.\n",
    "\n",
    "We've also added the `-s` (silent) flag to `ganda` to suppress the stderr output that shows the url and response codes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m:\u001b[0m\u001b[0;32m\"GET\"\u001b[0m\u001b[1;39m,\u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m:\u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m\u001b[1;39m}\u001b[0m\n",
      "\u001b[1;39m{\u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m:\u001b[0m\u001b[0;32m\"GET\"\u001b[0m\u001b[1;39m,\u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m:\u001b[0m\u001b[0;32m\"http://httpbin.org/anything/2\"\u001b[0m\u001b[1;39m\u001b[1;39m}\u001b[0m\n",
      "\u001b[1;39m{\u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m:\u001b[0m\u001b[0;32m\"GET\"\u001b[0m\u001b[1;39m,\u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m:\u001b[0m\u001b[0;32m\"http://httpbin.org/anything/3\"\u001b[0m\u001b[1;39m\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "seq 3 |\\\n",
    "  awk '{printf \"http://httpbin.org/anything/%s\\n\", $1}' |\\\n",
    "  ganda -s |\\\n",
    "  jq -c '{method, url}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "## JSON Output\n",
    "\n",
    "`ganda` uses the `-J` flag for JSON output.  This emits JSON with the response as the `\"body\"` field and includes other details about the request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"code\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m200\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"body\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"args\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"files\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"form\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"headers\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "      \u001b[0m\u001b[34;1m\"Accept-Encoding\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"gzip\"\u001b[0m\u001b[1;39m,\n",
      "      \u001b[0m\u001b[34;1m\"Host\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"httpbin.org\"\u001b[0m\u001b[1;39m,\n",
      "      \u001b[0m\u001b[34;1m\"User-Agent\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Go-http-client/1.1\"\u001b[0m\u001b[1;39m,\n",
      "      \u001b[0m\u001b[34;1m\"X-Amzn-Trace-Id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Root=1-66a91ed2-5bfa363b569ce54744523bab\"\u001b[0m\u001b[1;39m\n",
      "    \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"json\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"GET\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"origin\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"173.16.32.166\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "echo \"http://httpbin.org/anything/1\" |\\\n",
    "  ganda -s -J |\\\n",
    "  jq '.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "The body of the response is assumed to be JSON as a default.  This emits the `raw` response bytes after the `\"body\"` property.  If the response isn't JSON, you've got a few options for escaping/encoding the response using the `-B/--response-body <value>` flag:\n",
    "\n",
    "1. `raw` - the default, shown above\n",
    "2. `base64` - encode the bytes as a `base64` string, useful for binary content.\n",
    "3. `discard` - drop the bytes and set the body to `null` \n",
    "4. `escaped` - escape the JSON and emit the value as a String\n",
    "5. `sha256` - calculate the sha256 value of the body, useful for checking if the response has changed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"code\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m200\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"body\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ewogICJhcmdzIjoge30sIAogICJkYXRhIjogIiIsIAogICJmaWxlcyI6IHt9LCAKICAiZm9ybSI6IHt9LCAKICAiaGVhZGVycyI6IHsKICAgICJBY2NlcHQtRW5jb2RpbmciOiAiZ3ppcCIsIAogICAgIkhvc3QiOiAiaHR0cGJpbi5vcmciLCAKICAgICJVc2VyLUFnZW50IjogIkdvLWh0dHAtY2xpZW50LzEuMSIsIAogICAgIlgtQW16bi1UcmFjZS1JZCI6ICJSb290PTEtNjZhOTFlZDMtMDYwNTVmOTk0MDViNmYyOTBmOTA5ZTMzIgogIH0sIAogICJqc29uIjogbnVsbCwgCiAgIm1ldGhvZCI6ICJHRVQiLCAKICAib3JpZ2luIjogIjE3My4xNi4zMi4xNjYiLCAKICAidXJsIjogImh0dHA6Ly9odHRwYmluLm9yZy9hbnl0aGluZy8xIgp9Cg==\"\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# base64 encode the response body\n",
    "echo \"http://httpbin.org/anything/1\" |\\\n",
    "  ganda -s -J -B base64 |\\\n",
    "  jq '.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"code\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m200\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"body\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# discard the response body\n",
    "echo \"http://httpbin.org/anything/1\" |\\\n",
    "  ganda -s -J -B discard |\\\n",
    "  jq '.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"code\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m200\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"body\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"{\\n  \\\"args\\\": {}, \\n  \\\"data\\\": \\\"\\\", \\n  \\\"files\\\": {}, \\n  \\\"form\\\": {}, \\n  \\\"headers\\\": {\\n    \\\"Accept-Encoding\\\": \\\"gzip\\\", \\n    \\\"Host\\\": \\\"httpbin.org\\\", \\n    \\\"User-Agent\\\": \\\"Go-http-client/1.1\\\", \\n    \\\"X-Amzn-Trace-Id\\\": \\\"Root=1-66a91ed4-5001f6e743bce7dd42a1f5c8\\\"\\n  }, \\n  \\\"json\\\": null, \\n  \\\"method\\\": \\\"GET\\\", \\n  \\\"origin\\\": \\\"173.16.32.166\\\", \\n  \\\"url\\\": \\\"http://httpbin.org/anything/1\\\"\\n}\\n\"\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# JSON escape the response body\n",
    "echo \"http://httpbin.org/anything/1\" |\\\n",
    "  ganda -s -J -B escaped |\\\n",
    "  jq '.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"code\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m200\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"body\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"bdf9764f7059125ec2541df1e001e00a4590810e933e23c5513b0fe55962c36f\"\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# calculate the sha256 hash of the response body\n",
    "echo \"http://httpbin.org/anything/1\" |\\\n",
    "  ganda -s -J -B sha256 |\\\n",
    "  jq '.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "## Customizing Requests with JSON Request Syntax\n",
    "\n",
    "`ganda` supports an alternate JSON-lines syntax for requests.  The [JSON schema](../request.schema.json) is available, but the summary of fields it allows is:\n",
    "- `\"url\"` - required string - is the only required field - the request URL\n",
    "- `\"method\"` - optional string - a valid HTTP method (`GET|PUT|POST|DELETE|...`) - defaults to `GET`\n",
    "- `\"headers\"` - optional JSON object - string key/value pairs\n",
    "- `\"context\"` - optional JSON value - carried forward into the JSON output of the response, used to correlate requests and responses, can be a string, array, or object\n",
    "- `\"body\"` - optional JSON value - a string or valid JSON object, the body of the request\n",
    "- `\"bodyType\"` - optional enum - one of: `json` (default), `escaped`, or `base64`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding a Request Body\n",
    "\n",
    "What if you want to `POST` instead of `GET`?  `ganda` supports the same `-X <http method>` syntax that `curl` uses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m:\u001b[0m\u001b[0;32m\"POST\"\u001b[0m\u001b[1;39m,\u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m:\u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m\u001b[1;39m}\u001b[0m\n",
      "\u001b[1;39m{\u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m:\u001b[0m\u001b[0;32m\"POST\"\u001b[0m\u001b[1;39m,\u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m:\u001b[0m\u001b[0;32m\"http://httpbin.org/anything/2\"\u001b[0m\u001b[1;39m\u001b[1;39m}\u001b[0m\n",
      "\u001b[1;39m{\u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m:\u001b[0m\u001b[0;32m\"POST\"\u001b[0m\u001b[1;39m,\u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m:\u001b[0m\u001b[0;32m\"http://httpbin.org/anything/3\"\u001b[0m\u001b[1;39m\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "seq 3 |\\\n",
    "  awk '{printf \"http://httpbin.org/anything/%s\\n\", $1}' |\\\n",
    "  ganda -s -X POST |\\\n",
    "  jq -c '{method, url}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "But, along with most `POST` requests, you'll want to include a body.  `ganda` has an alternate JSON-lines syntax for requests that allows specifying the method and body:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"args\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"{\\\"key1\\\":\\\"value1\\\",\\\"key2\\\":\\\"value2\\\"}\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"files\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"form\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"headers\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"Accept-Encoding\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"gzip\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"Content-Length\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"33\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"Host\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"httpbin.org\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"User-Agent\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Go-http-client/1.1\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"X-Amzn-Trace-Id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Root=1-66a91ed6-4f8f6bd2228d0ee141fdaec3\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"json\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"key1\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"value1\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"key2\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"value2\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"POST\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"origin\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"173.16.32.166\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "echo '\n",
    "{ \n",
    "  \"method\": \"POST\", \n",
    "  \"url\": \"http://httpbin.org/anything/1\", \n",
    "  \"body\": { \"key1\": \"value1\", \"key2\": \"value2\" } \n",
    "}' |\\\n",
    "  # ganda wants JSON-lines input, use jq -c to compact the JSON to a single line\n",
    "  jq -c '.' |\\\n",
    "  ganda -s |\\\n",
    "  jq '.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, it assumes that the body in the request JSON is also a valid JSON object.\n",
    "\n",
    "If you have a body value that isn't valid JSON, `ganda` provides two different transformers via the `\"bodyType\"` JSON field:\n",
    "1. `\"bodyType\": \"escaped\"` - `ganda` will unescape the value before making the request.  This can be useful for escaped JSON (maybe that contains newlines), or for a uuencoded/base64 value that you store as a JSON string, but want the string quotes (`\"`) stripped from the ends.\n",
    "2. `\"bodyType\": \"base64\"` - `ganda` will `base64` decode the value before sending it as the body.  This is most often used with binary data that you want to `POST`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"args\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"{ \\\"key1\\\": \\\"value1\\\" }\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"files\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"form\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"headers\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"Accept-Encoding\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"gzip\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"Content-Length\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"20\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"Host\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"httpbin.org\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"User-Agent\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Go-http-client/1.1\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"X-Amzn-Trace-Id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Root=1-66a91ed7-19039110465d24d47e32e9ff\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"json\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"key1\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"value1\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"POST\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"origin\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"173.16.32.166\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# \"bodyType\": \"escaped\" - ganda will unescape before making the request\n",
    "echo '\n",
    "{ \n",
    "  \"url\": \"http://httpbin.org/anything/1\", \n",
    "  \"bodyType\": \"escaped\", \n",
    "  \"body\": \"{ \\\"key1\\\": \\\"value1\\\" }\" \n",
    "}' |\\\n",
    "  jq -c '.' |\\\n",
    "  ganda -s -X POST |\\\n",
    "  jq '.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So above, `escaped` told `ganda` to unescape the value before sending it, and this is the JSON object that was sent:\n",
    "```json\n",
    "{ \"key1\": \"value1\" }\n",
    "```\n",
    "\n",
    "next, we'll show how a `base64` encoded value, sent in the JSON as a string (with double quotes around it), will `base64` decode the string and send the literal JSON value of: \n",
    "\n",
    "```\n",
    "{ \"value\": \"was base64 escaped\" }\n",
    "```\n",
    "\n",
    "Normally, this is used for encoding binary files, but that doesn't render well in the response, so we're `base64` encoding JSON as an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"args\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"{ \\\"value\\\": \\\"was base64 escaped\\\" }\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"files\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"form\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"headers\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"Accept-Encoding\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"gzip\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"Content-Length\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"33\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"Host\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"httpbin.org\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"User-Agent\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Go-http-client/1.1\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"X-Amzn-Trace-Id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Root=1-66a91ed8-78025df11baf73b06bfeed38\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"json\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"was base64 escaped\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"POST\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"origin\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"173.16.32.166\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# \"bodyType\": \"base64\" - ganda will decode before making the request\n",
    "# generated with: echo -n '{ \"value\": \"was base64 escaped\" }' | base64\n",
    "echo '\n",
    "{ \n",
    "  \"url\": \"http://httpbin.org/anything/1\", \n",
    "  \"bodyType\": \"base64\", \n",
    "  \"body\": \"eyAidmFsdWUiOiAid2FzIGJhc2U2NCBlc2NhcGVkIiB9\" \n",
    "}' |\\\n",
    "  jq -c '.' |\\\n",
    "  ganda -s -X POST |\\\n",
    "  jq '.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "### Request Headers\n",
    "\n",
    "`ganda` allows adding static headers to every request with the `-H key:value` syntax.  Multiple headers can be specified, and they'll override defaulted values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"args\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"files\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"form\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"headers\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"Accept-Encoding\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"gzip\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"Host\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"httpbin.org\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"User-Agent\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"static-ganda\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"X-Amzn-Trace-Id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Root=1-66a91ed9-7c95dd6518c096e9249ae3ef\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"X-My-Header\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"1234\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"json\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"GET\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"origin\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"173.16.32.166\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "echo '{ \"url\": \"http://httpbin.org/anything/1\" }' |\\\n",
    "  ganda -s -H \"X-My-Header: 1234\" -H \"User-Agent: static-ganda\" |\\\n",
    "  jq '.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "The JSON-lines syntax can also specify per-request headers that will override static headers.\n",
    "\n",
    "Here, the `User-Agent` in the JSON overrides the static header `User-Agent` from the `-H` flag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"args\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"files\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"form\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"headers\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"Accept-Encoding\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"gzip\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"Host\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"httpbin.org\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"User-Agent\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"per-request-ganda\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"X-Amzn-Trace-Id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Root=1-66a91eda-3f42e5e7611abb683e39c0d4\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"X-My-Header\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"1234\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"X-Second-Header\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"5678\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"json\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;30mnull\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"GET\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"origin\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"173.16.32.166\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/anything/1\"\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "echo '\n",
    "{ \n",
    "  \"url\": \"http://httpbin.org/anything/1\", \n",
    "  \"headers\": {\"X-Second-Header\": \"5678\", \"User-Agent\": \"per-request-ganda\" } \n",
    "}' |\\\n",
    "  jq -c '.' |\\\n",
    "  ganda -s -H \"X-My-Header: 1234\" -H \"User-Agent: static-ganda\" |\\\n",
    "  jq '.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "## Request Context\n",
    "\n",
    "Your requests are part of a pipeline, what if you want to carry context through your pipeline that isn't part of the HTTP request/response?\n",
    "\n",
    "An example would be calling an HTTP endpoint to generate a new UUID, but the response does not include the ID that we want to associate with the UUID.\n",
    "\n",
    "`ganda` allows you to specify values along with the URL that will still be present in the JSON envelope output.\n",
    "\n",
    "This can be done with the simple request syntax by specifying tab-separated values after the URL:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://httpbin.org/uuid\t1\t\"single\tvalue\twith\ttabs\"\n"
     ]
    }
   ],
   "source": [
    "# echo can emit tab separated values for correlating requests and responses\n",
    "# here is what is being passed to ganda:\n",
    "echo -e 'http://httpbin.org/uuid\\t1\\t\"single\\tvalue\\twith\\ttabs\"'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/uuid\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"code\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m200\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"body\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"uuid\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"3046801f-8a41-435f-ac47-12ebf593d0c5\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"context\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m[\n",
      "    \u001b[0;32m\"1\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0;32m\"single\\tvalue\\twith\\ttabs\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m]\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "echo -e 'http://httpbin.org/uuid\\t1\\t\"single\\tvalue\\twith\\ttabs\"' |\\\n",
    "  ganda -s -J |\\\n",
    "  jq '.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the `\"context\"` emitted at the bottom of the JSON.\n",
    "\n",
    "The JSON-lines request format also allows context to be specified, and it can be any valid JSON object (string, array, or object):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"url\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"http://httpbin.org/uuid\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"code\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m200\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"body\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"uuid\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"108e190d-4d8b-4873-8bf5-4d608c5117a6\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"context\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m1\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"value\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"correlation value\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "echo '\n",
    "{ \n",
    "  \"url\": \"http://httpbin.org/uuid\", \n",
    "  \"context\": { \"id\": 1, \"value\": \"correlation value\"} \n",
    "}' |\\\n",
    "  jq -c '.' |\\\n",
    "  ganda -s -J |\\\n",
    "  jq '.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "## `ganda echoserver` - a simple server that echoes requests \n",
    "\n",
    "`ganda` comes with a built-in echo server to make verifying requests easier. \n",
    "\n",
    "We don't want to hammer the public `httpbin.org` server, so let's fire up `ganda echoserver` as a background process and use that instead. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:\n",
      "   ganda echoserver - Starts an echo server, --port <port> to override the default port of 8080\n",
      "\n",
      "USAGE:\n",
      "   ganda echoserver [command [command options]] \n",
      "\n",
      "OPTIONS:\n",
      "   --port value          Port number to start the echo server on (default: 8080)\n",
      "   --delay-millis value  Number of milliseconds to delay responding (default: 0)\n",
      "   --help, -h            show help (default: false)\n"
     ]
    }
   ],
   "source": [
    "ganda echoserver --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "Normally, we'd run `ganda echoserver` in another terminal window with a command like:\n",
    "\n",
    "```\n",
    "ganda echoserver --port 9090\n",
    "``` \n",
    "For this notebook, we'll run the echoserver in the background with a 1 second delay on every response.  We'll also suppress its logging output to stdout:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] 39607\n"
     ]
    }
   ],
   "source": [
    "# run the echoserver in the background. give it a 1000ms/1s delay for responding to each request.  \n",
    "# If you're running it in a separate terminal, you can omit the `>/dev/null &` part\n",
    "ganda echoserver --port 9090 --delay-millis 1000 >/dev/null &"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "Let's use `ganda` to make a single request to our echoserver so we can see its output.  It takes about a second because of the echoserver delay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"time\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"2024-07-30T12:11:59-05:00\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"remote_ip\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"::1\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"host\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"localhost:9090\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"GET\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"uri\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"/anything/1\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"user_agent\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Go-http-client/1.1\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"status\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m200\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"headers\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"Accept-Encoding\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"gzip\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"Connection\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"keep-alive\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"User-Agent\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Go-http-client/1.1\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"request_body\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"\"\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n",
      "\n",
      "real\t0m1.019s\n",
      "user\t0m0.027s\n",
      "sys\t0m0.008s\n"
     ]
    }
   ],
   "source": [
    "time echo \"http://localhost:9090/anything/1\" | ganda -s | jq '.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "## Parallelizing Requests\n",
    "\n",
    "By default, `ganda` using a single worker thread and a single connection to make requests.  This will guarantee that requests are made in the order they are received. \n",
    "\n",
    "If order doesn't matter, and you'd like to increase throughput, we can use the `-W <number of workers>` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response: 200 http://localhost:9090/slow-api/1\n",
      "Response: 200 http://localhost:9090/slow-api/2\n",
      "Response: 200 http://localhost:9090/slow-api/3\n",
      "Response: 200 http://localhost:9090/slow-api/4\n",
      "Response: 200 http://localhost:9090/slow-api/5\n",
      "Response: 200 http://localhost:9090/slow-api/6\n",
      "Response: 200 http://localhost:9090/slow-api/7\n",
      "Response: 200 http://localhost:9090/slow-api/8\n",
      "Response: 200 http://localhost:9090/slow-api/9\n",
      "Response: 200 http://localhost:9090/slow-api/10\n",
      "10.0  0:00:10 [ 995m/s] [ 995m/s]\n"
     ]
    }
   ],
   "source": [
    "# our echoserver is running with a 1000ms delay.  about 10 seconds to complete with the single default worker\n",
    "seq 10 |\\\n",
    "  awk '{printf \"http://localhost:9090/slow-api/%s\\n\", $1}' |\\\n",
    "  ganda |\\\n",
    "  # use pv - pipeviewer - to show the total number of requests, the total time taken, and the rate of requests per second\n",
    "  pv -albert > /dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "If we increase the number of workers to 10, we should finish in about a second"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response: 200 http://localhost:9090/slow-api/8\n",
      "Response: 200 http://localhost:9090/slow-api/2\n",
      "Response: 200 http://localhost:9090/slow-api/10\n",
      "Response: 200 http://localhost:9090/slow-api/5\n",
      "Response: 200 http://localhost:9090/slow-api/4\n",
      "Response: 200 http://localhost:9090/slow-api/9\n",
      "Response: 200 http://localhost:9090/slow-api/6\n",
      "Response: 200 http://localhost:9090/slow-api/1\n",
      "Response: 200 http://localhost:9090/slow-api/7\n",
      "Response: 200 http://localhost:9090/slow-api/3\n",
      "10.0  0:00:01 [9.78 /s] [9.78 /s]\n"
     ]
    }
   ],
   "source": [
    "# our echoserver is running with a 1 second delay so the 10 requests \n",
    "# should be handled by 10 workers in about 1 second\n",
    "seq 10 |\\\n",
    "  awk '{printf \"http://localhost:9090/slow-api/%s\\n\", $1}' |\\\n",
    "  ganda -W 10 |\\\n",
    "  pv -albert > /dev/null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.00k 0:00:10 [99.0 /s] [99.0 /s]\n"
     ]
    }
   ],
   "source": [
    "# if we use 100 parallel workers and make 1k requests that \n",
    "# each take 1 second, it should take about 10 seconds\n",
    "seq 1000 |\\\n",
    "  awk '{printf \"http://localhost:9090/slow-api/%s\\n\", $1}' |\\\n",
    "  ganda -s -W 100 | \n",
    "  pv -albert > /dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "`ganda` also supports throttling the number of requests its workers will make using the `--throttle-per-second <requests per second>` flag.\n",
    "\n",
    "If we use 100 parallel workers, but throttle them so that they can only make 5 requests per second, it should take about 20 seconds to complete 100 requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 100  0:00:21 [4.76 /s] [4.76 /s]\n"
     ]
    }
   ],
   "source": [
    "seq 100 |\\\n",
    "  awk '{printf \"http://localhost:9090/slow-api/%s\\n\", $1}' |\\\n",
    "  ganda -s -W 100 --throttle-per-second 5 |\\\n",
    "  pv -albert > /dev/null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "echoserver stopped\n"
     ]
    }
   ],
   "source": [
    "# clean up the delay echoserver that we'd previously run in the background\n",
    "pkill ganda && echo \"echoserver stopped\" || echo \"echoserver not stopped\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "## Saving Individual Responses to Files\n",
    "\n",
    "`ganda` also supports saving individual responses to files. This can be useful for debugging or for saving responses for later analysis. \n",
    "\n",
    "The `--output-directory <directory_name>` flag is used to specify the directory where the responses should be saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] 39629\n"
     ]
    }
   ],
   "source": [
    "# start up a echoserver that has no delay on port 9090 and put it in the background\n",
    "ganda echoserver --port 9090 >/dev/null &"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "Let's make 10 requests and save them as individual files in the `scratch/ten` directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response: 200 http://localhost:9090/fast-api/1 -> scratch/ten/http-localhost-9090-fast-api-1\n",
      "Response: 200 http://localhost:9090/fast-api/2 -> scratch/ten/http-localhost-9090-fast-api-2\n",
      "Response: 200 http://localhost:9090/fast-api/3 -> scratch/ten/http-localhost-9090-fast-api-3\n",
      "Response: 200 http://localhost:9090/fast-api/4 -> scratch/ten/http-localhost-9090-fast-api-4\n",
      "Response: 200 http://localhost:9090/fast-api/5 -> scratch/ten/http-localhost-9090-fast-api-5\n",
      "Response: 200 http://localhost:9090/fast-api/6 -> scratch/ten/http-localhost-9090-fast-api-6\n",
      "Response: 200 http://localhost:9090/fast-api/7 -> scratch/ten/http-localhost-9090-fast-api-7\n",
      "Response: 200 http://localhost:9090/fast-api/8 -> scratch/ten/http-localhost-9090-fast-api-8\n",
      "Response: 200 http://localhost:9090/fast-api/9 -> scratch/ten/http-localhost-9090-fast-api-9\n",
      "Response: 200 http://localhost:9090/fast-api/10 -> scratch/ten/http-localhost-9090-fast-api-10\n",
      "0.00  0:00:00 [0.00 /s] [0.00 /s]\n"
     ]
    }
   ],
   "source": [
    "seq 10 |\\\n",
    "  awk '{printf \"http://localhost:9090/fast-api/%s\\n\", $1}' |\\\n",
    "  ganda -W 1 --output-directory scratch/ten |\\\n",
    "  pv -albert > /dev/null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http-localhost-9090-fast-api-1\thttp-localhost-9090-fast-api-5\n",
      "http-localhost-9090-fast-api-10\thttp-localhost-9090-fast-api-6\n",
      "http-localhost-9090-fast-api-2\thttp-localhost-9090-fast-api-7\n",
      "http-localhost-9090-fast-api-3\thttp-localhost-9090-fast-api-8\n",
      "http-localhost-9090-fast-api-4\thttp-localhost-9090-fast-api-9\n"
     ]
    }
   ],
   "source": [
    "ls scratch/ten"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"time\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"2024-07-30T12:12:43-05:00\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"remote_ip\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"::1\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"host\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"localhost:9090\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"method\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"GET\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"uri\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"/fast-api/1\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"user_agent\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Go-http-client/1.1\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"status\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m200\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"headers\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
      "    \u001b[0m\u001b[34;1m\"Accept-Encoding\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"gzip\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"Connection\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"keep-alive\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0m\u001b[34;1m\"User-Agent\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Go-http-client/1.1\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"request_body\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"\"\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# show the JSON envelope response for one of the requests\n",
    "cat scratch/ten/http-localhost-9090-fast-api-1 | jq '.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's great for a low number of files, but filesystems get cranky when you get more than low thousands of files in a single directory.\n",
    "\n",
    "`ganda` supports a `--subdir-length <length>/-S <length>` flag that will hash the url and put the response in a subdirectory for that hash.\n",
    "\n",
    "So with a `--subdir-length 2` the `http://localhost:9090/fast-api/1` response gets hashed to the `d8` subdirectory under our `scratch/ten-subdir-length-two` output directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response: 200 http://localhost:9090/fast-api/3 -> scratch/ten-subdir-length-two/55/http-localhost-9090-fast-api-3\n",
      "Response: 200 http://localhost:9090/fast-api/9 -> scratch/ten-subdir-length-two/91/http-localhost-9090-fast-api-9\n",
      "Response: 200 http://localhost:9090/fast-api/2 -> scratch/ten-subdir-length-two/1d/http-localhost-9090-fast-api-2\n",
      "Response: 200 http://localhost:9090/fast-api/4 -> scratch/ten-subdir-length-two/47/http-localhost-9090-fast-api-4\n",
      "Response: 200 http://localhost:9090/fast-api/10 -> scratch/ten-subdir-length-two/eb/http-localhost-9090-fast-api-10\n",
      "Response: 200 http://localhost:9090/fast-api/1 -> scratch/ten-subdir-length-two/d8/http-localhost-9090-fast-api-1\n",
      "Response: 200 http://localhost:9090/fast-api/6 -> scratch/ten-subdir-length-two/3f/http-localhost-9090-fast-api-6\n",
      "Response: 200 http://localhost:9090/fast-api/8 -> scratch/ten-subdir-length-two/d1/http-localhost-9090-fast-api-8\n",
      "Response: 200 http://localhost:9090/fast-api/7 -> scratch/ten-subdir-length-two/80/http-localhost-9090-fast-api-7\n",
      "Response: 200 http://localhost:9090/fast-api/5 -> scratch/ten-subdir-length-two/44/http-localhost-9090-fast-api-5\n",
      "0.00  0:00:00 [0.00 /s] [0.00 /s]\n"
     ]
    }
   ],
   "source": [
    "seq 10 |\\\n",
    "  awk '{printf \"http://localhost:9090/fast-api/%s\\n\", $1}' |\\\n",
    "  ganda -W 10 --output-directory scratch/ten-subdir-length-two --subdir-length 2 |\\\n",
    "  pv -albert > /dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make 100k requests and see how many of them get hashed to the `d8` subdirectory that the `/fast-api/1` request hashed to above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 100k 0:00:13 [7.57k/s] [7.57k/s]\n"
     ]
    }
   ],
   "source": [
    "seq 100000 |\\\n",
    "  awk '{printf \"http://localhost:9090/fast-api/%s\\n\", $1}' |\\\n",
    "  ganda -W 3 --output-directory scratch/10k-subdir-length-two --subdir-length 2 2>&1 |\\\n",
    "  pv -albert > /dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "So it took less than 15 seconds on my machine to make 100k requests to the echoserver and save each response to its own file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     256\n"
     ]
    }
   ],
   "source": [
    "# how many subdirectories were created\n",
    "ls scratch/10k-subdir-length-two | wc -l        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00\n",
      "01\n",
      "02\n",
      "03\n",
      "04\n"
     ]
    }
   ],
   "source": [
    "# show the first 5 subdirectories - expect 00, 01, 02, 03, 04\n",
    "ls scratch/10k-subdir-length-two | head -n 5    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     406\n"
     ]
    }
   ],
   "source": [
    "# how many files are in the d8 subdirectory\n",
    "ls scratch/10k-subdir-length-two/d8 | wc -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http-localhost-9090-fast-api-1\n",
      "http-localhost-9090-fast-api-1000\n",
      "http-localhost-9090-fast-api-10057\n",
      "http-localhost-9090-fast-api-10125\n",
      "http-localhost-9090-fast-api-10135\n"
     ]
    }
   ],
   "source": [
    "# show the first 5 files in the d8 subdirectory\n",
    "ls scratch/10k-subdir-length-two/d8 | head -n 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "There are 256 subdirectories, so every hash value was hit, and our sample subdirectory had 406 files in it.  A nice distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "echoserver stopped\n"
     ]
    }
   ],
   "source": [
    "# clean up the scratch directory\n",
    "rm -rf scratch\n",
    "\n",
    "# clean up the background echoserver\n",
    "pkill ganda && echo \"echoserver stopped\" || echo \"echoserver not stopped\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "That's it!  A whirlwind tour of what `ganda` can do."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}