Lesson 01 Basics of Awk.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 01: Basics of Awk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you haven't read the Awk [man page](https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/awk.1.html), you should start there. It's helpful! Some highlights: \n",
    "\n",
    "> awk − pattern-directed scanning and processing language\n",
    "\n",
    "> `awk [ −F fs ] [ −v var=value ] [ ’prog’ | −f progfile ] [ file ... ]`\n",
    "\n",
    "> _Awk_ scans each input _file_ for lines that match any of a set of patterns specified literally in _prog_ or in one or more files specified as __−f__ _progfile_.\n",
    "\n",
    "> With each pattern there can be an associated action that will be performed when a line of a _file_ matches the pattern.\n",
    "\n",
    "> Each line is matched against the pattern portion of every pattern-action statement; the associated action is performed for each matched pattern\n",
    "\n",
    "> A pattern-action statement has the form `pattern {action}`.\n",
    "\n",
    "> A missing `{ action }` means print the line; a missing pattern always matches. \n",
    "\n",
    "I created an simple example file to demonstrate basic Awk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a\r\n",
      "bb\r\n",
      "ccc\r\n",
      "dddd\r\n",
      "ggg\r\n",
      "hh\r\n",
      "i"
     ]
    }
   ],
   "source": [
    "cat data/letters.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Basic Pattern\n",
    "\n",
    "If we match lines longer than two characters and use the implicit print action, we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bb\r\n",
      "ccc\r\n",
      "dddd\r\n",
      "ggg\r\n",
      "hh\r\n"
     ]
    }
   ],
   "source": [
    "awk 'length $0 > 2' data/letters.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`$0` is a built-in variable that contains the line.\n",
    "\n",
    "### A Basic Function\n",
    "\n",
    "If we leave out a pattern, we will match every line. A trivial action would be to print each line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a\r\n",
      "bb\r\n",
      "ccc\r\n",
      "dddd\r\n",
      "ggg\r\n",
      "hh\r\n",
      "i\r\n"
     ]
    }
   ],
   "source": [
    "awk '{ print }' data/letters.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `length` function as our action, we can get the length of each line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\r\n",
      "2\r\n",
      "3\r\n",
      "4\r\n",
      "3\r\n",
      "2\r\n",
      "1\r\n"
     ]
    }
   ],
   "source": [
    "awk '{ print length }' data/letters.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The action implicity acts on the whole line. We can be more explicit if we want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1a\r\n",
      "2bb\r\n",
      "3ccc\r\n",
      "4dddd\r\n",
      "3ggg\r\n",
      "2hh\r\n",
      "1i\r\n"
     ]
    }
   ],
   "source": [
    "awk '{ print length $0 }' data/letters.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awk has special controls for executing some code before the file input begins and after it is complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HI\r\n",
      "a\r\n",
      "bb\r\n",
      "ccc\r\n",
      "dddd\r\n",
      "ggg\r\n",
      "hh\r\n",
      "i\r\n",
      "BYE!\r\n"
     ]
    }
   ],
   "source": [
    "awk 'BEGIN { print \"HI\" } { print $0 } END { print \"BYE!\" }' data/letters.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can have more control over printing by using `printf`. The following example comes from the [GNU Awk User's Guide](https://www.gnu.org/software/gawk/manual/html_node/Printf-Examples.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name       Number\r\n",
      "----       ------\r\n",
      "Amelia     555-5553\r\n",
      "Anthony    555-3412\r\n",
      "Becky      555-7685\r\n",
      "Bill       555-1675\r\n",
      "Broderick  555-0542\r\n",
      "Camilla    555-2912\r\n",
      "Fabius     555-1234\r\n",
      "Julie      555-6699\r\n",
      "Martin     555-6480\r\n",
      "Samuel     555-3430\r\n",
      "Jean-Paul  555-2127\r\n"
     ]
    }
   ],
   "source": [
    "awk 'BEGIN { printf \"%-10s %s\\n\", \"Name\", \"Number\"\n",
    "             printf \"%-10s %s\\n\", \"----\", \"------\" }\n",
    "           { printf \"%-10s %s\\n\", $1, $2 }' data/mail-data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining Patterns and Functions\n",
    "\n",
    "Of course, patterns and functions can be combined so that the function is only applied when the pattern is matched. \n",
    "\n",
    "From the man page:\n",
    "\n",
    "> A pattern-action statement has the form\n",
    "\n",
    "> ```pattern { action }```\n",
    "\n",
    "We can print the length of all lines longer than 2 characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\r\n",
      "4\r\n",
      "3\r\n"
     ]
    }
   ],
   "source": [
    "awk 'length($0) > 2 { print length($0) }' data/letters.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually, we don't have to limit Awk to just one pattern! We can have arbitrarily many patterns separated by a semicolon or a new line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Short: 1\r\n",
      "Long:  3\r\n",
      "Long:  4\r\n",
      "Long:  3\r\n",
      "Short: 1\r\n"
     ]
    }
   ],
   "source": [
    "awk 'length($0) > 2 { print \"Long:  \" length($0) }; length($0) < 2 { print \"Short: \" length($0) }' data/letters.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiple Fields\n",
    "\n",
    "Awk is designed for easy handling of data with multiple fields per row. The field delimiter can be specified with the `-F` option.\n",
    "\n",
    "Here's a simple space-delimited file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Roses are red,\r\n",
      "Violets are blue,\r\n",
      "Sugar is sweet,\r\n",
      "And so are you.\r\n"
     ]
    }
   ],
   "source": [
    "awk '{ print }' data/field_data.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we specify the field seperator, we can print the second field from each row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "are\r\n",
      "are\r\n",
      "is\r\n",
      "so\r\n"
     ]
    }
   ],
   "source": [
    "awk -F \" \" '{ print $2 }' data/field_data.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't get an error if a line doesn't have the referenced field; it just shows up as blank:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\r\n",
      "\r\n",
      "you.\r\n"
     ]
    }
   ],
   "source": [
    "awk -F \" \" '{ print $4 }' data/field_data.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The seperator expression is interpreted as a regular expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Field 1: Roses \r\n",
      "Field 2: red,\r\n",
      "Field 1: Violets \r\n",
      "Field 2: blue,\r\n",
      "Field 1: Sugar \r\n",
      "Field 2: sweet,\r\n",
      "Field 1: And \r\n",
      "Field 2: you.\r\n"
     ]
    }
   ],
   "source": [
    "awk -F \"((so )?are|is) \" '{print \"Field 1: \" $1 \"\\nField 2: \" $2}' data/field_data.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regular Expressions\n",
    "\n",
    "Patterns can be regular expressions, not just built-in functions. From the man page:\n",
    "\n",
    "> Regular expressions are as defined in [re_format(7)](https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man7/re_format.7.html#//apple_ref/doc/man/7/re_format).  Isolated regular expressions in a pattern apply to the entire line.\n",
    "\n",
    "We can use a regular expression to find all words in the Unix words with 5 vowels in a row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cadiueio\r\n",
      "Chaouia\r\n",
      "euouae\r\n",
      "Guauaenok\r\n"
     ]
    }
   ],
   "source": [
    "awk '/[aeiou]{5}/' /usr/share/dict/words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Passing variables into program"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `-v` option for Awk allows us to pass variables it the program. For example, we could use it to hard code constants."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.1415\r\n"
     ]
    }
   ],
   "source": [
    "awk -v pi=3.1415 'BEGIN { print pi }'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use `-v` to pass Bash variables as Awk variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tdhopper\r\n"
     ]
    }
   ],
   "source": [
    "awk -v user=$USER 'BEGIN { print user }'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### If-else Statements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If-else statements in Awk are of the form:\n",
    "\n",
    "    if (condition) then-body [else else-body]\n",
    "  \n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 is odd\r\n",
      "2 is even\r\n",
      "3 is odd\r\n",
      "4 is even\r\n"
     ]
    }
   ],
   "source": [
    "printf \"1\\n2\\n3\\n4\" | awk \\\n",
    "    '{ \\\n",
    "        if ($1 % 2 == 0) print $1, \"is even\"; \\\n",
    "        else print $1, \"is odd\" \\\n",
    "     }'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Looping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awk includes several looping statements: `while`, `do while`, and `for`.\n",
    "\n",
    "They take the expected C-ish syntax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\r\n",
      "1\r\n",
      "2\r\n",
      "3\r\n",
      "4\r\n"
     ]
    }
   ],
   "source": [
    "awk \\\n",
    "    'BEGIN { \\\n",
    "        i = 0; \\\n",
    "        while (i < 5) { print i; i+=1; } \\\n",
    "     }'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\r\n"
     ]
    }
   ],
   "source": [
    "awk \\\n",
    "    'BEGIN { \\\n",
    "        i = 0; \\\n",
    "        do { print i; i+=1; } while(i < 0) \\\n",
    "     }'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\r\n",
      "1\r\n",
      "2\r\n",
      "3\r\n",
      "4\r\n"
     ]
    }
   ],
   "source": [
    "awk \\\n",
    "    'BEGIN { \\\n",
    "        i = 0; \\\n",
    "        for(i = 0; i<5; i++) print i \\\n",
    "     }'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`for` can also loop through the keys of an array, which we will see in the next lesson."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}