Submodule_00_background.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b2476ae8-9aad-4594-a003-f92ad8b0e126",
   "metadata": {},
   "source": [
    "# MDIBL Transcriptome Assembly Learning Module\n",
    "# Notebook 0: Background Material"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e6d2086-4dbf-4a61-a5bb-8f08a269f3fa",
   "metadata": {},
   "source": [
    "## Welcome!\n",
    "\n",
    "This is a series of notebooks that allows you to explore the biological and computational process of the transcriptome assembly. Through these notebooks, you will also learn to leverage the powerful capabilities of tools such as Nextflow and Google Life Science API to bring your computational capabilities to the next level!\n",
    "\n",
    "Before you get started, please take this prerequisite that checks existing knowledge that will be assumed to be known through the rest of these workbooks.\n",
    "\n",
    "Throughout the notebooks, there will be periodic quizzes and knowledge checks that you are encouraged to do.\n",
    "\n",
    "Good luck, and have fun!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22b95a28-fad7-4b6c-99ae-093c323f769c",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <i class=\"fa fa-pencil\" aria-hidden=\"true\"></i>\n",
    "    <b>Precheck:</b>\n",
    "</div>\n",
    "\n",
    ">Before you get started, please take this quick quiz that will verify some baseline knowledge on the ideas of DNA, RNA, Transcription, and Gene Expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c479507-0e54-44e1-a727-d13dceaa1c7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is an install that you need to run once to allow the quizes to be functional.\n",
    "!pip install jupyterquiz==2.0.7\n",
    "!pip install jupytercards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0a4331b-494c-4054-93c8-ef8433ca7b40",
   "metadata": {},
   "outputs": [],
   "source": [
    "from jupyterquiz import display_quiz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a89c8abf-fad1-4b23-b3fc-1a5933193fa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "display_quiz(\"quiz-material/00-pc1.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1d5ab0d-1671-47c6-888d-a8d2774df30c",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "    <i class=\"fa fa-hand-paper-o\" aria-hidden=\"true\"></i>\n",
    "    <b>Note: </b>  Some Resources\n",
    "</div>\n",
    "\n",
    ">If you feel unsure about your knowledge in any of these topics, please reference [Submodule_00_Glossary.md](./Submodule_00_Glossary.md) along with the National Human Genome Research Institute's [Glossary of Genomic and Genetic Terms](https://www.genome.gov/genetics-glossary)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39b204c4-847a-49b3-afa0-c4da4216ace4",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run the command below to watch the video\n",
    "from IPython.display import YouTubeVideo\n",
    "\n",
    "YouTubeVideo('abw2XAg1e_g', width=800, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45b85d9f-4115-446d-8120-02c988a7769f",
   "metadata": {},
   "source": [
    "## Why do we sequence RNA?\n",
    "RNA-sequencing (RNA-seq) is the most common means by which biological samples are characterized at the molecular level.  In brief, it is a means of measuring which genes have RNA copies (transcripts) present in a sample and in what relative abundance. The sample is prepared in such a way that DNA and proteins are degraded away, and then the remaining RNA is prepared such that it can be read (as a series of DNA bases A, C, G, and T) on a modern sequencer. Sequencing machines are generally classified as short read, which produces sequence read lengths of 50 to 150 nucleotides, or long-read, which can generate up to tens of thousands of bases. Short-read sequencers have been available for a longer time, and remain more capable of high throughput quantitative output, and these reads are the focus of our work here.\n",
    "\n",
    "The standard workflow analysis of RNA-seq data consists of these broad steps:\n",
    "- Quality assessment and pre-processing\n",
    "- Assignment of reads to transcripts/genes\n",
    "- Normalization of reads between samples\n",
    "- Assessment of differential expression of transcripts/genes between experimental conditions\n",
    "- Interpretation of the resulting differential expression profiles\n",
    "\n",
    "Implicit in the workflow above is the existence of a target transcriptome to which the RNA-seq reads can be compared, aligned, and assigned for quantification. For well-studied organisms, such as human, mouse, zebrafish, or other model organisms, there are abundant reference materials available from such sites as [Ensembl](https://www.ensembl.org/), [NCBI](https://ncbi.nlm.nih.gov/), and the [UCSC Genome Browser](https://genome.ucsc.edu/).\n",
    "\n",
    "For less well-studied organisms, no such references are generally available, however, the RNA-seq data contains the information necessary to infer not only abundance but also the transcript sequences from which the data was generated. The process of inferring the starting transcripts from the data, termed ***Transcriptome Assembly***, is the focus of this module."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e392c553-1831-4978-af06-83359f8de746",
   "metadata": {},
   "source": [
    "## Transcriptome Sequence Assembly\n",
    "As a first approximation, sequence assembly of a single molecule (*e.g.*, a chromosome) can be thought of as analogous to the process of reconstructing a picture from smaller, overlapping segments of the picture. Overlapping pieces are identified and matched, extending the construct until an estimation of the complete picture is generated. To make this metaphor a bit more realistic, the subsegments of the original picture are *imperfect*, such that successful construction of the complete picture will require error identification (or at least estimation) and correction.\n",
    "\n",
    "In order to extend this analog to transcriptome assembly, imagine that instead of one picture, our smaller segments instead are drawn from many pictures. Now the process of reconstruction will necessarily include a step that attempts to separate the smaller segments into related groups, after which the assembly procedure proceeds.\n",
    "\n",
    "#### Preprocessing and Data Cleaning\n",
    "For reasons described below, stringent quality assessment and filtering of the data is generally carried out before the assembly process is begun. The primary steps include:\n",
    "- Removal of low-quality score data\n",
    "- Removal of contaminant sequence data\n",
    "- Removal of known functional RNA\n",
    "\n",
    "#### Sequence Assembly\n",
    "<img  src=\"images/deBruijnGraph.png\" width=600 alt=\"Conceptual design of a de Bruijn Graph\">\n",
    "\n",
    "**Figure 1:** Conceptual diagram of a sequence-defined [de Bruijn graph](https://en.wikipedia.org/wiki/De_Bruijn_graph). (A) Each sequence in an RNA-seq is broken into overlapping *k*-mers. (B) Each *k*-mer becomes a node in the graph, shown in the example with *k*=6. Edges are drawn between nodes that match *k*-1 contiguous nucleotides. (C) Putative transcripts (shown in distinct colors) are represented as traversals of one of the many connected components of the graph generated by the starting sequence set.</p>\n",
    "\n",
    "#### Assembly Assessment\n",
    "- Internal consistency\n",
    "    - Use of a de Bruijn graph is computationally efficient (especially compared to exhaustive pairwise alignment of all sequence reads), but all \"long-range\" information is weakened.\n",
    "    - The weakening of the long-range information necessitates further QC. The problem is that building complete transcripts from just *k*-mers and their probabilities means that we can generate sequences that are computationally possible but don't exist in the input data. Internal consistency refers to the process of aligning the original input reads to the output transcriptome. Transcripts that do not get sufficient coverage are flagged as probable artifacts.\n",
    "- External consistency\n",
    "    - Studies of transcriptomes across many organisms have demonstrated common features. By \"external consistency\" we mean matching our new transcriptome to these expectations.\n",
    "    - [BUSCO](https://busco.ezlab.org/) is an innovative analysis and set of tools developed by the [EZlab at the Swiss Insitute of Bioinformatics](https://www.ezlab.org/). The fundamental idea behind BUSCO (**B**enchmarking **U**niversal **S**ingle-**C**opy **O**rthologs) derives from the Zdobnov group's analysis, which showed that for a defined phylogenetic range of organisms, there is a core set of protein-coding genes that are nearly universally present in only a single copy. The BUSCO tools test this assumption.\n",
    "    - The second standard process for external consistency is to align all predicted proteins for the new transcriptome to a complete set of proteins from a well-studied (e.g., fly or mouse) under the assumption that most of the proteins should match.\n",
    "\n",
    "#### Assembly Refinement\n",
    "Assemblies are refined in several different manners:\n",
    "- Removal of redundant (or likely so) transcripts, based on sequence similarity between assembled forms.\n",
    "- Limitation to transcripts with predicted/conceptual translated protein sequences that match known proteins in other organisms.\n",
    "\n",
    "For Assembly refinement, the TransPi workflow relies primarily on the \"EvidentialGene\" tool."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "725afa76-ab95-4ab5-8b85-6b6795436c0e",
   "metadata": {},
   "source": [
    "## Workflow Execution with Nextflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "394e8677-1f82-4e42-9e37-535777128a1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run the command below to watch the video\n",
    "from IPython.display import YouTubeVideo\n",
    "\n",
    "YouTubeVideo('FMcZD10Qrbs', width=800, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "624b90a4-732a-47a2-987e-e37d16f124bb",
   "metadata": {},
   "source": [
    "\n",
    "It is standard practice in modern biotechnology, bioinformatics, and computational biology that most complex analyses are carried out not by a single comprehensive program, but are instead carried out in a defined sequence of multiple programs. The process of running through these steps in the proper order is collectively called a ***workflow*** or ***pipeline***.\n",
    "\n",
    "<img align=\"right\" src=\"images/workflow_concept.png\" width=400>\n",
    "\n",
    "Workflow management systems, *e.g.*, Nextflow, provide a syntax for defining the order of steps and the associated flow of information between steps, while also providing management/control software that can read and carry out these workflows. The workflow control systems (which are generally platform-specific) are responsible for allocating resources, activating analysis steps, and also making sure that all steps occur in the proper order (e.g., only activating a sequence alignment program after the sequence quality control has been performed).\n",
    "\n",
    "<img align=\"left\" src=\"images/module_concept.png\" width=250>\n",
    "\n",
    "Workflows can be conceptually broken up into steps or modules (see the figure at left), which formalize the flow of information as inputs and outputs. A workflow conceptually ties the steps/modules together and enforces the dependencies (see the figure above), specifically in that if the output from one step is the input for a later step, the later step is blocked until the earlier step completes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77e96ae4-5e6e-4499-9f6e-bda00cbbabc5",
   "metadata": {},
   "source": [
    "## Running Individual Analysis Steps with Docker\n",
    "One of the most frustrating aspects of carrying out computational biology/bioinformatics programs is installing and maintaining the software programs that we use. These programs are built by a wide variety of research and industrial organizations, and they are built on a wide variety of platforms and utilize an even wider set of supporting libraries and auxiliary components. The reason this causes problems is that the underlying dependencies can conflict with those of other programs or the operating system.\n",
    "\n",
    "One of our primary tools for efficient maintenance is a container system such as [Docker](https://www.docker.com/).\n",
    "#### What are container systems and what are containers?\n",
    "A container system is a program that creates protected environments within your computer in which programs and their dependencies can be loaded only as long as they are needed to run the program of interest. The container system can load and unload containers as needed. One of the primary benefits of such systems is that once a container has been defined for a specific program, it can be reused repeatedly on the same computer or shared with others through online repositories.\n",
    "#### Why do we use containers?\n",
    "We use containers because they allow us to run a broad range of computer programs without having to manage all of their underlying programmatic dependencies. Having a program encapsulated in a container also preserves our ability to continue to use that version of the program, even if either the program or its dependencies are updated."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9497e93b-b398-41fb-87a5-f1082e13a0aa",
   "metadata": {},
   "source": [
    "## Running workflows using the Google Cloud Life Sciences API\n",
    "The [Google Cloud Life Sciences API (GLS)](https://cloud.google.com/life-sciences) is a service provided by Google that both understands workflows and also controls, including activation, program execution, and deactivation of Google Cloud computing servers.\n",
    "\n",
    "#### What do we gain by using GLS\n",
    "- The key to cost-efficient cloud computing is to only use the resources you need for as long as you need them. \n",
    "- GLS allows us to control our process from a modest, inexpensive machine that can interface with GLS to provision and use the more expensive machines needed for computing.\n",
    "- GLS explicitly supports the Nextflow workflow system that we are using, mapping computational tasks onto GCP computing resources."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97bd4c54-198a-4c94-a959-20d440a02156",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <i class=\"fa fa-pencil\" aria-hidden=\"true\"></i>\n",
    "    <b>Checkpoint 1:</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b64e6f8a-84ba-4279-b002-25ae4f0755ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "display_quiz(\"quiz-material/00-cp1.json\", shuffle_questions = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b46f12f-f439-425e-ba98-28bfe1b5ec77",
   "metadata": {},
   "source": [
    "## Jupyter Notebook Introduction\n",
    "\n",
    "All of the content contained within this module is presented in Jupyter notebooks which have the `.ipynb` file type. *You are in a Jupyter notebook right now.* Within each notebook is a series of cells that can be individually executed by pressing the `shift + enter` keys at the same time.\n",
    "\n",
    "Each cell has options as to how it is executed. For example, the text that you are reading right now in this cell is in the `Markdown` cell type, but there are also `code`, and `raw` cell types. In these modules, you will primarily be seeing `Markdown` and `code` cells. *You can choose what each cell type is by using the drop-down menu at the top of the notebook.*\n",
    "\n",
    "> <img src=\"images/cellMenu.png\" width=\"500\">\n",
    "\n",
    "For the code cells, information carries over between cells, but in execution order. This is important because when looking at a series of cells you may be expecting a specific output, but receive a different output due to the order of execution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02c8af23-fc70-4265-a07f-50ef77c12564",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <i class=\"fa fa-pencil\" aria-hidden=\"true\"></i>\n",
    "    <b>Example:</b> Follow the steps in the cells below\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da4d8800-63cb-4dd8-bc2e-b28a52513c76",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute 1st:\n",
    "var1 = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70147ffa-fe71-40eb-bf83-12e7cb0cfdce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute 2nd and 4th:\n",
    "print(var1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bb0a293-4feb-4ca3-b3fd-caf4df3ce6af",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute 3rd:\n",
    "var1 = 'not the same anymore'\n",
    "# And now run the cell above"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "927a8df2-87f9-45aa-a096-4e1133a97f64",
   "metadata": {},
   "source": [
    ">As you can see, `var1` got overwritten, and when you retroactively re-run the `print(var1)` cell, the output has changed, even though it is above the variable assignment.\n",
    "\n",
    "In the following notebooks, there will be some code cells that will take a long time to run. *Sometimes with no output.* So there are two ways to check if the cell is still executing:\n",
    "\n",
    "1. The first way to check is to look to the left of the code cell. There will be an indication that looks like this: `[ ]:` If it is empty, then the cell has never been executed yet. If it looks like this: `[*]:`, that means that it is actively executing. And if it looks like this `[53]:`, that means that it has completed executing.\n",
    "2. The second way, which will check to see if anything in the entire notebook is executing is in the top right of the notebook (Image Below). If the circle is empty, then nothing is actively executing. If the circle is grayed out, then there is something executing.\n",
    "\n",
    "> <img src=\"images/jupyterRuntimeCircle.png\" width=\"500\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20730a46-7d8c-4dec-a6de-3dcf10fdb888",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <i class=\"fa fa-pencil\" aria-hidden=\"true\"></i>\n",
    "    <b>Knowledge Check:</b> \n",
    "</div>\n",
    "\n",
    ">Change the cell below from a code cell to a markdown cell. *Don't forget to execute the cell.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ebda978-4968-46f6-a330-927a978b43ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "change our cell type\n",
    "# I WANT TO BE BIGGER\n",
    "*I want to be tilted*\n",
    "\n",
    "**I want to be bold**\n",
    "\n",
    "`And I want to have a grey background`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d473f3b2-3440-47a5-98b9-950ceb66704e",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <i class=\"fa fa-pencil\" aria-hidden=\"true\"></i>\n",
    "    <b>Checkpoint 2:</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a080a9dd-9ea6-4d6a-b5d5-ccd3382f09ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "display_quiz(\"quiz-material/00-cp2.json\", shuffle_questions = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "032f3aa2-8e73-4e64-bda7-a4a35e0f2a99",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <i class=\"fa fa-pencil\" aria-hidden=\"true\"></i>\n",
    "    <b>Glossary:</b> \n",
    "</div>\n",
    "\n",
    "> Within the the file [`Submodule_00_glossary.md`](./Submodule_00_Glossary.md) you will find a compilation of useful terms that will be beneficial to refer to throughout the rest of the learning module."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "489beca6-4a9e-4a2e-a646-6b276270d810",
   "metadata": {},
   "source": [
    "## When you are ready, proceed to the next notebook: [`Submodule_01_prog_setup.ipynb`](./Submodule_01_prog_setup.ipynb)."
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}