nmt-python-advanced-synthetic-data-generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "de4ac58e",
   "metadata": {},
   "source": [
    "<img src=\"\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# How to perform synthetic data generation using Riva NMT Multilingual model with Nvidia NeMo\n",
    "This tutorial walks you through how to perform synthetic data generation using a Riva NMT Multilingual model with Nvidia NeMo. The synthetic data generated in turn can be used for fine-tuning the models further."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3c4adc2",
   "metadata": {},
   "source": [
    "## NVIDIA Riva Overview\n",
    "\n",
    "NVIDIA Riva is a GPU-accelerated SDK for building speech AI applications that are customized for your use case and deliver real-time performance. <br/>\n",
    "Riva offers a rich set of speech and natural language understanding services such as:\n",
    "\n",
    "- Automated speech recognition (ASR)\n",
    "- Text-to-Speech synthesis (TTS)\n",
    "- Neural Machine Translation (NMT)\n",
    "- A collection of natural language processing (NLP) services, such as named entity recognition (NER), punctuation, and intent classification.\n",
    "\n",
    "In this tutorial, we will perform data generation using a Riva NMT Multilingual model with Nvidia NeMo. <br> \n",
    "To understand the basics of Riva NMT APIs, refer to the \"How do I perform Language Translation using Riva NMT APIs with out-of-the-box models?\" tutorial in [Riva NMT Tutorials](https://github.com/nvidia-riva/tutorials/blob/main/nmt-python-basics.ipynb). <br>\n",
    "\n",
    "For more information about Riva, refer to the [Riva developer documentation](https://developer.nvidia.com/riva). <br> For more information about Riva NMT, refer to the [Riva NMT documentation](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/translation/translation-overview.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9271159f",
   "metadata": {},
   "source": [
    "## NVIDIA NeMo Overview\n",
    "\n",
    "NVIDIA NeMo is a toolkit for building new state-of-the-art conversational AI models. NeMo has separate collections for Automatic Speech Recognition (ASR), Natural Language Processing (NLP), and Text-to-Speech (TTS) models. Each collection consists of prebuilt modules that include everything needed to train on your data. Every module can easily be customized, extended, and composed to create new conversational AI model architectures. \n",
    "<br>  \n",
    "For more information about NeMo, refer to the [NeMo product page](https://developer.nvidia.com/nvidia-nemo) and [documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/starthere/intro.html). The open-source NeMo repository can be found [here](https://github.com/NVIDIA/NeMo)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "124d8455",
   "metadata": {},
   "source": [
    "## Generating synthetic data using Riva NMT Multilingual model with NVIDIA NeMo\n",
    "\n",
    "For this tutorial, we will be using the Riva NMT Multilingual Any-to-En model on the [Scielo](https://data.scielo.org/) [English-Spanish dataset](https://figshare.com/articles/dataset/A_Large_Parallel_Corpus_of_Full-Text_Scientific_Articles/5382757) for generating data in french language.\n",
    "\n",
    "\n",
    "The process of synthetic data generation here can be split into following steps:\n",
    "1. Requirements and Setup.\n",
    "2. Data preprocessing(may vary based on actual data you use, please follow fine-tuning tutorial for more detailed pre-processing).\n",
    "3. Running inference using the NMT model with NeMo.\n",
    "4. Refer to the fine-tuning tutorial for using this data to customize the OOTB model.  \n",
    "\n",
    "Let's walk through each of these steps in detail."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bcea097",
   "metadata": {},
   "source": [
    "<a id='nmt_requirements_and_setup'></a>\n",
    "### Step 1. Requirements and Setup\n",
    "\n",
    "This tutorial needs to be run from inside a NeMo docker container. If you are not running this tutorial through a NeMo docker container, please refer to the [Riva NMT Tutorials](https://github.com/nvidia-riva/tutorials/blob/main/nmt-python-basics.ipynb) to get started.\n",
    "\n",
    "Before we get into the Requirements and Setup, let us create a base directory for our work here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "410da3c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "base_dir = \"NMTSynDataGeneration\"\n",
    "!mkdir $base_dir\n",
    "base_dir=os.path.abspath(\"NMTSynDataGeneration\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4b0cd97",
   "metadata": {},
   "source": [
    "1. Clone the [NeMo github repository](https://github.com/NVIDIA/NeMo)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2682295d",
   "metadata": {},
   "outputs": [],
   "source": [
    "NeMoBranch = \"v1.17.0_pt_23.04\"\n",
    "!git clone -b $NeMoBranch https://github.com/NVIDIA/NeMo $base_dir/NeMo\n",
    "!apt-get update && apt-get install -y libsndfile1 ffmpeg\n",
    "!pip3 install \"cython<3.0.0\" wheel && pip3 install pyyaml==5.4.1 --no-build-isolation\n",
    "%cd $base_dir/NeMo\n",
    "!./reinstall.sh\n",
    "!pip install torchmetrics==0.11.4\n",
    "%cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "349e2f52",
   "metadata": {},
   "source": [
    "2. Check CUDA installation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28113a53",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "122a1aab",
   "metadata": {},
   "source": [
    "3. Install Apex (if not using NeMo container)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a69b834",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/NVIDIA/apex.git\n",
    "%cd apex\n",
    "!git checkout a32d7a6dddcf4e39d241b0d139c222a97c91887d\n",
    "!pip install -v --disable-pip-version-check --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" --global-option=\"--fast_layer_norm\" --global-option=\"--distributed_adam\" --global-option=\"--deprecated_fused_adam\" ./\n",
    "%cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c713ecd",
   "metadata": {},
   "source": [
    "#### Data download\n",
    "Let us download the [Scielo](https://data.scielo.org/) [English-Spanish dataset](https://figshare.com/articles/dataset/A_Large_Parallel_Corpus_of_Full-Text_Scientific_Articles/5382757). Specifically we are going to download the Moses's version of the dataset, which consist of 2 files, `en_es.en` and `en_es.es`. Each newline-separated entry in the `en_es.en` file is a translation of the corresponding entry in the `en_es.es` file, and vice-versa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4248defd",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = base_dir + \"/data\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc6eeb24",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir $data_dir\n",
    "\n",
    "# Download the Scielo dataset\n",
    "!wget -P $data_dir https://figshare.com/ndownloader/files/14019287\n",
    "# Untar the downloaded the Scielo dataset\n",
    "!tar -xvf $data_dir/14019287 -C $data_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12b75442",
   "metadata": {},
   "source": [
    "### Step 2. Data preprocessing\n",
    "\n",
    "Data preprocessing consists of multiple steps to improve the quality of the dataset. [NeMo documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/machine_translation.html#data-cleaning-normalization-tokenization) provides detailed instructions about the 8-step data preprocessing for NMT. NeMo also provides a [jupyter notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Data_Preprocessing_and_Cleaning_for_NMT.ipynb) that takes users programatically through the different preprocessing steps. Note that depending on the dataset, some or all preprocessing steps can be skipped.\n",
    "\n",
    "To simplify the process in the Riva NMT program, we are only performing lang id filtering before data generation to get rid of any noise that maybe present in raw dataset. The input to these scripts will be a parallel corpus (i.e., source and target language) data files. In this tutorial, we are using the Moses' version of the Scielo dataset, which directly provides us the source (`en_es.en`) and target (`en_es.es`) data files. If the dataset does not directly provide these files, then we first need to generate these 2 files from the dataset before using the preprocessing scripts.\n",
    "\n",
    "#### Language filtering\n",
    "The language filtering preprocessing script is used for verifying language in machine translation data sets, using the [Fasttext Language Identification model](https://fasttext.cc/docs/en/language-identification.html). If the script is used on a parallel corpus, it verifies both a source and a target language. Filtered data is stored into the files specified by `output_src` and `output-tgt`, and the removed lines are put into the files specified by `removed_src` and `removed-tgt`. If language cannot be detected (e.g. date), the line is removed.\n",
    "\n",
    "This script exposes a number of parameters, the most common of which are:\n",
    "- input-src: Path to the input file which contains text in source language.\n",
    "- input-tgt: Path to the input file which contains text in target language.\n",
    "- output-src: File path where the source language's filtered data is to be saved.\n",
    "- output-tgt: File path where the target language's filtered data is to be saved.\n",
    "- removed-src: File path where the discarded data from source language is to be saved.\n",
    "- removed-tgt: File path where the discarded data from target language is to be saved.\n",
    "- source-lang: Source language's language code.\n",
    "- target-lang: Target language's language code.\n",
    "- fasttext-model: Path to fasttext model. The description and download links are [here](https://fasttext.cc/docs/en/language-identification.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b00f3c56",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let us first download the fasttext model.\n",
    "!wget https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin -O $data_dir/lid.176.bin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf52bda5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Running the language filtering preprocessing script.\n",
    "!python $base_dir/NeMo/scripts/neural_machine_translation/filter_langs_nmt.py \\\n",
    "    --input-src $data_dir/en_es.en \\\n",
    "    --input-tgt $data_dir/en_es.es \\\n",
    "    --output-src $data_dir/en_es_preprocessed.en \\\n",
    "    --output-tgt $data_dir/en_es_preprocessed.es \\\n",
    "    --removed-src $data_dir/en_es_garbage.en \\\n",
    "    --removed-tgt $data_dir/en_es_garbage.es \\\n",
    "    --source-lang en \\\n",
    "    --target-lang es \\\n",
    "    --fasttext-model $data_dir/lid.176.bin\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f57d1a56",
   "metadata": {},
   "source": [
    "#### Download the ootb model to perform data generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be7d5c4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create directory to hold model\n",
    "model_dir = base_dir + \"/model\"\n",
    "!mkdir $model_dir\n",
    "\n",
    "# Download the NMT model from NGC using wget command\n",
    "!wget -O $model_dir/megatronnmt_en_any_500m_1.0.0.zip --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/nemo/megatronnmt_en_any_500m/versions/1.0.0/zip \n",
    "\n",
    "# Unzip the downloaded model zip file.\n",
    "!unzip -o $model_dir/megatronnmt_en_any_500m_1.0.0.zip -d $model_dir/pretrained_ckpt\n",
    "\n",
    "# Alternate way to download the model from NGC using NGC CLI (Please make sure to install and setup NGC CLI):\n",
    "#!cd $model_dir && ngc registry model download-version \"nvidia/nemo/megatronnmt_any_en_500m:1.0.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fd496a2",
   "metadata": {},
   "source": [
    "### Step 3. Running inference using the NMT model with NeMo for data generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "4f7d46e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[NeMo W 2023-12-15 10:48:14 experimental:27] Module <class 'nemo.collections.nlp.data.language_modeling.megatron.megatron_batch_samplers.MegatronPretrainingRandomBatchSampler'> is experimental, not ready for production and is not fully supported. Use at your own risk.\n",
      "[NeMo W 2023-12-15 10:48:14 experimental:27] Module <class 'nemo.collections.nlp.models.text_normalization_as_tagging.thutmose_tagger.ThutmoseTaggerModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.\n",
      "[NeMo W 2023-12-15 10:48:15 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.\n",
      "LexerNoViableAltException: \\\n",
      "                           ^\n",
      "See https://hydra.cc/docs/next/advanced/override_grammar/basic for details\n",
      "\n",
      "Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python $base_dir/NeMo/examples/nlp/machine_translation/nmt_transformer_infer_megatron.py \\\n",
    "     model_file=$model_dir/pretrained_ckpt/megatronnmt_en_any_500m.nemo \\\n",
    "     srctext=$data_dir/en_es_preprocessed.en \\\n",
    "     tgtout=$data_dir/en_fr2.fr \\\n",
    "     source_lang=en \\\n",
    "     target_lang=fr \\\n",
    "     batch_size=10 \\\n",
    "     trainer.precision=32\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f48ad0a",
   "metadata": {},
   "source": [
    "### Step 4. Refer to the fine-tuning tutorial for using this data to customize the OOTB model.\n",
    "\n",
    "Lastly, follow the steps in \" in [Riva NMT Tutorials](https://ngc.nvidia.com/resources/riem1phmzvud:riva:riva_nmt_ea_tutorials) to use this data for customizing the OOTB model."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}