9. Understanding VideoBERT, BART, and more/9.10. Computing sentence representation using BERT as service.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "85YqSRLI5kLR"
   },
   "source": [
    "# BERT as Service \n",
    "The bert-as-service is another popularly used library for BERT. It maps the variable-length sentence to a fixed-length representation. It is simple, scalable, and easy to use. Besides, bert-as-service also has excellent documentation with clear details about how the library works. We can check the documentation here - https://bert-as-service.readthedocs.io/en/latest/index.html\n",
    "\n",
    "In this section, let us get a basic overview of how to use bert-as-service for obtaining fixed-length sentence representation. \n",
    "\n",
    "#### Make sure to run this notebook in GPU \n",
    "\n",
    "Installing the library \n",
    "The bert-as-service can be installed directly using pip. As shown below, we install the bert-serving-client and bert-serving-server :\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "id": "zCo8t8Ay5kLc"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install tensorflow==1.14\n",
    "!pip install bert-serving-client\n",
    "!pip install -U bert-serving-server[http]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YdnVOeLd5kLd"
   },
   "source": [
    "## Computing sentence representation \n",
    "Now, let us explore how to use bert-as-service to obtain a fixed-length representation of the variable-length sentences. Let's compute the representation of two sentences using bert-as-service and finds the similarity between the two sentences. \n",
    "\n",
    "First, we will download and unzip the pre-trained BERT model which we want to use. In this example, we use the pre-trained BERT-base uncased model. We can also try any other pre-trained BERT models: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "jXsxyXZP5kLd",
    "outputId": "798632bb-66af-47bc-faf0-8dc2f58fe788"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-12-30 23:07:47--  https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip\n",
      "Resolving storage.googleapis.com (storage.googleapis.com)... 172.253.122.128, 142.250.73.208, 172.253.63.128, ...\n",
      "Connecting to storage.googleapis.com (storage.googleapis.com)|172.253.122.128|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 407727028 (389M) [application/zip]\n",
      "Saving to: ‘uncased_L-12_H-768_A-12.zip’\n",
      "\n",
      "uncased_L-12_H-768_ 100%[===================>] 388.84M   254MB/s    in 1.5s    \n",
      "\n",
      "2020-12-30 23:07:48 (254 MB/s) - ‘uncased_L-12_H-768_A-12.zip’ saved [407727028/407727028]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "COK3nAV15kLe",
    "outputId": "90efb871-7a16-4b8c-e336-a446c8ded1e6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  uncased_L-12_H-768_A-12.zip\n",
      "   creating: uncased_L-12_H-768_A-12/\n",
      "  inflating: uncased_L-12_H-768_A-12/bert_model.ckpt.meta  \n",
      "  inflating: uncased_L-12_H-768_A-12/bert_model.ckpt.data-00000-of-00001  \n",
      "  inflating: uncased_L-12_H-768_A-12/vocab.txt  \n",
      "  inflating: uncased_L-12_H-768_A-12/bert_model.ckpt.index  \n",
      "  inflating: uncased_L-12_H-768_A-12/bert_config.json  \n"
     ]
    }
   ],
   "source": [
    "!unzip uncased_L-12_H-768_A-12.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Eq-VOPyx5kLe"
   },
   "source": [
    "\n",
    "Next, we start bert-server. While starting the BERT server, we also pass the pooling strategy that we prefer. That is, we learned that BERT returns the representation for each word in the sentence, and to obtain the representation of a complete sentence, we use the pooling method. Thus, we pass the pooling strategy which we want to use. In this example, we use the mean pooling strategy:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "id": "AcAS-BIA5kLf"
   },
   "outputs": [],
   "source": [
    "!nohup bert-serving-start -pooling_strategy REDUCE_MEAN -model_dir=./uncased_L-12_H-768_A-12 > out.file 2>&1 &"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UCu6q4Ev5kLf"
   },
   "source": [
    "\n",
    "Next, import the BERT client: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "id": "iROSq1nt5kLf"
   },
   "outputs": [],
   "source": [
    "from bert_serving.client import BertClient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qH9yKw005kLg"
   },
   "source": [
    "\n",
    "Start the BERT client: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "id": "rFtWg3yG5kLg"
   },
   "outputs": [],
   "source": [
    "bc = BertClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vM8pJOll5kLg"
   },
   "source": [
    "\n",
    "Define the sentences to which we need to compute representations: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "id": "ifgMsEHF5kLh"
   },
   "outputs": [],
   "source": [
    "sentence1 = 'the weather is great today'\n",
    "sentence2 = 'it looks like today the weather is pretty nice'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8AOAwYVQ5kLh"
   },
   "source": [
    "\n",
    "\n",
    "Compute the representation of the sentence using our BERT client: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "id": "VrVt2lEu5kLh"
   },
   "outputs": [],
   "source": [
    "sent_rep1 = bc.encode([sentence1])\n",
    "sent_rep2 = bc.encode([sentence2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "daRJoRIc5kLi"
   },
   "source": [
    "\n",
    "Now let us check the size of the representation of the given two sentences: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OU-Svy-l5kLi",
    "outputId": "4f76613e-0d88-4912-c269-44ae53a4b115"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 768) (1, 768)\n"
     ]
    }
   ],
   "source": [
    "print(sent_rep1.shape, sent_rep2.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RtOP05ak5kLi"
   },
   "source": [
    "\n",
    "As we notice the shape of both the sentences is (1, 768) which denotes that we mapped the given variable-length sentences to a fixed length representations. \n",
    "\n",
    "Next, we can compute the similarity between the vector representations of the given sentences: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "hpU2zRwk5kLi",
    "outputId": "6f391af7-8d9d-4494-8990-8fb8cbf310a2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.8532591]], dtype=float32)"
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "cosine_similarity(sent_rep1,sent_rep2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rHOpJ8yL5kLj"
   },
   "source": [
    "\n",
    "\n",
    "Thus, we can understand that our given two sentences are 85% similar. "
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "9.10. Computing sentence representation using BERT as service.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}