08_자연어_처리의_전처리.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "08.자연어 처리의 전처리.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyP8+d8tWF11WiM4nN9gr5g/",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/threegenie/studying_pytorch/blob/main/08_%EC%9E%90%EC%97%B0%EC%96%B4_%EC%B2%98%EB%A6%AC%EC%9D%98_%EC%A0%84%EC%B2%98%EB%A6%AC.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XD8f5WncHTfR"
      },
      "source": [
        "import spacy\n",
        "import torch\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "import os, sys"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f0BEetbjHsNH"
      },
      "source": [
        "%%capture\n",
        "!pip install nltk"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9ZT69ekyHvco"
      },
      "source": [
        "import nltk\n",
        "from nltk.tokenize import word_tokenize"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pIfStX7kH7lU",
        "outputId": "f8c5a216-43b7-41ab-9971-f102e5bcf6c2"
      },
      "source": [
        "nltk.download('punkt')"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CpA2oqfkYAIz"
      },
      "source": [
        "> 한국어 예시 문장"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qBaHS9sJIfTh"
      },
      "source": [
        "kor_text = '인터넷뱅킹을 통해 이체한도 감액 신청 가능한 이용 경로 안내해드리겠습니다. 회원님의 이메일 주소를 알려주시면 바로 답신 드릴테니 문자 주시면 감사하겠습니다.'"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4qLo270xX8Pw"
      },
      "source": [
        "> 한국어 띄어쓰기 토큰화"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Qj6IDOD6IyAm",
        "outputId": "ca04058a-84a2-45bc-f0a1-7d54e0784f4b"
      },
      "source": [
        "# 한국어 띄어쓰기 토큰화\n",
        "print(kor_text.split())"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['인터넷뱅킹을', '통해', '이체한도', '감액', '신청', '가능한', '이용', '경로', '안내해드리겠습니다.', '회원님의', '이메일', '주소를', '알려주시면', '바로', '답신', '드릴테니', '문자', '주시면', '감사하겠습니다.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M5OL3IjSX6aZ"
      },
      "source": [
        "> 한국어 형태소 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ykz9uC8NXfhQ"
      },
      "source": [
        "colab에 konlpy + mecab 설치 - \n",
        "https://zetawiki.com/wiki/Colab%EC%97%90_konlpy_%2B_mecab_%EC%84%A4%EC%B9%98"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nZXy1cxLXZnk"
      },
      "source": [
        "%%capture\n",
        "!set -x \\\n",
        "&& pip install konlpy \\\n",
        "&& curl -s https://raw.githubusercontent.com/konlpy/konlpy/master/scripts/mecab.sh | bash -x"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xn5uuTRxXn4K"
      },
      "source": [
        "import konlpy\n",
        "from konlpy.tag import Mecab"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1YKdEUpdXtbC",
        "outputId": "0388771f-c106-412c-ec9f-a2ad9e69e3c5"
      },
      "source": [
        "tokenizer = Mecab()\n",
        "print(tokenizer.morphs(kor_text))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['인터넷', '뱅킹', '을', '통해', '이', '체한', '도', '감액', '신청', '가능', '한', '이용', '경로', '안내', '해', '드리', '겠', '습니다', '.', '회원', '님', '의', '이메일', '주소', '를', '알려', '주', '시', '면', '바로', '답신', '드릴', '테', '니', '문자', '주', '시', '면', '감사', '하', '겠', '습니다', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RRJe7wycYPCm"
      },
      "source": [
        "> 한국어 단어 집합 생성\n",
        "- 네이버 영화 리뷰 분류 데이터 이용 - 총 20만개 영화 리뷰, 긍정 1 부정 0으로 라벨링"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bw2V737UYZGu"
      },
      "source": [
        "import urllib.request\n",
        "from nltk import FreqDist"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 358
        },
        "id": "fiGc8WjqYd3J",
        "outputId": "adc21e05-c90e-4e50-f849-6ca26e22997a"
      },
      "source": [
        "urllib.request.urlretrieve(\"https://raw.githubusercontent.com/e9t/nsmc/master/ratings.txt\", filename=\"ratings.txt\")\n",
        "data = pd.read_table('ratings.txt') # 데이터프레임에 저장\n",
        "data[:10]"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>document</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>8112052</td>\n",
              "      <td>어릴때보고 지금다시봐도 재밌어요ㅋㅋ</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>8132799</td>\n",
              "      <td>디자인을 배우는 학생으로, 외국디자이너와 그들이 일군 전통을 통해 발전해가는 문화산...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>4655635</td>\n",
              "      <td>폴리스스토리 시리즈는 1부터 뉴까지 버릴께 하나도 없음.. 최고.</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9251303</td>\n",
              "      <td>와.. 연기가 진짜 개쩔구나.. 지루할거라고 생각했는데 몰입해서 봤다.. 그래 이런...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>10067386</td>\n",
              "      <td>안개 자욱한 밤하늘에 떠 있는 초승달 같은 영화.</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>2190435</td>\n",
              "      <td>사랑을 해본사람이라면 처음부터 끝까지 웃을수 있는영화</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>9279041</td>\n",
              "      <td>완전 감동입니다 다시봐도 감동</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>7865729</td>\n",
              "      <td>개들의 전쟁2 나오나요? 나오면 1빠로 보고 싶음</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>7477618</td>\n",
              "      <td>굿</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>9250537</td>\n",
              "      <td>바보가 아니라 병 쉰 인듯</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         id                                           document  label\n",
              "0   8112052                                어릴때보고 지금다시봐도 재밌어요ㅋㅋ      1\n",
              "1   8132799  디자인을 배우는 학생으로, 외국디자이너와 그들이 일군 전통을 통해 발전해가는 문화산...      1\n",
              "2   4655635               폴리스스토리 시리즈는 1부터 뉴까지 버릴께 하나도 없음.. 최고.      1\n",
              "3   9251303  와.. 연기가 진짜 개쩔구나.. 지루할거라고 생각했는데 몰입해서 봤다.. 그래 이런...      1\n",
              "4  10067386                        안개 자욱한 밤하늘에 떠 있는 초승달 같은 영화.      1\n",
              "5   2190435                      사랑을 해본사람이라면 처음부터 끝까지 웃을수 있는영화      1\n",
              "6   9279041                                   완전 감동입니다 다시봐도 감동      1\n",
              "7   7865729                        개들의 전쟁2 나오나요? 나오면 1빠로 보고 싶음      1\n",
              "8   7477618                                                  굿      1\n",
              "9   9250537                                     바보가 아니라 병 쉰 인듯      1"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IcdyoB6BYiFa"
      },
      "source": [
        "sample_data = data[:100] # 임의로 100개만 저장"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "90obSR0-YjlI"
      },
      "source": [
        "# 불용어 정의\n",
        "stopwords=['의','가','이','은','들','는','좀','잘','걍','과','도','를','으로','자','에','와','한','하다']"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "02Y2DGn6ZPva"
      },
      "source": [
        "tokenizer = Mecab()"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DyJSFj_DZRl8"
      },
      "source": [
        "tokenized=[]\n",
        "for sentence in sample_data['document']:\n",
        "    temp = tokenizer.morphs(sentence) # 토큰화\n",
        "    temp = [word for word in temp if not word in stopwords] # 불용어 제거\n",
        "    tokenized.append(temp)"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xHN8U5nTZSP-",
        "outputId": "613ed114-1566-4ecc-9855-0ac7244fd361"
      },
      "source": [
        "print(tokenized[:10])"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[['어릴', '때', '보', '고', '지금', '다시', '봐도', '재밌', '어요', 'ㅋㅋ'], ['디자인', '을', '배우', '학생', ',', '외국', '디자이너', '그', '일군', '전통', '을', '통해', '발전', '해', '문화', '산업', '부러웠', '는데', '.', '사실', '우리', '나라', '에서', '그', '어려운', '시절', '끝', '까지', '열정', '을', '지킨', '노라노', '같', '전통', '있', '어', '저', '같', '사람', '꿈', '을', '꾸', '고', '이뤄나갈', '수', '있', '다는', '것', '감사', '합니다', '.'], ['폴리스', '스토리', '시리즈', '1', '부터', '뉴', '까지', '버릴', '께', '하나', '없', '음', '.', '.', '최고', '.'], ['.', '.', '연기', '진짜', '개', '쩔', '구나', '.', '.', '지루', '할거', '라고', '생각', '했', '는데', '몰입', '해서', '봤', '다', '.', '.', '그래', '이런', '게', '진짜', '영화', '지'], ['안개', '자욱', '밤하늘', '떠', '있', '초승달', '같', '영화', '.'], ['사랑', '을', '해', '본', '사람', '라면', '처음', '부터', '끝', '까지', '웃', '을', '수', '있', '영화'], ['완전', '감동', '입니다', '다시', '봐도', '감동'], ['개', '전쟁', '2', '나오', '나요', '?', '나오', '면', '1', '빠', '로', '보', '고', '싶', '음'], ['굿'], ['바보', '아니', '라', '병', '쉰', '인', '듯']]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "owCeUzWHZUIT",
        "outputId": "8e89c32a-5628-4396-ba4b-887fe41647dc"
      },
      "source": [
        "# 단어 집합 형성 - freqdist를 사용하여 빈도수 계산\n",
        "vocab = FreqDist(np.hstack(tokenized))\n",
        "print('단어 집합의 크기 : {}'.format(len(vocab)))"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "단어 집합의 크기 : 698\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6FZjQ2NOZgcL",
        "outputId": "789c0237-3390-422a-db27-864c920250dc"
      },
      "source": [
        "vocab_size = 500\n",
        "# 상위 vocab_size개의 단어만 보존 - 상위 500개로 설정 \n",
        "vocab = vocab.most_common(vocab_size)\n",
        "print('단어 집합의 크기 : {}'.format(len(vocab)))"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "단어 집합의 크기 : 500\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LiVo8EYVZl1y"
      },
      "source": [
        "word_to_index = {word[0] : index + 2 for index, word in enumerate(vocab)} #2부터 501까지 인덱싱\n",
        "word_to_index['pad'] = 1\n",
        "word_to_index['unk'] = 0"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-_T418qYEVYZ"
      },
      "source": [
        "encoded = []\n",
        "for line in tokenized: #입력 데이터에서 1줄씩 문장을 읽음\n",
        "    temp = []\n",
        "    for w in line: #각 줄에서 1개씩 글자를 읽음\n",
        "      try:\n",
        "        temp.append(word_to_index[w]) # 글자를 해당되는 정수로 변환\n",
        "      except KeyError: # 단어 집합에 없는 단어일 경우 unk로 대체된다.\n",
        "        temp.append(word_to_index['unk']) # unk의 인덱스로 변환\n",
        "\n",
        "    encoded.append(temp)"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q3KRDFxXEm6L",
        "outputId": "afb6df18-10e5-493d-8d4b-19fdb1900f96"
      },
      "source": [
        "print(encoded[:10])"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[84, 32, 10, 5, 54, 46, 85, 18, 33, 34], [201, 6, 86, 202, 19, 203, 204, 55, 205, 122, 6, 206, 207, 20, 123, 208, 209, 15, 2, 56, 87, 124, 35, 55, 210, 125, 126, 36, 211, 6, 212, 213, 21, 122, 8, 72, 57, 21, 47, 214, 6, 215, 5, 216, 16, 8, 88, 37, 217, 89, 2], [218, 127, 219, 90, 58, 220, 36, 221, 222, 59, 11, 28, 2, 2, 12, 2], [2, 2, 48, 38, 128, 223, 224, 2, 2, 225, 226, 73, 49, 39, 15, 227, 129, 17, 3, 2, 2, 228, 74, 9, 38, 4, 40], [229, 230, 231, 232, 8, 233, 21, 4, 2], [130, 6, 20, 41, 47, 131, 132, 58, 126, 36, 91, 6, 16, 8, 4], [133, 42, 234, 46, 85, 42], [128, 235, 134, 60, 236, 29, 60, 92, 90, 237, 50, 10, 5, 51, 28], [61], [238, 93, 94, 239, 240, 62, 95]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 332
        },
        "id": "rD5FQClUEqui",
        "outputId": "b5fbe874-5c08-40c7-c510-db092ec37eab"
      },
      "source": [
        "max_len = max(len(l) for l in encoded)\n",
        "print('리뷰의 최대 길이 : %d' % max_len)\n",
        "print('리뷰의 최소 길이 : %d' % min(len(l) for l in encoded))\n",
        "print('리뷰의 평균 길이 : %f' % (sum(map(len, encoded))/len(encoded)))\n",
        "plt.hist([len(s) for s in encoded], bins=50)\n",
        "plt.xlabel('length of sample')\n",
        "plt.ylabel('number of sample')\n",
        "plt.show()"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "리뷰의 최대 길이 : 63\n",
            "리뷰의 최소 길이 : 1\n",
            "리뷰의 평균 길이 : 15.620000\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVD0lEQVR4nO3df5BlZX3n8fdHQPAHCmQ6FIrjYKRQYxBw8MdKDEq0CBJ/7JIoKxEVnVrXCO76Y6FMKbE2K5YuMdFddRSEMoTdrIq66AoTBIkrAWZghAEkGgGFoIxR+WUEBr77xzm9tu30zJnuPvf27fN+VZ3qe557+zzfZ/rOt08/9znfk6pCkjQcDxt3AJKk0TLxS9LAmPglaWBM/JI0MCZ+SRqYnccdQBcrVqyoVatWjTsMSZooGzZs+FFVTc1un4jEv2rVKtavXz/uMCRpoiS5ZWvtTvVI0sCY+CVpYEz8kjQwJn5JGhgTvyQNjIlfkgbGxC9JA2Pil6SBMfFL0sBMxJW7S92qk7+01fabT3vJiCORpO3zjF+SBsbEL0kDY+KXpIEx8UvSwJj4JWlgTPySNDAmfkkaGBO/JA2MiV+SBsbEL0kDY+KXpIHpLfEnOTPJHUk2zWj7QJJvJbkmyXlJ9uirf0nS1vV5xn8WcOSstnXA06vqQOAfgFN67F+StBW9Jf6quhT48ay2C6tqS7v798C+ffUvSdq6cc7xvx74P2PsX5IGaSyJP8m7gC3AOdt4zZok65Os37x58+iCk6RlbuSJP8lrgaOBV1dVzfW6qlpbVauravXU1NTI4pOk5W6kd+BKciTwTuB3qupno+xbktTocznnucBlwAFJbk1yAvARYHdgXZKNST7WV/+SpK3r7Yy/qo7dSvMZffUnSerGK3claWBM/JI0MCZ+SRoYE78kDYyJX5IGxsQvSQNj4pekgTHxS9LAmPglaWBM/JI0MCZ+SRoYE78kDYyJX5IGxsQvSQNj4pekgTHxS9LAmPglaWBM/JI0MCZ+SRoYE78kDYyJX5IGxsQvSQNj4pekgekt8Sc5M8kdSTbNaNsrybok326/7tlX/5KkrevzjP8s4MhZbScDF1XV/sBF7b4kaYR6S/xVdSnw41nNLwPObh+fDby8r/4lSVs36jn+vavq9vbxD4C9R9y/JA3e2D7craoCaq7nk6xJsj7J+s2bN48wMkla3kad+H+YZB+A9usdc72wqtZW1eqqWj01NTWyACVpuRt14v8icHz7+HjgCyPuX5IGr8/lnOcClwEHJLk1yQnAacCLknwb+N12X5I0Qjv3deCqOnaOp47oq09J0vZ55a4kDUynxJ/ksCSvax9PJdmv37AkSX3ZbuJP8h7gPwGntE27AH/VZ1CSpP50OeN/BfBS4F6AqvonYPc+g5Ik9adL4r9/5sVWSR7Vb0iSpD51Sfx/k+TjwB5J3gj8LfCJfsOSJPVlu8s5q+qDSV4E3AUcALy7qtb1HpkkqRed1vG3id5kL0nLwJyJP8ndbL2IWmhqrD2mt6gkSb2ZM/FXlSt3JGkZ6jTVk+QQ4DCavwC+XlVX9xqVRmrVyV/aavvNp71kxJFIGoUuF3C9m+ZuWb8GrADOSvInfQcmSepHlzP+VwPPqKqfAyQ5DdgI/Oc+A5Mk9aPLOv5/Anabsb8rcFs/4UiS+tbljP9O4Lok62jm+F8EXJHkLwGq6sQe45MkLbIuif+8dpt2ST+hSJJGocuVu2ePIhBJ0mh0WdVzdJKrk/w4yV1J7k5y1yiCkyQtvi5TPR8C/jVwbVulU5I0wbqs6vk+sMmkL0nLQ5cz/ncCX07yNeC+6caqOr23qCRJvemS+P8MuIdmLf/D+w1HktS3Lon/cVX19N4jkSSNRJc5/i8nefFidprkPyS5LsmmJOcm2W373yVJWgxdEv+bgK8k+ZfFWM6Z5PHAicDq9i+JnYBXzfd4kqQd0+UCrj7q8u8MPCLJA8AjaeoBSZJGoGs9/j2B/ZlRrK2qLp1Ph1V1W5IPAt8D/gW4sKou3Eqfa4A1ACtXrpxPV2O31OrczxWPpGHpcuXuG4BLgQuAP22/njrfDttfIi8D9gMeBzwqyXGzX1dVa6tqdVWtnpqamm93kqRZuszxnwQcCtxSVS8ADgZ+uoA+fxe4qao2V9UDwOeAf7WA40mSdkCXxP/zGTdh2bWqvgUcsIA+vwc8J8kjkwQ4ArhhAceTJO2ALnP8tybZA/g8sC7JT4Bb5tthVV2e5DPAVcAW4Gpg7XyPJ0naMV1W9byifXhqkouBxwJfWUinVfUe4D0LOYYkaX66fLj7G0l2nd4FVtEswZQkTaAuc/yfBR5M8mSaKZknAH/da1SSpN50SfwPVdUW4BXAh6vqHcA+/YYlSepLl8T/QJJjgeOB89u2XfoLSZLUpy6J/3XAc4E/q6qbkuwHfLrfsCRJfemyqud6mqJq0/s3Ae/vMyhJUn+6nPFLkpYRE78kDcyciT/Jp9uvJ40uHElS37Z1xv/MJI8DXp9kzyR7zdxGFaAkaXFt68PdjwEXAU8CNtBctTut2nbNw1Kr0y9pWOY846+qv6yqpwJnVtWTqmq/GZtJX5ImVJflnG9K8gzgt9umS6vqmn7DkiT1pUuRthOBc4Bfb7dzkryl78AkSf3oUo//DcCzq+pegCTvBy4DPtxnYJKkfnRZxx/gwRn7D/LLH/RKkiZIlzP+TwGXJzmv3X85cEZ/IUmS+tTlw93Tk1wCHNY2va6qru41KklSb7qc8VNVV9HcI1eSNOGs1SNJA2Pil6SB2WbiT7JTkotHFYwkqX/bTPxV9SDwUJLHjigeSVLPuny4ew9wbZJ1wL3TjVV14tzfsm1J9gA+CTydpuDb66vqsvkeT5LUXZfE/7l2W0x/AXylqo5J8nDgkYt8fEnSHLqs4z87ySOAlVV140I7bKeNng+8tj3+/cD9Cz2uJKmb7Sb+JL8PfBB4OLBfkoOA91bVS+fZ537AZuBTbdXPDcBJ07WAZvS7BlgDsHLlynl2tXjmqqE/zr6t3y9pPros5zwVeBbwU4Cq2sjCbsKyM3AI8NGqOpjmc4OTZ7+oqtZW1eqqWj01NbWA7iRJM3VJ/A9U1Z2z2h5aQJ+3ArdW1eXt/mdofhFIkkagS+K/Lsm/BXZKsn+SDwPfmG+HVfUD4PtJDmibjgCun+/xJEk7pkvifwvwm8B9wLnAXcBbF9jvW2hu6HINcBDwXxZ4PElSR11W9fwMeFd7A5aqqrsX2mn7OcHqhR5HkrTjutx68dAk1wLX0FzI9c0kz+w/NElSH7pcwHUG8O+r6u8AkhxGc3OWA/sMTJLUjy5z/A9OJ32Aqvo6sKW/kCRJfZrzjD/J9BLLryX5OM0HuwW8Erik/9AkSX3Y1lTPf521/54Zj6uHWCRJIzBn4q+qF4wyEEnSaHSp1bMH8Bpg1czXL6QssyRpfLqs6vky8PfAtSysVIMkaQnokvh3q6r/2HskkqSR6LKc89NJ3phknyR7TW+9RyZJ6kWXM/77gQ8A7+IXq3mKhZVmHhlr2UvSL+uS+N8GPLmqftR3MJKk/nWZ6vkO8LO+A5EkjUaXM/57gY1JLqYpzQy4nFOSJlWXxP/5dpMkLQNd6vGfPYpAJEmj0eXK3ZvYSm2eqpqIVT2SpF/WZapn5p2ydgP+AHAdvyRNqO2u6qmqf56x3VZVHwJcBC9JE6rLVM8hM3YfRvMXQJe/FCRJS1CXBD6zLv8W4GbgD3uJRpLUuy6reqzLL0nLSJepnl2Bf8Ov1uN/b39hSZL60mWq5wvAncAGZly5u1BJdgLWA7dV1dGLdVxJ0rZ1Sfz7VtWRPfR9EnAD8Jgeji1JmkOXIm3fSPJbi9lpkn1ploR+cjGPK0navi5n/IcBr22v4L0PCFBVdeAC+v0Q8E5g97lekGQNsAZg5cqVC+hK8+W9DKTlqUvi/73F7DDJ0cAdVbUhyeFzva6q1gJrAVavXv0rJSMkSfPTZTnnLYvc5/OAlyY5iqYExGOS/FVVHbfI/UiStqLLHP+iqqpTqmrfqloFvAr4qklfkkZn5IlfkjReY625U1WXAJeMMwZJGhrP+CVpYEz8kjQwJn5JGhgTvyQNjIlfkgbGxC9JA2Pil6SBMfFL0sCY+CVpYEz8kjQwYy3ZoIUZV738xex3sY7lvQOk7jzjl6SBMfFL0sCY+CVpYEz8kjQwJn5JGhgTvyQNjIlfkgbGxC9JA2Pil6SBMfFL0sCY+CVpYEz8kjQwI0/8SZ6Q5OIk1ye5LslJo45BkoZsHNU5twBvq6qrkuwObEiyrqquH0MskjQ4Iz/jr6rbq+qq9vHdwA3A40cdhyQN1Vjr8SdZBRwMXL6V59YAawBWrly56H0vxfrtc8W0HCzW2Ib4bzRJ91fQZBjbh7tJHg18FnhrVd01+/mqWltVq6tq9dTU1OgDlKRlaiyJP8kuNEn/nKr63DhikKShGseqngBnADdU1emj7l+Shm4cZ/zPA/4IeGGSje121BjikKRBGvmHu1X1dSCj7leS1PDKXUkaGBO/JA2MiV+SBsbEL0kDY+KXpIEx8UvSwJj4JWlgTPySNDAmfkkaGBO/JA3MWOvxL0XLod77uMawFP/t+q41by370drR99iO/hx29Oe5WD//bY2rj/eSZ/ySNDAmfkkaGBO/JA2MiV+SBsbEL0kDY+KXpIEx8UvSwJj4JWlgTPySNDAmfkkaGBO/JA2MiV+SBmYsiT/JkUluTPKdJCePIwZJGqqRJ/4kOwH/Dfg94GnAsUmeNuo4JGmoxnHG/yzgO1X13aq6H/gfwMvGEIckDVKqarQdJscAR1bVG9r9PwKeXVV/POt1a4A17e4BwI0dDr8C+NEihjsOjmHpWA7jcAxLw7jG8MSqmprduGRvxFJVa4G1O/I9SdZX1eqeQhoJx7B0LIdxOIalYamNYRxTPbcBT5ixv2/bJkkagXEk/iuB/ZPsl+ThwKuAL44hDkkapJFP9VTVliR/DFwA7AScWVXXLdLhd2hqaIlyDEvHchiHY1galtQYRv7hriRpvLxyV5IGxsQvSQOzbBL/JJaBSHJmkjuSbJrRtleSdUm+3X7dc5wxbk+SJyS5OMn1Sa5LclLbPjHjSLJbkiuSfLMdw5+27fslubx9T/3PdjHCkpZkpyRXJzm/3Z+oMSS5Ocm1STYmWd+2Tcx7CSDJHkk+k+RbSW5I8tylNoZlkfgnuAzEWcCRs9pOBi6qqv2Bi9r9pWwL8LaqehrwHODN7b/9JI3jPuCFVfUM4CDgyCTPAd4P/HlVPRn4CXDCGGPs6iTghhn7kziGF1TVQTPWvU/SewngL4CvVNVTgGfQ/DyW1hiqauI34LnABTP2TwFOGXdcHWNfBWyasX8jsE/7eB/gxnHHuIPj+QLwokkdB/BI4Crg2TRXWu7ctv/Se2wpbjTXxFwEvBA4H8gEjuFmYMWstol5LwGPBW6iXTizVMewLM74gccD35+xf2vbNon2rqrb28c/APYeZzA7Iskq4GDgciZsHO0UyUbgDmAd8I/AT6tqS/uSSXhPfQh4J/BQu/9rTN4YCrgwyYa2bAtM1ntpP2Az8Kl2yu2TSR7FEhvDckn8y1I1pwcTsd42yaOBzwJvraq7Zj43CeOoqger6iCas+ZnAU8Zc0g7JMnRwB1VtWHcsSzQYVV1CM207ZuTPH/mkxPwXtoZOAT4aFUdDNzLrGmdpTCG5ZL4l1MZiB8m2Qeg/XrHmOPZriS70CT9c6rqc23zxI0DoKp+ClxMMy2yR5LpixyX+nvqecBLk9xMU/H2hTRzzZM0BqrqtvbrHcB5NL+EJ+m9dCtwa1Vd3u5/huYXwZIaw3JJ/MupDMQXgePbx8fTzJkvWUkCnAHcUFWnz3hqYsaRZCrJHu3jR9B8RnEDzS+AY9qXLekxVNUpVbVvVa2ief9/tapezQSNIcmjkuw+/Rh4MbCJCXovVdUPgO8nOaBtOgK4nqU2hnF/GLKIH6ocBfwDzdzsu8YdT8eYzwVuBx6gOVM4gWZe9iLg28DfAnuNO87tjOEwmj9brwE2tttRkzQO4EDg6nYMm4B3t+1PAq4AvgP8L2DXccfacTyHA+dP2hjaWL/ZbtdN/z+epPdSG+9BwPr2/fR5YM+lNgZLNkjSwCyXqR5JUkcmfkkaGBO/JA2MiV+SBsbEL0kDY+LXkpXknh6OeVCSo2bsn5rk7Qs43h+0FRgvXpwI5x3HzUlWjDMGTQ4Tv4bmIJrrDBbLCcAbq+oFi3hMqVcmfk2EJO9IcmWSa2bUy1/Vnm1/oq2jf2F75S1JDm1fuzHJB5Jsaq/qfi/wyrb9le3hn5bkkiTfTXLiHP0f29aJ35Tk/W3bu2kuYDsjyQdmvX6fJJe2/WxK8ttt+0eTrJ9Z979tvznJ+6br0Cc5JMkFSf4xyb9rX3N4e8wvpbn3xMeS/Mr/4STHpbm/wMYkH2/Llku/MO6r3Nzc5tqAe9qvL6a5WXVoTlbOB55PU9J6C3BQ+7q/AY5rH28Cnts+Po229DXwWuAjM/o4FfgGsCuwAvhnYJdZcTwO+B4wRVOE66vAy9vnLgFWbyX2t/GLK093AnZvH+81o+0S4MB2/2bgTe3jP6e56nP3ts8ftu2HAz+nucJ1J5oqosfM+P4VwFOB/z09BuC/A68Z98/SbWltnvFrEry43a6mqZX/FGD/9rmbqmpj+3gDsKqtu7N7VV3Wtv/1do7/paq6r6p+RFM8a3bJ3EOBS6pqczUljs+h+cWzLVcCr0tyKvBbVXV32/6HSa5qx/KbNDcOmjZdX+pa4PKquruqNgP3TdcSAq6oqu9W1YM0JT8Om9XvEcAzgSvbMtNH0PyikP6/nbf/EmnsAryvqj7+S41N/f/7ZjQ9CDxiHseffYwF/7+oqkvbksIvAc5Kcjrwd8DbgUOr6idJzgJ220ocD82K6aEZMc2usTJ7P8DZVXXKQseg5cszfk2CC4DXtzX/SfL4JL8+14urKa18d5Jnt02vmvH03TRTKDviCuB3kqxo58uPBb62rW9I8kSaKZpPAJ+kKc37GJr67Hcm2Zum5vyOelZbhfZhwCuBr896/iLgmOl/nzT3en3iPPrRMuYZv5a8qrowyVOBy5oq0NwDHEdzdj6XE4BPJHmIJknf2bZfDJzcToO8r2P/tyc5uf3e0EwNba+s7uHAO5I80Mb7mqq6KcnVwLdo7hj3f7v0P8uVwEeAJ7fxnDcr1uuT/AnNXaweRlP59c3ALfPoS8uU1Tm1LCV5dFXd0z4+meZ+pyeNOawFSXI48PaqOnrcsWiyecav5eolSU6heY/fQrOaRxKe8UvS4PjhriQNjIlfkgbGxC9JA2Pil6SBMfFL0sD8P6krRfAZkjqsAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tIEhiDCpEuS_"
      },
      "source": [
        "# 길이가 다른 문장들을 전부 동일한 길이로 변환 - 패딩\n",
        "# 여기서는 가장 길이가 긴 리뷰의 길이가 63이므로 63으로 통일\n",
        "for line in encoded:\n",
        "    if len(line) < max_len: # 현재 샘플이 정해준 길이보다 짧으면\n",
        "        line += [word_to_index['pad']] * (max_len - len(line)) # 나머지는 전부 'pad' 토큰으로 채운다."
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3_AMikiTE2A4",
        "outputId": "f9751f73-b8ee-46de-d944-a39e56dfbc73"
      },
      "source": [
        "print('리뷰의 최대 길이 : %d' % max(len(l) for l in encoded))\n",
        "print('리뷰의 최소 길이 : %d' % min(len(l) for l in encoded))\n",
        "print('리뷰의 평균 길이 : %f' % (sum(map(len, encoded))/len(encoded)))"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "리뷰의 최대 길이 : 63\n",
            "리뷰의 최소 길이 : 63\n",
            "리뷰의 평균 길이 : 63.000000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VHk8cCoXE770",
        "outputId": "a68a4fed-c3e0-46d1-9eca-fda3ca59b941"
      },
      "source": [
        "print(encoded[:3])"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[84, 32, 10, 5, 54, 46, 85, 18, 33, 34, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [201, 6, 86, 202, 19, 203, 204, 55, 205, 122, 6, 206, 207, 20, 123, 208, 209, 15, 2, 56, 87, 124, 35, 55, 210, 125, 126, 36, 211, 6, 212, 213, 21, 122, 8, 72, 57, 21, 47, 214, 6, 215, 5, 216, 16, 8, 88, 37, 217, 89, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [218, 127, 219, 90, 58, 220, 36, 221, 222, 59, 11, 28, 2, 2, 12, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]\n"
          ]
        }
      ]
    }
  ]
}