{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "time: 174 µs (started: 2022-09-26 14:40:06 -07:00)\n" ] } ], "source": [ "from typing import List, Optional, Dict, Union\n", "import os\n", "import random\n", "\n", "import pandas as pd\n", "import numpy as np\n", "from pydantic import Field\n", "from datatile.summary.df import DataFrameSummary\n", "from sklearn.metrics import average_precision_score, f1_score, precision_score, recall_score, classification_report, roc_auc_score\n", "from catboost import CatBoostClassifier\n", "from sklearn.utils import shuffle\n", "from sklearn.model_selection import train_test_split\n", "from transformers import DistilBertTokenizer\n", "\n", "import tldextract\n", "\n", "%load_ext autotime" ] }, { "cell_type": "markdown", "metadata": { "ExecuteTime": { "end_time": "2020-09-15T14:54:56.893866Z", "start_time": "2020-09-15T14:54:56.883866Z" } }, "source": [ "# Initial Research\n", "## Docs and Existed Solutions\n", "1. [Domain generation algorithm / Detection](https://en.wikipedia.org/wiki/Domain_generation_algorithm#Detection)\n", "2. [\"Character Level based Detection of DGA Domain Names\"](http://faculty.washington.edu/mdecock/papers/byu2018a.pdf) - a perfect review to step into this problem.\n", "3. GAN perspective:\n", " - [\"DeepDGA: Adversarially-Tuned Domain Generation and Detection\"](https://arxiv.org/pdf/1610.01969.pdf) It has a [code](https://github.com/roreagan/DeepDGA).\n", " - [\"MaskDGA: A Black-box Evasion Technique Against DGA Classifiers and Adversarial Defenses\"](https://arxiv.org/pdf/1902.08909.pdf)\n", "4. [\"Inline Detection of Domain Generation Algorithms with Context-Sensitive Word Embeddings\"](https://arxiv.org/pdf/1811.08705.pdf) - non-typical approach with splitting DN to words and applying a pretrained ELMO model.\n", "5. https://github.com/jayjacobs/dga On R :(\n", "6. https://github.com/andrewaeva/DGA 5 years old :( Has a big DGA data (18 MB)\n", "7. https://github.com/nickwallen/botnet-dga-classifier On R. 6 years old :(\n", "8. https://github.com/Daniellee1990/Language-Model-based-Detection-Approach-of-Algorithmically-Generated-Malicious-Domain-Names old methods :(\n" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "# Initial Approach" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I'm selecting one of the existing solutions for the start, based on these criteria: \n", "- it should be \"good enough\" and \"near state-of-art\" (up-to-date)\n", "- it should be fast to start, not complex for implementation\n", "\n", "A good candidate is the 4th solution based on a pretrained model.\n", "\n", "## Pro & Cons\n", "### Pro\n", "- It is simple enough to create a baseline solution and get the first result.\n", "- There are many points of improvement including many pretrained models.\n", "- Fast training, which allows the precise hyperparameter tuning.\n", "\n", "### Cons\n", "- No source code to start right away (but there is no good code base for other candidates either)\n", "\n", "\n", "## Open Questions with current solutions and ways to experiments\n", "1. `word2vec` vs. `ELMO` and `BERT`. Is there any \"long text' correlation between \"words\" inside DNS domain names (DN)? If no such correlation, then we don't need the long text context hence word2vec would work just fine.\n", "1. Splitting DN to words. Does it help? Verify this hypothesis. \n", "1. Word Embedding. The BPE encoding can be better. Verify with the FB FastText model (and BERT model).\n", "1. Best Pretrained Model. The BERT or custom ELMO (flair pretrained) can be better. Verify with the pretrained ELMO, BERT and, maybe, train a custom ELMO model.\n", "1. More complex trained classification layers. The paper didn't mention any hyperparameter tuning. If the proposed model trained fast, it makes sense to play with hyperparameters: several layers-units; normalization; dropout; batch size; learning rate progress; etc.\n", "1. Classification objective. Do we need a multilabel classification while business requires only the binary classification? Does multilabel classification improve binary results?\n", "1. Symmetrical Loss function. Are FPs and TPs important at the same rate? \n", "1. TLD. Is TLD helpful? " ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "# Data Preparations\n", "* download correct DNs:\n", " 1. http://s3.amazonaws.com/alexa-static/top-1m.csv.zip\n", "* download malignant DNs (clone these projects):\n", " 1. https://github.com/baderj/domain_generation_algorithms\n", " 1. https://github.com/liorsidi/Adversarial-DGA-Datasets\n", " \n", "* download pretrained models (see code below): \n", " 1. word embedding, defined by the pretrained model. GloVe from [spaCy](https://spacy.io/models/en) : md model: 685k keys, 20k unique vectors (300 dimensions) or lg model: 685k keys, 685k unique vectors (300 dimensions); original GloVe. We will create an input data of fixed size (in number of words).\n", " 2. BPE embedding: [flair](https://github.com/flairNLP/flair), [FastText](https://fasttext.cc/docs/en/python-module.html#text-classification-model); code examples: https://github.com/facebookresearch/fastText/tree/master/python/doc/examples, https://fasttext.cc/docs/en/english-vectors.html; models: https://fasttext.cc/docs/en/english-vectors.html. \n", "- Input Data: we convert the combined benign and malignant DNs into embedding from pretrained models\n", "- Output Data: DGA code (multiclass probability) or Good/Bad (binary)." ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "## Input raw data\n", "Download http://s3.amazonaws.com/alexa-static/top-1m.csv.zip - 500K benign DNs.\n", "\n", "Clone https://github.com/baderj/domain_generation_algorithms project with malignant DNs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### benign" ] }, { "cell_type": "code", "execution_count": 90, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T20:39:09.173076Z", "start_time": "2020-09-17T20:39:08.050000Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Load 525,215 from file \"data/Alex_top-0.5M.csv\"\n", "525215 ['525211,webtrics.ch', '525212,weshfee.com', '525213,yosofunny.com', '525214,yougogirlz.com', '525215,zehabesha.com']\n", "525215 ['webtrics.ch', 'weshfee.com', 'yosofunny.com', 'yougogirlz.com', 'zehabesha.com']\n", "525215 ['webtrics', 'weshfee', 'yosofunny', 'yougogirlz', 'zehabesha']\n", "525215 [{'dn': 'webtrics', 'label': 'good'}, {'dn': 'weshfee', 'label': 'good'}, {'dn': 'yosofunny', 'label': 'good'}, {'dn': 'yougogirlz', 'label': 'good'}, {'dn': 'zehabesha', 'label': 'good'}]\n" ] } ], "source": [ "def read_data(file_name):\n", " ret = open(file_name, encoding='utf-8').read().splitlines()\n", " print(f'Load {len(ret):,} from file \"{file_name}\"')\n", " return ret\n", " \n", "def remove_TLD(dns):\n", " return [el.split('.')[0] for el in dns]\n", "\n", "file_name = 'data/Alex_top-0.5M.csv'\n", "good_dn = read_data(file_name)\n", "\n", "print(len(good_dn), good_dn[-5:])\n", "good_dn = [el.split(',')[1] for el in good_dn]\n", "print(len(good_dn), good_dn[-5:])\n", "\n", "good_dn = remove_TLD(good_dn)\n", "print(len(good_dn), good_dn[-5:])\n", "\n", "good_dn = [{'dn': el, 'label': 'good'} for el in good_dn]\n", "print(len(good_dn), good_dn[-5:])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### malignant" ] }, { "cell_type": "code", "execution_count": 92, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T20:39:41.941945Z", "start_time": "2020-09-17T20:39:41.921932Z" }, "hidden": true }, "outputs": [ { "data": { "text/plain": [ "31" ] }, "execution_count": 92, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# clone https://github.com/baderj/domain_generation_algorithms project then:\n", "\n", "dr = 'domain_generation_algorithms/*/example_domains.txt'\n", "\n", "import glob\n", "\n", "# root_dir needs a trailing slash (i.e. /root/dir/)\n", "files = glob.iglob(dr, recursive=True)\n", "len(list(files))" ] }, { "cell_type": "code", "execution_count": 93, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T20:39:44.029688Z", "start_time": "2020-09-17T20:39:44.018690Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 []\n" ] } ], "source": [ "# files = ['data/banjori.txt', 'data/bazarbackdoor.txt', 'data/chinad.txt', 'data/corebot.txt']\n", "bad_dn = []\n", "for f in files:\n", " label = f.split('/')[1].split('.')[0]\n", " bad_dn += [{'dn': el, 'label': label} for el in remove_TLD(read_data(f))]\n", " \n", " \n", "print(len(bad_dn), bad_dn[-5:])\n" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved in domains.csv\n", "Saved in domains.csv\n" ] } ], "source": [ "def gen_domain(max):\n", " for domain in range(max):\n", " yield str(domain)\n", " \n", "import pandas as pd\n", "\n", "def write_file(mx):\n", " df = pd.DataFrame(data={\"domain\": gen_domain(mx)})\n", " df = df.drop_duplicates()\n", " df.to_csv(\"domains.csv\", index=False)\n", " print(\"Saved in domains.csv\")\n", "\n", "mx = 100\n", "write_file(mx) \n", "print(\"Saved in domains.csv\")" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pattern='domain_generation_algorithms/*/example_domains.txt'\n", "files=<generator object _iglob at 0x7f163b1e4ac0>\n", "Loaded domain_generation_algorithms/unnamed_downloader/example_domains.txt, 120. Example: ddknt.github.io\n", " Deduplicated: 60\n", " Concatenated 60 -> 60\n", "Loaded domain_generation_algorithms/pitou/example_domains.txt, 306. Example: --------------+\n", " Deduplicated: 283\n", " Concatenated 283 -> 343\n", "Loaded domain_generation_algorithms/chinad/example_domains.txt, 256. Example: 8f6bacmw30xxv6sc.cn\n", " Deduplicated: 256\n", " Concatenated 256 -> 599\n", "Loaded domain_generation_algorithms/qadars/example_domains.txt, 200. Example: jk9enwhansl2.org\n", " Deduplicated: 200\n", " Concatenated 200 -> 799\n", "Loaded domain_generation_algorithms/mydoom/example_domains.txt, 99. Example: qehspqnmrn.info\n", " Deduplicated: 99\n", " Concatenated 99 -> 898\n", "Loaded domain_generation_algorithms/monerodownloader/example_domains.txt, 2,500. Example: 31b4bd31fg1x2.org\n", " Deduplicated: 2,500\n", " Concatenated 2,500 -> 3,398\n", "Loaded domain_generation_algorithms/sisron/example_domains.txt, 40. Example: mdiwnjiwmtya.com\n", " Deduplicated: 40\n", " Concatenated 40 -> 3,438\n", "Loaded domain_generation_algorithms/necurs/example_domains.txt, 2,048. Example: nccojqvabqvkiwhj.mx\n", " Deduplicated: 2,048\n", " Concatenated 2,048 -> 5,486\n", "Loaded domain_generation_algorithms/ramnit/example_domains.txt, 100. Example: knpqxlxcwtlvgrdyhd.com\n", " Deduplicated: 100\n", " Concatenated 100 -> 5,586\n", "Loaded domain_generation_algorithms/nymaim/example_domains.txt, 128. Example: oftbpec.com\n", " Deduplicated: 128\n", " Concatenated 128 -> 5,714\n", "Loaded domain_generation_algorithms/dircrypt/example_domains.txt, 30. Example: rauggyguyp.com\n", " Deduplicated: 30\n", " Concatenated 30 -> 5,744\n", "Loaded domain_generation_algorithms/tinba/example_domains.txt, 401. Example: blackfreeqazyio.cc\n", " Deduplicated: 401\n", " Concatenated 401 -> 6,145\n", "Loaded domain_generation_algorithms/shiotob/example_domains.txt, 2,001. Example: wtipubctwiekhir.net\n", " Deduplicated: 2,001\n", " Concatenated 2,001 -> 8,146\n", "Loaded domain_generation_algorithms/bazarbackdoor/example_domains.txt, 2,160. Example: acegikahggim.bazar\n", " Deduplicated: 2,160\n", " Concatenated 2,160 -> 10,306\n", "Loaded domain_generation_algorithms/nymaim2/example_domains.txt, 704. Example: surfaces-drawing.com\n", " Deduplicated: 704\n", " Concatenated 704 -> 11,010\n", "Loaded domain_generation_algorithms/qakbot/example_domains.txt, 5,000. Example: bqkrtxgkmriwsiwcngtivpx.info\n", " Deduplicated: 5,000\n", " Concatenated 5,000 -> 16,010\n", "Loaded domain_generation_algorithms/newgoz/example_domains.txt, 1,000. Example: xzz3ug32bale1uo60y7xj6rge.com\n", " Deduplicated: 1,000\n", " Concatenated 1,000 -> 17,010\n", "Loaded domain_generation_algorithms/corebot/example_domains.txt, 40. Example: lkhylm0mhyfuhg.ddns.net\n", " Deduplicated: 40\n", " Concatenated 40 -> 17,050\n", "Loaded domain_generation_algorithms/locky/example_domains.txt, 8. Example: gegjiimqmlgtdmk.tf\n", " Deduplicated: 8\n", " Concatenated 8 -> 17,058\n", "Loaded domain_generation_algorithms/padcrypt/example_domains.txt, 24. Example: elkfcfnacacmofdf.com\n", " Deduplicated: 24\n", " Concatenated 24 -> 17,082\n", "Loaded domain_generation_algorithms/pizd/example_domains.txt, 85. Example: difficultnearly.net\n", " Deduplicated: 85\n", " Concatenated 85 -> 17,167\n", "Loaded domain_generation_algorithms/simda/example_domains.txt, 1,000. Example: gatyfus.com\n", " Deduplicated: 1,000\n", " Concatenated 1,000 -> 18,167\n", "Loaded domain_generation_algorithms/unnamed_javascript_dga/example_domains.txt, 30. Example: rxxeqcoy.cc\n", " Deduplicated: 30\n", " Concatenated 30 -> 18,197\n", "Loaded domain_generation_algorithms/dnschanger/example_domains.txt, 5. Example: aktklyvbiu.com\n", " Deduplicated: 5\n", " Concatenated 5 -> 18,202\n", "Loaded domain_generation_algorithms/proslikefan/example_domains.txt, 100. Example: flarvcpk.eu\n", " Deduplicated: 100\n", " Concatenated 100 -> 18,302\n", "Loaded domain_generation_algorithms/pushdo/example_domains.txt, 1,350. Example: weafokuggeir.kz\n", " Deduplicated: 1,350\n", " Concatenated 1,350 -> 19,652\n", "Loaded domain_generation_algorithms/gozi/example_domains.txt, 12. Example: quodpresidentemaxsagit.com\n", " Deduplicated: 12\n", " Concatenated 12 -> 19,664\n", "Loaded domain_generation_algorithms/symmi/example_domains.txt, 64. Example: ogovugtuipawi.ddns.net\n", " Deduplicated: 64\n", " Concatenated 64 -> 19,728\n", "Loaded domain_generation_algorithms/banjori/example_domains.txt, 1,000. Example: earnestnessbiophysicalohax.com\n", " Deduplicated: 1,000\n", " Concatenated 1,000 -> 20,728\n", "Loaded domain_generation_algorithms/reconyc/example_domains.txt, 100. Example: E5zHail0Mw.com\n", " Deduplicated: 100\n", " Concatenated 100 -> 20,828\n", "Loaded domain_generation_algorithms/tempedreve/example_domains.txt, 1,746. Example: dlbebsga.net\n", " Deduplicated: 1,746\n", " Concatenated 1,746 -> 22,574\n", "Result 22,574\n", "Depuplicated by domains 22,574\n", "Saved 22,574 into domain_generation_algorithms/malignant_domains.external.all.0M.csv\n" ] } ], "source": [ "import glob\n", "import pandas as pd\n", "\n", "\n", "def input_malignant(output_file, data_dir, nrows=None):\n", " domain_field = 'domain'\n", " pattern = f\"{data_dir}/*/example_domains.txt\"\n", " files = glob.iglob(pattern, recursive=True)\n", " print(f\"{pattern=}\")\n", " print(f\"{files=}\")\n", " out_df = pd.DataFrame()\n", " for file in files:\n", " file_name = file\n", " df = pd.read_csv(file_name, header=None, nrows=nrows, usecols=[0]).rename(columns={0: domain_field}) \n", " print(f\"Loaded {file}, {df.shape[0]:,}. Example: {df.loc[0, domain_field]}\")\n", " df = df.drop_duplicates()\n", " print(f\" Deduplicated: {df.shape[0]:,}\") \n", " \n", " # df['subdomain'] = df[domain_field].apply(extract_subdomain)\n", " # df = df.dropna(subset=['subdomain'])\n", " # df = df.drop_duplicates(subset=['subdomain'])\n", " # print(f\" Deduplicated by subdomains: {df.shape[0]:,}\") \n", " \n", " out_df = pd.concat([out_df, df])\n", " print(f\" Concatenated {df.shape[0]:,} -> {out_df.shape[0]:,}\")\n", " \n", " print(f\"Result {out_df.shape[0]:,}\") \n", " out_df = out_df.drop_duplicates(subset=[domain_field])\n", " print(f\"Depuplicated by domains {out_df.shape[0]:,}\") \n", " output_file = f\"{data_dir}/{output_file[:-4]}.{out_df.shape[0]//1_000_000}M.csv\"\n", " out_df.to_csv(output_file, index=False)\n", " print(f\"Saved {out_df.shape[0]:,} into {output_file}\") \n", " return out_df\n", "\n", "\n", "data_dir = \"domain_generation_algorithms\" # /external\n", "output_file = \"malignant_domains.external.all.csv\"\n", "nrows = None\n", "df = input_malignant(output_file, data_dir=data_dir, nrows=nrows)" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['RandomAttack',\n", " 'SearchAttack',\n", " 'MaskDGA',\n", " 'AppendAttack',\n", " 'unigrams_distribution',\n", " 'CharBot',\n", " 'DeepDGA']" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from glob import glob\n", "[f.replace(\"Adversarial-DGA-Datasets/\", \"\")[:-1] for f in glob(\"Adversarial-DGA-Datasets/*/\", recursive = True)]" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "## splitting DN to words (Optional)\n", "Calculate the max word of the test data, we can use these max for defining the fixed size of the input windows, if we use the word embeddings" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "ExecuteTime": { "end_time": "2020-09-16T00:20:36.344556Z", "start_time": "2020-09-16T00:19:56.823390Z" }, "hidden": true }, "outputs": [ { "data": { "text/plain": [ "(19, 33)" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import wordninja\n", "\n", "bad_len_max, good_len_max = max([len(wordninja.split(el['dn'])) for el in bad_dn]), max([len(wordninja.split(el['dn'])) for el in good_dn])\n", "bad_len_max, good_len_max # (19, 33)\n", "\n", "# we can use these max for defining the fixed size of the input windows, if we use the word embeddings." ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "## SpaCy tokenizer, Embeddings" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "### GloVe" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "ExecuteTime": { "end_time": "2020-09-15T23:32:23.441936Z", "start_time": "2020-09-15T23:32:21.877419Z" }, "hidden": true }, "outputs": [], "source": [ "import spacy\n", "from spacy.tokenizer import Tokenizer\n", "\n", "\n", "class Model_vectors():\n", " def __init__(self, model_size='M'):\n", " models = {'S': 'en_core_web_sm', 'M': 'en_core_web_md', 'L': 'en_core_web_lg', 'XL': 'en_vectors_web_lg'}\n", " self.model = model_size\n", " self.nlp = spacy.load(models[self.model], disable=[\"tagger\", \"parser\", 'ner'])\n", " self._vocab = self.nlp.vocab\n", " self._tokenizer = Tokenizer(self.nlp.vocab)\n", " self.get_vecs = self._get_spacy_vecs\n", " self.get_vec = self._get_spacy_vec\n", " print(f'Initialized \"{models[self.model]}\" model.')\n", "\n", " def is_oov(self, word):\n", " return False if word in self._vocab else True\n", "\n", " def remove_oov(self, text):\n", " \"\"\"\n", " It takes text and remove (or replace) words that are out-of-vocabulary.\n", " Note: it splits text only with spaces.\n", " :param text: any text\n", " :return: text without oov words.\n", " \"\"\"\n", " return (' '.join(str(w) for w in text.split(' ') if not self.is_oov(str(w)))).strip()\n", "\n", " def _get_spacy_vecs(self, texts):\n", " return [(doc.text, doc.vector) for doc in self._tokenizer.pipe(texts, batch_size=1000)]\n", "\n", " def _get_spacy_vec(self, text):\n", " return text, self._tokenizer(text).vector\n", "\n", " def doc_without_stop_words(self, text):\n", " return self.nlp(self.text_without_stop_words(text))\n", "\n", " def text_without_stop_words(self, text):\n", " return ' '.join([str(t).lower() for t in self.nlp(text) if not t.is_stop and not t.is_punct and not t.is_oov])\n" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "ExecuteTime": { "end_time": "2020-09-15T23:34:04.639997Z", "start_time": "2020-09-15T23:33:47.751349Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Initialized \"en_core_web_md\" model.\n" ] } ], "source": [ "model_vectors = Model_vectors('M')" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "ExecuteTime": { "end_time": "2020-09-15T23:35:05.950839Z", "start_time": "2020-09-15T23:35:05.927865Z" }, "collapsed": true, "hidden": true, "jupyter": { "outputs_hidden": true }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "[('text', array([ 0.037103 , -0.31259 , -0.17857 , 0.30001 , 0.078154 ,\n", " 0.17958 , 0.12048 , -0.11879 , -0.20601 , 1.2849 ,\n", " -0.20409 , 0.80613 , 0.34344 , -0.19191 , -0.084511 ,\n", " 0.17339 , 0.042483 , 2.0282 , -0.16278 , -0.60306 ,\n", " -0.53766 , 0.35711 , 0.22882 , 0.1171 , 0.42983 ,\n", " 0.16165 , 0.407 , 0.036476 , 0.52636 , -0.13524 ,\n", " -0.016897 , 0.029259 , -0.079115 , -0.32305 , 0.052255 ,\n", " -0.3617 , -0.18355 , -0.34717 , -0.3691 , 0.16881 ,\n", " 0.21018 , -0.38376 , -0.096909 , -0.36296 , -0.37319 ,\n", " 0.0021152, 0.32512 , 0.063977 , 0.36249 , -0.26935 ,\n", " -0.59341 , -0.13625 , 0.016425 , -0.2474 , -0.07498 ,\n", " 0.034708 , -0.01476 , -0.11648 , 0.25559 , -0.35002 ,\n", " -0.52707 , 0.21221 , 0.062456 , 0.26184 , 0.53149 ,\n", " 0.34957 , -0.22692 , 0.44076 , 0.4438 , 0.6335 ,\n", " -0.049757 , -0.08134 , 0.65618 , -0.4716 , 0.090675 ,\n", " -0.084873 , 0.31455 , -0.38495 , -0.19247 , 0.48064 ,\n", " 0.26688 , 0.095743 , 0.13024 , 0.37023 , 0.46269 ,\n", " -0.32844 , 0.17375 , -0.36325 , 0.30672 , -0.075042 ,\n", " -0.64684 , -0.49822 , 0.12372 , -0.28547 , 0.61811 ,\n", " -0.19228 , 0.0040473, 0.1774 , 0.033154 , -0.54862 ,\n", " 0.34695 , -0.53506 , -0.013381 , 0.085712 , -0.054447 ,\n", " -0.64673 , 0.016749 , 0.47676 , 0.037803 , -0.10066 ,\n", " -0.4165 , -0.20252 , 0.2794 , 0.10852 , -0.40154 ,\n", " -0.037365 , -0.2169 , 0.15695 , -0.30861 , 0.13608 ,\n", " -0.054815 , -0.2745 , 0.33972 , 1.0216 , -0.15806 ,\n", " 0.072194 , 0.41601 , -0.17963 , 0.2317 , -0.26406 ,\n", " -0.19499 , 0.17078 , 0.20094 , 0.22253 , -0.081561 ,\n", " 0.066999 , -0.14185 , 0.10981 , -0.78997 , -0.36028 ,\n", " -1.0506 , 0.13359 , -0.32002 , 0.2477 , 0.51096 ,\n", " -0.2931 , -0.041309 , -0.43818 , -0.14525 , 0.16848 ,\n", " 0.13057 , 0.28067 , -0.083986 , 0.33443 , -0.089388 ,\n", " 0.39786 , -0.23063 , -0.42105 , -0.11821 , -0.015988 ,\n", " 0.24295 , -0.15992 , 0.17772 , -0.18388 , -0.94167 ,\n", " -0.044425 , -0.29272 , 0.42616 , 0.19122 , 0.02808 ,\n", " -0.24205 , 0.50342 , 0.036795 , -0.46235 , 0.85868 ,\n", " 0.052041 , 0.3903 , -0.60877 , -0.057832 , 0.27056 ,\n", " -0.33748 , -0.46636 , 0.3445 , -0.0084502, -0.28073 ,\n", " -0.087422 , -0.48422 , -0.38597 , 0.1432 , -0.84204 ,\n", " -0.070929 , -0.023476 , 0.2386 , 0.35143 , 0.19108 ,\n", " 0.40031 , -0.32652 , -0.41601 , -0.083684 , -0.058436 ,\n", " -0.25185 , -0.65487 , -0.28796 , -0.17906 , 0.09493 ,\n", " 0.064797 , 0.15409 , -0.083054 , -0.59515 , 0.31789 ,\n", " 0.60796 , -0.11354 , 0.19747 , -0.62954 , 0.097006 ,\n", " -0.091876 , 0.35441 , -0.030365 , -0.93405 , 0.093392 ,\n", " 0.60332 , -0.33488 , 0.32505 , 0.57094 , -0.21796 ,\n", " 0.34112 , 0.082672 , -0.38271 , 0.61988 , -0.29306 ,\n", " -0.23168 , 0.075344 , -0.11834 , 0.18269 , -0.37256 ,\n", " 0.65191 , 0.2294 , 0.393 , 0.44951 , 0.15056 ,\n", " 0.029826 , 0.12707 , -0.0026709, -0.46874 , 0.34049 ,\n", " -0.68405 , 0.26635 , -0.050739 , 0.2014 , -0.4979 ,\n", " -0.26716 , 0.048927 , -0.40881 , 0.18379 , 0.59551 ,\n", " 0.48461 , -0.28415 , 0.19632 , 0.1419 , -0.35254 ,\n", " 0.60201 , 0.015565 , 0.33977 , 0.10444 , 0.2258 ,\n", " -0.20714 , 0.031911 , 0.32246 , 1.7953 , -0.36103 ,\n", " -0.99446 , -0.56578 , 0.17655 , 0.025559 , 0.14056 ,\n", " -0.57778 , -0.17401 , 0.27002 , -0.09088 , -0.11778 ,\n", " 0.17015 , 0.029416 , 0.54829 , -0.79043 , -0.31254 ,\n", " -0.16837 , -0.48399 , 0.94058 , 0.076599 , -0.18597 ,\n", " 0.12836 , -0.20373 , 0.025537 , 0.55825 , 0.040429 ,\n", " 0.19721 , -0.48547 , -0.17655 , -0.069085 , 0.38212 ],\n", " dtype=float32)),\n", " ('title', array([-0.27409 , -0.10899 , -0.44377 , 0.055029 , 0.073908 ,\n", " -0.085274 , -0.023821 , -0.22339 , 0.015482 , 1.8443 ,\n", " -0.50172 , 0.73327 , 0.39423 , -0.0080002, 0.4899 ,\n", " 0.13731 , 0.86044 , 0.54383 , 0.3618 , -0.13255 ,\n", " -0.31003 , -0.1694 , 0.576 , -0.21519 , -0.020777 ,\n", " 0.45715 , 0.19577 , -0.30796 , 0.027145 , 0.3346 ,\n", " -0.15108 , -0.13252 , 0.15023 , 0.43497 , -0.37727 ,\n", " -0.30116 , -0.055824 , -0.21436 , -0.15635 , 0.18757 ,\n", " 0.17995 , 0.11149 , 0.5965 , -0.4197 , 0.19509 ,\n", " -0.4496 , -0.04078 , 0.57386 , -0.15958 , -0.023845 ,\n", " -0.2726 , 0.45388 , 0.14053 , -0.12576 , -0.46797 ,\n", " 0.21373 , 0.46424 , -0.2253 , 0.074821 , -0.4059 ,\n", " 0.072153 , -0.21921 , 0.32058 , 0.54763 , 0.11891 ,\n", " 0.32064 , 0.18676 , -0.43277 , 0.11908 , 0.31752 ,\n", " 0.11414 , -0.01404 , -0.56966 , 0.41143 , -0.26228 ,\n", " 0.37589 , -0.051109 , -0.26042 , 0.38441 , 0.068188 ,\n", " -0.25595 , 0.31704 , -0.25638 , 0.29252 , 0.31983 ,\n", " -0.024231 , -0.38089 , -0.37096 , 0.038813 , 0.002022 ,\n", " -0.79531 , -0.38653 , -0.38212 , 0.21827 , 0.4776 ,\n", " -0.2231 , 0.13736 , 0.064696 , 0.4148 , 0.12869 ,\n", " -0.35552 , -0.26823 , 0.040341 , -0.25815 , 0.27563 ,\n", " 0.49419 , -0.1758 , 0.25811 , -0.3401 , -0.031005 ,\n", " 0.15656 , -0.016722 , 0.22983 , 0.36909 , -0.060794 ,\n", " 0.099257 , -0.13895 , 0.34194 , 0.024073 , -0.34706 ,\n", " -0.2418 , -0.13364 , -0.086025 , 0.055324 , -0.34563 ,\n", " 0.53991 , 0.36551 , -0.090766 , -0.1346 , 0.21202 ,\n", " -0.22155 , -0.030866 , 0.12743 , -0.28135 , 0.17701 ,\n", " -0.15173 , -0.018258 , -0.023369 , -0.00638 , -0.42322 ,\n", " -1.3251 , -0.13133 , -0.37617 , -0.088589 , -0.168 ,\n", " -0.45516 , 0.66677 , 0.16292 , 0.21581 , 0.026985 ,\n", " -0.14255 , -0.6044 , -0.3743 , 0.25466 , -0.24371 ,\n", " -0.26281 , -0.45333 , 0.069396 , 0.16025 , 0.50201 ,\n", " -0.24571 , 0.15171 , 0.62315 , -0.19009 , -0.72376 ,\n", " -0.86903 , -0.087613 , 0.14561 , 0.35485 , 0.30589 ,\n", " -0.16767 , 0.26717 , -0.5878 , -0.47573 , 0.38508 ,\n", " -0.17917 , 0.44554 , 0.15533 , -0.55442 , 0.14262 ,\n", " 0.024764 , -0.30727 , 0.0042029, -0.30366 , -0.083989 ,\n", " -0.12939 , -0.17858 , 0.62597 , -0.17988 , -0.036059 ,\n", " 0.080507 , 0.19203 , 0.50822 , 0.3949 , 0.81686 ,\n", " 0.32855 , -0.026594 , -0.57621 , -0.17436 , 0.34332 ,\n", " 0.0046707, -0.50848 , -0.52363 , 0.019293 , -0.056288 ,\n", " -0.069271 , 0.018551 , 0.17636 , 0.23585 , 0.20668 ,\n", " 0.065329 , 0.30025 , 0.54376 , -0.54993 , -0.55273 ,\n", " 0.81333 , 0.30956 , 0.11698 , -0.4283 , 0.19276 ,\n", " 0.51739 , -0.24216 , -0.12089 , 0.36578 , 0.045 ,\n", " -0.45702 , -0.078447 , 0.45101 , 0.069623 , 0.046022 ,\n", " -0.20743 , 0.052619 , -0.22687 , 0.11581 , -0.35349 ,\n", " -0.033085 , -0.35541 , 0.36638 , -0.37858 , -0.29241 ,\n", " 0.043515 , -0.41882 , -0.10521 , -0.26249 , 0.18502 ,\n", " -0.86412 , 0.2517 , 0.0964 , 0.29931 , -0.33146 ,\n", " 0.54186 , -0.11851 , -0.2104 , 0.057476 , 0.014694 ,\n", " 0.45289 , -0.20416 , 0.14658 , -0.14917 , -0.10567 ,\n", " 0.38655 , 0.081924 , 0.17391 , 0.7822 , 0.4994 ,\n", " -0.12012 , -0.33085 , -0.13365 , 0.93653 , -0.32158 ,\n", " 0.013466 , -0.56513 , -0.20204 , -0.19847 , 0.29712 ,\n", " -0.036764 , -0.12111 , 0.27224 , -0.047855 , -0.20563 ,\n", " 0.18713 , 0.048798 , 0.0061171, -0.50512 , -0.55017 ,\n", " -0.056496 , -0.26158 , 0.31507 , 0.25999 , -0.56265 ,\n", " 0.21355 , -0.46628 , -0.39165 , -0.17727 , 0.50744 ,\n", " 0.083136 , -0.25254 , -0.66409 , 0.29799 , -0.36593 ],\n", " dtype=float32))]" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_vectors.get_vecs(['text', 'title'])" ] }, { "cell_type": "markdown", "metadata": { "hidden": true }, "source": [ "### ELMO embedding" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "ExecuteTime": { "end_time": "2020-09-16T02:27:21.243552Z", "start_time": "2020-09-16T02:27:21.225518Z" }, "hidden": true }, "outputs": [], "source": [ "from flair.data import Sentence\n", "from flair.embeddings import WordEmbeddings, FlairEmbeddings, StackedEmbeddings, CharacterEmbeddings\n", "from flair.embeddings import BertEmbeddings, ELMoEmbeddings, CharLMEmbeddings" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "ExecuteTime": { "end_time": "2020-09-16T00:48:33.498788Z", "start_time": "2020-09-16T00:48:32.016529Z" }, "hidden": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "I0915 17:48:32.024530 21016 utils.py:422] loading Word2VecKeyedVectors object from C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim\n", "I0915 17:48:33.275043 21016 utils.py:461] loading vectors from C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim.vectors.npy with mmap=None\n", "I0915 17:48:33.492771 21016 utils.py:494] setting ignored attribute vectors_norm to None\n", "I0915 17:48:33.493755 21016 utils.py:428] loaded C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim\n" ] } ], "source": [ "glove_emb = WordEmbeddings('glove')" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "ExecuteTime": { "end_time": "2020-09-16T00:50:45.764678Z", "start_time": "2020-09-16T00:50:41.866877Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Token: 1 This torch.Size([100])\n", "Token: 2 is torch.Size([100])\n", "Token: 3 a torch.Size([100])\n", "Token: 4 system torch.Size([100])\n", "Token: 5 integration. torch.Size([100])\n" ] } ], "source": [ "sent = Sentence('This is a system integration.')\n", "glove_emb.embed(sent)\n", "_ = [print(t, t.embedding.shape) for t in sent]" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "ExecuteTime": { "end_time": "2020-09-16T00:53:08.467214Z", "start_time": "2020-09-16T00:53:07.438764Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Token: 1 ig torch.Size([100])\n", "Token: 2 j torch.Size([100])\n", "Token: 3 yest torch.Size([100])\n", "Token: 4 ness torch.Size([100])\n", "Token: 5 bio torch.Size([100])\n", "Token: 6 physical torch.Size([100])\n", "Token: 7 oha torch.Size([100])\n", "Token: 8 x torch.Size([100])\n" ] } ], "source": [ "sent = Sentence(' '.join(wordninja.split('igjyestnessbiophysicalohax')))\n", "glove_emb.embed(sent)\n", "_ = [print(t, t.embedding.shape) for t in sent]" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "ExecuteTime": { "end_time": "2020-09-16T02:31:36.517610Z", "start_time": "2020-09-16T02:31:36.481609Z" }, "hidden": true }, "outputs": [ { "data": { "text/plain": [ "(torch.Size([100]),\n", " tensor([ 0.3876, 0.1173, -0.1786, -0.5518, -1.3703, -1.7363, -0.4303, 0.2215,\n", " -0.1897, -0.2305, -0.2904, -0.9843, -0.6379, 0.8521, -0.4181, 0.7708,\n", " 0.1105, -0.6815, -0.1501, 0.0790, 1.1104, -0.1524, -0.0356, -0.5496,\n", " 0.6761, -1.1164, 0.3294, 0.3358, -0.1761, 0.3174, 0.2326, -0.1749,\n", " -0.1531, -0.2616, -0.1671, -0.1590, 0.9442, 0.3235, -0.1124, -0.2102,\n", " -0.2055, 0.2575, -0.3094, 0.0053, -0.3699, 0.0685, 0.1430, -0.0455,\n", " -0.2702, 0.4782, 0.3559, -0.3921, -0.2094, 0.0851, -1.3836, 0.4357,\n", " -0.4373, 0.6922, -0.2560, 0.4856, 0.7953, -0.4402, -0.6072, 0.8645,\n", " 0.6602, -0.1795, -0.2840, 0.6075, 1.1740, 0.0348, 0.0285, 0.1810,\n", " -0.8015, -1.0579, -0.3046, -0.5177, -0.3481, -1.2305, -0.9520, 0.0256,\n", " 0.6460, 0.2348, -0.8135, -0.4275, -0.2730, -0.3802, 0.5882, 0.0488,\n", " 0.7306, -0.4720, 0.7308, -0.0583, 0.1116, 0.0297, 1.1622, 0.2942,\n", " 1.0848, 0.7977, 0.2797, 0.7269], device='cuda:0'))" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sent = Sentence(' '.join(wordninja.split('igjyestnessbiophysicalohax')))\n", "glove_emb.embed(sent)\n", "sent[0].embedding.shape, sent[0].embedding" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "ExecuteTime": { "end_time": "2020-09-16T02:29:36.106320Z", "start_time": "2020-09-16T02:29:30.354597Z" }, "hidden": true }, "outputs": [], "source": [ "flair_emb = FlairEmbeddings('d:/Program Files/flair/embeddings/lm-news-english-forward-v0.2rc.pt')" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "ExecuteTime": { "end_time": "2020-09-16T02:31:21.482003Z", "start_time": "2020-09-16T02:31:21.129345Z" }, "hidden": true }, "outputs": [ { "data": { "text/plain": [ "(torch.Size([2048]),\n", " tensor([ 1.4106e-05, 3.7738e-02, -5.4646e-02, ..., -1.2723e-04,\n", " 1.2393e-01, -2.3951e-03], device='cuda:0'))" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sent = Sentence(' '.join(wordninja.split('igjyestnessbiophysicalohax')))\n", "flair_emb.embed(sent)\n", "sent[0].embedding.shape, sent[0].embedding" ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "ExecuteTime": { "end_time": "2020-09-16T02:33:36.277263Z", "start_time": "2020-09-16T02:33:32.800498Z" }, "hidden": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "I0915 19:33:33.334479 21016 tokenization_utils.py:380] loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at C:\\Users\\leo_g\\.cache\\torch\\transformers\\26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084\n", "I0915 19:33:33.884137 21016 configuration_utils.py:157] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at C:\\Users\\leo_g\\.cache\\torch\\transformers\\4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517\n", "I0915 19:33:33.885138 21016 configuration_utils.py:174] Model config {\n", " \"architectures\": [\n", " \"BertForMaskedLM\"\n", " ],\n", " \"attention_probs_dropout_prob\": 0.1,\n", " \"finetuning_task\": null,\n", " \"hidden_act\": \"gelu\",\n", " \"hidden_dropout_prob\": 0.1,\n", " \"hidden_size\": 768,\n", " \"initializer_range\": 0.02,\n", " \"intermediate_size\": 3072,\n", " \"is_decoder\": false,\n", " \"layer_norm_eps\": 1e-12,\n", " \"max_position_embeddings\": 512,\n", " \"model_type\": \"bert\",\n", " \"num_attention_heads\": 12,\n", " \"num_hidden_layers\": 12,\n", " \"num_labels\": 2,\n", " \"output_attentions\": false,\n", " \"output_hidden_states\": true,\n", " \"output_past\": true,\n", " \"pad_token_id\": 0,\n", " \"pruned_heads\": {},\n", " \"torchscript\": false,\n", " \"type_vocab_size\": 2,\n", " \"use_bfloat16\": false,\n", " \"vocab_size\": 30522\n", "}\n", "\n", "I0915 19:33:34.332040 21016 modeling_utils.py:393] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin from cache at C:\\Users\\leo_g\\.cache\\torch\\transformers\\aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157\n" ] } ], "source": [ "bert_base_emb = BertEmbeddings('bert-base-uncased')\n" ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "ExecuteTime": { "end_time": "2020-09-16T02:33:54.625478Z", "start_time": "2020-09-16T02:33:53.993297Z" }, "hidden": true }, "outputs": [ { "data": { "text/plain": [ "(torch.Size([3072]),\n", " tensor([-0.4745, 0.2342, 0.8207, ..., -0.2806, 0.5300, 0.3944],\n", " device='cuda:0'))" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sent = Sentence(' '.join(wordninja.split('igjyestnessbiophysicalohax')))\n", "bert_base_emb.embed(sent)\n", "sent[0].embedding.shape, sent[0].embedding" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "# flair models" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "## classification example" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "ExecuteTime": { "end_time": "2020-09-18T03:47:09.926395Z", "start_time": "2020-09-18T03:47:09.913399Z" }, "hidden": true }, "outputs": [], "source": [ "from flair.data import Corpus\n", "from flair.datasets import TREC_6\n", "from flair.embeddings import WordEmbeddings, FlairEmbeddings, DocumentRNNEmbeddings\n", "from flair.models import TextClassifier\n", "from flair.trainers import ModelTrainer" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-09-18T03:31:59.850656Z", "start_time": "2020-09-18T03:31:59.809658Z" }, "hidden": true }, "outputs": [], "source": [ "# # 1. get the corpus\n", "# corpus: Corpus = TREC_6()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T15:34:33.203440Z", "start_time": "2020-09-17T15:34:33.188470Z" }, "hidden": true }, "outputs": [ { "data": { "text/plain": [ "['__add__',\n", " '__class__',\n", " '__delattr__',\n", " '__dict__',\n", " '__dir__',\n", " '__doc__',\n", " '__eq__',\n", " '__format__',\n", " '__ge__',\n", " '__getattribute__',\n", " '__getitem__',\n", " '__gt__',\n", " '__hash__',\n", " '__init__',\n", " '__init_subclass__',\n", " '__le__',\n", " '__len__',\n", " '__lt__',\n", " '__module__',\n", " '__ne__',\n", " '__new__',\n", " '__reduce__',\n", " '__reduce_ex__',\n", " '__repr__',\n", " '__setattr__',\n", " '__sizeof__',\n", " '__str__',\n", " '__subclasshook__',\n", " '__weakref__',\n", " '_parse_line_to_sentence',\n", " 'in_memory',\n", " 'is_in_memory',\n", " 'label_prefix',\n", " 'max_chars_per_doc',\n", " 'max_tokens_per_doc',\n", " 'path_to_file',\n", " 'sentences',\n", " 'tokenizer',\n", " 'total_sentence_count']" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = corpus.test\n", "dir(ds)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T15:35:48.408217Z", "start_time": "2020-09-17T15:35:48.355219Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 08:35:48,357 Computing label dictionary. Progress:\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████████████████████████████████████████████████████████████████████| 4907/4907 [00:00<00:00, 213334.54it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 08:35:48,394 [b'ENTY', b'DESC', b'NUM', b'ABBR', b'LOC', b'HUM']\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "# 2. create the label dictionary\n", "label_dict = corpus.make_label_dictionary()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2020-09-18T03:37:54.211276Z", "start_time": "2020-09-18T03:37:54.169231Z" }, "hidden": true }, "outputs": [ { "ename": "AttributeError", "evalue": "module 'flair' has no attribute 'nn'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mAttributeError\u001b[0m Traceback (most recent call last)", "\u001b[1;32m<ipython-input-7-c7335f7ee386>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m 6\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mflair\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0membeddings\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mTransformerDocumentEmbeddings\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 7\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mflair\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmodels\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mTextClassifier\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 8\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mflair\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtrainers\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mModelTrainer\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[1;32m~\\AppData\\Roaming\\Python\\Python36\\site-packages\\flair\\__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m 19\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mmodels\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 20\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mvisual\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 21\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtrainers\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 22\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mnn\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 23\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m\u001b[0mtraining_utils\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mAnnealOnPlateau\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", "\u001b[1;32m~\\AppData\\Roaming\\Python\\Python36\\site-packages\\flair\\trainers\\__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m\u001b[0mtrainer\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mModelTrainer\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[1;32m~\\AppData\\Roaming\\Python\\Python36\\site-packages\\flair\\trainers\\trainer.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m 38\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 39\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 40\u001b[1;33m \u001b[1;32mclass\u001b[0m \u001b[0mModelTrainer\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 41\u001b[0m def __init__(\n\u001b[0;32m 42\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", "\u001b[1;32m~\\AppData\\Roaming\\Python\\Python36\\site-packages\\flair\\trainers\\trainer.py\u001b[0m in \u001b[0;36mModelTrainer\u001b[1;34m()\u001b[0m\n\u001b[0;32m 45\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mtorch\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0moptim\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mOptimizer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mSGD\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 46\u001b[0m \u001b[0mepoch\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mint\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 47\u001b[1;33m \u001b[0muse_tensorboard\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mFalse\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 48\u001b[0m ):\n\u001b[0;32m 49\u001b[0m \"\"\"\n", "\u001b[1;31mAttributeError\u001b[0m: module 'flair' has no attribute 'nn'" ] } ], "source": [ "# len(label_dict), type(label_dict), dir(label_dict)\n", "from torch.optim.adam import Adam\n", "\n", "from flair.data import Corpus\n", "from flair.datasets import TREC_6\n", "from flair.embeddings import TransformerDocumentEmbeddings\n", "from flair.models import TextClassifier\n", "from flair.trainers import ModelTrainer\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T15:37:06.643008Z", "start_time": "2020-09-17T15:37:05.395375Z" }, "hidden": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "I0917 08:37:05.399397 11616 utils.py:422] loading Word2VecKeyedVectors object from C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim\n", "I0917 08:37:06.428536 11616 utils.py:461] loading vectors from C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim.vectors.npy with mmap=None\n", "I0917 08:37:06.633977 11616 utils.py:494] setting ignored attribute vectors_norm to None\n", "I0917 08:37:06.634947 11616 utils.py:428] loaded C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim\n" ] } ], "source": [ "# 3. make a list of word embeddings\n", "word_embeddings = [WordEmbeddings('glove')]" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T15:40:17.030158Z", "start_time": "2020-09-17T15:40:17.020150Z" }, "hidden": true }, "outputs": [], "source": [ "# len(word_embeddings), type(word_embeddings), dir(word_embeddings)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T15:39:39.595678Z", "start_time": "2020-09-17T15:39:33.300257Z" }, "hidden": true }, "outputs": [], "source": [ "# 4. initialize document embedding by passing list of word embeddings\n", "# Can choose between many RNN types (GRU by default, to change use rnn_type parameter)\n", "document_embeddings = DocumentRNNEmbeddings(word_embeddings, hidden_size=256)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T15:40:12.284954Z", "start_time": "2020-09-17T15:40:12.266955Z" }, "hidden": true }, "outputs": [], "source": [ "# type(document_embeddings), dir(document_embeddings)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T15:40:33.212197Z", "start_time": "2020-09-17T15:40:32.891382Z" }, "hidden": true }, "outputs": [], "source": [ "# 5. create the text classifier\n", "classifier = TextClassifier(document_embeddings, label_dictionary=label_dict)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T15:40:43.556563Z", "start_time": "2020-09-17T15:40:43.541542Z" }, "hidden": true }, "outputs": [], "source": [ "# 6. initialize the text classifier trainer\n", "trainer = ModelTrainer(classifier, corpus)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T16:03:23.833516Z", "start_time": "2020-09-17T15:40:55.387351Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 08:40:57,074 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:40:57,076 Model: \"TextClassifier(\n", " (document_embeddings): DocumentRNNEmbeddings(\n", " (embeddings): StackedEmbeddings(\n", " (list_embedding_0): WordEmbeddings('glove')\n", " )\n", " (word_reprojection_map): Linear(in_features=100, out_features=100, bias=True)\n", " (rnn): GRU(100, 256, batch_first=True)\n", " (dropout): Dropout(p=0.5, inplace=False)\n", " )\n", " (decoder): Linear(in_features=256, out_features=6, bias=True)\n", " (loss_function): CrossEntropyLoss()\n", ")\"\n", "2020-09-17 08:40:57,077 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:40:57,094 Corpus: \"Corpus: 4907 train + 545 dev + 500 test sentences\"\n", "2020-09-17 08:40:57,096 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:40:57,097 Parameters:\n", "2020-09-17 08:40:57,098 - learning_rate: \"0.1\"\n", "2020-09-17 08:40:57,101 - mini_batch_size: \"32\"\n", "2020-09-17 08:40:57,103 - patience: \"5\"\n", "2020-09-17 08:40:57,106 - anneal_factor: \"0.5\"\n", "2020-09-17 08:40:57,108 - max_epochs: \"150\"\n", "2020-09-17 08:40:57,110 - shuffle: \"True\"\n", "2020-09-17 08:40:57,116 - train_with_dev: \"False\"\n", "2020-09-17 08:40:57,118 - batch_growth_annealing: \"False\"\n", "2020-09-17 08:40:57,120 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:40:57,123 Model training base path: \"resources\\taggers\\trec\"\n", "2020-09-17 08:40:57,125 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:40:57,128 Device: cuda:0\n", "2020-09-17 08:40:57,129 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:40:57,131 Embeddings storage mode: cpu\n", "2020-09-17 08:40:57,151 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:40:58,435 epoch 1 - iter 0/154 - loss 1.83018792 - samples/sec: 374.80\n", "2020-09-17 08:40:59,729 epoch 1 - iter 15/154 - loss 1.74221355 - samples/sec: 385.39\n", "2020-09-17 08:41:00,894 epoch 1 - iter 30/154 - loss 1.71043461 - samples/sec: 427.02\n", "2020-09-17 08:41:02,265 epoch 1 - iter 45/154 - loss 1.70260032 - samples/sec: 359.49\n", "2020-09-17 08:41:03,430 epoch 1 - iter 60/154 - loss 1.69388972 - samples/sec: 427.04\n", "2020-09-17 08:41:04,660 epoch 1 - iter 75/154 - loss 1.67512008 - samples/sec: 400.76\n", "2020-09-17 08:41:05,819 epoch 1 - iter 90/154 - loss 1.65221116 - samples/sec: 427.56\n", "2020-09-17 08:41:07,037 epoch 1 - iter 105/154 - loss 1.64822389 - samples/sec: 405.40\n", "2020-09-17 08:41:08,401 epoch 1 - iter 120/154 - loss 1.64258651 - samples/sec: 361.49\n", "2020-09-17 08:41:09,658 epoch 1 - iter 135/154 - loss 1.63527459 - samples/sec: 393.43\n", "2020-09-17 08:41:10,867 epoch 1 - iter 150/154 - loss 1.62601529 - samples/sec: 407.47\n", "2020-09-17 08:41:11,084 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:41:11,085 EPOCH 1 done: loss 1.6238 - lr 0.1000\n", "2020-09-17 08:41:12,298 DEV : loss 1.4586095809936523 - score 0.411\n", "2020-09-17 08:41:12,325 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:41:16,795 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:41:16,860 epoch 2 - iter 0/154 - loss 1.38213122 - samples/sec: 7741.84\n", "2020-09-17 08:41:17,789 epoch 2 - iter 15/154 - loss 1.45874841 - samples/sec: 535.85\n", "2020-09-17 08:41:18,749 epoch 2 - iter 30/154 - loss 1.49112390 - samples/sec: 519.94\n", "2020-09-17 08:41:19,669 epoch 2 - iter 45/154 - loss 1.48928932 - samples/sec: 544.18\n", "2020-09-17 08:41:20,660 epoch 2 - iter 60/154 - loss 1.49432389 - samples/sec: 502.42\n", "2020-09-17 08:41:21,605 epoch 2 - iter 75/154 - loss 1.49229584 - samples/sec: 530.39\n", "2020-09-17 08:41:22,516 epoch 2 - iter 90/154 - loss 1.49269020 - samples/sec: 546.65\n", "2020-09-17 08:41:23,488 epoch 2 - iter 105/154 - loss 1.48966470 - samples/sec: 516.65\n", "2020-09-17 08:41:24,396 epoch 2 - iter 120/154 - loss 1.49143307 - samples/sec: 549.74\n", "2020-09-17 08:41:25,376 epoch 2 - iter 135/154 - loss 1.47819957 - samples/sec: 512.22\n", "2020-09-17 08:41:26,321 epoch 2 - iter 150/154 - loss 1.47582191 - samples/sec: 528.65\n", "2020-09-17 08:41:26,502 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:41:26,503 EPOCH 2 done: loss 1.4752 - lr 0.1000\n", "2020-09-17 08:41:27,362 DEV : loss 1.5128141641616821 - score 0.3743\n", "2020-09-17 08:41:27,378 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:41:27,380 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:41:27,467 epoch 3 - iter 0/154 - loss 1.54040027 - samples/sec: 5581.26\n", "2020-09-17 08:41:28,388 epoch 3 - iter 15/154 - loss 1.42931751 - samples/sec: 544.16\n", "2020-09-17 08:41:29,296 epoch 3 - iter 30/154 - loss 1.38326055 - samples/sec: 551.99\n", "2020-09-17 08:41:30,287 epoch 3 - iter 45/154 - loss 1.38690084 - samples/sec: 503.16\n", "2020-09-17 08:41:31,232 epoch 3 - iter 60/154 - loss 1.39076413 - samples/sec: 528.06\n", "2020-09-17 08:41:32,211 epoch 3 - iter 75/154 - loss 1.38759932 - samples/sec: 510.73\n", "2020-09-17 08:41:33,112 epoch 3 - iter 90/154 - loss 1.38098083 - samples/sec: 553.30\n", "2020-09-17 08:41:34,017 epoch 3 - iter 105/154 - loss 1.37900963 - samples/sec: 550.35\n", "2020-09-17 08:41:34,993 epoch 3 - iter 120/154 - loss 1.36568035 - samples/sec: 510.70\n", "2020-09-17 08:41:35,918 epoch 3 - iter 135/154 - loss 1.35870855 - samples/sec: 538.80\n", "2020-09-17 08:41:36,939 epoch 3 - iter 150/154 - loss 1.35459640 - samples/sec: 487.61\n", "2020-09-17 08:41:37,150 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:41:37,151 EPOCH 3 done: loss 1.3539 - lr 0.1000\n", "2020-09-17 08:41:38,083 DEV : loss 1.4478060007095337 - score 0.4294\n", "2020-09-17 08:41:38,100 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:41:42,429 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:41:42,492 epoch 4 - iter 0/154 - loss 1.37430179 - samples/sec: 7994.77\n", "2020-09-17 08:41:43,428 epoch 4 - iter 15/154 - loss 1.22524202 - samples/sec: 529.68\n", "2020-09-17 08:41:44,356 epoch 4 - iter 30/154 - loss 1.24898812 - samples/sec: 539.44\n", "2020-09-17 08:41:45,288 epoch 4 - iter 45/154 - loss 1.28158134 - samples/sec: 534.45\n", "2020-09-17 08:41:46,241 epoch 4 - iter 60/154 - loss 1.28029170 - samples/sec: 520.12\n", "2020-09-17 08:41:47,159 epoch 4 - iter 75/154 - loss 1.28344258 - samples/sec: 546.09\n", "2020-09-17 08:41:48,154 epoch 4 - iter 90/154 - loss 1.28573294 - samples/sec: 498.18\n", "2020-09-17 08:41:49,075 epoch 4 - iter 105/154 - loss 1.27805208 - samples/sec: 546.20\n", "2020-09-17 08:41:49,962 epoch 4 - iter 120/154 - loss 1.27061762 - samples/sec: 560.82\n", "2020-09-17 08:41:50,931 epoch 4 - iter 135/154 - loss 1.27844691 - samples/sec: 515.52\n", "2020-09-17 08:41:51,849 epoch 4 - iter 150/154 - loss 1.26840330 - samples/sec: 541.64\n", "2020-09-17 08:41:52,031 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:41:52,032 EPOCH 4 done: loss 1.2666 - lr 0.1000\n", "2020-09-17 08:41:52,916 DEV : loss 1.459532618522644 - score 0.4147\n", "2020-09-17 08:41:52,930 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:41:52,932 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:41:52,991 epoch 5 - iter 0/154 - loss 1.66579664 - samples/sec: 8416.59\n", "2020-09-17 08:41:53,927 epoch 5 - iter 15/154 - loss 1.19705896 - samples/sec: 531.32\n", "2020-09-17 08:41:54,877 epoch 5 - iter 30/154 - loss 1.20760085 - samples/sec: 522.71\n", "2020-09-17 08:41:55,812 epoch 5 - iter 45/154 - loss 1.21064639 - samples/sec: 539.16\n", "2020-09-17 08:41:56,743 epoch 5 - iter 60/154 - loss 1.21762990 - samples/sec: 533.90\n", "2020-09-17 08:41:57,720 epoch 5 - iter 75/154 - loss 1.21822268 - samples/sec: 510.13\n", "2020-09-17 08:41:58,635 epoch 5 - iter 90/154 - loss 1.22344246 - samples/sec: 546.23\n", "2020-09-17 08:41:59,582 epoch 5 - iter 105/154 - loss 1.22330612 - samples/sec: 526.28\n", "2020-09-17 08:42:00,575 epoch 5 - iter 120/154 - loss 1.21609633 - samples/sec: 504.53\n", "2020-09-17 08:42:01,496 epoch 5 - iter 135/154 - loss 1.21323682 - samples/sec: 542.16\n", "2020-09-17 08:42:02,440 epoch 5 - iter 150/154 - loss 1.20384763 - samples/sec: 527.20\n", "2020-09-17 08:42:02,625 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:42:02,626 EPOCH 5 done: loss 1.2023 - lr 0.1000\n", "2020-09-17 08:42:03,387 DEV : loss 1.0862823724746704 - score 0.5817\n", "2020-09-17 08:42:03,401 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:42:07,851 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:42:07,914 epoch 6 - iter 0/154 - loss 1.17252481 - samples/sec: 7868.99\n", "2020-09-17 08:42:08,923 epoch 6 - iter 15/154 - loss 1.07886096 - samples/sec: 491.81\n", "2020-09-17 08:42:09,918 epoch 6 - iter 30/154 - loss 1.07490783 - samples/sec: 499.89\n", "2020-09-17 08:42:10,947 epoch 6 - iter 45/154 - loss 1.06794010 - samples/sec: 482.82\n", "2020-09-17 08:42:11,919 epoch 6 - iter 60/154 - loss 1.09830035 - samples/sec: 515.03\n", "2020-09-17 08:42:12,848 epoch 6 - iter 75/154 - loss 1.10102981 - samples/sec: 536.19\n", "2020-09-17 08:42:13,834 epoch 6 - iter 90/154 - loss 1.10023795 - samples/sec: 506.03\n", "2020-09-17 08:42:14,756 epoch 6 - iter 105/154 - loss 1.09884320 - samples/sec: 543.85\n", "2020-09-17 08:42:15,721 epoch 6 - iter 120/154 - loss 1.09788988 - samples/sec: 518.04\n", "2020-09-17 08:42:16,681 epoch 6 - iter 135/154 - loss 1.09325209 - samples/sec: 523.28\n", "2020-09-17 08:42:17,631 epoch 6 - iter 150/154 - loss 1.09716275 - samples/sec: 525.88\n", "2020-09-17 08:42:17,820 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:42:17,821 EPOCH 6 done: loss 1.0989 - lr 0.1000\n", "2020-09-17 08:42:18,669 DEV : loss 1.117511510848999 - score 0.5358\n", "2020-09-17 08:42:18,686 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:42:18,687 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:42:18,754 epoch 7 - iter 0/154 - loss 1.14436972 - samples/sec: 7494.71\n", "2020-09-17 08:42:19,689 epoch 7 - iter 15/154 - loss 1.05698588 - samples/sec: 533.87\n", "2020-09-17 08:42:20,674 epoch 7 - iter 30/154 - loss 1.01580172 - samples/sec: 503.30\n", "2020-09-17 08:42:21,664 epoch 7 - iter 45/154 - loss 1.01566661 - samples/sec: 505.82\n", "2020-09-17 08:42:22,587 epoch 7 - iter 60/154 - loss 1.02257143 - samples/sec: 542.31\n", "2020-09-17 08:42:23,511 epoch 7 - iter 75/154 - loss 1.03337308 - samples/sec: 546.51\n", "2020-09-17 08:42:24,417 epoch 7 - iter 90/154 - loss 1.03853320 - samples/sec: 552.60\n", "2020-09-17 08:42:25,399 epoch 7 - iter 105/154 - loss 1.02362797 - samples/sec: 513.38\n", "2020-09-17 08:42:26,335 epoch 7 - iter 120/154 - loss 1.02437582 - samples/sec: 533.92\n", "2020-09-17 08:42:27,282 epoch 7 - iter 135/154 - loss 1.02272838 - samples/sec: 526.32\n", "2020-09-17 08:42:28,226 epoch 7 - iter 150/154 - loss 1.02280438 - samples/sec: 532.41\n", "2020-09-17 08:42:28,415 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:42:28,416 EPOCH 7 done: loss 1.0325 - lr 0.1000\n", "2020-09-17 08:42:29,215 DEV : loss 0.9613503217697144 - score 0.6422\n", "2020-09-17 08:42:29,232 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:42:33,753 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:42:33,829 epoch 8 - iter 0/154 - loss 1.18564963 - samples/sec: 6503.87\n", "2020-09-17 08:42:34,875 epoch 8 - iter 15/154 - loss 0.97893873 - samples/sec: 477.97\n", "2020-09-17 08:42:35,886 epoch 8 - iter 30/154 - loss 1.00841281 - samples/sec: 493.38\n", "2020-09-17 08:42:36,850 epoch 8 - iter 45/154 - loss 0.97391248 - samples/sec: 515.92\n", "2020-09-17 08:42:37,733 epoch 8 - iter 60/154 - loss 0.96790417 - samples/sec: 566.50\n", "2020-09-17 08:42:38,677 epoch 8 - iter 75/154 - loss 0.97422929 - samples/sec: 531.59\n", "2020-09-17 08:42:39,635 epoch 8 - iter 90/154 - loss 0.97338708 - samples/sec: 523.99\n", "2020-09-17 08:42:40,539 epoch 8 - iter 105/154 - loss 0.96709728 - samples/sec: 555.12\n", "2020-09-17 08:42:41,537 epoch 8 - iter 120/154 - loss 0.96682342 - samples/sec: 500.41\n", "2020-09-17 08:42:42,426 epoch 8 - iter 135/154 - loss 0.95721136 - samples/sec: 564.02\n", "2020-09-17 08:42:43,358 epoch 8 - iter 150/154 - loss 0.94798565 - samples/sec: 535.24\n", "2020-09-17 08:42:43,594 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:42:43,594 EPOCH 8 done: loss 0.9454 - lr 0.1000\n", "2020-09-17 08:42:44,354 DEV : loss 0.9537383317947388 - score 0.6349\n", "2020-09-17 08:42:44,366 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:42:44,369 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:42:44,436 epoch 9 - iter 0/154 - loss 0.89374548 - samples/sec: 7383.50\n", "2020-09-17 08:42:45,419 epoch 9 - iter 15/154 - loss 0.92112946 - samples/sec: 506.41\n", "2020-09-17 08:42:46,365 epoch 9 - iter 30/154 - loss 0.89975801 - samples/sec: 528.16\n", "2020-09-17 08:42:47,365 epoch 9 - iter 45/154 - loss 0.91358758 - samples/sec: 496.30\n", "2020-09-17 08:42:48,305 epoch 9 - iter 60/154 - loss 0.93426236 - samples/sec: 530.17\n", "2020-09-17 08:42:49,251 epoch 9 - iter 75/154 - loss 0.93322996 - samples/sec: 530.06\n", "2020-09-17 08:42:50,296 epoch 9 - iter 90/154 - loss 0.93047885 - samples/sec: 476.99\n", "2020-09-17 08:42:51,305 epoch 9 - iter 105/154 - loss 0.93570090 - samples/sec: 497.92\n", "2020-09-17 08:42:52,290 epoch 9 - iter 120/154 - loss 0.92295540 - samples/sec: 505.70\n", "2020-09-17 08:42:53,312 epoch 9 - iter 135/154 - loss 0.91840233 - samples/sec: 490.05\n", "2020-09-17 08:42:54,414 epoch 9 - iter 150/154 - loss 0.91143699 - samples/sec: 452.80\n", "2020-09-17 08:42:54,635 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:42:54,637 EPOCH 9 done: loss 0.9073 - lr 0.1000\n", "2020-09-17 08:42:55,499 DEV : loss 0.7254136800765991 - score 0.7193\n", "2020-09-17 08:42:55,515 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:42:59,968 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:43:00,041 epoch 10 - iter 0/154 - loss 0.62184030 - samples/sec: 6668.23\n", "2020-09-17 08:43:01,015 epoch 10 - iter 15/154 - loss 0.86726173 - samples/sec: 510.70\n", "2020-09-17 08:43:01,975 epoch 10 - iter 30/154 - loss 0.84851758 - samples/sec: 520.10\n", "2020-09-17 08:43:02,886 epoch 10 - iter 45/154 - loss 0.83090996 - samples/sec: 551.51\n", "2020-09-17 08:43:03,833 epoch 10 - iter 60/154 - loss 0.86082481 - samples/sec: 530.20\n", "2020-09-17 08:43:04,791 epoch 10 - iter 75/154 - loss 0.88071626 - samples/sec: 522.04\n", "2020-09-17 08:43:05,714 epoch 10 - iter 90/154 - loss 0.87638579 - samples/sec: 540.93\n", "2020-09-17 08:43:06,695 epoch 10 - iter 105/154 - loss 0.86520858 - samples/sec: 507.27\n", "2020-09-17 08:43:07,663 epoch 10 - iter 120/154 - loss 0.85527089 - samples/sec: 516.60\n", "2020-09-17 08:43:08,614 epoch 10 - iter 135/154 - loss 0.85346116 - samples/sec: 525.95\n", "2020-09-17 08:43:09,639 epoch 10 - iter 150/154 - loss 0.84259231 - samples/sec: 492.18\n", "2020-09-17 08:43:09,816 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:43:09,817 EPOCH 10 done: loss 0.8449 - lr 0.1000\n", "2020-09-17 08:43:10,599 DEV : loss 0.7877208590507507 - score 0.7046\n", "2020-09-17 08:43:10,613 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:43:10,615 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:43:10,694 epoch 11 - iter 0/154 - loss 1.04077613 - samples/sec: 6233.87\n", "2020-09-17 08:43:11,657 epoch 11 - iter 15/154 - loss 0.79468361 - samples/sec: 518.25\n", "2020-09-17 08:43:12,575 epoch 11 - iter 30/154 - loss 0.72452083 - samples/sec: 545.36\n", "2020-09-17 08:43:13,549 epoch 11 - iter 45/154 - loss 0.76410428 - samples/sec: 513.94\n", "2020-09-17 08:43:14,495 epoch 11 - iter 60/154 - loss 0.81160275 - samples/sec: 535.75\n", "2020-09-17 08:43:15,437 epoch 11 - iter 75/154 - loss 0.80992556 - samples/sec: 530.38\n", "2020-09-17 08:43:16,603 epoch 11 - iter 90/154 - loss 0.80919972 - samples/sec: 496.31\n", "2020-09-17 08:43:17,528 epoch 11 - iter 105/154 - loss 0.80230314 - samples/sec: 542.18\n", "2020-09-17 08:43:18,528 epoch 11 - iter 120/154 - loss 0.80666312 - samples/sec: 498.53\n", "2020-09-17 08:43:19,525 epoch 11 - iter 135/154 - loss 0.79973664 - samples/sec: 499.45\n", "2020-09-17 08:43:20,575 epoch 11 - iter 150/154 - loss 0.79486089 - samples/sec: 480.01\n", "2020-09-17 08:43:20,756 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:43:20,758 EPOCH 11 done: loss 0.7944 - lr 0.1000\n", "2020-09-17 08:43:21,599 DEV : loss 1.0040833950042725 - score 0.6312\n", "2020-09-17 08:43:21,613 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:43:21,616 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:43:21,684 epoch 12 - iter 0/154 - loss 1.42439318 - samples/sec: 7382.20\n", "2020-09-17 08:43:22,616 epoch 12 - iter 15/154 - loss 0.78031881 - samples/sec: 534.45\n", "2020-09-17 08:43:23,537 epoch 12 - iter 30/154 - loss 0.77762859 - samples/sec: 549.16\n", "2020-09-17 08:43:24,484 epoch 12 - iter 45/154 - loss 0.77193386 - samples/sec: 526.89\n", "2020-09-17 08:43:25,459 epoch 12 - iter 60/154 - loss 0.77484313 - samples/sec: 510.32\n", "2020-09-17 08:43:26,389 epoch 12 - iter 75/154 - loss 0.77354109 - samples/sec: 537.23\n", "2020-09-17 08:43:27,346 epoch 12 - iter 90/154 - loss 0.76224761 - samples/sec: 522.95\n", "2020-09-17 08:43:28,311 epoch 12 - iter 105/154 - loss 0.76232956 - samples/sec: 520.01\n", "2020-09-17 08:43:29,222 epoch 12 - iter 120/154 - loss 0.76197255 - samples/sec: 548.53\n", "2020-09-17 08:43:30,220 epoch 12 - iter 135/154 - loss 0.75723405 - samples/sec: 501.13\n", "2020-09-17 08:43:31,144 epoch 12 - iter 150/154 - loss 0.76219605 - samples/sec: 541.35\n", "2020-09-17 08:43:31,344 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:43:31,345 EPOCH 12 done: loss 0.7624 - lr 0.1000\n", "2020-09-17 08:43:32,168 DEV : loss 0.7417812943458557 - score 0.7394\n", "2020-09-17 08:43:32,180 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:43:36,538 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:43:36,620 epoch 13 - iter 0/154 - loss 1.01700652 - samples/sec: 6075.95\n", "2020-09-17 08:43:37,515 epoch 13 - iter 15/154 - loss 0.77986348 - samples/sec: 558.82\n", "2020-09-17 08:43:38,428 epoch 13 - iter 30/154 - loss 0.73918162 - samples/sec: 549.98\n", "2020-09-17 08:43:39,409 epoch 13 - iter 45/154 - loss 0.71001165 - samples/sec: 509.29\n", "2020-09-17 08:43:40,341 epoch 13 - iter 60/154 - loss 0.70411422 - samples/sec: 535.92\n", "2020-09-17 08:43:41,326 epoch 13 - iter 75/154 - loss 0.70767895 - samples/sec: 508.63\n", "2020-09-17 08:43:42,268 epoch 13 - iter 90/154 - loss 0.71394394 - samples/sec: 529.45\n", "2020-09-17 08:43:43,216 epoch 13 - iter 105/154 - loss 0.72905175 - samples/sec: 531.97\n", "2020-09-17 08:43:44,181 epoch 13 - iter 120/154 - loss 0.72157644 - samples/sec: 516.36\n", "2020-09-17 08:43:45,143 epoch 13 - iter 135/154 - loss 0.72615036 - samples/sec: 519.46\n", "2020-09-17 08:43:46,135 epoch 13 - iter 150/154 - loss 0.72067405 - samples/sec: 502.60\n", "2020-09-17 08:43:46,332 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:43:46,333 EPOCH 13 done: loss 0.7177 - lr 0.1000\n", "2020-09-17 08:43:47,155 DEV : loss 0.6499255299568176 - score 0.7394\n", "2020-09-17 08:43:47,176 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:43:51,476 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:43:51,549 epoch 14 - iter 0/154 - loss 1.18047774 - samples/sec: 6857.29\n", "2020-09-17 08:43:52,454 epoch 14 - iter 15/154 - loss 0.74128841 - samples/sec: 553.12\n", "2020-09-17 08:43:53,488 epoch 14 - iter 30/154 - loss 0.72311837 - samples/sec: 483.76\n", "2020-09-17 08:43:54,478 epoch 14 - iter 45/154 - loss 0.71235278 - samples/sec: 505.45\n", "2020-09-17 08:43:55,440 epoch 14 - iter 60/154 - loss 0.71622231 - samples/sec: 519.51\n", "2020-09-17 08:43:56,368 epoch 14 - iter 75/154 - loss 0.73051200 - samples/sec: 541.05\n", "2020-09-17 08:43:57,347 epoch 14 - iter 90/154 - loss 0.73342301 - samples/sec: 510.26\n", "2020-09-17 08:43:58,224 epoch 14 - iter 105/154 - loss 0.72841264 - samples/sec: 568.11\n", "2020-09-17 08:43:59,110 epoch 14 - iter 120/154 - loss 0.72012054 - samples/sec: 567.30\n", "2020-09-17 08:44:00,082 epoch 14 - iter 135/154 - loss 0.72616448 - samples/sec: 513.47\n", "2020-09-17 08:44:01,033 epoch 14 - iter 150/154 - loss 0.71809935 - samples/sec: 524.52\n", "2020-09-17 08:44:01,250 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:44:01,251 EPOCH 14 done: loss 0.7222 - lr 0.1000\n", "2020-09-17 08:44:02,166 DEV : loss 0.6644572019577026 - score 0.7358\n", "2020-09-17 08:44:02,181 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:44:02,183 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:44:02,244 epoch 15 - iter 0/154 - loss 0.56726515 - samples/sec: 8132.07\n", "2020-09-17 08:44:03,200 epoch 15 - iter 15/154 - loss 0.64099861 - samples/sec: 527.47\n", "2020-09-17 08:44:04,200 epoch 15 - iter 30/154 - loss 0.67168225 - samples/sec: 500.51\n", "2020-09-17 08:44:05,148 epoch 15 - iter 45/154 - loss 0.70463899 - samples/sec: 531.26\n", "2020-09-17 08:44:06,218 epoch 15 - iter 60/154 - loss 0.69836672 - samples/sec: 465.50\n", "2020-09-17 08:44:07,210 epoch 15 - iter 75/154 - loss 0.69513514 - samples/sec: 502.59\n", "2020-09-17 08:44:08,222 epoch 15 - iter 90/154 - loss 0.69867321 - samples/sec: 490.61\n", "2020-09-17 08:44:09,250 epoch 15 - iter 105/154 - loss 0.70425679 - samples/sec: 485.14\n", "2020-09-17 08:44:10,229 epoch 15 - iter 120/154 - loss 0.69529294 - samples/sec: 511.03\n", "2020-09-17 08:44:11,249 epoch 15 - iter 135/154 - loss 0.69432109 - samples/sec: 488.80\n", "2020-09-17 08:44:12,207 epoch 15 - iter 150/154 - loss 0.68699504 - samples/sec: 520.26\n", "2020-09-17 08:44:12,388 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:44:12,389 EPOCH 15 done: loss 0.6910 - lr 0.1000\n", "2020-09-17 08:44:13,212 DEV : loss 0.655533492565155 - score 0.7284\n", "2020-09-17 08:44:13,227 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:44:13,228 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:44:13,320 epoch 16 - iter 0/154 - loss 0.53883034 - samples/sec: 5333.47\n", "2020-09-17 08:44:14,319 epoch 16 - iter 15/154 - loss 0.60707116 - samples/sec: 503.94\n", "2020-09-17 08:44:15,226 epoch 16 - iter 30/154 - loss 0.64390820 - samples/sec: 552.54\n", "2020-09-17 08:44:16,192 epoch 16 - iter 45/154 - loss 0.64652951 - samples/sec: 515.27\n", "2020-09-17 08:44:17,066 epoch 16 - iter 60/154 - loss 0.64580723 - samples/sec: 570.58\n", "2020-09-17 08:44:18,071 epoch 16 - iter 75/154 - loss 0.65157384 - samples/sec: 497.43\n", "2020-09-17 08:44:19,056 epoch 16 - iter 90/154 - loss 0.64791287 - samples/sec: 515.03\n", "2020-09-17 08:44:19,964 epoch 16 - iter 105/154 - loss 0.65261372 - samples/sec: 554.48\n", "2020-09-17 08:44:20,906 epoch 16 - iter 120/154 - loss 0.65917246 - samples/sec: 529.81\n", "2020-09-17 08:44:21,867 epoch 16 - iter 135/154 - loss 0.66221129 - samples/sec: 519.66\n", "2020-09-17 08:44:22,887 epoch 16 - iter 150/154 - loss 0.65582889 - samples/sec: 489.38\n", "2020-09-17 08:44:23,080 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:44:23,081 EPOCH 16 done: loss 0.6557 - lr 0.1000\n", "2020-09-17 08:44:23,907 DEV : loss 0.5636053085327148 - score 0.7706\n", "2020-09-17 08:44:23,921 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:44:28,254 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:44:28,331 epoch 17 - iter 0/154 - loss 0.43598625 - samples/sec: 6432.57\n", "2020-09-17 08:44:29,258 epoch 17 - iter 15/154 - loss 0.60036130 - samples/sec: 538.76\n", "2020-09-17 08:44:30,229 epoch 17 - iter 30/154 - loss 0.62158019 - samples/sec: 515.79\n", "2020-09-17 08:44:31,116 epoch 17 - iter 45/154 - loss 0.66702972 - samples/sec: 567.38\n", "2020-09-17 08:44:32,118 epoch 17 - iter 60/154 - loss 0.64715165 - samples/sec: 497.51\n", "2020-09-17 08:44:33,036 epoch 17 - iter 75/154 - loss 0.64839264 - samples/sec: 545.74\n", "2020-09-17 08:44:33,941 epoch 17 - iter 90/154 - loss 0.64458025 - samples/sec: 551.78\n", "2020-09-17 08:44:34,951 epoch 17 - iter 105/154 - loss 0.64538896 - samples/sec: 492.52\n", "2020-09-17 08:44:35,933 epoch 17 - iter 120/154 - loss 0.63559149 - samples/sec: 512.01\n", "2020-09-17 08:44:36,927 epoch 17 - iter 135/154 - loss 0.63272227 - samples/sec: 504.44\n", "2020-09-17 08:44:37,859 epoch 17 - iter 150/154 - loss 0.63573848 - samples/sec: 538.08\n", "2020-09-17 08:44:38,048 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:44:38,049 EPOCH 17 done: loss 0.6340 - lr 0.1000\n", "2020-09-17 08:44:38,944 DEV : loss 0.5324942469596863 - score 0.7963\n", "2020-09-17 08:44:38,958 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:44:43,235 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:44:43,325 epoch 18 - iter 0/154 - loss 0.52554232 - samples/sec: 5517.79\n", "2020-09-17 08:44:44,331 epoch 18 - iter 15/154 - loss 0.66534532 - samples/sec: 497.81\n", "2020-09-17 08:44:45,274 epoch 18 - iter 30/154 - loss 0.59145527 - samples/sec: 531.82\n", "2020-09-17 08:44:46,232 epoch 18 - iter 45/154 - loss 0.61919989 - samples/sec: 523.10\n", "2020-09-17 08:44:47,168 epoch 18 - iter 60/154 - loss 0.63118371 - samples/sec: 534.82\n", "2020-09-17 08:44:48,164 epoch 18 - iter 75/154 - loss 0.63492458 - samples/sec: 498.73\n", "2020-09-17 08:44:49,082 epoch 18 - iter 90/154 - loss 0.63354703 - samples/sec: 546.00\n", "2020-09-17 08:44:50,023 epoch 18 - iter 105/154 - loss 0.64153443 - samples/sec: 531.98\n", "2020-09-17 08:44:50,968 epoch 18 - iter 120/154 - loss 0.63208055 - samples/sec: 528.50\n", "2020-09-17 08:44:51,908 epoch 18 - iter 135/154 - loss 0.62546947 - samples/sec: 533.14\n", "2020-09-17 08:44:52,899 epoch 18 - iter 150/154 - loss 0.62399244 - samples/sec: 503.28\n", "2020-09-17 08:44:53,085 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:44:53,086 EPOCH 18 done: loss 0.6207 - lr 0.1000\n", "2020-09-17 08:44:53,848 DEV : loss 0.5046014785766602 - score 0.8\n", "2020-09-17 08:44:53,862 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:44:58,249 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:44:58,318 epoch 19 - iter 0/154 - loss 0.51401347 - samples/sec: 7163.04\n", "2020-09-17 08:44:59,258 epoch 19 - iter 15/154 - loss 0.56021131 - samples/sec: 531.58\n", "2020-09-17 08:45:00,281 epoch 19 - iter 30/154 - loss 0.55715307 - samples/sec: 489.78\n", "2020-09-17 08:45:01,234 epoch 19 - iter 45/154 - loss 0.56592278 - samples/sec: 524.80\n", "2020-09-17 08:45:02,197 epoch 19 - iter 60/154 - loss 0.56632017 - samples/sec: 521.20\n", "2020-09-17 08:45:03,112 epoch 19 - iter 75/154 - loss 0.58037908 - samples/sec: 545.01\n", "2020-09-17 08:45:04,065 epoch 19 - iter 90/154 - loss 0.57893606 - samples/sec: 525.16\n", "2020-09-17 08:45:05,037 epoch 19 - iter 105/154 - loss 0.57952498 - samples/sec: 520.26\n", "2020-09-17 08:45:05,957 epoch 19 - iter 120/154 - loss 0.59105469 - samples/sec: 542.25\n", "2020-09-17 08:45:06,957 epoch 19 - iter 135/154 - loss 0.59264780 - samples/sec: 499.84\n", "2020-09-17 08:45:07,883 epoch 19 - iter 150/154 - loss 0.59842980 - samples/sec: 539.43\n", "2020-09-17 08:45:08,078 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:45:08,080 EPOCH 19 done: loss 0.5979 - lr 0.1000\n", "2020-09-17 08:45:08,942 DEV : loss 0.5927051305770874 - score 0.7725\n", "2020-09-17 08:45:08,958 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:45:08,960 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:45:09,029 epoch 20 - iter 0/154 - loss 0.71233368 - samples/sec: 7166.05\n", "2020-09-17 08:45:09,992 epoch 20 - iter 15/154 - loss 0.55507455 - samples/sec: 519.80\n", "2020-09-17 08:45:11,052 epoch 20 - iter 30/154 - loss 0.58391772 - samples/sec: 468.15\n", "2020-09-17 08:45:11,986 epoch 20 - iter 45/154 - loss 0.56636989 - samples/sec: 538.93\n", "2020-09-17 08:45:12,909 epoch 20 - iter 60/154 - loss 0.57968388 - samples/sec: 539.68\n", "2020-09-17 08:45:13,887 epoch 20 - iter 75/154 - loss 0.57898138 - samples/sec: 507.90\n", "2020-09-17 08:45:14,816 epoch 20 - iter 90/154 - loss 0.59078578 - samples/sec: 536.28\n", "2020-09-17 08:45:15,759 epoch 20 - iter 105/154 - loss 0.59594110 - samples/sec: 524.34\n", "2020-09-17 08:45:16,719 epoch 20 - iter 120/154 - loss 0.59488263 - samples/sec: 524.78\n", "2020-09-17 08:45:17,655 epoch 20 - iter 135/154 - loss 0.59283041 - samples/sec: 533.76\n", "2020-09-17 08:45:18,644 epoch 20 - iter 150/154 - loss 0.59209025 - samples/sec: 501.38\n", "2020-09-17 08:45:18,826 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:45:18,827 EPOCH 20 done: loss 0.5879 - lr 0.1000\n", "2020-09-17 08:45:19,606 DEV : loss 0.6165120601654053 - score 0.778\n", "2020-09-17 08:45:19,619 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:45:19,621 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:45:19,691 epoch 21 - iter 0/154 - loss 0.93119234 - samples/sec: 7162.15\n", "2020-09-17 08:45:20,678 epoch 21 - iter 15/154 - loss 0.63224182 - samples/sec: 501.04\n", "2020-09-17 08:45:21,634 epoch 21 - iter 30/154 - loss 0.55611975 - samples/sec: 518.48\n", "2020-09-17 08:45:22,551 epoch 21 - iter 45/154 - loss 0.54532969 - samples/sec: 549.88\n", "2020-09-17 08:45:23,504 epoch 21 - iter 60/154 - loss 0.55916429 - samples/sec: 521.16\n", "2020-09-17 08:45:24,428 epoch 21 - iter 75/154 - loss 0.57627438 - samples/sec: 535.54\n", "2020-09-17 08:45:25,434 epoch 21 - iter 90/154 - loss 0.58807621 - samples/sec: 493.80\n", "2020-09-17 08:45:26,355 epoch 21 - iter 105/154 - loss 0.57921984 - samples/sec: 539.11\n", "2020-09-17 08:45:27,298 epoch 21 - iter 120/154 - loss 0.57751538 - samples/sec: 528.16\n", "2020-09-17 08:45:28,207 epoch 21 - iter 135/154 - loss 0.57375883 - samples/sec: 553.00\n", "2020-09-17 08:45:29,123 epoch 21 - iter 150/154 - loss 0.57472525 - samples/sec: 542.78\n", "2020-09-17 08:45:29,300 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:45:29,302 EPOCH 21 done: loss 0.5743 - lr 0.1000\n", "2020-09-17 08:45:30,194 DEV : loss 0.5154196619987488 - score 0.7945\n", "2020-09-17 08:45:30,209 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:45:30,212 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:45:30,286 epoch 22 - iter 0/154 - loss 0.70903379 - samples/sec: 6667.66\n", "2020-09-17 08:45:31,220 epoch 22 - iter 15/154 - loss 0.54347884 - samples/sec: 533.84\n", "2020-09-17 08:45:32,201 epoch 22 - iter 30/154 - loss 0.57461896 - samples/sec: 504.73\n", "2020-09-17 08:45:33,103 epoch 22 - iter 45/154 - loss 0.56042972 - samples/sec: 552.41\n", "2020-09-17 08:45:34,004 epoch 22 - iter 60/154 - loss 0.55355490 - samples/sec: 555.84\n", "2020-09-17 08:45:34,983 epoch 22 - iter 75/154 - loss 0.56557354 - samples/sec: 506.74\n", "2020-09-17 08:45:35,894 epoch 22 - iter 90/154 - loss 0.56247158 - samples/sec: 545.28\n", "2020-09-17 08:45:36,881 epoch 22 - iter 105/154 - loss 0.56202839 - samples/sec: 516.09\n", "2020-09-17 08:45:37,797 epoch 22 - iter 120/154 - loss 0.56674196 - samples/sec: 543.46\n", "2020-09-17 08:45:38,735 epoch 22 - iter 135/154 - loss 0.57501798 - samples/sec: 531.00\n", "2020-09-17 08:45:39,712 epoch 22 - iter 150/154 - loss 0.57224388 - samples/sec: 513.98\n", "2020-09-17 08:45:39,901 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:45:39,902 EPOCH 22 done: loss 0.5721 - lr 0.1000\n", "2020-09-17 08:45:40,709 DEV : loss 0.6011821627616882 - score 0.789\n", "2020-09-17 08:45:40,722 BAD EPOCHS (no improvement): 4\n", "2020-09-17 08:45:40,724 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:45:40,789 epoch 23 - iter 0/154 - loss 0.60241961 - samples/sec: 7736.04\n", "2020-09-17 08:45:41,848 epoch 23 - iter 15/154 - loss 0.53316347 - samples/sec: 470.29\n", "2020-09-17 08:45:42,768 epoch 23 - iter 30/154 - loss 0.53839592 - samples/sec: 545.72\n", "2020-09-17 08:45:43,780 epoch 23 - iter 45/154 - loss 0.53638160 - samples/sec: 491.58\n", "2020-09-17 08:45:44,728 epoch 23 - iter 60/154 - loss 0.53681191 - samples/sec: 525.70\n", "2020-09-17 08:45:45,714 epoch 23 - iter 75/154 - loss 0.53815041 - samples/sec: 504.57\n", "2020-09-17 08:45:46,718 epoch 23 - iter 90/154 - loss 0.54550043 - samples/sec: 503.30\n", "2020-09-17 08:45:47,680 epoch 23 - iter 105/154 - loss 0.54051415 - samples/sec: 520.20\n", "2020-09-17 08:45:48,629 epoch 23 - iter 120/154 - loss 0.53782664 - samples/sec: 525.90\n", "2020-09-17 08:45:49,584 epoch 23 - iter 135/154 - loss 0.54638279 - samples/sec: 521.81\n", "2020-09-17 08:45:50,559 epoch 23 - iter 150/154 - loss 0.54693036 - samples/sec: 510.33\n", "2020-09-17 08:45:50,750 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:45:50,751 EPOCH 23 done: loss 0.5485 - lr 0.1000\n", "2020-09-17 08:45:51,588 DEV : loss 0.6499155759811401 - score 0.7523\n", "2020-09-17 08:45:51,607 BAD EPOCHS (no improvement): 5\n", "2020-09-17 08:45:51,609 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:45:51,698 epoch 24 - iter 0/154 - loss 0.83179027 - samples/sec: 5579.29\n", "2020-09-17 08:45:52,639 epoch 24 - iter 15/154 - loss 0.48986275 - samples/sec: 531.78\n", "2020-09-17 08:45:53,591 epoch 24 - iter 30/154 - loss 0.52403153 - samples/sec: 531.52\n", "2020-09-17 08:45:54,525 epoch 24 - iter 45/154 - loss 0.54648625 - samples/sec: 536.67\n", "2020-09-17 08:45:55,509 epoch 24 - iter 60/154 - loss 0.53876125 - samples/sec: 505.85\n", "2020-09-17 08:45:56,453 epoch 24 - iter 75/154 - loss 0.53964699 - samples/sec: 529.21\n", "2020-09-17 08:45:57,479 epoch 24 - iter 90/154 - loss 0.53238639 - samples/sec: 487.28\n", "2020-09-17 08:45:58,394 epoch 24 - iter 105/154 - loss 0.53403296 - samples/sec: 558.08\n", "2020-09-17 08:45:59,321 epoch 24 - iter 120/154 - loss 0.52870968 - samples/sec: 539.47\n", "2020-09-17 08:46:00,348 epoch 24 - iter 135/154 - loss 0.52469435 - samples/sec: 485.84\n", "2020-09-17 08:46:01,304 epoch 24 - iter 150/154 - loss 0.52845992 - samples/sec: 522.84\n", "2020-09-17 08:46:01,527 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:46:01,528 EPOCH 24 done: loss 0.5257 - lr 0.1000\n", "2020-09-17 08:46:02,441 DEV : loss 0.4620254337787628 - score 0.8312\n", "2020-09-17 08:46:02,456 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:46:06,630 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:46:06,720 epoch 25 - iter 0/154 - loss 0.55696088 - samples/sec: 5517.26\n", "2020-09-17 08:46:07,643 epoch 25 - iter 15/154 - loss 0.51809840 - samples/sec: 545.77\n", "2020-09-17 08:46:08,591 epoch 25 - iter 30/154 - loss 0.52472088 - samples/sec: 527.55\n", "2020-09-17 08:46:09,557 epoch 25 - iter 45/154 - loss 0.52192854 - samples/sec: 517.59\n", "2020-09-17 08:46:10,495 epoch 25 - iter 60/154 - loss 0.53030461 - samples/sec: 532.52\n", "2020-09-17 08:46:11,547 epoch 25 - iter 75/154 - loss 0.53951604 - samples/sec: 475.26\n", "2020-09-17 08:46:12,565 epoch 25 - iter 90/154 - loss 0.53338523 - samples/sec: 487.81\n", "2020-09-17 08:46:13,603 epoch 25 - iter 105/154 - loss 0.53237445 - samples/sec: 481.45\n", "2020-09-17 08:46:14,510 epoch 25 - iter 120/154 - loss 0.53026811 - samples/sec: 555.27\n", "2020-09-17 08:46:15,474 epoch 25 - iter 135/154 - loss 0.52988114 - samples/sec: 518.25\n", "2020-09-17 08:46:16,541 epoch 25 - iter 150/154 - loss 0.53818894 - samples/sec: 469.72\n", "2020-09-17 08:46:16,714 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:46:16,715 EPOCH 25 done: loss 0.5364 - lr 0.1000\n", "2020-09-17 08:46:17,487 DEV : loss 0.46257925033569336 - score 0.8257\n", "2020-09-17 08:46:17,500 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:46:17,502 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:46:17,559 epoch 26 - iter 0/154 - loss 0.45907670 - samples/sec: 8566.76\n", "2020-09-17 08:46:18,506 epoch 26 - iter 15/154 - loss 0.49223812 - samples/sec: 524.96\n", "2020-09-17 08:46:19,420 epoch 26 - iter 30/154 - loss 0.50107834 - samples/sec: 542.43\n", "2020-09-17 08:46:20,349 epoch 26 - iter 45/154 - loss 0.50700427 - samples/sec: 536.83\n", "2020-09-17 08:46:21,323 epoch 26 - iter 60/154 - loss 0.51780127 - samples/sec: 512.19\n", "2020-09-17 08:46:22,301 epoch 26 - iter 75/154 - loss 0.50829049 - samples/sec: 506.82\n", "2020-09-17 08:46:23,342 epoch 26 - iter 90/154 - loss 0.52077179 - samples/sec: 477.38\n", "2020-09-17 08:46:24,290 epoch 26 - iter 105/154 - loss 0.51301469 - samples/sec: 524.91\n", "2020-09-17 08:46:25,310 epoch 26 - iter 120/154 - loss 0.51020353 - samples/sec: 485.82\n", "2020-09-17 08:46:26,301 epoch 26 - iter 135/154 - loss 0.51511223 - samples/sec: 502.85\n", "2020-09-17 08:46:27,238 epoch 26 - iter 150/154 - loss 0.51234917 - samples/sec: 533.89\n", "2020-09-17 08:46:27,439 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:46:27,441 EPOCH 26 done: loss 0.5103 - lr 0.1000\n", "2020-09-17 08:46:28,261 DEV : loss 0.41604509949684143 - score 0.8404\n", "2020-09-17 08:46:28,276 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:46:32,937 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:46:32,999 epoch 27 - iter 0/154 - loss 0.69577843 - samples/sec: 8000.10\n", "2020-09-17 08:46:33,980 epoch 27 - iter 15/154 - loss 0.49832411 - samples/sec: 503.35\n", "2020-09-17 08:46:35,019 epoch 27 - iter 30/154 - loss 0.51707672 - samples/sec: 480.23\n", "2020-09-17 08:46:35,977 epoch 27 - iter 45/154 - loss 0.50164674 - samples/sec: 521.42\n", "2020-09-17 08:46:36,966 epoch 27 - iter 60/154 - loss 0.49837460 - samples/sec: 501.57\n", "2020-09-17 08:46:37,954 epoch 27 - iter 75/154 - loss 0.50330281 - samples/sec: 505.14\n", "2020-09-17 08:46:38,989 epoch 27 - iter 90/154 - loss 0.50162993 - samples/sec: 478.09\n", "2020-09-17 08:46:39,939 epoch 27 - iter 105/154 - loss 0.50041539 - samples/sec: 530.16\n", "2020-09-17 08:46:40,941 epoch 27 - iter 120/154 - loss 0.49140836 - samples/sec: 494.41\n", "2020-09-17 08:46:41,991 epoch 27 - iter 135/154 - loss 0.49350700 - samples/sec: 472.99\n", "2020-09-17 08:46:43,133 epoch 27 - iter 150/154 - loss 0.49069217 - samples/sec: 446.34\n", "2020-09-17 08:46:43,328 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:46:43,329 EPOCH 27 done: loss 0.4867 - lr 0.1000\n", "2020-09-17 08:46:44,325 DEV : loss 0.433791846036911 - score 0.8367\n", "2020-09-17 08:46:44,340 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:46:44,342 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:46:44,402 epoch 28 - iter 0/154 - loss 0.44624346 - samples/sec: 8136.02\n", "2020-09-17 08:46:45,499 epoch 28 - iter 15/154 - loss 0.51238604 - samples/sec: 453.30\n", "2020-09-17 08:46:46,585 epoch 28 - iter 30/154 - loss 0.48322163 - samples/sec: 459.31\n", "2020-09-17 08:46:47,645 epoch 28 - iter 45/154 - loss 0.48184486 - samples/sec: 466.84\n", "2020-09-17 08:46:48,672 epoch 28 - iter 60/154 - loss 0.48179187 - samples/sec: 481.93\n", "2020-09-17 08:46:49,706 epoch 28 - iter 75/154 - loss 0.49155031 - samples/sec: 479.04\n", "2020-09-17 08:46:50,779 epoch 28 - iter 90/154 - loss 0.49598998 - samples/sec: 465.12\n", "2020-09-17 08:46:51,748 epoch 28 - iter 105/154 - loss 0.49363552 - samples/sec: 520.49\n", "2020-09-17 08:46:52,694 epoch 28 - iter 120/154 - loss 0.49700043 - samples/sec: 525.98\n", "2020-09-17 08:46:53,667 epoch 28 - iter 135/154 - loss 0.49334761 - samples/sec: 510.08\n", "2020-09-17 08:46:54,609 epoch 28 - iter 150/154 - loss 0.49557419 - samples/sec: 532.94\n", "2020-09-17 08:46:54,791 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:46:54,793 EPOCH 28 done: loss 0.4935 - lr 0.1000\n", "2020-09-17 08:46:55,637 DEV : loss 0.43277764320373535 - score 0.8294\n", "2020-09-17 08:46:55,650 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:46:55,652 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:46:55,716 epoch 29 - iter 0/154 - loss 0.40680206 - samples/sec: 7741.78\n", "2020-09-17 08:46:56,641 epoch 29 - iter 15/154 - loss 0.44022097 - samples/sec: 537.52\n", "2020-09-17 08:46:57,561 epoch 29 - iter 30/154 - loss 0.47939367 - samples/sec: 541.17\n", "2020-09-17 08:46:58,498 epoch 29 - iter 45/154 - loss 0.48069310 - samples/sec: 536.69\n", "2020-09-17 08:46:59,394 epoch 29 - iter 60/154 - loss 0.49674024 - samples/sec: 555.50\n", "2020-09-17 08:47:00,391 epoch 29 - iter 75/154 - loss 0.50233355 - samples/sec: 496.94\n", "2020-09-17 08:47:01,304 epoch 29 - iter 90/154 - loss 0.49602299 - samples/sec: 545.64\n", "2020-09-17 08:47:02,224 epoch 29 - iter 105/154 - loss 0.48464133 - samples/sec: 540.71\n", "2020-09-17 08:47:03,151 epoch 29 - iter 120/154 - loss 0.48655941 - samples/sec: 542.55\n", "2020-09-17 08:47:04,095 epoch 29 - iter 135/154 - loss 0.48840677 - samples/sec: 526.18\n", "2020-09-17 08:47:05,051 epoch 29 - iter 150/154 - loss 0.47931675 - samples/sec: 520.01\n", "2020-09-17 08:47:05,241 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:47:05,242 EPOCH 29 done: loss 0.4785 - lr 0.1000\n", "2020-09-17 08:47:05,998 DEV : loss 0.546108067035675 - score 0.8092\n", "2020-09-17 08:47:06,013 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:47:06,015 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:47:06,080 epoch 30 - iter 0/154 - loss 0.81846243 - samples/sec: 7619.60\n", "2020-09-17 08:47:07,066 epoch 30 - iter 15/154 - loss 0.55593885 - samples/sec: 503.98\n", "2020-09-17 08:47:07,991 epoch 30 - iter 30/154 - loss 0.51351806 - samples/sec: 537.66\n", "2020-09-17 08:47:09,065 epoch 30 - iter 45/154 - loss 0.51211901 - samples/sec: 460.90\n", "2020-09-17 08:47:10,120 epoch 30 - iter 60/154 - loss 0.50305846 - samples/sec: 473.40\n", "2020-09-17 08:47:11,077 epoch 30 - iter 75/154 - loss 0.49925294 - samples/sec: 520.60\n", "2020-09-17 08:47:12,050 epoch 30 - iter 90/154 - loss 0.48953974 - samples/sec: 511.22\n", "2020-09-17 08:47:12,967 epoch 30 - iter 105/154 - loss 0.47788822 - samples/sec: 540.60\n", "2020-09-17 08:47:14,005 epoch 30 - iter 120/154 - loss 0.47886987 - samples/sec: 475.48\n", "2020-09-17 08:47:14,943 epoch 30 - iter 135/154 - loss 0.47788799 - samples/sec: 530.41\n", "2020-09-17 08:47:15,954 epoch 30 - iter 150/154 - loss 0.47735038 - samples/sec: 491.37\n", "2020-09-17 08:47:16,219 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:47:16,220 EPOCH 30 done: loss 0.4789 - lr 0.1000\n", "2020-09-17 08:47:17,128 DEV : loss 0.49495548009872437 - score 0.8202\n", "2020-09-17 08:47:17,141 BAD EPOCHS (no improvement): 4\n", "2020-09-17 08:47:17,143 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:47:17,219 epoch 31 - iter 0/154 - loss 0.49845910 - samples/sec: 6487.86\n", "2020-09-17 08:47:18,454 epoch 31 - iter 15/154 - loss 0.47123308 - samples/sec: 407.48\n", "2020-09-17 08:47:19,545 epoch 31 - iter 30/154 - loss 0.50877005 - samples/sec: 468.27\n", "2020-09-17 08:47:20,435 epoch 31 - iter 45/154 - loss 0.48678339 - samples/sec: 560.66\n", "2020-09-17 08:47:21,484 epoch 31 - iter 60/154 - loss 0.47714004 - samples/sec: 471.94\n", "2020-09-17 08:47:22,587 epoch 31 - iter 75/154 - loss 0.48625103 - samples/sec: 449.13\n", "2020-09-17 08:47:23,588 epoch 31 - iter 90/154 - loss 0.47073268 - samples/sec: 496.90\n", "2020-09-17 08:47:24,479 epoch 31 - iter 105/154 - loss 0.47550846 - samples/sec: 557.97\n", "2020-09-17 08:47:25,413 epoch 31 - iter 120/154 - loss 0.46906275 - samples/sec: 532.07\n", "2020-09-17 08:47:26,365 epoch 31 - iter 135/154 - loss 0.46213152 - samples/sec: 528.16\n", "2020-09-17 08:47:27,294 epoch 31 - iter 150/154 - loss 0.46191752 - samples/sec: 536.33\n", "2020-09-17 08:47:27,469 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:47:27,470 EPOCH 31 done: loss 0.4618 - lr 0.1000\n", "2020-09-17 08:47:28,299 DEV : loss 0.4721139967441559 - score 0.844\n", "2020-09-17 08:47:28,313 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:47:32,630 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:47:32,691 epoch 32 - iter 0/154 - loss 0.54501790 - samples/sec: 8136.08\n", "2020-09-17 08:47:33,617 epoch 32 - iter 15/154 - loss 0.44498506 - samples/sec: 535.92\n", "2020-09-17 08:47:34,522 epoch 32 - iter 30/154 - loss 0.43836637 - samples/sec: 549.71\n", "2020-09-17 08:47:35,491 epoch 32 - iter 45/154 - loss 0.43942052 - samples/sec: 512.82\n", "2020-09-17 08:47:36,404 epoch 32 - iter 60/154 - loss 0.43968216 - samples/sec: 546.25\n", "2020-09-17 08:47:37,402 epoch 32 - iter 75/154 - loss 0.42970712 - samples/sec: 495.79\n", "2020-09-17 08:47:38,358 epoch 32 - iter 90/154 - loss 0.44014940 - samples/sec: 518.78\n", "2020-09-17 08:47:39,349 epoch 32 - iter 105/154 - loss 0.44215414 - samples/sec: 500.01\n", "2020-09-17 08:47:40,241 epoch 32 - iter 120/154 - loss 0.44504288 - samples/sec: 563.30\n", "2020-09-17 08:47:41,202 epoch 32 - iter 135/154 - loss 0.44061964 - samples/sec: 521.19\n", "2020-09-17 08:47:42,178 epoch 32 - iter 150/154 - loss 0.44158221 - samples/sec: 507.95\n", "2020-09-17 08:47:42,359 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:47:42,360 EPOCH 32 done: loss 0.4445 - lr 0.1000\n", "2020-09-17 08:47:43,117 DEV : loss 0.4730919897556305 - score 0.8385\n", "2020-09-17 08:47:43,130 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:47:43,131 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:47:43,193 epoch 33 - iter 0/154 - loss 0.43239948 - samples/sec: 8003.03\n", "2020-09-17 08:47:44,172 epoch 33 - iter 15/154 - loss 0.41870546 - samples/sec: 507.41\n", "2020-09-17 08:47:45,086 epoch 33 - iter 30/154 - loss 0.44082512 - samples/sec: 546.33\n", "2020-09-17 08:47:46,012 epoch 33 - iter 45/154 - loss 0.43864475 - samples/sec: 535.01\n", "2020-09-17 08:47:46,970 epoch 33 - iter 60/154 - loss 0.43275807 - samples/sec: 519.47\n", "2020-09-17 08:47:47,910 epoch 33 - iter 75/154 - loss 0.43301847 - samples/sec: 528.90\n", "2020-09-17 08:47:48,901 epoch 33 - iter 90/154 - loss 0.42551151 - samples/sec: 502.44\n", "2020-09-17 08:47:49,819 epoch 33 - iter 105/154 - loss 0.43756980 - samples/sec: 541.72\n", "2020-09-17 08:47:50,844 epoch 33 - iter 120/154 - loss 0.44410921 - samples/sec: 484.11\n", "2020-09-17 08:47:51,962 epoch 33 - iter 135/154 - loss 0.44772948 - samples/sec: 451.13\n", "2020-09-17 08:47:52,964 epoch 33 - iter 150/154 - loss 0.44790418 - samples/sec: 493.83\n", "2020-09-17 08:47:53,138 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:47:53,139 EPOCH 33 done: loss 0.4506 - lr 0.1000\n", "2020-09-17 08:47:54,011 DEV : loss 0.468923956155777 - score 0.822\n", "2020-09-17 08:47:54,024 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:47:54,026 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:47:54,082 epoch 34 - iter 0/154 - loss 0.44897088 - samples/sec: 9050.09\n", "2020-09-17 08:47:55,177 epoch 34 - iter 15/154 - loss 0.44007531 - samples/sec: 451.98\n", "2020-09-17 08:47:56,201 epoch 34 - iter 30/154 - loss 0.43833273 - samples/sec: 486.77\n", "2020-09-17 08:47:57,172 epoch 34 - iter 45/154 - loss 0.44410494 - samples/sec: 517.34\n", "2020-09-17 08:47:58,183 epoch 34 - iter 60/154 - loss 0.43993343 - samples/sec: 492.88\n", "2020-09-17 08:47:59,129 epoch 34 - iter 75/154 - loss 0.43784125 - samples/sec: 525.57\n", "2020-09-17 08:48:00,065 epoch 34 - iter 90/154 - loss 0.43374302 - samples/sec: 530.01\n", "2020-09-17 08:48:01,078 epoch 34 - iter 105/154 - loss 0.44217296 - samples/sec: 489.85\n", "2020-09-17 08:48:02,117 epoch 34 - iter 120/154 - loss 0.44522162 - samples/sec: 478.56\n", "2020-09-17 08:48:03,184 epoch 34 - iter 135/154 - loss 0.44969685 - samples/sec: 465.13\n", "2020-09-17 08:48:04,064 epoch 34 - iter 150/154 - loss 0.44260877 - samples/sec: 564.70\n", "2020-09-17 08:48:04,230 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:48:04,231 EPOCH 34 done: loss 0.4433 - lr 0.1000\n", "2020-09-17 08:48:05,087 DEV : loss 0.4542025625705719 - score 0.8294\n", "2020-09-17 08:48:05,104 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:48:05,106 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:48:05,184 epoch 35 - iter 0/154 - loss 0.57633495 - samples/sec: 6316.13\n", "2020-09-17 08:48:06,062 epoch 35 - iter 15/154 - loss 0.43384211 - samples/sec: 570.25\n", "2020-09-17 08:48:06,976 epoch 35 - iter 30/154 - loss 0.47151881 - samples/sec: 543.61\n", "2020-09-17 08:48:07,937 epoch 35 - iter 45/154 - loss 0.46743700 - samples/sec: 515.09\n", "2020-09-17 08:48:08,906 epoch 35 - iter 60/154 - loss 0.45566785 - samples/sec: 514.59\n", "2020-09-17 08:48:09,872 epoch 35 - iter 75/154 - loss 0.45172675 - samples/sec: 512.00\n", "2020-09-17 08:48:10,832 epoch 35 - iter 90/154 - loss 0.44153968 - samples/sec: 519.00\n", "2020-09-17 08:48:11,764 epoch 35 - iter 105/154 - loss 0.43877675 - samples/sec: 537.88\n", "2020-09-17 08:48:12,747 epoch 35 - iter 120/154 - loss 0.43347406 - samples/sec: 502.98\n", "2020-09-17 08:48:13,684 epoch 35 - iter 135/154 - loss 0.43915027 - samples/sec: 530.69\n", "2020-09-17 08:48:14,662 epoch 35 - iter 150/154 - loss 0.43874495 - samples/sec: 505.30\n", "2020-09-17 08:48:14,871 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:48:14,873 EPOCH 35 done: loss 0.4417 - lr 0.1000\n", "2020-09-17 08:48:15,663 DEV : loss 0.39028945565223694 - score 0.8587\n", "2020-09-17 08:48:15,683 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:48:19,871 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:48:19,931 epoch 36 - iter 0/154 - loss 0.38922396 - samples/sec: 8278.37\n", "2020-09-17 08:48:20,856 epoch 36 - iter 15/154 - loss 0.40600947 - samples/sec: 536.50\n", "2020-09-17 08:48:21,838 epoch 36 - iter 30/154 - loss 0.38329836 - samples/sec: 506.83\n", "2020-09-17 08:48:22,771 epoch 36 - iter 45/154 - loss 0.39186755 - samples/sec: 531.60\n", "2020-09-17 08:48:23,736 epoch 36 - iter 60/154 - loss 0.38678852 - samples/sec: 512.66\n", "2020-09-17 08:48:24,643 epoch 36 - iter 75/154 - loss 0.39947429 - samples/sec: 547.89\n", "2020-09-17 08:48:25,571 epoch 36 - iter 90/154 - loss 0.42013099 - samples/sec: 536.13\n", "2020-09-17 08:48:26,518 epoch 36 - iter 105/154 - loss 0.42182017 - samples/sec: 532.35\n", "2020-09-17 08:48:27,449 epoch 36 - iter 120/154 - loss 0.41898657 - samples/sec: 533.03\n", "2020-09-17 08:48:28,393 epoch 36 - iter 135/154 - loss 0.42142977 - samples/sec: 525.96\n", "2020-09-17 08:48:29,275 epoch 36 - iter 150/154 - loss 0.42159840 - samples/sec: 564.33\n", "2020-09-17 08:48:29,459 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:48:29,460 EPOCH 36 done: loss 0.4220 - lr 0.1000\n", "2020-09-17 08:48:30,219 DEV : loss 0.37047436833381653 - score 0.8606\n", "2020-09-17 08:48:30,246 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:48:34,478 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:48:34,538 epoch 37 - iter 0/154 - loss 0.29895619 - samples/sec: 8136.25\n", "2020-09-17 08:48:35,491 epoch 37 - iter 15/154 - loss 0.45863818 - samples/sec: 521.80\n", "2020-09-17 08:48:36,416 epoch 37 - iter 30/154 - loss 0.45950284 - samples/sec: 538.47\n", "2020-09-17 08:48:37,416 epoch 37 - iter 45/154 - loss 0.44264391 - samples/sec: 494.64\n", "2020-09-17 08:48:38,319 epoch 37 - iter 60/154 - loss 0.43508944 - samples/sec: 550.58\n", "2020-09-17 08:48:39,215 epoch 37 - iter 75/154 - loss 0.42580054 - samples/sec: 556.49\n", "2020-09-17 08:48:40,229 epoch 37 - iter 90/154 - loss 0.43280298 - samples/sec: 488.24\n", "2020-09-17 08:48:41,386 epoch 37 - iter 105/154 - loss 0.43072277 - samples/sec: 430.09\n", "2020-09-17 08:48:42,388 epoch 37 - iter 120/154 - loss 0.43090597 - samples/sec: 500.00\n", "2020-09-17 08:48:43,407 epoch 37 - iter 135/154 - loss 0.43067498 - samples/sec: 487.36\n", "2020-09-17 08:48:44,467 epoch 37 - iter 150/154 - loss 0.42474392 - samples/sec: 467.99\n", "2020-09-17 08:48:44,689 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:48:44,690 EPOCH 37 done: loss 0.4241 - lr 0.1000\n", "2020-09-17 08:48:45,566 DEV : loss 0.42703256011009216 - score 0.8385\n", "2020-09-17 08:48:45,581 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:48:45,582 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:48:45,650 epoch 38 - iter 0/154 - loss 0.39214289 - samples/sec: 7272.70\n", "2020-09-17 08:48:46,688 epoch 38 - iter 15/154 - loss 0.35143316 - samples/sec: 478.53\n", "2020-09-17 08:48:47,634 epoch 38 - iter 30/154 - loss 0.39467813 - samples/sec: 525.57\n", "2020-09-17 08:48:48,647 epoch 38 - iter 45/154 - loss 0.40601419 - samples/sec: 489.30\n", "2020-09-17 08:48:49,901 epoch 38 - iter 60/154 - loss 0.39878281 - samples/sec: 395.72\n", "2020-09-17 08:48:51,198 epoch 38 - iter 75/154 - loss 0.39802152 - samples/sec: 388.67\n", "2020-09-17 08:48:52,414 epoch 38 - iter 90/154 - loss 0.40220144 - samples/sec: 407.81\n", "2020-09-17 08:48:53,533 epoch 38 - iter 105/154 - loss 0.40309720 - samples/sec: 449.86\n", "2020-09-17 08:48:54,771 epoch 38 - iter 120/154 - loss 0.40142922 - samples/sec: 403.19\n", "2020-09-17 08:48:55,787 epoch 38 - iter 135/154 - loss 0.40133742 - samples/sec: 489.29\n", "2020-09-17 08:48:56,774 epoch 38 - iter 150/154 - loss 0.40275137 - samples/sec: 503.26\n", "2020-09-17 08:48:56,950 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:48:56,952 EPOCH 38 done: loss 0.4041 - lr 0.1000\n", "2020-09-17 08:48:57,729 DEV : loss 0.4972012937068939 - score 0.8202\n", "2020-09-17 08:48:57,742 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:48:57,744 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:48:57,808 epoch 39 - iter 0/154 - loss 0.36125883 - samples/sec: 7869.18\n", "2020-09-17 08:48:58,797 epoch 39 - iter 15/154 - loss 0.40315570 - samples/sec: 501.05\n", "2020-09-17 08:48:59,740 epoch 39 - iter 30/154 - loss 0.38752554 - samples/sec: 527.17\n", "2020-09-17 08:49:00,675 epoch 39 - iter 45/154 - loss 0.40196277 - samples/sec: 531.29\n", "2020-09-17 08:49:01,628 epoch 39 - iter 60/154 - loss 0.39565423 - samples/sec: 521.70\n", "2020-09-17 08:49:02,621 epoch 39 - iter 75/154 - loss 0.39338529 - samples/sec: 499.09\n", "2020-09-17 08:49:03,616 epoch 39 - iter 90/154 - loss 0.39469775 - samples/sec: 505.26\n", "2020-09-17 08:49:04,600 epoch 39 - iter 105/154 - loss 0.38963491 - samples/sec: 506.10\n", "2020-09-17 08:49:05,575 epoch 39 - iter 120/154 - loss 0.39312212 - samples/sec: 508.47\n", "2020-09-17 08:49:06,493 epoch 39 - iter 135/154 - loss 0.39499878 - samples/sec: 540.83\n", "2020-09-17 08:49:07,405 epoch 39 - iter 150/154 - loss 0.39413205 - samples/sec: 547.57\n", "2020-09-17 08:49:07,607 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:49:07,608 EPOCH 39 done: loss 0.3917 - lr 0.1000\n", "2020-09-17 08:49:08,391 DEV : loss 0.3936179280281067 - score 0.8606\n", "2020-09-17 08:49:08,405 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:49:13,131 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:49:13,200 epoch 40 - iter 0/154 - loss 0.41467845 - samples/sec: 7165.51\n", "2020-09-17 08:49:14,224 epoch 40 - iter 15/154 - loss 0.39588843 - samples/sec: 524.79\n", "2020-09-17 08:49:15,248 epoch 40 - iter 30/154 - loss 0.39132004 - samples/sec: 482.54\n", "2020-09-17 08:49:16,144 epoch 40 - iter 45/154 - loss 0.40803963 - samples/sec: 555.66\n", "2020-09-17 08:49:17,144 epoch 40 - iter 60/154 - loss 0.39573950 - samples/sec: 494.34\n", "2020-09-17 08:49:18,069 epoch 40 - iter 75/154 - loss 0.40052708 - samples/sec: 536.32\n", "2020-09-17 08:49:19,019 epoch 40 - iter 90/154 - loss 0.40529449 - samples/sec: 522.88\n", "2020-09-17 08:49:19,993 epoch 40 - iter 105/154 - loss 0.40905478 - samples/sec: 509.52\n", "2020-09-17 08:49:20,882 epoch 40 - iter 120/154 - loss 0.40924649 - samples/sec: 558.94\n", "2020-09-17 08:49:21,863 epoch 40 - iter 135/154 - loss 0.40589784 - samples/sec: 506.34\n", "2020-09-17 08:49:22,802 epoch 40 - iter 150/154 - loss 0.40904066 - samples/sec: 530.95\n", "2020-09-17 08:49:22,983 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:49:22,985 EPOCH 40 done: loss 0.4066 - lr 0.1000\n", "2020-09-17 08:49:23,749 DEV : loss 0.4119681417942047 - score 0.8532\n", "2020-09-17 08:49:23,766 BAD EPOCHS (no improvement): 4\n", "2020-09-17 08:49:23,769 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:49:23,843 epoch 41 - iter 0/154 - loss 0.38837168 - samples/sec: 6667.77\n", "2020-09-17 08:49:24,781 epoch 41 - iter 15/154 - loss 0.36806412 - samples/sec: 533.92\n", "2020-09-17 08:49:25,698 epoch 41 - iter 30/154 - loss 0.36472470 - samples/sec: 540.83\n", "2020-09-17 08:49:26,687 epoch 41 - iter 45/154 - loss 0.39301705 - samples/sec: 499.87\n", "2020-09-17 08:49:27,606 epoch 41 - iter 60/154 - loss 0.39009325 - samples/sec: 542.95\n", "2020-09-17 08:49:28,670 epoch 41 - iter 75/154 - loss 0.38945091 - samples/sec: 466.91\n", "2020-09-17 08:49:29,589 epoch 41 - iter 90/154 - loss 0.39189105 - samples/sec: 551.67\n", "2020-09-17 08:49:30,498 epoch 41 - iter 105/154 - loss 0.39144634 - samples/sec: 546.68\n", "2020-09-17 08:49:31,484 epoch 41 - iter 120/154 - loss 0.39374552 - samples/sec: 504.71\n", "2020-09-17 08:49:32,460 epoch 41 - iter 135/154 - loss 0.38782135 - samples/sec: 506.86\n", "2020-09-17 08:49:33,441 epoch 41 - iter 150/154 - loss 0.38928701 - samples/sec: 505.78\n", "2020-09-17 08:49:33,617 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:49:33,618 EPOCH 41 done: loss 0.3880 - lr 0.1000\n", "2020-09-17 08:49:34,371 DEV : loss 0.3772420883178711 - score 0.8697\n", "2020-09-17 08:49:34,384 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:49:38,764 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:49:38,828 epoch 42 - iter 0/154 - loss 0.52248478 - samples/sec: 7620.35\n", "2020-09-17 08:49:39,764 epoch 42 - iter 15/154 - loss 0.41299151 - samples/sec: 532.94\n", "2020-09-17 08:49:40,787 epoch 42 - iter 30/154 - loss 0.40359560 - samples/sec: 483.26\n", "2020-09-17 08:49:41,745 epoch 42 - iter 45/154 - loss 0.38539715 - samples/sec: 519.46\n", "2020-09-17 08:49:42,738 epoch 42 - iter 60/154 - loss 0.39350714 - samples/sec: 497.93\n", "2020-09-17 08:49:43,631 epoch 42 - iter 75/154 - loss 0.39461647 - samples/sec: 555.52\n", "2020-09-17 08:49:44,549 epoch 42 - iter 90/154 - loss 0.39094642 - samples/sec: 540.85\n", "2020-09-17 08:49:45,476 epoch 42 - iter 105/154 - loss 0.39472459 - samples/sec: 541.76\n", "2020-09-17 08:49:46,379 epoch 42 - iter 120/154 - loss 0.39737605 - samples/sec: 549.07\n", "2020-09-17 08:49:47,337 epoch 42 - iter 135/154 - loss 0.39712915 - samples/sec: 516.69\n", "2020-09-17 08:49:48,249 epoch 42 - iter 150/154 - loss 0.39275896 - samples/sec: 544.94\n", "2020-09-17 08:49:48,427 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:49:48,427 EPOCH 42 done: loss 0.3930 - lr 0.1000\n", "2020-09-17 08:49:49,214 DEV : loss 0.43211784958839417 - score 0.8422\n", "2020-09-17 08:49:49,231 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:49:49,233 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:49:49,309 epoch 43 - iter 0/154 - loss 0.43500471 - samples/sec: 6575.56\n", "2020-09-17 08:49:50,260 epoch 43 - iter 15/154 - loss 0.40754305 - samples/sec: 527.54\n", "2020-09-17 08:49:51,211 epoch 43 - iter 30/154 - loss 0.40379776 - samples/sec: 524.19\n", "2020-09-17 08:49:52,213 epoch 43 - iter 45/154 - loss 0.39281249 - samples/sec: 494.43\n", "2020-09-17 08:49:53,179 epoch 43 - iter 60/154 - loss 0.39351714 - samples/sec: 517.52\n", "2020-09-17 08:49:54,115 epoch 43 - iter 75/154 - loss 0.39931669 - samples/sec: 530.37\n", "2020-09-17 08:49:55,075 epoch 43 - iter 90/154 - loss 0.39565854 - samples/sec: 517.06\n", "2020-09-17 08:49:56,025 epoch 43 - iter 105/154 - loss 0.38506051 - samples/sec: 521.17\n", "2020-09-17 08:49:57,038 epoch 43 - iter 120/154 - loss 0.38291511 - samples/sec: 493.88\n", "2020-09-17 08:49:57,966 epoch 43 - iter 135/154 - loss 0.38275040 - samples/sec: 536.60\n", "2020-09-17 08:49:58,927 epoch 43 - iter 150/154 - loss 0.38717832 - samples/sec: 513.90\n", "2020-09-17 08:49:59,096 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:49:59,098 EPOCH 43 done: loss 0.3848 - lr 0.1000\n", "2020-09-17 08:49:59,861 DEV : loss 0.4324362576007843 - score 0.8569\n", "2020-09-17 08:49:59,874 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:49:59,876 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:49:59,943 epoch 44 - iter 0/154 - loss 0.43254471 - samples/sec: 7281.57\n", "2020-09-17 08:50:00,911 epoch 44 - iter 15/154 - loss 0.43415625 - samples/sec: 513.23\n", "2020-09-17 08:50:01,839 epoch 44 - iter 30/154 - loss 0.39839831 - samples/sec: 537.73\n", "2020-09-17 08:50:02,741 epoch 44 - iter 45/154 - loss 0.36987896 - samples/sec: 551.81\n", "2020-09-17 08:50:03,746 epoch 44 - iter 60/154 - loss 0.35972939 - samples/sec: 495.87\n", "2020-09-17 08:50:04,779 epoch 44 - iter 75/154 - loss 0.36069147 - samples/sec: 479.80\n", "2020-09-17 08:50:05,798 epoch 44 - iter 90/154 - loss 0.36080223 - samples/sec: 485.82\n", "2020-09-17 08:50:06,724 epoch 44 - iter 105/154 - loss 0.37354131 - samples/sec: 537.50\n", "2020-09-17 08:50:07,647 epoch 44 - iter 120/154 - loss 0.37080982 - samples/sec: 536.91\n", "2020-09-17 08:50:08,631 epoch 44 - iter 135/154 - loss 0.37207173 - samples/sec: 509.02\n", "2020-09-17 08:50:09,596 epoch 44 - iter 150/154 - loss 0.36956299 - samples/sec: 518.64\n", "2020-09-17 08:50:09,770 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:50:09,771 EPOCH 44 done: loss 0.3732 - lr 0.1000\n", "2020-09-17 08:50:10,612 DEV : loss 0.3915727734565735 - score 0.855\n", "2020-09-17 08:50:10,631 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:50:10,633 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:50:10,699 epoch 45 - iter 0/154 - loss 0.60903239 - samples/sec: 7617.15\n", "2020-09-17 08:50:11,642 epoch 45 - iter 15/154 - loss 0.38437290 - samples/sec: 528.26\n", "2020-09-17 08:50:12,640 epoch 45 - iter 30/154 - loss 0.37751857 - samples/sec: 497.15\n", "2020-09-17 08:50:13,559 epoch 45 - iter 45/154 - loss 0.36982405 - samples/sec: 540.35\n", "2020-09-17 08:50:14,447 epoch 45 - iter 60/154 - loss 0.35534891 - samples/sec: 561.73\n", "2020-09-17 08:50:15,405 epoch 45 - iter 75/154 - loss 0.36877875 - samples/sec: 517.18\n", "2020-09-17 08:50:16,325 epoch 45 - iter 90/154 - loss 0.37239165 - samples/sec: 542.89\n", "2020-09-17 08:50:17,292 epoch 45 - iter 105/154 - loss 0.37295442 - samples/sec: 512.46\n", "2020-09-17 08:50:18,173 epoch 45 - iter 120/154 - loss 0.37229287 - samples/sec: 565.35\n", "2020-09-17 08:50:19,078 epoch 45 - iter 135/154 - loss 0.36850916 - samples/sec: 547.93\n", "2020-09-17 08:50:20,059 epoch 45 - iter 150/154 - loss 0.36866490 - samples/sec: 506.70\n", "2020-09-17 08:50:20,248 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:50:20,249 EPOCH 45 done: loss 0.3675 - lr 0.1000\n", "2020-09-17 08:50:21,039 DEV : loss 0.378752738237381 - score 0.8679\n", "2020-09-17 08:50:21,054 BAD EPOCHS (no improvement): 4\n", "2020-09-17 08:50:21,056 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:50:21,122 epoch 46 - iter 0/154 - loss 0.30052584 - samples/sec: 7381.68\n", "2020-09-17 08:50:22,114 epoch 46 - iter 15/154 - loss 0.30641786 - samples/sec: 504.20\n", "2020-09-17 08:50:23,023 epoch 46 - iter 30/154 - loss 0.37171853 - samples/sec: 546.70\n", "2020-09-17 08:50:23,958 epoch 46 - iter 45/154 - loss 0.37412933 - samples/sec: 531.29\n", "2020-09-17 08:50:24,869 epoch 46 - iter 60/154 - loss 0.37437653 - samples/sec: 549.23\n", "2020-09-17 08:50:25,806 epoch 46 - iter 75/154 - loss 0.37392916 - samples/sec: 530.20\n", "2020-09-17 08:50:26,845 epoch 46 - iter 90/154 - loss 0.36314499 - samples/sec: 502.10\n", "2020-09-17 08:50:27,790 epoch 46 - iter 105/154 - loss 0.35674683 - samples/sec: 524.53\n", "2020-09-17 08:50:28,787 epoch 46 - iter 120/154 - loss 0.35358607 - samples/sec: 495.05\n", "2020-09-17 08:50:29,709 epoch 46 - iter 135/154 - loss 0.35964300 - samples/sec: 540.10\n", "2020-09-17 08:50:30,661 epoch 46 - iter 150/154 - loss 0.35930149 - samples/sec: 525.16\n", "2020-09-17 08:50:30,881 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:50:30,882 EPOCH 46 done: loss 0.3577 - lr 0.1000\n", "2020-09-17 08:50:31,670 DEV : loss 0.39096799492836 - score 0.8734\n", "2020-09-17 08:50:31,683 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:50:36,800 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:50:36,859 epoch 47 - iter 0/154 - loss 0.46727926 - samples/sec: 8269.90\n", "2020-09-17 08:50:37,907 epoch 47 - iter 15/154 - loss 0.37793587 - samples/sec: 471.51\n", "2020-09-17 08:50:38,873 epoch 47 - iter 30/154 - loss 0.35528016 - samples/sec: 515.01\n", "2020-09-17 08:50:39,792 epoch 47 - iter 45/154 - loss 0.35359808 - samples/sec: 541.30\n", "2020-09-17 08:50:40,837 epoch 47 - iter 60/154 - loss 0.35843113 - samples/sec: 471.95\n", "2020-09-17 08:50:41,764 epoch 47 - iter 75/154 - loss 0.35125349 - samples/sec: 537.01\n", "2020-09-17 08:50:42,718 epoch 47 - iter 90/154 - loss 0.34349647 - samples/sec: 519.32\n", "2020-09-17 08:50:43,657 epoch 47 - iter 105/154 - loss 0.34915640 - samples/sec: 529.14\n", "2020-09-17 08:50:44,610 epoch 47 - iter 120/154 - loss 0.35038980 - samples/sec: 521.79\n", "2020-09-17 08:50:45,550 epoch 47 - iter 135/154 - loss 0.35368398 - samples/sec: 535.69\n", "2020-09-17 08:50:46,465 epoch 47 - iter 150/154 - loss 0.35238965 - samples/sec: 542.97\n", "2020-09-17 08:50:46,643 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:50:46,644 EPOCH 47 done: loss 0.3491 - lr 0.1000\n", "2020-09-17 08:50:47,462 DEV : loss 0.4490405321121216 - score 0.8569\n", "2020-09-17 08:50:47,478 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:50:47,480 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:50:47,542 epoch 48 - iter 0/154 - loss 0.14084174 - samples/sec: 7871.45\n", "2020-09-17 08:50:48,439 epoch 48 - iter 15/154 - loss 0.36163055 - samples/sec: 554.15\n", "2020-09-17 08:50:49,402 epoch 48 - iter 30/154 - loss 0.39187807 - samples/sec: 517.25\n", "2020-09-17 08:50:50,321 epoch 48 - iter 45/154 - loss 0.37210059 - samples/sec: 543.54\n", "2020-09-17 08:50:51,284 epoch 48 - iter 60/154 - loss 0.37698173 - samples/sec: 514.08\n", "2020-09-17 08:50:52,332 epoch 48 - iter 75/154 - loss 0.37332328 - samples/sec: 475.23\n", "2020-09-17 08:50:53,327 epoch 48 - iter 90/154 - loss 0.36563177 - samples/sec: 497.92\n", "2020-09-17 08:50:54,249 epoch 48 - iter 105/154 - loss 0.37023313 - samples/sec: 539.72\n", "2020-09-17 08:50:55,191 epoch 48 - iter 120/154 - loss 0.36535563 - samples/sec: 538.70\n", "2020-09-17 08:50:56,167 epoch 48 - iter 135/154 - loss 0.36228978 - samples/sec: 509.00\n", "2020-09-17 08:50:57,178 epoch 48 - iter 150/154 - loss 0.36276235 - samples/sec: 493.54\n", "2020-09-17 08:50:57,362 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:50:57,363 EPOCH 48 done: loss 0.3629 - lr 0.1000\n", "2020-09-17 08:50:58,174 DEV : loss 0.3892788290977478 - score 0.8606\n", "2020-09-17 08:50:58,189 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:50:58,191 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:50:58,247 epoch 49 - iter 0/154 - loss 0.59900618 - samples/sec: 8881.30\n", "2020-09-17 08:50:59,248 epoch 49 - iter 15/154 - loss 0.35171173 - samples/sec: 498.01\n", "2020-09-17 08:51:00,189 epoch 49 - iter 30/154 - loss 0.35720723 - samples/sec: 536.69\n", "2020-09-17 08:51:01,131 epoch 49 - iter 45/154 - loss 0.35257635 - samples/sec: 527.51\n", "2020-09-17 08:51:02,055 epoch 49 - iter 60/154 - loss 0.35713275 - samples/sec: 538.72\n", "2020-09-17 08:51:02,953 epoch 49 - iter 75/154 - loss 0.36974847 - samples/sec: 553.36\n", "2020-09-17 08:51:03,887 epoch 49 - iter 90/154 - loss 0.36365927 - samples/sec: 531.80\n", "2020-09-17 08:51:04,808 epoch 49 - iter 105/154 - loss 0.36668871 - samples/sec: 538.70\n", "2020-09-17 08:51:05,782 epoch 49 - iter 120/154 - loss 0.36575468 - samples/sec: 509.01\n", "2020-09-17 08:51:06,677 epoch 49 - iter 135/154 - loss 0.36866167 - samples/sec: 557.31\n", "2020-09-17 08:51:07,566 epoch 49 - iter 150/154 - loss 0.36949473 - samples/sec: 562.83\n", "2020-09-17 08:51:07,769 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:51:07,770 EPOCH 49 done: loss 0.3673 - lr 0.1000\n", "2020-09-17 08:51:08,638 DEV : loss 0.36433517932891846 - score 0.8881\n", "2020-09-17 08:51:08,653 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:51:13,626 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:51:13,683 epoch 50 - iter 0/154 - loss 0.54789340 - samples/sec: 8575.59\n", "2020-09-17 08:51:14,689 epoch 50 - iter 15/154 - loss 0.34220473 - samples/sec: 496.91\n", "2020-09-17 08:51:15,621 epoch 50 - iter 30/154 - loss 0.33811911 - samples/sec: 535.14\n", "2020-09-17 08:51:16,637 epoch 50 - iter 45/154 - loss 0.32961376 - samples/sec: 494.30\n", "2020-09-17 08:51:17,611 epoch 50 - iter 60/154 - loss 0.33067656 - samples/sec: 513.87\n", "2020-09-17 08:51:18,511 epoch 50 - iter 75/154 - loss 0.32820575 - samples/sec: 550.47\n", "2020-09-17 08:51:19,497 epoch 50 - iter 90/154 - loss 0.32224094 - samples/sec: 510.05\n", "2020-09-17 08:51:20,438 epoch 50 - iter 105/154 - loss 0.33220453 - samples/sec: 530.26\n", "2020-09-17 08:51:21,414 epoch 50 - iter 120/154 - loss 0.33175481 - samples/sec: 508.45\n", "2020-09-17 08:51:22,405 epoch 50 - iter 135/154 - loss 0.32834956 - samples/sec: 503.15\n", "2020-09-17 08:51:23,351 epoch 50 - iter 150/154 - loss 0.33410992 - samples/sec: 524.57\n", "2020-09-17 08:51:23,536 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:51:23,538 EPOCH 50 done: loss 0.3330 - lr 0.1000\n", "2020-09-17 08:51:24,376 DEV : loss 0.37489500641822815 - score 0.8716\n", "2020-09-17 08:51:24,390 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:51:24,392 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:51:24,458 epoch 51 - iter 0/154 - loss 0.12465048 - samples/sec: 7546.57\n", "2020-09-17 08:51:25,476 epoch 51 - iter 15/154 - loss 0.33994321 - samples/sec: 488.29\n", "2020-09-17 08:51:26,568 epoch 51 - iter 30/154 - loss 0.33690864 - samples/sec: 451.75\n", "2020-09-17 08:51:27,687 epoch 51 - iter 45/154 - loss 0.34716499 - samples/sec: 446.03\n", "2020-09-17 08:51:28,695 epoch 51 - iter 60/154 - loss 0.33953664 - samples/sec: 493.63\n", "2020-09-17 08:51:29,755 epoch 51 - iter 75/154 - loss 0.35201706 - samples/sec: 469.57\n", "2020-09-17 08:51:30,869 epoch 51 - iter 90/154 - loss 0.35377296 - samples/sec: 447.31\n", "2020-09-17 08:51:31,983 epoch 51 - iter 105/154 - loss 0.35087511 - samples/sec: 454.13\n", "2020-09-17 08:51:32,992 epoch 51 - iter 120/154 - loss 0.34828146 - samples/sec: 492.64\n", "2020-09-17 08:51:34,084 epoch 51 - iter 135/154 - loss 0.35282188 - samples/sec: 454.61\n", "2020-09-17 08:51:35,057 epoch 51 - iter 150/154 - loss 0.35906638 - samples/sec: 515.56\n", "2020-09-17 08:51:35,244 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:51:35,246 EPOCH 51 done: loss 0.3620 - lr 0.1000\n", "2020-09-17 08:51:36,303 DEV : loss 0.45599284768104553 - score 0.8477\n", "2020-09-17 08:51:36,317 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:51:36,319 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:51:36,386 epoch 52 - iter 0/154 - loss 0.11871673 - samples/sec: 7385.26\n", "2020-09-17 08:51:37,334 epoch 52 - iter 15/154 - loss 0.33132869 - samples/sec: 529.81\n", "2020-09-17 08:51:38,297 epoch 52 - iter 30/154 - loss 0.31975951 - samples/sec: 516.87\n", "2020-09-17 08:51:39,271 epoch 52 - iter 45/154 - loss 0.31728138 - samples/sec: 516.65\n", "2020-09-17 08:51:40,208 epoch 52 - iter 60/154 - loss 0.34764947 - samples/sec: 535.21\n", "2020-09-17 08:51:41,222 epoch 52 - iter 75/154 - loss 0.34451735 - samples/sec: 491.04\n", "2020-09-17 08:51:42,319 epoch 52 - iter 90/154 - loss 0.34211353 - samples/sec: 455.36\n", "2020-09-17 08:51:43,370 epoch 52 - iter 105/154 - loss 0.33762686 - samples/sec: 477.61\n", "2020-09-17 08:51:44,355 epoch 52 - iter 120/154 - loss 0.33597280 - samples/sec: 507.94\n", "2020-09-17 08:51:45,389 epoch 52 - iter 135/154 - loss 0.33347086 - samples/sec: 488.80\n", "2020-09-17 08:51:46,352 epoch 52 - iter 150/154 - loss 0.33696979 - samples/sec: 521.36\n", "2020-09-17 08:51:46,528 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:51:46,530 EPOCH 52 done: loss 0.3340 - lr 0.1000\n", "2020-09-17 08:51:47,314 DEV : loss 0.3762478828430176 - score 0.8771\n", "2020-09-17 08:51:47,330 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:51:47,333 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:51:47,419 epoch 53 - iter 0/154 - loss 0.48142138 - samples/sec: 5869.72\n", "2020-09-17 08:51:48,400 epoch 53 - iter 15/154 - loss 0.27760891 - samples/sec: 515.94\n", "2020-09-17 08:51:49,307 epoch 53 - iter 30/154 - loss 0.30837768 - samples/sec: 552.99\n", "2020-09-17 08:51:50,258 epoch 53 - iter 45/154 - loss 0.32087717 - samples/sec: 525.13\n", "2020-09-17 08:51:51,160 epoch 53 - iter 60/154 - loss 0.34035765 - samples/sec: 554.90\n", "2020-09-17 08:51:52,137 epoch 53 - iter 75/154 - loss 0.33086195 - samples/sec: 509.01\n", "2020-09-17 08:51:53,067 epoch 53 - iter 90/154 - loss 0.34502017 - samples/sec: 544.21\n", "2020-09-17 08:51:53,988 epoch 53 - iter 105/154 - loss 0.34869911 - samples/sec: 542.97\n", "2020-09-17 08:51:54,953 epoch 53 - iter 120/154 - loss 0.34130369 - samples/sec: 516.78\n", "2020-09-17 08:51:55,867 epoch 53 - iter 135/154 - loss 0.34331727 - samples/sec: 547.93\n", "2020-09-17 08:51:56,815 epoch 53 - iter 150/154 - loss 0.34353032 - samples/sec: 527.13\n", "2020-09-17 08:51:57,019 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:51:57,020 EPOCH 53 done: loss 0.3443 - lr 0.1000\n", "2020-09-17 08:51:57,789 DEV : loss 0.38995975255966187 - score 0.8679\n", "2020-09-17 08:51:57,803 BAD EPOCHS (no improvement): 4\n", "2020-09-17 08:51:57,805 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:51:57,875 epoch 54 - iter 0/154 - loss 0.21391124 - samples/sec: 7059.88\n", "2020-09-17 08:51:58,796 epoch 54 - iter 15/154 - loss 0.32370317 - samples/sec: 540.05\n", "2020-09-17 08:51:59,706 epoch 54 - iter 30/154 - loss 0.33872759 - samples/sec: 550.44\n", "2020-09-17 08:52:00,637 epoch 54 - iter 45/154 - loss 0.33474311 - samples/sec: 536.29\n", "2020-09-17 08:52:01,756 epoch 54 - iter 60/154 - loss 0.33857185 - samples/sec: 446.65\n", "2020-09-17 08:52:02,754 epoch 54 - iter 75/154 - loss 0.33338570 - samples/sec: 503.50\n", "2020-09-17 08:52:03,758 epoch 54 - iter 90/154 - loss 0.33290045 - samples/sec: 503.22\n", "2020-09-17 08:52:04,665 epoch 54 - iter 105/154 - loss 0.32782732 - samples/sec: 553.08\n", "2020-09-17 08:52:05,615 epoch 54 - iter 120/154 - loss 0.33225818 - samples/sec: 526.28\n", "2020-09-17 08:52:06,597 epoch 54 - iter 135/154 - loss 0.33045860 - samples/sec: 508.45\n", "2020-09-17 08:52:07,548 epoch 54 - iter 150/154 - loss 0.33561366 - samples/sec: 525.71\n", "2020-09-17 08:52:07,727 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:52:07,728 EPOCH 54 done: loss 0.3334 - lr 0.1000\n", "2020-09-17 08:52:08,618 DEV : loss 0.3526367247104645 - score 0.8789\n", "2020-09-17 08:52:08,633 BAD EPOCHS (no improvement): 5\n", "2020-09-17 08:52:08,634 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:52:08,702 epoch 55 - iter 0/154 - loss 0.31604201 - samples/sec: 7269.68\n", "2020-09-17 08:52:09,621 epoch 55 - iter 15/154 - loss 0.28141161 - samples/sec: 546.04\n", "2020-09-17 08:52:10,612 epoch 55 - iter 30/154 - loss 0.27537713 - samples/sec: 507.40\n", "2020-09-17 08:52:11,560 epoch 55 - iter 45/154 - loss 0.27569440 - samples/sec: 537.52\n", "2020-09-17 08:52:12,575 epoch 55 - iter 60/154 - loss 0.27727258 - samples/sec: 488.28\n", "2020-09-17 08:52:13,553 epoch 55 - iter 75/154 - loss 0.28160846 - samples/sec: 512.22\n", "2020-09-17 08:52:14,545 epoch 55 - iter 90/154 - loss 0.29800578 - samples/sec: 507.02\n", "2020-09-17 08:52:15,514 epoch 55 - iter 105/154 - loss 0.29691673 - samples/sec: 513.37\n", "2020-09-17 08:52:16,419 epoch 55 - iter 120/154 - loss 0.29386108 - samples/sec: 552.37\n", "2020-09-17 08:52:17,351 epoch 55 - iter 135/154 - loss 0.29757500 - samples/sec: 537.07\n", "2020-09-17 08:52:18,318 epoch 55 - iter 150/154 - loss 0.30398050 - samples/sec: 517.38\n", "2020-09-17 08:52:18,505 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:52:18,507 EPOCH 55 done: loss 0.3071 - lr 0.1000\n", "2020-09-17 08:52:19,323 DEV : loss 0.3829459249973297 - score 0.8789\n", "Epoch 54: reducing learning rate of group 0 to 5.0000e-02.\n", "2020-09-17 08:52:19,336 BAD EPOCHS (no improvement): 6\n", "2020-09-17 08:52:19,338 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:52:19,414 epoch 56 - iter 0/154 - loss 0.22897518 - samples/sec: 6487.12\n", "2020-09-17 08:52:20,383 epoch 56 - iter 15/154 - loss 0.30990435 - samples/sec: 516.77\n", "2020-09-17 08:52:21,286 epoch 56 - iter 30/154 - loss 0.28774249 - samples/sec: 554.72\n", "2020-09-17 08:52:22,220 epoch 56 - iter 45/154 - loss 0.28517475 - samples/sec: 533.93\n", "2020-09-17 08:52:23,232 epoch 56 - iter 60/154 - loss 0.28408732 - samples/sec: 495.85\n", "2020-09-17 08:52:24,162 epoch 56 - iter 75/154 - loss 0.29552690 - samples/sec: 538.44\n", "2020-09-17 08:52:25,140 epoch 56 - iter 90/154 - loss 0.29557508 - samples/sec: 508.39\n", "2020-09-17 08:52:26,102 epoch 56 - iter 105/154 - loss 0.29381103 - samples/sec: 519.19\n", "2020-09-17 08:52:27,130 epoch 56 - iter 120/154 - loss 0.29248250 - samples/sec: 487.81\n", "2020-09-17 08:52:28,077 epoch 56 - iter 135/154 - loss 0.28380341 - samples/sec: 531.03\n", "2020-09-17 08:52:29,001 epoch 56 - iter 150/154 - loss 0.28375113 - samples/sec: 541.45\n", "2020-09-17 08:52:29,207 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:52:29,208 EPOCH 56 done: loss 0.2861 - lr 0.0500\n", "2020-09-17 08:52:30,040 DEV : loss 0.3735714256763458 - score 0.8789\n", "2020-09-17 08:52:30,054 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:52:30,056 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:52:30,128 epoch 57 - iter 0/154 - loss 0.39974803 - samples/sec: 6953.95\n", "2020-09-17 08:52:31,066 epoch 57 - iter 15/154 - loss 0.27352504 - samples/sec: 540.07\n", "2020-09-17 08:52:32,052 epoch 57 - iter 30/154 - loss 0.26931395 - samples/sec: 507.29\n", "2020-09-17 08:52:32,953 epoch 57 - iter 45/154 - loss 0.25971343 - samples/sec: 552.34\n", "2020-09-17 08:52:33,880 epoch 57 - iter 60/154 - loss 0.24984893 - samples/sec: 541.76\n", "2020-09-17 08:52:34,816 epoch 57 - iter 75/154 - loss 0.24574264 - samples/sec: 537.28\n", "2020-09-17 08:52:35,734 epoch 57 - iter 90/154 - loss 0.25254078 - samples/sec: 545.17\n", "2020-09-17 08:52:36,692 epoch 57 - iter 105/154 - loss 0.25668447 - samples/sec: 523.43\n", "2020-09-17 08:52:37,681 epoch 57 - iter 120/154 - loss 0.25958828 - samples/sec: 502.56\n", "2020-09-17 08:52:38,667 epoch 57 - iter 135/154 - loss 0.25917622 - samples/sec: 506.35\n", "2020-09-17 08:52:39,600 epoch 57 - iter 150/154 - loss 0.25973554 - samples/sec: 537.86\n", "2020-09-17 08:52:39,787 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:52:39,788 EPOCH 57 done: loss 0.2603 - lr 0.0500\n", "2020-09-17 08:52:40,581 DEV : loss 0.37158524990081787 - score 0.8752\n", "2020-09-17 08:52:40,596 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:52:40,598 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:52:40,680 epoch 58 - iter 0/154 - loss 0.22636844 - samples/sec: 6156.44\n", "2020-09-17 08:52:41,748 epoch 58 - iter 15/154 - loss 0.28068735 - samples/sec: 468.71\n", "2020-09-17 08:52:42,737 epoch 58 - iter 30/154 - loss 0.28504852 - samples/sec: 507.35\n", "2020-09-17 08:52:43,725 epoch 58 - iter 45/154 - loss 0.28233559 - samples/sec: 503.75\n", "2020-09-17 08:52:44,699 epoch 58 - iter 60/154 - loss 0.27533098 - samples/sec: 512.08\n", "2020-09-17 08:52:45,827 epoch 58 - iter 75/154 - loss 0.27359242 - samples/sec: 441.09\n", "2020-09-17 08:52:46,855 epoch 58 - iter 90/154 - loss 0.27256921 - samples/sec: 483.51\n", "2020-09-17 08:52:47,870 epoch 58 - iter 105/154 - loss 0.28101661 - samples/sec: 494.34\n", "2020-09-17 08:52:49,048 epoch 58 - iter 120/154 - loss 0.28040870 - samples/sec: 421.42\n", "2020-09-17 08:52:50,439 epoch 58 - iter 135/154 - loss 0.28370266 - samples/sec: 355.67\n", "2020-09-17 08:52:51,853 epoch 58 - iter 150/154 - loss 0.27889721 - samples/sec: 349.44\n", "2020-09-17 08:52:52,055 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:52:52,056 EPOCH 58 done: loss 0.2840 - lr 0.0500\n", "2020-09-17 08:52:52,945 DEV : loss 0.3527340888977051 - score 0.8807\n", "2020-09-17 08:52:52,960 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:52:52,962 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:52:53,017 epoch 59 - iter 0/154 - loss 0.20902500 - samples/sec: 9058.56\n", "2020-09-17 08:52:54,010 epoch 59 - iter 15/154 - loss 0.27836555 - samples/sec: 499.45\n", "2020-09-17 08:52:54,974 epoch 59 - iter 30/154 - loss 0.24628820 - samples/sec: 514.47\n", "2020-09-17 08:52:55,934 epoch 59 - iter 45/154 - loss 0.26264143 - samples/sec: 519.48\n", "2020-09-17 08:52:56,884 epoch 59 - iter 60/154 - loss 0.25863020 - samples/sec: 521.62\n", "2020-09-17 08:52:57,865 epoch 59 - iter 75/154 - loss 0.26332473 - samples/sec: 506.09\n", "2020-09-17 08:52:58,799 epoch 59 - iter 90/154 - loss 0.26296828 - samples/sec: 532.25\n", "2020-09-17 08:52:59,784 epoch 59 - iter 105/154 - loss 0.26644945 - samples/sec: 503.67\n", "2020-09-17 08:53:00,724 epoch 59 - iter 120/154 - loss 0.26497711 - samples/sec: 528.62\n", "2020-09-17 08:53:01,640 epoch 59 - iter 135/154 - loss 0.27256814 - samples/sec: 544.49\n", "2020-09-17 08:53:02,671 epoch 59 - iter 150/154 - loss 0.27077568 - samples/sec: 480.17\n", "2020-09-17 08:53:02,856 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:53:02,857 EPOCH 59 done: loss 0.2713 - lr 0.0500\n", "2020-09-17 08:53:03,637 DEV : loss 0.37298503518104553 - score 0.8936\n", "2020-09-17 08:53:03,650 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:53:08,278 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:53:08,368 epoch 60 - iter 0/154 - loss 0.27515122 - samples/sec: 5517.31\n", "2020-09-17 08:53:09,773 epoch 60 - iter 15/154 - loss 0.26034528 - samples/sec: 356.45\n", "2020-09-17 08:53:10,920 epoch 60 - iter 30/154 - loss 0.28378616 - samples/sec: 432.83\n", "2020-09-17 08:53:11,993 epoch 60 - iter 45/154 - loss 0.28071733 - samples/sec: 461.77\n", "2020-09-17 08:53:12,944 epoch 60 - iter 60/154 - loss 0.28525573 - samples/sec: 525.86\n", "2020-09-17 08:53:13,912 epoch 60 - iter 75/154 - loss 0.28510268 - samples/sec: 512.28\n", "2020-09-17 08:53:14,877 epoch 60 - iter 90/154 - loss 0.28465652 - samples/sec: 516.13\n", "2020-09-17 08:53:15,843 epoch 60 - iter 105/154 - loss 0.28590082 - samples/sec: 516.12\n", "2020-09-17 08:53:16,822 epoch 60 - iter 120/154 - loss 0.28308470 - samples/sec: 507.94\n", "2020-09-17 08:53:17,771 epoch 60 - iter 135/154 - loss 0.28666363 - samples/sec: 521.75\n", "2020-09-17 08:53:18,814 epoch 60 - iter 150/154 - loss 0.28366151 - samples/sec: 476.67\n", "2020-09-17 08:53:18,995 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:53:18,996 EPOCH 60 done: loss 0.2825 - lr 0.0500\n", "2020-09-17 08:53:19,794 DEV : loss 0.3572901785373688 - score 0.9009\n", "2020-09-17 08:53:19,808 BAD EPOCHS (no improvement): 0\n", "2020-09-17 08:53:26,401 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:53:26,489 epoch 61 - iter 0/154 - loss 0.09127169 - samples/sec: 5646.69\n", "2020-09-17 08:53:27,530 epoch 61 - iter 15/154 - loss 0.30648667 - samples/sec: 478.09\n", "2020-09-17 08:53:28,571 epoch 61 - iter 30/154 - loss 0.28841792 - samples/sec: 477.15\n", "2020-09-17 08:53:29,618 epoch 61 - iter 45/154 - loss 0.28260109 - samples/sec: 474.83\n", "2020-09-17 08:53:30,653 epoch 61 - iter 60/154 - loss 0.28083605 - samples/sec: 482.90\n", "2020-09-17 08:53:31,679 epoch 61 - iter 75/154 - loss 0.27343147 - samples/sec: 486.81\n", "2020-09-17 08:53:32,733 epoch 61 - iter 90/154 - loss 0.26959555 - samples/sec: 470.22\n", "2020-09-17 08:53:33,690 epoch 61 - iter 105/154 - loss 0.26932343 - samples/sec: 523.97\n", "2020-09-17 08:53:34,635 epoch 61 - iter 120/154 - loss 0.26320646 - samples/sec: 525.37\n", "2020-09-17 08:53:35,589 epoch 61 - iter 135/154 - loss 0.26273987 - samples/sec: 520.59\n", "2020-09-17 08:53:36,553 epoch 61 - iter 150/154 - loss 0.26708357 - samples/sec: 515.02\n", "2020-09-17 08:53:36,757 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:53:36,758 EPOCH 61 done: loss 0.2679 - lr 0.0500\n", "2020-09-17 08:53:37,609 DEV : loss 0.3652314841747284 - score 0.8789\n", "2020-09-17 08:53:37,624 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:53:37,626 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:53:37,685 epoch 62 - iter 0/154 - loss 0.38391525 - samples/sec: 8571.76\n", "2020-09-17 08:53:38,626 epoch 62 - iter 15/154 - loss 0.27334811 - samples/sec: 528.63\n", "2020-09-17 08:53:39,635 epoch 62 - iter 30/154 - loss 0.27422994 - samples/sec: 492.35\n", "2020-09-17 08:53:40,588 epoch 62 - iter 45/154 - loss 0.26579075 - samples/sec: 526.85\n", "2020-09-17 08:53:41,570 epoch 62 - iter 60/154 - loss 0.26881177 - samples/sec: 507.82\n", "2020-09-17 08:53:42,538 epoch 62 - iter 75/154 - loss 0.27154484 - samples/sec: 513.35\n", "2020-09-17 08:53:43,525 epoch 62 - iter 90/154 - loss 0.27139610 - samples/sec: 502.37\n", "2020-09-17 08:53:44,548 epoch 62 - iter 105/154 - loss 0.27190984 - samples/sec: 497.91\n", "2020-09-17 08:53:45,490 epoch 62 - iter 120/154 - loss 0.27158706 - samples/sec: 527.05\n", "2020-09-17 08:53:46,490 epoch 62 - iter 135/154 - loss 0.26663899 - samples/sec: 498.42\n", "2020-09-17 08:53:47,446 epoch 62 - iter 150/154 - loss 0.26131100 - samples/sec: 523.36\n", "2020-09-17 08:53:47,631 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:53:47,633 EPOCH 62 done: loss 0.2624 - lr 0.0500\n", "2020-09-17 08:53:48,430 DEV : loss 0.37082332372665405 - score 0.8807\n", "2020-09-17 08:53:48,445 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:53:48,446 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:53:48,511 epoch 63 - iter 0/154 - loss 0.19815043 - samples/sec: 7619.23\n", "2020-09-17 08:53:49,518 epoch 63 - iter 15/154 - loss 0.26004793 - samples/sec: 491.15\n", "2020-09-17 08:53:50,504 epoch 63 - iter 30/154 - loss 0.30798576 - samples/sec: 503.49\n", "2020-09-17 08:53:51,804 epoch 63 - iter 45/154 - loss 0.31683742 - samples/sec: 378.25\n", "2020-09-17 08:53:53,036 epoch 63 - iter 60/154 - loss 0.30306068 - samples/sec: 414.51\n", "2020-09-17 08:53:54,417 epoch 63 - iter 75/154 - loss 0.30331372 - samples/sec: 356.87\n", "2020-09-17 08:53:55,478 epoch 63 - iter 90/154 - loss 0.29736875 - samples/sec: 476.41\n", "2020-09-17 08:53:56,612 epoch 63 - iter 105/154 - loss 0.29185457 - samples/sec: 435.92\n", "2020-09-17 08:53:57,629 epoch 63 - iter 120/154 - loss 0.28197640 - samples/sec: 487.95\n", "2020-09-17 08:53:58,628 epoch 63 - iter 135/154 - loss 0.27716415 - samples/sec: 498.63\n", "2020-09-17 08:53:59,646 epoch 63 - iter 150/154 - loss 0.27283440 - samples/sec: 492.42\n", "2020-09-17 08:53:59,848 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:53:59,850 EPOCH 63 done: loss 0.2734 - lr 0.0500\n", "2020-09-17 08:54:00,695 DEV : loss 0.4010722041130066 - score 0.8624\n", "2020-09-17 08:54:00,710 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:54:00,712 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:54:00,771 epoch 64 - iter 0/154 - loss 0.25437412 - samples/sec: 8277.48\n", "2020-09-17 08:54:01,930 epoch 64 - iter 15/154 - loss 0.26292134 - samples/sec: 429.49\n", "2020-09-17 08:54:03,124 epoch 64 - iter 30/154 - loss 0.25481109 - samples/sec: 416.93\n", "2020-09-17 08:54:05,003 epoch 64 - iter 45/154 - loss 0.23337130 - samples/sec: 263.63\n", "2020-09-17 08:54:06,576 epoch 64 - iter 60/154 - loss 0.23958304 - samples/sec: 319.84\n", "2020-09-17 08:54:08,372 epoch 64 - iter 75/154 - loss 0.24395224 - samples/sec: 277.17\n", "2020-09-17 08:54:10,085 epoch 64 - iter 90/154 - loss 0.23512809 - samples/sec: 293.92\n", "2020-09-17 08:54:11,803 epoch 64 - iter 105/154 - loss 0.24191262 - samples/sec: 291.70\n", "2020-09-17 08:54:13,661 epoch 64 - iter 120/154 - loss 0.23917348 - samples/sec: 268.43\n", "2020-09-17 08:54:15,365 epoch 64 - iter 135/154 - loss 0.24495526 - samples/sec: 295.80\n", "2020-09-17 08:54:16,960 epoch 64 - iter 150/154 - loss 0.24768724 - samples/sec: 318.38\n", "2020-09-17 08:54:17,242 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:54:17,244 EPOCH 64 done: loss 0.2483 - lr 0.0500\n", "2020-09-17 08:54:18,296 DEV : loss 0.33853450417518616 - score 0.8899\n", "2020-09-17 08:54:18,328 BAD EPOCHS (no improvement): 4\n", "2020-09-17 08:54:18,330 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:54:18,427 epoch 65 - iter 0/154 - loss 0.38951007 - samples/sec: 5052.57\n", "2020-09-17 08:54:19,504 epoch 65 - iter 15/154 - loss 0.30967177 - samples/sec: 467.02\n", "2020-09-17 08:54:20,585 epoch 65 - iter 30/154 - loss 0.29811764 - samples/sec: 460.31\n", "2020-09-17 08:54:21,768 epoch 65 - iter 45/154 - loss 0.27447067 - samples/sec: 421.80\n", "2020-09-17 08:54:23,068 epoch 65 - iter 60/154 - loss 0.25989303 - samples/sec: 380.34\n", "2020-09-17 08:54:24,582 epoch 65 - iter 75/154 - loss 0.26249945 - samples/sec: 349.46\n", "2020-09-17 08:54:26,126 epoch 65 - iter 90/154 - loss 0.26223842 - samples/sec: 318.58\n", "2020-09-17 08:54:27,308 epoch 65 - iter 105/154 - loss 0.26422607 - samples/sec: 428.55\n", "2020-09-17 08:54:28,389 epoch 65 - iter 120/154 - loss 0.26300967 - samples/sec: 462.52\n", "2020-09-17 08:54:29,333 epoch 65 - iter 135/154 - loss 0.25361973 - samples/sec: 527.51\n", "2020-09-17 08:54:30,321 epoch 65 - iter 150/154 - loss 0.25295122 - samples/sec: 503.96\n", "2020-09-17 08:54:30,510 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:54:30,511 EPOCH 65 done: loss 0.2556 - lr 0.0500\n", "2020-09-17 08:54:31,355 DEV : loss 0.3943146765232086 - score 0.8716\n", "2020-09-17 08:54:31,370 BAD EPOCHS (no improvement): 5\n", "2020-09-17 08:54:31,371 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:54:31,437 epoch 66 - iter 0/154 - loss 0.22021146 - samples/sec: 7496.77\n", "2020-09-17 08:54:32,393 epoch 66 - iter 15/154 - loss 0.25722831 - samples/sec: 519.51\n", "2020-09-17 08:54:33,374 epoch 66 - iter 30/154 - loss 0.24618746 - samples/sec: 505.24\n", "2020-09-17 08:54:34,299 epoch 66 - iter 45/154 - loss 0.24222479 - samples/sec: 542.97\n", "2020-09-17 08:54:35,294 epoch 66 - iter 60/154 - loss 0.23939163 - samples/sec: 499.49\n", "2020-09-17 08:54:36,239 epoch 66 - iter 75/154 - loss 0.23595286 - samples/sec: 531.54\n", "2020-09-17 08:54:37,213 epoch 66 - iter 90/154 - loss 0.23955643 - samples/sec: 509.52\n", "2020-09-17 08:54:38,217 epoch 66 - iter 105/154 - loss 0.24026117 - samples/sec: 493.04\n", "2020-09-17 08:54:39,230 epoch 66 - iter 120/154 - loss 0.24494816 - samples/sec: 490.80\n", "2020-09-17 08:54:40,293 epoch 66 - iter 135/154 - loss 0.24467233 - samples/sec: 468.74\n", "2020-09-17 08:54:41,386 epoch 66 - iter 150/154 - loss 0.24190240 - samples/sec: 458.63\n", "2020-09-17 08:54:41,584 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:54:41,584 EPOCH 66 done: loss 0.2396 - lr 0.0500\n", "2020-09-17 08:54:42,515 DEV : loss 0.37364688515663147 - score 0.8862\n", "Epoch 65: reducing learning rate of group 0 to 2.5000e-02.\n", "2020-09-17 08:54:42,534 BAD EPOCHS (no improvement): 6\n", "2020-09-17 08:54:42,535 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:54:42,600 epoch 67 - iter 0/154 - loss 0.37509233 - samples/sec: 7740.77\n", "2020-09-17 08:54:43,522 epoch 67 - iter 15/154 - loss 0.24311952 - samples/sec: 541.10\n", "2020-09-17 08:54:44,559 epoch 67 - iter 30/154 - loss 0.22554752 - samples/sec: 481.44\n", "2020-09-17 08:54:45,578 epoch 67 - iter 45/154 - loss 0.23749554 - samples/sec: 491.29\n", "2020-09-17 08:54:46,514 epoch 67 - iter 60/154 - loss 0.23672992 - samples/sec: 531.16\n", "2020-09-17 08:54:47,500 epoch 67 - iter 75/154 - loss 0.24263239 - samples/sec: 503.15\n", "2020-09-17 08:54:48,451 epoch 67 - iter 90/154 - loss 0.24685186 - samples/sec: 519.48\n", "2020-09-17 08:54:49,520 epoch 67 - iter 105/154 - loss 0.24544490 - samples/sec: 462.89\n", "2020-09-17 08:54:50,527 epoch 67 - iter 120/154 - loss 0.24211873 - samples/sec: 495.38\n", "2020-09-17 08:54:51,456 epoch 67 - iter 135/154 - loss 0.23608169 - samples/sec: 537.51\n", "2020-09-17 08:54:52,637 epoch 67 - iter 150/154 - loss 0.24287955 - samples/sec: 416.64\n", "2020-09-17 08:54:52,834 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:54:52,836 EPOCH 67 done: loss 0.2412 - lr 0.0250\n", "2020-09-17 08:54:53,739 DEV : loss 0.35277608036994934 - score 0.8936\n", "2020-09-17 08:54:53,766 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:54:53,769 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:54:53,879 epoch 68 - iter 0/154 - loss 0.23245344 - samples/sec: 4486.00\n", "2020-09-17 08:54:55,360 epoch 68 - iter 15/154 - loss 0.20064148 - samples/sec: 337.25\n", "2020-09-17 08:54:56,674 epoch 68 - iter 30/154 - loss 0.22466057 - samples/sec: 380.74\n", "2020-09-17 08:54:57,964 epoch 68 - iter 45/154 - loss 0.22375448 - samples/sec: 389.23\n", "2020-09-17 08:54:59,104 epoch 68 - iter 60/154 - loss 0.22787052 - samples/sec: 448.60\n", "2020-09-17 08:55:00,332 epoch 68 - iter 75/154 - loss 0.23881833 - samples/sec: 408.80\n", "2020-09-17 08:55:01,375 epoch 68 - iter 90/154 - loss 0.23417654 - samples/sec: 475.38\n", "2020-09-17 08:55:02,628 epoch 68 - iter 105/154 - loss 0.23373793 - samples/sec: 393.33\n", "2020-09-17 08:55:03,773 epoch 68 - iter 120/154 - loss 0.23844401 - samples/sec: 444.32\n", "2020-09-17 08:55:04,767 epoch 68 - iter 135/154 - loss 0.23572081 - samples/sec: 498.97\n", "2020-09-17 08:55:05,726 epoch 68 - iter 150/154 - loss 0.23424753 - samples/sec: 518.38\n", "2020-09-17 08:55:05,897 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:55:05,898 EPOCH 68 done: loss 0.2345 - lr 0.0250\n", "2020-09-17 08:55:06,694 DEV : loss 0.36143583059310913 - score 0.8826\n", "2020-09-17 08:55:06,710 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:55:06,712 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:55:06,796 epoch 69 - iter 0/154 - loss 0.10268354 - samples/sec: 5928.75\n", "2020-09-17 08:55:07,746 epoch 69 - iter 15/154 - loss 0.25771395 - samples/sec: 527.77\n", "2020-09-17 08:55:08,689 epoch 69 - iter 30/154 - loss 0.23529439 - samples/sec: 528.91\n", "2020-09-17 08:55:09,650 epoch 69 - iter 45/154 - loss 0.23124479 - samples/sec: 523.66\n", "2020-09-17 08:55:10,561 epoch 69 - iter 60/154 - loss 0.23458318 - samples/sec: 545.35\n", "2020-09-17 08:55:11,514 epoch 69 - iter 75/154 - loss 0.23698447 - samples/sec: 522.33\n", "2020-09-17 08:55:12,524 epoch 69 - iter 90/154 - loss 0.23384923 - samples/sec: 502.44\n", "2020-09-17 08:55:13,429 epoch 69 - iter 105/154 - loss 0.23387504 - samples/sec: 551.06\n", "2020-09-17 08:55:14,393 epoch 69 - iter 120/154 - loss 0.23224689 - samples/sec: 515.31\n", "2020-09-17 08:55:15,261 epoch 69 - iter 135/154 - loss 0.23304298 - samples/sec: 573.22\n", "2020-09-17 08:55:16,219 epoch 69 - iter 150/154 - loss 0.22895676 - samples/sec: 520.12\n", "2020-09-17 08:55:16,435 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:55:16,436 EPOCH 69 done: loss 0.2279 - lr 0.0250\n", "2020-09-17 08:55:17,202 DEV : loss 0.360670268535614 - score 0.8862\n", "2020-09-17 08:55:17,218 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:55:17,220 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:55:17,287 epoch 70 - iter 0/154 - loss 0.20619360 - samples/sec: 7304.71\n", "2020-09-17 08:55:18,174 epoch 70 - iter 15/154 - loss 0.20947369 - samples/sec: 559.07\n", "2020-09-17 08:55:19,097 epoch 70 - iter 30/154 - loss 0.21165238 - samples/sec: 537.29\n", "2020-09-17 08:55:19,973 epoch 70 - iter 45/154 - loss 0.21157911 - samples/sec: 566.72\n", "2020-09-17 08:55:20,959 epoch 70 - iter 60/154 - loss 0.22567784 - samples/sec: 503.12\n", "2020-09-17 08:55:21,892 epoch 70 - iter 75/154 - loss 0.22276114 - samples/sec: 535.14\n", "2020-09-17 08:55:22,806 epoch 70 - iter 90/154 - loss 0.21913063 - samples/sec: 544.94\n", "2020-09-17 08:55:23,752 epoch 70 - iter 105/154 - loss 0.22461381 - samples/sec: 524.91\n", "2020-09-17 08:55:24,667 epoch 70 - iter 120/154 - loss 0.22929669 - samples/sec: 542.47\n", "2020-09-17 08:55:25,717 epoch 70 - iter 135/154 - loss 0.23195596 - samples/sec: 473.11\n", "2020-09-17 08:55:26,640 epoch 70 - iter 150/154 - loss 0.22737986 - samples/sec: 538.45\n", "2020-09-17 08:55:26,816 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:55:26,817 EPOCH 70 done: loss 0.2271 - lr 0.0250\n", "2020-09-17 08:55:27,580 DEV : loss 0.35208263993263245 - score 0.8899\n", "2020-09-17 08:55:27,594 BAD EPOCHS (no improvement): 4\n", "2020-09-17 08:55:27,596 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:55:27,660 epoch 71 - iter 0/154 - loss 0.08306511 - samples/sec: 7744.08\n", "2020-09-17 08:55:28,595 epoch 71 - iter 15/154 - loss 0.20720360 - samples/sec: 538.07\n", "2020-09-17 08:55:29,500 epoch 71 - iter 30/154 - loss 0.21155740 - samples/sec: 551.72\n", "2020-09-17 08:55:30,457 epoch 71 - iter 45/154 - loss 0.22466321 - samples/sec: 517.17\n", "2020-09-17 08:55:31,402 epoch 71 - iter 60/154 - loss 0.21935588 - samples/sec: 524.57\n", "2020-09-17 08:55:32,309 epoch 71 - iter 75/154 - loss 0.22426653 - samples/sec: 546.53\n", "2020-09-17 08:55:33,260 epoch 71 - iter 90/154 - loss 0.22005030 - samples/sec: 528.48\n", "2020-09-17 08:55:34,194 epoch 71 - iter 105/154 - loss 0.22279145 - samples/sec: 535.09\n", "2020-09-17 08:55:35,122 epoch 71 - iter 120/154 - loss 0.22614614 - samples/sec: 533.97\n", "2020-09-17 08:55:36,037 epoch 71 - iter 135/154 - loss 0.22313135 - samples/sec: 543.62\n", "2020-09-17 08:55:36,987 epoch 71 - iter 150/154 - loss 0.22198352 - samples/sec: 521.87\n", "2020-09-17 08:55:37,214 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:55:37,215 EPOCH 71 done: loss 0.2205 - lr 0.0250\n", "2020-09-17 08:55:37,998 DEV : loss 0.38568681478500366 - score 0.8881\n", "2020-09-17 08:55:38,012 BAD EPOCHS (no improvement): 5\n", "2020-09-17 08:55:38,013 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:55:38,081 epoch 72 - iter 0/154 - loss 0.11887620 - samples/sec: 7194.32\n", "2020-09-17 08:55:39,027 epoch 72 - iter 15/154 - loss 0.20052867 - samples/sec: 526.02\n", "2020-09-17 08:55:39,998 epoch 72 - iter 30/154 - loss 0.22208617 - samples/sec: 513.43\n", "2020-09-17 08:55:40,987 epoch 72 - iter 45/154 - loss 0.23462349 - samples/sec: 500.57\n", "2020-09-17 08:55:41,941 epoch 72 - iter 60/154 - loss 0.23762766 - samples/sec: 521.51\n", "2020-09-17 08:55:42,842 epoch 72 - iter 75/154 - loss 0.23126596 - samples/sec: 550.36\n", "2020-09-17 08:55:43,728 epoch 72 - iter 90/154 - loss 0.23347258 - samples/sec: 562.22\n", "2020-09-17 08:55:44,689 epoch 72 - iter 105/154 - loss 0.22878091 - samples/sec: 516.62\n", "2020-09-17 08:55:45,590 epoch 72 - iter 120/154 - loss 0.22738589 - samples/sec: 553.41\n", "2020-09-17 08:55:46,570 epoch 72 - iter 135/154 - loss 0.22463990 - samples/sec: 505.26\n", "2020-09-17 08:55:47,497 epoch 72 - iter 150/154 - loss 0.22563891 - samples/sec: 537.81\n", "2020-09-17 08:55:47,675 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:55:47,676 EPOCH 72 done: loss 0.2239 - lr 0.0250\n", "2020-09-17 08:55:48,544 DEV : loss 0.37153229117393494 - score 0.8899\n", "Epoch 71: reducing learning rate of group 0 to 1.2500e-02.\n", "2020-09-17 08:55:48,561 BAD EPOCHS (no improvement): 6\n", "2020-09-17 08:55:48,563 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:55:48,631 epoch 73 - iter 0/154 - loss 0.15476924 - samples/sec: 7272.94\n", "2020-09-17 08:55:49,582 epoch 73 - iter 15/154 - loss 0.20823604 - samples/sec: 524.74\n", "2020-09-17 08:55:50,549 epoch 73 - iter 30/154 - loss 0.23406009 - samples/sec: 514.44\n", "2020-09-17 08:55:51,582 epoch 73 - iter 45/154 - loss 0.23261078 - samples/sec: 478.40\n", "2020-09-17 08:55:52,575 epoch 73 - iter 60/154 - loss 0.22985054 - samples/sec: 498.97\n", "2020-09-17 08:55:53,581 epoch 73 - iter 75/154 - loss 0.21651013 - samples/sec: 493.31\n", "2020-09-17 08:55:54,544 epoch 73 - iter 90/154 - loss 0.21418249 - samples/sec: 515.47\n", "2020-09-17 08:55:55,546 epoch 73 - iter 105/154 - loss 0.21057431 - samples/sec: 495.83\n", "2020-09-17 08:55:56,518 epoch 73 - iter 120/154 - loss 0.21374645 - samples/sec: 512.52\n", "2020-09-17 08:55:57,507 epoch 73 - iter 135/154 - loss 0.21857236 - samples/sec: 500.50\n", "2020-09-17 08:55:58,624 epoch 73 - iter 150/154 - loss 0.21589783 - samples/sec: 445.27\n", "2020-09-17 08:55:58,809 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:55:58,810 EPOCH 73 done: loss 0.2140 - lr 0.0125\n", "2020-09-17 08:55:59,576 DEV : loss 0.35357382893562317 - score 0.8936\n", "2020-09-17 08:55:59,589 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:55:59,591 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:55:59,646 epoch 74 - iter 0/154 - loss 0.34098747 - samples/sec: 9061.46\n", "2020-09-17 08:56:00,655 epoch 74 - iter 15/154 - loss 0.23593240 - samples/sec: 490.62\n", "2020-09-17 08:56:01,606 epoch 74 - iter 30/154 - loss 0.23053448 - samples/sec: 523.99\n", "2020-09-17 08:56:02,552 epoch 74 - iter 45/154 - loss 0.21738356 - samples/sec: 525.32\n", "2020-09-17 08:56:03,524 epoch 74 - iter 60/154 - loss 0.20441832 - samples/sec: 522.15\n", "2020-09-17 08:56:04,482 epoch 74 - iter 75/154 - loss 0.20221644 - samples/sec: 525.39\n", "2020-09-17 08:56:05,443 epoch 74 - iter 90/154 - loss 0.20285186 - samples/sec: 513.54\n", "2020-09-17 08:56:06,407 epoch 74 - iter 105/154 - loss 0.19968325 - samples/sec: 521.92\n", "2020-09-17 08:56:07,349 epoch 74 - iter 120/154 - loss 0.20215845 - samples/sec: 526.21\n", "2020-09-17 08:56:08,256 epoch 74 - iter 135/154 - loss 0.20557900 - samples/sec: 547.69\n", "2020-09-17 08:56:09,187 epoch 74 - iter 150/154 - loss 0.21120611 - samples/sec: 537.74\n", "2020-09-17 08:56:09,374 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:56:09,376 EPOCH 74 done: loss 0.2110 - lr 0.0125\n", "2020-09-17 08:56:10,251 DEV : loss 0.3510146737098694 - score 0.8972\n", "2020-09-17 08:56:10,266 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:56:10,268 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:56:10,334 epoch 75 - iter 0/154 - loss 0.15859744 - samples/sec: 7500.15\n", "2020-09-17 08:56:11,348 epoch 75 - iter 15/154 - loss 0.24293265 - samples/sec: 490.67\n", "2020-09-17 08:56:12,385 epoch 75 - iter 30/154 - loss 0.25338470 - samples/sec: 477.63\n", "2020-09-17 08:56:13,388 epoch 75 - iter 45/154 - loss 0.22505214 - samples/sec: 495.85\n", "2020-09-17 08:56:14,395 epoch 75 - iter 60/154 - loss 0.23234620 - samples/sec: 492.49\n", "2020-09-17 08:56:15,370 epoch 75 - iter 75/154 - loss 0.23325246 - samples/sec: 511.12\n", "2020-09-17 08:56:16,360 epoch 75 - iter 90/154 - loss 0.23460935 - samples/sec: 507.79\n", "2020-09-17 08:56:17,307 epoch 75 - iter 105/154 - loss 0.22310798 - samples/sec: 530.95\n", "2020-09-17 08:56:18,287 epoch 75 - iter 120/154 - loss 0.22333743 - samples/sec: 506.55\n", "2020-09-17 08:56:19,317 epoch 75 - iter 135/154 - loss 0.22185267 - samples/sec: 482.32\n", "2020-09-17 08:56:20,242 epoch 75 - iter 150/154 - loss 0.21748585 - samples/sec: 538.56\n", "2020-09-17 08:56:20,442 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:56:20,443 EPOCH 75 done: loss 0.2263 - lr 0.0125\n", "2020-09-17 08:56:21,361 DEV : loss 0.339412122964859 - score 0.8991\n", "2020-09-17 08:56:21,374 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:56:21,378 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:56:21,458 epoch 76 - iter 0/154 - loss 0.31175730 - samples/sec: 6168.34\n", "2020-09-17 08:56:22,380 epoch 76 - iter 15/154 - loss 0.22490774 - samples/sec: 544.04\n", "2020-09-17 08:56:23,316 epoch 76 - iter 30/154 - loss 0.22450734 - samples/sec: 532.74\n", "2020-09-17 08:56:24,294 epoch 76 - iter 45/154 - loss 0.21318417 - samples/sec: 515.87\n", "2020-09-17 08:56:25,212 epoch 76 - iter 60/154 - loss 0.20195785 - samples/sec: 543.11\n", "2020-09-17 08:56:26,254 epoch 76 - iter 75/154 - loss 0.20341362 - samples/sec: 475.71\n", "2020-09-17 08:56:27,235 epoch 76 - iter 90/154 - loss 0.20074117 - samples/sec: 512.02\n", "2020-09-17 08:56:28,271 epoch 76 - iter 105/154 - loss 0.20272644 - samples/sec: 478.77\n", "2020-09-17 08:56:29,225 epoch 76 - iter 120/154 - loss 0.20762871 - samples/sec: 522.72\n", "2020-09-17 08:56:30,181 epoch 76 - iter 135/154 - loss 0.21053792 - samples/sec: 517.70\n", "2020-09-17 08:56:31,171 epoch 76 - iter 150/154 - loss 0.21257777 - samples/sec: 504.61\n", "2020-09-17 08:56:31,426 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:56:31,427 EPOCH 76 done: loss 0.2111 - lr 0.0125\n", "2020-09-17 08:56:32,202 DEV : loss 0.35670655965805054 - score 0.8881\n", "2020-09-17 08:56:32,215 BAD EPOCHS (no improvement): 4\n", "2020-09-17 08:56:32,217 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:56:32,284 epoch 77 - iter 0/154 - loss 0.25497928 - samples/sec: 7384.64\n", "2020-09-17 08:56:33,291 epoch 77 - iter 15/154 - loss 0.22214673 - samples/sec: 495.86\n", "2020-09-17 08:56:34,339 epoch 77 - iter 30/154 - loss 0.20344659 - samples/sec: 475.95\n", "2020-09-17 08:56:35,370 epoch 77 - iter 45/154 - loss 0.20966993 - samples/sec: 481.66\n", "2020-09-17 08:56:36,297 epoch 77 - iter 60/154 - loss 0.21670036 - samples/sec: 535.05\n", "2020-09-17 08:56:37,224 epoch 77 - iter 75/154 - loss 0.21499322 - samples/sec: 536.63\n", "2020-09-17 08:56:38,167 epoch 77 - iter 90/154 - loss 0.21315958 - samples/sec: 527.17\n", "2020-09-17 08:56:39,090 epoch 77 - iter 105/154 - loss 0.21493115 - samples/sec: 540.96\n", "2020-09-17 08:56:40,060 epoch 77 - iter 120/154 - loss 0.21143677 - samples/sec: 511.70\n", "2020-09-17 08:56:40,973 epoch 77 - iter 135/154 - loss 0.21205567 - samples/sec: 546.42\n", "2020-09-17 08:56:41,931 epoch 77 - iter 150/154 - loss 0.21154538 - samples/sec: 518.54\n", "2020-09-17 08:56:42,138 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:56:42,139 EPOCH 77 done: loss 0.2101 - lr 0.0125\n", "2020-09-17 08:56:42,929 DEV : loss 0.36940571665763855 - score 0.8881\n", "2020-09-17 08:56:42,944 BAD EPOCHS (no improvement): 5\n", "2020-09-17 08:56:42,945 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:56:43,011 epoch 78 - iter 0/154 - loss 0.10457741 - samples/sec: 7500.29\n", "2020-09-17 08:56:43,910 epoch 78 - iter 15/154 - loss 0.19965306 - samples/sec: 553.96\n", "2020-09-17 08:56:44,885 epoch 78 - iter 30/154 - loss 0.19438911 - samples/sec: 510.54\n", "2020-09-17 08:56:45,797 epoch 78 - iter 45/154 - loss 0.21651495 - samples/sec: 544.19\n", "2020-09-17 08:56:46,734 epoch 78 - iter 60/154 - loss 0.22343050 - samples/sec: 530.04\n", "2020-09-17 08:56:47,674 epoch 78 - iter 75/154 - loss 0.21957306 - samples/sec: 533.16\n", "2020-09-17 08:56:48,579 epoch 78 - iter 90/154 - loss 0.21583481 - samples/sec: 550.16\n", "2020-09-17 08:56:49,509 epoch 78 - iter 105/154 - loss 0.20975643 - samples/sec: 533.84\n", "2020-09-17 08:56:50,400 epoch 78 - iter 120/154 - loss 0.21030260 - samples/sec: 558.76\n", "2020-09-17 08:56:51,357 epoch 78 - iter 135/154 - loss 0.21114964 - samples/sec: 518.54\n", "2020-09-17 08:56:52,277 epoch 78 - iter 150/154 - loss 0.21061305 - samples/sec: 550.35\n", "2020-09-17 08:56:52,454 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:56:52,455 EPOCH 78 done: loss 0.2143 - lr 0.0125\n", "2020-09-17 08:56:53,211 DEV : loss 0.3582773804664612 - score 0.8972\n", "Epoch 77: reducing learning rate of group 0 to 6.2500e-03.\n", "2020-09-17 08:56:53,224 BAD EPOCHS (no improvement): 6\n", "2020-09-17 08:56:53,226 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:56:53,288 epoch 79 - iter 0/154 - loss 0.18350580 - samples/sec: 7861.71\n", "2020-09-17 08:56:54,251 epoch 79 - iter 15/154 - loss 0.23737658 - samples/sec: 518.26\n", "2020-09-17 08:56:55,163 epoch 79 - iter 30/154 - loss 0.22363136 - samples/sec: 550.01\n", "2020-09-17 08:56:56,133 epoch 79 - iter 45/154 - loss 0.22533384 - samples/sec: 518.19\n", "2020-09-17 08:56:57,082 epoch 79 - iter 60/154 - loss 0.22603961 - samples/sec: 523.14\n", "2020-09-17 08:56:58,034 epoch 79 - iter 75/154 - loss 0.22093921 - samples/sec: 523.23\n", "2020-09-17 08:56:58,993 epoch 79 - iter 90/154 - loss 0.21855001 - samples/sec: 522.29\n", "2020-09-17 08:56:59,955 epoch 79 - iter 105/154 - loss 0.21553115 - samples/sec: 521.24\n", "2020-09-17 08:57:00,928 epoch 79 - iter 120/154 - loss 0.21118295 - samples/sec: 510.65\n", "2020-09-17 08:57:01,837 epoch 79 - iter 135/154 - loss 0.20963630 - samples/sec: 548.67\n", "2020-09-17 08:57:02,819 epoch 79 - iter 150/154 - loss 0.20748087 - samples/sec: 504.61\n", "2020-09-17 08:57:03,017 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:57:03,018 EPOCH 79 done: loss 0.2108 - lr 0.0063\n", "2020-09-17 08:57:03,831 DEV : loss 0.34375301003456116 - score 0.8972\n", "2020-09-17 08:57:03,844 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:57:03,846 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:57:03,907 epoch 80 - iter 0/154 - loss 0.31946254 - samples/sec: 8135.82\n", "2020-09-17 08:57:04,868 epoch 80 - iter 15/154 - loss 0.22119417 - samples/sec: 516.09\n", "2020-09-17 08:57:05,909 epoch 80 - iter 30/154 - loss 0.22698818 - samples/sec: 475.43\n", "2020-09-17 08:57:06,836 epoch 80 - iter 45/154 - loss 0.20867588 - samples/sec: 536.33\n", "2020-09-17 08:57:07,786 epoch 80 - iter 60/154 - loss 0.20241761 - samples/sec: 523.64\n", "2020-09-17 08:57:08,697 epoch 80 - iter 75/154 - loss 0.20945768 - samples/sec: 545.36\n", "2020-09-17 08:57:09,598 epoch 80 - iter 90/154 - loss 0.20732284 - samples/sec: 554.17\n", "2020-09-17 08:57:10,537 epoch 80 - iter 105/154 - loss 0.20360599 - samples/sec: 529.83\n", "2020-09-17 08:57:11,501 epoch 80 - iter 120/154 - loss 0.20277215 - samples/sec: 515.80\n", "2020-09-17 08:57:12,477 epoch 80 - iter 135/154 - loss 0.20411814 - samples/sec: 509.91\n", "2020-09-17 08:57:13,414 epoch 80 - iter 150/154 - loss 0.20686989 - samples/sec: 529.75\n", "2020-09-17 08:57:13,589 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:57:13,590 EPOCH 80 done: loss 0.2074 - lr 0.0063\n", "2020-09-17 08:57:14,354 DEV : loss 0.35142049193382263 - score 0.8954\n", "2020-09-17 08:57:14,371 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:57:14,373 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:57:14,453 epoch 81 - iter 0/154 - loss 0.12080472 - samples/sec: 6233.85\n", "2020-09-17 08:57:15,379 epoch 81 - iter 15/154 - loss 0.18498626 - samples/sec: 541.21\n", "2020-09-17 08:57:16,312 epoch 81 - iter 30/154 - loss 0.18826937 - samples/sec: 532.72\n", "2020-09-17 08:57:17,295 epoch 81 - iter 45/154 - loss 0.20416427 - samples/sec: 512.28\n", "2020-09-17 08:57:18,196 epoch 81 - iter 60/154 - loss 0.21237964 - samples/sec: 550.55\n", "2020-09-17 08:57:19,154 epoch 81 - iter 75/154 - loss 0.20344920 - samples/sec: 518.92\n", "2020-09-17 08:57:20,051 epoch 81 - iter 90/154 - loss 0.20501442 - samples/sec: 557.40\n", "2020-09-17 08:57:20,961 epoch 81 - iter 105/154 - loss 0.20377749 - samples/sec: 550.27\n", "2020-09-17 08:57:21,859 epoch 81 - iter 120/154 - loss 0.20821361 - samples/sec: 554.24\n", "2020-09-17 08:57:22,773 epoch 81 - iter 135/154 - loss 0.20718710 - samples/sec: 543.82\n", "2020-09-17 08:57:23,728 epoch 81 - iter 150/154 - loss 0.21167129 - samples/sec: 522.00\n", "2020-09-17 08:57:23,932 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:57:23,933 EPOCH 81 done: loss 0.2122 - lr 0.0063\n", "2020-09-17 08:57:24,737 DEV : loss 0.3522891402244568 - score 0.8954\n", "2020-09-17 08:57:24,750 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:57:24,751 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:57:24,812 epoch 82 - iter 0/154 - loss 0.11023014 - samples/sec: 8217.71\n", "2020-09-17 08:57:25,744 epoch 82 - iter 15/154 - loss 0.23517658 - samples/sec: 532.31\n", "2020-09-17 08:57:26,748 epoch 82 - iter 30/154 - loss 0.24410558 - samples/sec: 493.62\n", "2020-09-17 08:57:27,711 epoch 82 - iter 45/154 - loss 0.23347459 - samples/sec: 515.57\n", "2020-09-17 08:57:28,733 epoch 82 - iter 60/154 - loss 0.23880031 - samples/sec: 485.10\n", "2020-09-17 08:57:29,645 epoch 82 - iter 75/154 - loss 0.23812296 - samples/sec: 546.59\n", "2020-09-17 08:57:30,545 epoch 82 - iter 90/154 - loss 0.22962674 - samples/sec: 551.09\n", "2020-09-17 08:57:31,489 epoch 82 - iter 105/154 - loss 0.22737497 - samples/sec: 531.58\n", "2020-09-17 08:57:32,449 epoch 82 - iter 120/154 - loss 0.21517936 - samples/sec: 518.33\n", "2020-09-17 08:57:33,466 epoch 82 - iter 135/154 - loss 0.21439298 - samples/sec: 492.62\n", "2020-09-17 08:57:34,367 epoch 82 - iter 150/154 - loss 0.21359836 - samples/sec: 552.38\n", "2020-09-17 08:57:34,556 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:57:34,557 EPOCH 82 done: loss 0.2149 - lr 0.0063\n", "2020-09-17 08:57:35,401 DEV : loss 0.3488753139972687 - score 0.8991\n", "2020-09-17 08:57:35,414 BAD EPOCHS (no improvement): 4\n", "2020-09-17 08:57:35,417 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:57:35,477 epoch 83 - iter 0/154 - loss 0.27051380 - samples/sec: 8135.89\n", "2020-09-17 08:57:36,383 epoch 83 - iter 15/154 - loss 0.19806362 - samples/sec: 549.64\n", "2020-09-17 08:57:37,320 epoch 83 - iter 30/154 - loss 0.19443054 - samples/sec: 534.48\n", "2020-09-17 08:57:38,250 epoch 83 - iter 45/154 - loss 0.19665426 - samples/sec: 536.97\n", "2020-09-17 08:57:39,163 epoch 83 - iter 60/154 - loss 0.19540179 - samples/sec: 547.32\n", "2020-09-17 08:57:40,138 epoch 83 - iter 75/154 - loss 0.19948987 - samples/sec: 509.01\n", "2020-09-17 08:57:41,096 epoch 83 - iter 90/154 - loss 0.20185553 - samples/sec: 517.24\n", "2020-09-17 08:57:42,040 epoch 83 - iter 105/154 - loss 0.20864306 - samples/sec: 526.04\n", "2020-09-17 08:57:43,027 epoch 83 - iter 120/154 - loss 0.20639581 - samples/sec: 508.46\n", "2020-09-17 08:57:43,999 epoch 83 - iter 135/154 - loss 0.20778958 - samples/sec: 513.34\n", "2020-09-17 08:57:45,023 epoch 83 - iter 150/154 - loss 0.20722802 - samples/sec: 485.65\n", "2020-09-17 08:57:45,204 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:57:45,206 EPOCH 83 done: loss 0.2071 - lr 0.0063\n", "2020-09-17 08:57:46,051 DEV : loss 0.34938549995422363 - score 0.8936\n", "2020-09-17 08:57:46,067 BAD EPOCHS (no improvement): 5\n", "2020-09-17 08:57:46,069 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:57:46,133 epoch 84 - iter 0/154 - loss 0.30073690 - samples/sec: 7618.82\n", "2020-09-17 08:57:47,130 epoch 84 - iter 15/154 - loss 0.17434245 - samples/sec: 497.95\n", "2020-09-17 08:57:48,067 epoch 84 - iter 30/154 - loss 0.17652462 - samples/sec: 534.85\n", "2020-09-17 08:57:49,015 epoch 84 - iter 45/154 - loss 0.17599707 - samples/sec: 525.13\n", "2020-09-17 08:57:50,022 epoch 84 - iter 60/154 - loss 0.19367359 - samples/sec: 491.11\n", "2020-09-17 08:57:50,939 epoch 84 - iter 75/154 - loss 0.19998231 - samples/sec: 542.79\n", "2020-09-17 08:57:51,905 epoch 84 - iter 90/154 - loss 0.20433673 - samples/sec: 512.31\n", "2020-09-17 08:57:52,812 epoch 84 - iter 105/154 - loss 0.20109053 - samples/sec: 546.29\n", "2020-09-17 08:57:53,753 epoch 84 - iter 120/154 - loss 0.20539388 - samples/sec: 527.11\n", "2020-09-17 08:57:54,821 epoch 84 - iter 135/154 - loss 0.20491907 - samples/sec: 509.07\n", "2020-09-17 08:57:55,731 epoch 84 - iter 150/154 - loss 0.20144969 - samples/sec: 546.84\n", "2020-09-17 08:57:55,906 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:57:55,908 EPOCH 84 done: loss 0.2008 - lr 0.0063\n", "2020-09-17 08:57:56,733 DEV : loss 0.34447845816612244 - score 0.8954\n", "Epoch 83: reducing learning rate of group 0 to 3.1250e-03.\n", "2020-09-17 08:57:56,747 BAD EPOCHS (no improvement): 6\n", "2020-09-17 08:57:56,749 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:57:56,811 epoch 85 - iter 0/154 - loss 0.25332263 - samples/sec: 8000.10\n", "2020-09-17 08:57:57,732 epoch 85 - iter 15/154 - loss 0.22802983 - samples/sec: 542.30\n", "2020-09-17 08:57:58,703 epoch 85 - iter 30/154 - loss 0.24149939 - samples/sec: 511.71\n", "2020-09-17 08:57:59,604 epoch 85 - iter 45/154 - loss 0.22362376 - samples/sec: 551.40\n", "2020-09-17 08:58:00,566 epoch 85 - iter 60/154 - loss 0.22173792 - samples/sec: 517.92\n", "2020-09-17 08:58:01,502 epoch 85 - iter 75/154 - loss 0.21326863 - samples/sec: 531.00\n", "2020-09-17 08:58:02,411 epoch 85 - iter 90/154 - loss 0.20674201 - samples/sec: 548.03\n", "2020-09-17 08:58:03,390 epoch 85 - iter 105/154 - loss 0.20086463 - samples/sec: 507.15\n", "2020-09-17 08:58:04,338 epoch 85 - iter 120/154 - loss 0.19930819 - samples/sec: 522.67\n", "2020-09-17 08:58:05,253 epoch 85 - iter 135/154 - loss 0.19934071 - samples/sec: 542.73\n", "2020-09-17 08:58:06,221 epoch 85 - iter 150/154 - loss 0.20141996 - samples/sec: 519.02\n", "2020-09-17 08:58:06,406 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:58:06,407 EPOCH 85 done: loss 0.2005 - lr 0.0031\n", "2020-09-17 08:58:07,168 DEV : loss 0.348294734954834 - score 0.8954\n", "2020-09-17 08:58:07,182 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:58:07,183 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:58:07,242 epoch 86 - iter 0/154 - loss 0.14703789 - samples/sec: 8417.02\n", "2020-09-17 08:58:08,173 epoch 86 - iter 15/154 - loss 0.22611129 - samples/sec: 533.73\n", "2020-09-17 08:58:09,075 epoch 86 - iter 30/154 - loss 0.21641227 - samples/sec: 551.87\n", "2020-09-17 08:58:10,044 epoch 86 - iter 45/154 - loss 0.21459843 - samples/sec: 514.84\n", "2020-09-17 08:58:11,038 epoch 86 - iter 60/154 - loss 0.21298746 - samples/sec: 506.86\n", "2020-09-17 08:58:11,984 epoch 86 - iter 75/154 - loss 0.22560209 - samples/sec: 526.27\n", "2020-09-17 08:58:12,941 epoch 86 - iter 90/154 - loss 0.22529634 - samples/sec: 517.43\n", "2020-09-17 08:58:13,851 epoch 86 - iter 105/154 - loss 0.22412135 - samples/sec: 546.94\n", "2020-09-17 08:58:14,831 epoch 86 - iter 120/154 - loss 0.22025260 - samples/sec: 505.25\n", "2020-09-17 08:58:15,711 epoch 86 - iter 135/154 - loss 0.21751637 - samples/sec: 564.14\n", "2020-09-17 08:58:16,654 epoch 86 - iter 150/154 - loss 0.22112219 - samples/sec: 525.16\n", "2020-09-17 08:58:16,839 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:58:16,840 EPOCH 86 done: loss 0.2189 - lr 0.0031\n", "2020-09-17 08:58:17,716 DEV : loss 0.3464142978191376 - score 0.8991\n", "2020-09-17 08:58:17,730 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:58:17,731 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:58:17,797 epoch 87 - iter 0/154 - loss 0.17093441 - samples/sec: 7493.82\n", "2020-09-17 08:58:18,755 epoch 87 - iter 15/154 - loss 0.17656192 - samples/sec: 518.49\n", "2020-09-17 08:58:19,693 epoch 87 - iter 30/154 - loss 0.17774919 - samples/sec: 529.77\n", "2020-09-17 08:58:20,617 epoch 87 - iter 45/154 - loss 0.17035296 - samples/sec: 537.27\n", "2020-09-17 08:58:21,585 epoch 87 - iter 60/154 - loss 0.18705282 - samples/sec: 514.73\n", "2020-09-17 08:58:22,527 epoch 87 - iter 75/154 - loss 0.18801695 - samples/sec: 534.49\n", "2020-09-17 08:58:23,447 epoch 87 - iter 90/154 - loss 0.17975359 - samples/sec: 542.14\n", "2020-09-17 08:58:24,363 epoch 87 - iter 105/154 - loss 0.18130305 - samples/sec: 543.91\n", "2020-09-17 08:58:25,261 epoch 87 - iter 120/154 - loss 0.18589255 - samples/sec: 554.24\n", "2020-09-17 08:58:26,187 epoch 87 - iter 135/154 - loss 0.18957998 - samples/sec: 537.51\n", "2020-09-17 08:58:27,147 epoch 87 - iter 150/154 - loss 0.19379076 - samples/sec: 518.92\n", "2020-09-17 08:58:27,335 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:58:27,337 EPOCH 87 done: loss 0.1950 - lr 0.0031\n", "2020-09-17 08:58:28,105 DEV : loss 0.35061216354370117 - score 0.8936\n", "2020-09-17 08:58:28,119 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:58:28,121 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:58:28,189 epoch 88 - iter 0/154 - loss 0.17264447 - samples/sec: 7272.55\n", "2020-09-17 08:58:29,177 epoch 88 - iter 15/154 - loss 0.19665443 - samples/sec: 504.70\n", "2020-09-17 08:58:30,073 epoch 88 - iter 30/154 - loss 0.19054690 - samples/sec: 556.37\n", "2020-09-17 08:58:31,044 epoch 88 - iter 45/154 - loss 0.19325137 - samples/sec: 511.18\n", "2020-09-17 08:58:31,950 epoch 88 - iter 60/154 - loss 0.19255885 - samples/sec: 550.49\n", "2020-09-17 08:58:32,891 epoch 88 - iter 75/154 - loss 0.19657003 - samples/sec: 527.56\n", "2020-09-17 08:58:33,838 epoch 88 - iter 90/154 - loss 0.20072648 - samples/sec: 525.20\n", "2020-09-17 08:58:34,747 epoch 88 - iter 105/154 - loss 0.20646886 - samples/sec: 549.49\n", "2020-09-17 08:58:35,693 epoch 88 - iter 120/154 - loss 0.20614481 - samples/sec: 524.00\n", "2020-09-17 08:58:36,610 epoch 88 - iter 135/154 - loss 0.20215706 - samples/sec: 542.30\n", "2020-09-17 08:58:37,554 epoch 88 - iter 150/154 - loss 0.20164935 - samples/sec: 531.65\n", "2020-09-17 08:58:37,773 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:58:37,774 EPOCH 88 done: loss 0.2017 - lr 0.0031\n", "2020-09-17 08:58:38,588 DEV : loss 0.34918200969696045 - score 0.8954\n", "2020-09-17 08:58:38,601 BAD EPOCHS (no improvement): 4\n", "2020-09-17 08:58:38,603 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:58:38,665 epoch 89 - iter 0/154 - loss 0.19630349 - samples/sec: 7862.35\n", "2020-09-17 08:58:39,615 epoch 89 - iter 15/154 - loss 0.23045901 - samples/sec: 523.54\n", "2020-09-17 08:58:40,616 epoch 89 - iter 30/154 - loss 0.21113894 - samples/sec: 497.29\n", "2020-09-17 08:58:41,524 epoch 89 - iter 45/154 - loss 0.20677737 - samples/sec: 553.93\n", "2020-09-17 08:58:42,529 epoch 89 - iter 60/154 - loss 0.20668796 - samples/sec: 493.83\n", "2020-09-17 08:58:43,468 epoch 89 - iter 75/154 - loss 0.20248039 - samples/sec: 533.31\n", "2020-09-17 08:58:44,371 epoch 89 - iter 90/154 - loss 0.19903296 - samples/sec: 550.46\n", "2020-09-17 08:58:45,344 epoch 89 - iter 105/154 - loss 0.20027709 - samples/sec: 510.08\n", "2020-09-17 08:58:46,251 epoch 89 - iter 120/154 - loss 0.19923817 - samples/sec: 546.09\n", "2020-09-17 08:58:47,229 epoch 89 - iter 135/154 - loss 0.20008632 - samples/sec: 514.50\n", "2020-09-17 08:58:48,122 epoch 89 - iter 150/154 - loss 0.20008628 - samples/sec: 558.10\n", "2020-09-17 08:58:48,304 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:58:48,305 EPOCH 89 done: loss 0.2003 - lr 0.0031\n", "2020-09-17 08:58:49,061 DEV : loss 0.34965890645980835 - score 0.8899\n", "2020-09-17 08:58:49,075 BAD EPOCHS (no improvement): 5\n", "2020-09-17 08:58:49,077 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:58:49,139 epoch 90 - iter 0/154 - loss 0.10052601 - samples/sec: 7862.17\n", "2020-09-17 08:58:50,107 epoch 90 - iter 15/154 - loss 0.18891970 - samples/sec: 511.51\n", "2020-09-17 08:58:51,037 epoch 90 - iter 30/154 - loss 0.19010226 - samples/sec: 532.58\n", "2020-09-17 08:58:52,020 epoch 90 - iter 45/154 - loss 0.18241612 - samples/sec: 504.50\n", "2020-09-17 08:58:52,937 epoch 90 - iter 60/154 - loss 0.18142625 - samples/sec: 542.72\n", "2020-09-17 08:58:53,852 epoch 90 - iter 75/154 - loss 0.19480905 - samples/sec: 543.55\n", "2020-09-17 08:58:54,839 epoch 90 - iter 90/154 - loss 0.19135572 - samples/sec: 506.86\n", "2020-09-17 08:58:55,747 epoch 90 - iter 105/154 - loss 0.19648329 - samples/sec: 547.37\n", "2020-09-17 08:58:56,725 epoch 90 - iter 120/154 - loss 0.19760559 - samples/sec: 507.01\n", "2020-09-17 08:58:57,636 epoch 90 - iter 135/154 - loss 0.19952531 - samples/sec: 544.87\n", "2020-09-17 08:58:58,572 epoch 90 - iter 150/154 - loss 0.19973082 - samples/sec: 530.99\n", "2020-09-17 08:58:58,793 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:58:58,795 EPOCH 90 done: loss 0.1989 - lr 0.0031\n", "2020-09-17 08:58:59,545 DEV : loss 0.3538850247859955 - score 0.8917\n", "Epoch 89: reducing learning rate of group 0 to 1.5625e-03.\n", "2020-09-17 08:58:59,559 BAD EPOCHS (no improvement): 6\n", "2020-09-17 08:58:59,560 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:58:59,618 epoch 91 - iter 0/154 - loss 0.12061109 - samples/sec: 8567.13\n", "2020-09-17 08:59:00,521 epoch 91 - iter 15/154 - loss 0.17572917 - samples/sec: 548.87\n", "2020-09-17 08:59:01,463 epoch 91 - iter 30/154 - loss 0.18700310 - samples/sec: 529.20\n", "2020-09-17 08:59:02,340 epoch 91 - iter 45/154 - loss 0.20565371 - samples/sec: 567.84\n", "2020-09-17 08:59:03,315 epoch 91 - iter 60/154 - loss 0.20700233 - samples/sec: 515.35\n", "2020-09-17 08:59:04,207 epoch 91 - iter 75/154 - loss 0.20668810 - samples/sec: 561.59\n", "2020-09-17 08:59:05,181 epoch 91 - iter 90/154 - loss 0.20461336 - samples/sec: 511.77\n", "2020-09-17 08:59:06,196 epoch 91 - iter 105/154 - loss 0.20521466 - samples/sec: 487.30\n", "2020-09-17 08:59:07,075 epoch 91 - iter 120/154 - loss 0.20385320 - samples/sec: 566.35\n", "2020-09-17 08:59:08,026 epoch 91 - iter 135/154 - loss 0.20582380 - samples/sec: 522.65\n", "2020-09-17 08:59:08,937 epoch 91 - iter 150/154 - loss 0.20782015 - samples/sec: 551.60\n", "2020-09-17 08:59:09,123 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:59:09,124 EPOCH 91 done: loss 0.2090 - lr 0.0016\n", "2020-09-17 08:59:09,873 DEV : loss 0.35197845101356506 - score 0.8936\n", "2020-09-17 08:59:09,887 BAD EPOCHS (no improvement): 1\n", "2020-09-17 08:59:09,888 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:59:09,952 epoch 92 - iter 0/154 - loss 0.16349667 - samples/sec: 7742.50\n", "2020-09-17 08:59:10,898 epoch 92 - iter 15/154 - loss 0.21994169 - samples/sec: 523.02\n", "2020-09-17 08:59:11,832 epoch 92 - iter 30/154 - loss 0.20306743 - samples/sec: 532.88\n", "2020-09-17 08:59:12,818 epoch 92 - iter 45/154 - loss 0.18917445 - samples/sec: 502.05\n", "2020-09-17 08:59:13,759 epoch 92 - iter 60/154 - loss 0.20238637 - samples/sec: 531.57\n", "2020-09-17 08:59:14,681 epoch 92 - iter 75/154 - loss 0.21084404 - samples/sec: 541.12\n", "2020-09-17 08:59:15,695 epoch 92 - iter 90/154 - loss 0.21676890 - samples/sec: 490.52\n", "2020-09-17 08:59:16,653 epoch 92 - iter 105/154 - loss 0.21388395 - samples/sec: 516.66\n", "2020-09-17 08:59:17,650 epoch 92 - iter 120/154 - loss 0.21374192 - samples/sec: 497.83\n", "2020-09-17 08:59:18,549 epoch 92 - iter 135/154 - loss 0.20831294 - samples/sec: 555.51\n", "2020-09-17 08:59:19,460 epoch 92 - iter 150/154 - loss 0.21132739 - samples/sec: 544.92\n", "2020-09-17 08:59:19,691 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:59:19,693 EPOCH 92 done: loss 0.2117 - lr 0.0016\n", "2020-09-17 08:59:20,457 DEV : loss 0.35343530774116516 - score 0.8936\n", "2020-09-17 08:59:20,471 BAD EPOCHS (no improvement): 2\n", "2020-09-17 08:59:20,472 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:59:20,532 epoch 93 - iter 0/154 - loss 0.35816035 - samples/sec: 8272.01\n", "2020-09-17 08:59:21,448 epoch 93 - iter 15/154 - loss 0.22168525 - samples/sec: 541.01\n", "2020-09-17 08:59:22,387 epoch 93 - iter 30/154 - loss 0.20326122 - samples/sec: 526.86\n", "2020-09-17 08:59:23,341 epoch 93 - iter 45/154 - loss 0.21046913 - samples/sec: 522.98\n", "2020-09-17 08:59:24,327 epoch 93 - iter 60/154 - loss 0.21000417 - samples/sec: 502.09\n", "2020-09-17 08:59:25,270 epoch 93 - iter 75/154 - loss 0.19734261 - samples/sec: 532.71\n", "2020-09-17 08:59:26,183 epoch 93 - iter 90/154 - loss 0.19554539 - samples/sec: 543.23\n", "2020-09-17 08:59:27,126 epoch 93 - iter 105/154 - loss 0.20435749 - samples/sec: 527.00\n", "2020-09-17 08:59:28,054 epoch 93 - iter 120/154 - loss 0.20583527 - samples/sec: 534.29\n", "2020-09-17 08:59:28,994 epoch 93 - iter 135/154 - loss 0.20423599 - samples/sec: 527.81\n", "2020-09-17 08:59:29,903 epoch 93 - iter 150/154 - loss 0.20439773 - samples/sec: 545.21\n", "2020-09-17 08:59:30,078 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:59:30,079 EPOCH 93 done: loss 0.2036 - lr 0.0016\n", "2020-09-17 08:59:30,850 DEV : loss 0.3528648614883423 - score 0.8936\n", "2020-09-17 08:59:30,863 BAD EPOCHS (no improvement): 3\n", "2020-09-17 08:59:30,866 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:59:30,936 epoch 94 - iter 0/154 - loss 0.20543510 - samples/sec: 7056.17\n", "2020-09-17 08:59:31,920 epoch 94 - iter 15/154 - loss 0.18314623 - samples/sec: 506.37\n", "2020-09-17 08:59:32,868 epoch 94 - iter 30/154 - loss 0.17561469 - samples/sec: 524.60\n", "2020-09-17 08:59:33,805 epoch 94 - iter 45/154 - loss 0.18612487 - samples/sec: 530.38\n", "2020-09-17 08:59:34,732 epoch 94 - iter 60/154 - loss 0.18909314 - samples/sec: 535.32\n", "2020-09-17 08:59:35,679 epoch 94 - iter 75/154 - loss 0.19540384 - samples/sec: 523.97\n", "2020-09-17 08:59:36,672 epoch 94 - iter 90/154 - loss 0.19395719 - samples/sec: 508.95\n", "2020-09-17 08:59:37,582 epoch 94 - iter 105/154 - loss 0.19682153 - samples/sec: 547.74\n", "2020-09-17 08:59:38,760 epoch 94 - iter 120/154 - loss 0.19775217 - samples/sec: 424.95\n", "2020-09-17 08:59:39,720 epoch 94 - iter 135/154 - loss 0.19396744 - samples/sec: 521.15\n", "2020-09-17 08:59:40,792 epoch 94 - iter 150/154 - loss 0.19078726 - samples/sec: 461.53\n", "2020-09-17 08:59:40,975 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:59:40,977 EPOCH 94 done: loss 0.1904 - lr 0.0016\n", "2020-09-17 08:59:41,877 DEV : loss 0.3519933521747589 - score 0.8936\n", "2020-09-17 08:59:41,892 BAD EPOCHS (no improvement): 4\n", "2020-09-17 08:59:41,894 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:59:41,965 epoch 95 - iter 0/154 - loss 0.48848724 - samples/sec: 6952.63\n", "2020-09-17 08:59:43,037 epoch 95 - iter 15/154 - loss 0.26462717 - samples/sec: 461.07\n", "2020-09-17 08:59:44,062 epoch 95 - iter 30/154 - loss 0.25377122 - samples/sec: 486.84\n", "2020-09-17 08:59:45,098 epoch 95 - iter 45/154 - loss 0.22464356 - samples/sec: 480.49\n", "2020-09-17 08:59:46,069 epoch 95 - iter 60/154 - loss 0.21843755 - samples/sec: 520.15\n", "2020-09-17 08:59:47,011 epoch 95 - iter 75/154 - loss 0.21002987 - samples/sec: 530.20\n", "2020-09-17 08:59:48,009 epoch 95 - iter 90/154 - loss 0.20568888 - samples/sec: 497.90\n", "2020-09-17 08:59:48,999 epoch 95 - iter 105/154 - loss 0.21108971 - samples/sec: 504.13\n", "2020-09-17 08:59:49,987 epoch 95 - iter 120/154 - loss 0.21297037 - samples/sec: 503.62\n", "2020-09-17 08:59:50,921 epoch 95 - iter 135/154 - loss 0.21761241 - samples/sec: 530.44\n", "2020-09-17 08:59:51,879 epoch 95 - iter 150/154 - loss 0.21757455 - samples/sec: 523.08\n", "2020-09-17 08:59:52,102 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:59:52,104 EPOCH 95 done: loss 0.2158 - lr 0.0016\n", "2020-09-17 08:59:52,912 DEV : loss 0.353048175573349 - score 0.8954\n", "2020-09-17 08:59:52,932 BAD EPOCHS (no improvement): 5\n", "2020-09-17 08:59:52,933 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 08:59:53,005 epoch 96 - iter 0/154 - loss 0.40940005 - samples/sec: 6857.48\n", "2020-09-17 08:59:53,894 epoch 96 - iter 15/154 - loss 0.19266923 - samples/sec: 560.32\n", "2020-09-17 08:59:54,853 epoch 96 - iter 30/154 - loss 0.20844424 - samples/sec: 521.54\n", "2020-09-17 08:59:55,838 epoch 96 - iter 45/154 - loss 0.20949856 - samples/sec: 506.57\n", "2020-09-17 08:59:56,900 epoch 96 - iter 60/154 - loss 0.20419879 - samples/sec: 474.60\n", "2020-09-17 08:59:57,888 epoch 96 - iter 75/154 - loss 0.20943078 - samples/sec: 505.17\n", "2020-09-17 08:59:58,841 epoch 96 - iter 90/154 - loss 0.20326085 - samples/sec: 526.99\n", "2020-09-17 08:59:59,899 epoch 96 - iter 105/154 - loss 0.20599897 - samples/sec: 471.57\n", "2020-09-17 09:00:00,997 epoch 96 - iter 120/154 - loss 0.20523324 - samples/sec: 458.24\n", "2020-09-17 09:00:02,057 epoch 96 - iter 135/154 - loss 0.20261321 - samples/sec: 474.78\n", "2020-09-17 09:00:03,049 epoch 96 - iter 150/154 - loss 0.20770752 - samples/sec: 503.16\n", "2020-09-17 09:00:03,241 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:00:03,242 EPOCH 96 done: loss 0.2064 - lr 0.0016\n", "2020-09-17 09:00:04,071 DEV : loss 0.3509064316749573 - score 0.8954\n", "Epoch 95: reducing learning rate of group 0 to 7.8125e-04.\n", "2020-09-17 09:00:04,086 BAD EPOCHS (no improvement): 6\n", "2020-09-17 09:00:04,088 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:00:04,154 epoch 97 - iter 0/154 - loss 0.16026522 - samples/sec: 7497.14\n", "2020-09-17 09:00:05,076 epoch 97 - iter 15/154 - loss 0.20708696 - samples/sec: 543.71\n", "2020-09-17 09:00:06,038 epoch 97 - iter 30/154 - loss 0.22716866 - samples/sec: 518.94\n", "2020-09-17 09:00:06,984 epoch 97 - iter 45/154 - loss 0.21697830 - samples/sec: 530.01\n", "2020-09-17 09:00:07,915 epoch 97 - iter 60/154 - loss 0.21227694 - samples/sec: 537.52\n", "2020-09-17 09:00:08,918 epoch 97 - iter 75/154 - loss 0.20827895 - samples/sec: 495.57\n", "2020-09-17 09:00:09,966 epoch 97 - iter 90/154 - loss 0.20933206 - samples/sec: 482.40\n", "2020-09-17 09:00:11,098 epoch 97 - iter 105/154 - loss 0.21164651 - samples/sec: 438.36\n", "2020-09-17 09:00:12,139 epoch 97 - iter 120/154 - loss 0.20936433 - samples/sec: 481.91\n", "2020-09-17 09:00:13,168 epoch 97 - iter 135/154 - loss 0.20622279 - samples/sec: 485.04\n", "2020-09-17 09:00:14,098 epoch 97 - iter 150/154 - loss 0.20289364 - samples/sec: 537.11\n", "2020-09-17 09:00:14,275 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:00:14,276 EPOCH 97 done: loss 0.2016 - lr 0.0008\n", "2020-09-17 09:00:15,077 DEV : loss 0.35326170921325684 - score 0.8936\n", "2020-09-17 09:00:15,091 BAD EPOCHS (no improvement): 1\n", "2020-09-17 09:00:15,093 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:00:15,171 epoch 98 - iter 0/154 - loss 0.13484402 - samples/sec: 6399.14\n", "2020-09-17 09:00:16,151 epoch 98 - iter 15/154 - loss 0.17710817 - samples/sec: 512.78\n", "2020-09-17 09:00:17,103 epoch 98 - iter 30/154 - loss 0.22453021 - samples/sec: 526.97\n", "2020-09-17 09:00:18,112 epoch 98 - iter 45/154 - loss 0.22531349 - samples/sec: 494.79\n", "2020-09-17 09:00:19,105 epoch 98 - iter 60/154 - loss 0.21059833 - samples/sec: 501.99\n", "2020-09-17 09:00:20,132 epoch 98 - iter 75/154 - loss 0.20904318 - samples/sec: 484.82\n", "2020-09-17 09:00:21,090 epoch 98 - iter 90/154 - loss 0.20969410 - samples/sec: 526.25\n", "2020-09-17 09:00:22,033 epoch 98 - iter 105/154 - loss 0.20717529 - samples/sec: 531.48\n", "2020-09-17 09:00:23,017 epoch 98 - iter 120/154 - loss 0.19614112 - samples/sec: 506.22\n", "2020-09-17 09:00:23,933 epoch 98 - iter 135/154 - loss 0.19418144 - samples/sec: 545.24\n", "2020-09-17 09:00:24,952 epoch 98 - iter 150/154 - loss 0.19993429 - samples/sec: 488.82\n", "2020-09-17 09:00:25,180 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:00:25,182 EPOCH 98 done: loss 0.1996 - lr 0.0008\n", "2020-09-17 09:00:25,949 DEV : loss 0.3547529876232147 - score 0.8917\n", "2020-09-17 09:00:25,964 BAD EPOCHS (no improvement): 2\n", "2020-09-17 09:00:25,966 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:00:26,044 epoch 99 - iter 0/154 - loss 0.17142858 - samples/sec: 6315.97\n", "2020-09-17 09:00:27,030 epoch 99 - iter 15/154 - loss 0.18261656 - samples/sec: 506.33\n", "2020-09-17 09:00:27,956 epoch 99 - iter 30/154 - loss 0.17872032 - samples/sec: 544.98\n", "2020-09-17 09:00:28,894 epoch 99 - iter 45/154 - loss 0.18891201 - samples/sec: 531.07\n", "2020-09-17 09:00:29,845 epoch 99 - iter 60/154 - loss 0.18904214 - samples/sec: 524.78\n", "2020-09-17 09:00:30,743 epoch 99 - iter 75/154 - loss 0.18713544 - samples/sec: 555.55\n", "2020-09-17 09:00:31,739 epoch 99 - iter 90/154 - loss 0.18290257 - samples/sec: 501.49\n", "2020-09-17 09:00:32,669 epoch 99 - iter 105/154 - loss 0.19177409 - samples/sec: 538.84\n", "2020-09-17 09:00:33,591 epoch 99 - iter 120/154 - loss 0.19137253 - samples/sec: 546.68\n", "2020-09-17 09:00:34,573 epoch 99 - iter 135/154 - loss 0.19713857 - samples/sec: 509.07\n", "2020-09-17 09:00:35,512 epoch 99 - iter 150/154 - loss 0.19641551 - samples/sec: 535.38\n", "2020-09-17 09:00:35,705 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:00:35,706 EPOCH 99 done: loss 0.1975 - lr 0.0008\n", "2020-09-17 09:00:36,527 DEV : loss 0.35381194949150085 - score 0.8954\n", "2020-09-17 09:00:36,541 BAD EPOCHS (no improvement): 3\n", "2020-09-17 09:00:36,542 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:00:36,611 epoch 100 - iter 0/154 - loss 0.18002027 - samples/sec: 7165.18\n", "2020-09-17 09:00:37,543 epoch 100 - iter 15/154 - loss 0.22265865 - samples/sec: 535.60\n", "2020-09-17 09:00:38,511 epoch 100 - iter 30/154 - loss 0.19932022 - samples/sec: 515.49\n", "2020-09-17 09:00:39,437 epoch 100 - iter 45/154 - loss 0.19427421 - samples/sec: 541.71\n", "2020-09-17 09:00:40,367 epoch 100 - iter 60/154 - loss 0.19469786 - samples/sec: 538.02\n", "2020-09-17 09:00:41,346 epoch 100 - iter 75/154 - loss 0.20100573 - samples/sec: 508.02\n", "2020-09-17 09:00:42,254 epoch 100 - iter 90/154 - loss 0.19596637 - samples/sec: 550.24\n", "2020-09-17 09:00:43,218 epoch 100 - iter 105/154 - loss 0.19620196 - samples/sec: 519.59\n", "2020-09-17 09:00:44,183 epoch 100 - iter 120/154 - loss 0.19848352 - samples/sec: 516.21\n", "2020-09-17 09:00:45,091 epoch 100 - iter 135/154 - loss 0.19770504 - samples/sec: 550.42\n", "2020-09-17 09:00:46,113 epoch 100 - iter 150/154 - loss 0.19677007 - samples/sec: 487.80\n", "2020-09-17 09:00:46,302 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:00:46,303 EPOCH 100 done: loss 0.1947 - lr 0.0008\n", "2020-09-17 09:00:47,102 DEV : loss 0.35407784581184387 - score 0.8954\n", "2020-09-17 09:00:47,117 BAD EPOCHS (no improvement): 4\n", "2020-09-17 09:00:47,120 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:00:47,192 epoch 101 - iter 0/154 - loss 0.50545204 - samples/sec: 6859.79\n", "2020-09-17 09:00:48,202 epoch 101 - iter 15/154 - loss 0.22828249 - samples/sec: 493.32\n", "2020-09-17 09:00:49,177 epoch 101 - iter 30/154 - loss 0.21774821 - samples/sec: 512.14\n", "2020-09-17 09:00:50,151 epoch 101 - iter 45/154 - loss 0.20723694 - samples/sec: 512.77\n", "2020-09-17 09:00:51,083 epoch 101 - iter 60/154 - loss 0.19370842 - samples/sec: 535.61\n", "2020-09-17 09:00:52,069 epoch 101 - iter 75/154 - loss 0.20111699 - samples/sec: 506.34\n", "2020-09-17 09:00:53,053 epoch 101 - iter 90/154 - loss 0.19283405 - samples/sec: 506.82\n", "2020-09-17 09:00:53,992 epoch 101 - iter 105/154 - loss 0.19919413 - samples/sec: 534.97\n", "2020-09-17 09:00:54,998 epoch 101 - iter 120/154 - loss 0.20383884 - samples/sec: 493.99\n", "2020-09-17 09:00:55,902 epoch 101 - iter 135/154 - loss 0.20101488 - samples/sec: 550.40\n", "2020-09-17 09:00:56,818 epoch 101 - iter 150/154 - loss 0.20306750 - samples/sec: 548.23\n", "2020-09-17 09:00:57,024 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:00:57,026 EPOCH 101 done: loss 0.2024 - lr 0.0008\n", "2020-09-17 09:00:57,794 DEV : loss 0.3529554307460785 - score 0.8954\n", "2020-09-17 09:00:57,808 BAD EPOCHS (no improvement): 5\n", "2020-09-17 09:00:57,810 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:00:57,878 epoch 102 - iter 0/154 - loss 0.45573252 - samples/sec: 7269.66\n", "2020-09-17 09:00:58,825 epoch 102 - iter 15/154 - loss 0.18777354 - samples/sec: 527.62\n", "2020-09-17 09:00:59,784 epoch 102 - iter 30/154 - loss 0.20331153 - samples/sec: 522.33\n", "2020-09-17 09:01:00,708 epoch 102 - iter 45/154 - loss 0.20296005 - samples/sec: 542.27\n", "2020-09-17 09:01:01,718 epoch 102 - iter 60/154 - loss 0.20543727 - samples/sec: 492.16\n", "2020-09-17 09:01:02,710 epoch 102 - iter 75/154 - loss 0.21048815 - samples/sec: 502.49\n", "2020-09-17 09:01:03,668 epoch 102 - iter 90/154 - loss 0.20406329 - samples/sec: 522.86\n", "2020-09-17 09:01:04,638 epoch 102 - iter 105/154 - loss 0.20410561 - samples/sec: 519.21\n", "2020-09-17 09:01:05,606 epoch 102 - iter 120/154 - loss 0.20167945 - samples/sec: 522.99\n", "2020-09-17 09:01:06,633 epoch 102 - iter 135/154 - loss 0.20338036 - samples/sec: 484.63\n", "2020-09-17 09:01:07,690 epoch 102 - iter 150/154 - loss 0.20687685 - samples/sec: 472.89\n", "2020-09-17 09:01:07,924 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:01:07,926 EPOCH 102 done: loss 0.2065 - lr 0.0008\n", "2020-09-17 09:01:08,881 DEV : loss 0.3523719310760498 - score 0.8954\n", "Epoch 101: reducing learning rate of group 0 to 3.9063e-04.\n", "2020-09-17 09:01:08,899 BAD EPOCHS (no improvement): 6\n", "2020-09-17 09:01:08,900 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:01:08,973 epoch 103 - iter 0/154 - loss 0.38171446 - samples/sec: 6857.08\n", "2020-09-17 09:01:10,079 epoch 103 - iter 15/154 - loss 0.25291864 - samples/sec: 456.28\n", "2020-09-17 09:01:11,046 epoch 103 - iter 30/154 - loss 0.21467162 - samples/sec: 515.56\n", "2020-09-17 09:01:11,972 epoch 103 - iter 45/154 - loss 0.20174437 - samples/sec: 539.21\n", "2020-09-17 09:01:12,913 epoch 103 - iter 60/154 - loss 0.20632305 - samples/sec: 529.25\n", "2020-09-17 09:01:13,998 epoch 103 - iter 75/154 - loss 0.20848461 - samples/sec: 458.45\n", "2020-09-17 09:01:14,985 epoch 103 - iter 90/154 - loss 0.20136205 - samples/sec: 508.45\n", "2020-09-17 09:01:16,096 epoch 103 - iter 105/154 - loss 0.20075919 - samples/sec: 447.74\n", "2020-09-17 09:01:17,055 epoch 103 - iter 120/154 - loss 0.20038580 - samples/sec: 522.83\n", "2020-09-17 09:01:18,017 epoch 103 - iter 135/154 - loss 0.20300972 - samples/sec: 520.73\n", "2020-09-17 09:01:18,953 epoch 103 - iter 150/154 - loss 0.20085003 - samples/sec: 534.53\n", "2020-09-17 09:01:19,131 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:01:19,132 EPOCH 103 done: loss 0.2003 - lr 0.0004\n", "2020-09-17 09:01:19,924 DEV : loss 0.3530525267124176 - score 0.8954\n", "2020-09-17 09:01:19,942 BAD EPOCHS (no improvement): 1\n", "2020-09-17 09:01:19,945 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:01:20,022 epoch 104 - iter 0/154 - loss 0.18856579 - samples/sec: 6400.04\n", "2020-09-17 09:01:20,936 epoch 104 - iter 15/154 - loss 0.19653357 - samples/sec: 549.81\n", "2020-09-17 09:01:21,854 epoch 104 - iter 30/154 - loss 0.19587232 - samples/sec: 547.29\n", "2020-09-17 09:01:22,847 epoch 104 - iter 45/154 - loss 0.20175948 - samples/sec: 500.89\n", "2020-09-17 09:01:23,760 epoch 104 - iter 60/154 - loss 0.19812471 - samples/sec: 546.66\n", "2020-09-17 09:01:24,739 epoch 104 - iter 75/154 - loss 0.20303145 - samples/sec: 509.44\n", "2020-09-17 09:01:25,640 epoch 104 - iter 90/154 - loss 0.19988181 - samples/sec: 557.38\n", "2020-09-17 09:01:26,529 epoch 104 - iter 105/154 - loss 0.20011058 - samples/sec: 560.53\n", "2020-09-17 09:01:27,510 epoch 104 - iter 120/154 - loss 0.19760894 - samples/sec: 507.94\n", "2020-09-17 09:01:28,446 epoch 104 - iter 135/154 - loss 0.19243152 - samples/sec: 532.97\n", "2020-09-17 09:01:29,439 epoch 104 - iter 150/154 - loss 0.18997412 - samples/sec: 500.05\n", "2020-09-17 09:01:29,638 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:01:29,639 EPOCH 104 done: loss 0.1906 - lr 0.0004\n", "2020-09-17 09:01:30,406 DEV : loss 0.3522569239139557 - score 0.8954\n", "2020-09-17 09:01:30,422 BAD EPOCHS (no improvement): 2\n", "2020-09-17 09:01:30,424 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:01:30,494 epoch 105 - iter 0/154 - loss 0.37780488 - samples/sec: 7055.28\n", "2020-09-17 09:01:31,401 epoch 105 - iter 15/154 - loss 0.23863202 - samples/sec: 550.77\n", "2020-09-17 09:01:32,355 epoch 105 - iter 30/154 - loss 0.19937770 - samples/sec: 528.63\n", "2020-09-17 09:01:33,281 epoch 105 - iter 45/154 - loss 0.19227112 - samples/sec: 540.41\n", "2020-09-17 09:01:34,236 epoch 105 - iter 60/154 - loss 0.19698372 - samples/sec: 520.62\n", "2020-09-17 09:01:35,150 epoch 105 - iter 75/154 - loss 0.19655155 - samples/sec: 548.27\n", "2020-09-17 09:01:36,079 epoch 105 - iter 90/154 - loss 0.19189787 - samples/sec: 537.81\n", "2020-09-17 09:01:37,046 epoch 105 - iter 105/154 - loss 0.19658792 - samples/sec: 521.93\n", "2020-09-17 09:01:37,951 epoch 105 - iter 120/154 - loss 0.19845687 - samples/sec: 553.65\n", "2020-09-17 09:01:38,932 epoch 105 - iter 135/154 - loss 0.19771562 - samples/sec: 505.76\n", "2020-09-17 09:01:39,865 epoch 105 - iter 150/154 - loss 0.19875597 - samples/sec: 535.06\n", "2020-09-17 09:01:40,045 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:01:40,047 EPOCH 105 done: loss 0.1999 - lr 0.0004\n", "2020-09-17 09:01:40,881 DEV : loss 0.3521348536014557 - score 0.8954\n", "2020-09-17 09:01:40,904 BAD EPOCHS (no improvement): 3\n", "2020-09-17 09:01:40,906 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:01:40,966 epoch 106 - iter 0/154 - loss 0.29063112 - samples/sec: 8127.97\n", "2020-09-17 09:01:41,887 epoch 106 - iter 15/154 - loss 0.23007246 - samples/sec: 539.89\n", "2020-09-17 09:01:42,825 epoch 106 - iter 30/154 - loss 0.21038671 - samples/sec: 532.80\n", "2020-09-17 09:01:43,795 epoch 106 - iter 45/154 - loss 0.19507220 - samples/sec: 515.02\n", "2020-09-17 09:01:44,706 epoch 106 - iter 60/154 - loss 0.19885855 - samples/sec: 550.52\n", "2020-09-17 09:01:45,684 epoch 106 - iter 75/154 - loss 0.20451843 - samples/sec: 507.95\n", "2020-09-17 09:01:46,637 epoch 106 - iter 90/154 - loss 0.21467098 - samples/sec: 522.80\n", "2020-09-17 09:01:47,673 epoch 106 - iter 105/154 - loss 0.20852575 - samples/sec: 479.53\n", "2020-09-17 09:01:48,647 epoch 106 - iter 120/154 - loss 0.20485463 - samples/sec: 520.17\n", "2020-09-17 09:01:49,573 epoch 106 - iter 135/154 - loss 0.20740653 - samples/sec: 547.17\n", "2020-09-17 09:01:50,538 epoch 106 - iter 150/154 - loss 0.20943517 - samples/sec: 518.97\n", "2020-09-17 09:01:50,756 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:01:50,757 EPOCH 106 done: loss 0.2070 - lr 0.0004\n", "2020-09-17 09:01:51,594 DEV : loss 0.35290828347206116 - score 0.8954\n", "2020-09-17 09:01:51,609 BAD EPOCHS (no improvement): 4\n", "2020-09-17 09:01:51,611 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:01:51,681 epoch 107 - iter 0/154 - loss 0.25630486 - samples/sec: 7089.74\n", "2020-09-17 09:01:52,643 epoch 107 - iter 15/154 - loss 0.20092819 - samples/sec: 519.84\n", "2020-09-17 09:01:53,577 epoch 107 - iter 30/154 - loss 0.22173839 - samples/sec: 533.26\n", "2020-09-17 09:01:54,572 epoch 107 - iter 45/154 - loss 0.21267619 - samples/sec: 499.54\n", "2020-09-17 09:01:55,549 epoch 107 - iter 60/154 - loss 0.21484912 - samples/sec: 511.97\n", "2020-09-17 09:01:56,532 epoch 107 - iter 75/154 - loss 0.21780817 - samples/sec: 506.74\n", "2020-09-17 09:01:57,565 epoch 107 - iter 90/154 - loss 0.22065768 - samples/sec: 480.23\n", "2020-09-17 09:01:58,568 epoch 107 - iter 105/154 - loss 0.21931410 - samples/sec: 496.85\n", "2020-09-17 09:01:59,578 epoch 107 - iter 120/154 - loss 0.22108015 - samples/sec: 495.38\n", "2020-09-17 09:02:00,625 epoch 107 - iter 135/154 - loss 0.21933473 - samples/sec: 484.09\n", "2020-09-17 09:02:01,722 epoch 107 - iter 150/154 - loss 0.21376625 - samples/sec: 459.33\n", "2020-09-17 09:02:01,904 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:02:01,905 EPOCH 107 done: loss 0.2122 - lr 0.0004\n", "2020-09-17 09:02:02,727 DEV : loss 0.35225534439086914 - score 0.8954\n", "2020-09-17 09:02:02,742 BAD EPOCHS (no improvement): 5\n", "2020-09-17 09:02:02,744 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:02:02,821 epoch 108 - iter 0/154 - loss 0.14110419 - samples/sec: 6481.69\n", "2020-09-17 09:02:03,793 epoch 108 - iter 15/154 - loss 0.21955505 - samples/sec: 514.92\n", "2020-09-17 09:02:04,755 epoch 108 - iter 30/154 - loss 0.22040493 - samples/sec: 523.69\n", "2020-09-17 09:02:05,688 epoch 108 - iter 45/154 - loss 0.23444393 - samples/sec: 534.13\n", "2020-09-17 09:02:06,688 epoch 108 - iter 60/154 - loss 0.22844362 - samples/sec: 501.38\n", "2020-09-17 09:02:07,671 epoch 108 - iter 75/154 - loss 0.22735524 - samples/sec: 508.41\n", "2020-09-17 09:02:08,661 epoch 108 - iter 90/154 - loss 0.21661457 - samples/sec: 503.15\n", "2020-09-17 09:02:09,573 epoch 108 - iter 105/154 - loss 0.21507017 - samples/sec: 545.79\n", "2020-09-17 09:02:10,539 epoch 108 - iter 120/154 - loss 0.21802813 - samples/sec: 515.54\n", "2020-09-17 09:02:11,552 epoch 108 - iter 135/154 - loss 0.21656520 - samples/sec: 492.84\n", "2020-09-17 09:02:12,487 epoch 108 - iter 150/154 - loss 0.21677523 - samples/sec: 537.28\n", "2020-09-17 09:02:12,675 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:02:12,677 EPOCH 108 done: loss 0.2168 - lr 0.0004\n", "2020-09-17 09:02:13,520 DEV : loss 0.3523738980293274 - score 0.8936\n", "Epoch 107: reducing learning rate of group 0 to 1.9531e-04.\n", "2020-09-17 09:02:13,534 BAD EPOCHS (no improvement): 6\n", "2020-09-17 09:02:13,536 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:02:13,606 epoch 109 - iter 0/154 - loss 0.25964090 - samples/sec: 7057.33\n", "2020-09-17 09:02:14,549 epoch 109 - iter 15/154 - loss 0.18513508 - samples/sec: 530.25\n", "2020-09-17 09:02:15,555 epoch 109 - iter 30/154 - loss 0.21405375 - samples/sec: 499.47\n", "2020-09-17 09:02:16,503 epoch 109 - iter 45/154 - loss 0.20898294 - samples/sec: 529.81\n", "2020-09-17 09:02:17,440 epoch 109 - iter 60/154 - loss 0.20981438 - samples/sec: 533.82\n", "2020-09-17 09:02:18,367 epoch 109 - iter 75/154 - loss 0.20632632 - samples/sec: 538.54\n", "2020-09-17 09:02:19,280 epoch 109 - iter 90/154 - loss 0.20713501 - samples/sec: 546.92\n", "2020-09-17 09:02:20,262 epoch 109 - iter 105/154 - loss 0.20603333 - samples/sec: 506.82\n", "2020-09-17 09:02:21,192 epoch 109 - iter 120/154 - loss 0.20989937 - samples/sec: 536.57\n", "2020-09-17 09:02:22,100 epoch 109 - iter 135/154 - loss 0.20783171 - samples/sec: 549.20\n", "2020-09-17 09:02:23,058 epoch 109 - iter 150/154 - loss 0.21021594 - samples/sec: 525.45\n", "2020-09-17 09:02:23,241 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:02:23,242 EPOCH 109 done: loss 0.2080 - lr 0.0002\n", "2020-09-17 09:02:24,025 DEV : loss 0.35326170921325684 - score 0.8936\n", "2020-09-17 09:02:24,042 BAD EPOCHS (no improvement): 1\n", "2020-09-17 09:02:24,044 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:02:24,109 epoch 110 - iter 0/154 - loss 0.07102735 - samples/sec: 7614.84\n", "2020-09-17 09:02:25,070 epoch 110 - iter 15/154 - loss 0.19047088 - samples/sec: 518.73\n", "2020-09-17 09:02:26,013 epoch 110 - iter 30/154 - loss 0.20597582 - samples/sec: 530.15\n", "2020-09-17 09:02:26,991 epoch 110 - iter 45/154 - loss 0.22487505 - samples/sec: 509.74\n", "2020-09-17 09:02:27,889 epoch 110 - iter 60/154 - loss 0.21618803 - samples/sec: 553.81\n", "2020-09-17 09:02:28,821 epoch 110 - iter 75/154 - loss 0.21780754 - samples/sec: 539.07\n", "2020-09-17 09:02:29,787 epoch 110 - iter 90/154 - loss 0.21333692 - samples/sec: 517.83\n", "2020-09-17 09:02:30,719 epoch 110 - iter 105/154 - loss 0.20640813 - samples/sec: 536.57\n", "2020-09-17 09:02:31,727 epoch 110 - iter 120/154 - loss 0.21005841 - samples/sec: 491.81\n", "2020-09-17 09:02:32,660 epoch 110 - iter 135/154 - loss 0.20720438 - samples/sec: 537.45\n", "2020-09-17 09:02:33,625 epoch 110 - iter 150/154 - loss 0.20507608 - samples/sec: 517.26\n", "2020-09-17 09:02:33,853 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:02:33,854 EPOCH 110 done: loss 0.2041 - lr 0.0002\n", "2020-09-17 09:02:34,610 DEV : loss 0.35287684202194214 - score 0.8936\n", "2020-09-17 09:02:34,629 BAD EPOCHS (no improvement): 2\n", "2020-09-17 09:02:34,631 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:02:34,707 epoch 111 - iter 0/154 - loss 0.05041994 - samples/sec: 6576.40\n", "2020-09-17 09:02:35,601 epoch 111 - iter 15/154 - loss 0.21664469 - samples/sec: 558.78\n", "2020-09-17 09:02:36,586 epoch 111 - iter 30/154 - loss 0.18628614 - samples/sec: 505.58\n", "2020-09-17 09:02:37,509 epoch 111 - iter 45/154 - loss 0.18844842 - samples/sec: 537.79\n", "2020-09-17 09:02:38,520 epoch 111 - iter 60/154 - loss 0.19923696 - samples/sec: 494.89\n", "2020-09-17 09:02:39,437 epoch 111 - iter 75/154 - loss 0.19439198 - samples/sec: 541.82\n", "2020-09-17 09:02:40,380 epoch 111 - iter 90/154 - loss 0.20310201 - samples/sec: 526.67\n", "2020-09-17 09:02:41,390 epoch 111 - iter 105/154 - loss 0.20613366 - samples/sec: 495.86\n", "2020-09-17 09:02:42,345 epoch 111 - iter 120/154 - loss 0.20856594 - samples/sec: 525.44\n", "2020-09-17 09:02:43,284 epoch 111 - iter 135/154 - loss 0.21149184 - samples/sec: 531.02\n", "2020-09-17 09:02:44,214 epoch 111 - iter 150/154 - loss 0.20643389 - samples/sec: 536.66\n", "2020-09-17 09:02:44,399 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:02:44,401 EPOCH 111 done: loss 0.2079 - lr 0.0002\n", "2020-09-17 09:02:45,193 DEV : loss 0.3528764247894287 - score 0.8936\n", "2020-09-17 09:02:45,209 BAD EPOCHS (no improvement): 3\n", "2020-09-17 09:02:45,212 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:02:45,290 epoch 112 - iter 0/154 - loss 0.14459394 - samples/sec: 6315.55\n", "2020-09-17 09:02:46,216 epoch 112 - iter 15/154 - loss 0.23116280 - samples/sec: 543.48\n", "2020-09-17 09:02:47,146 epoch 112 - iter 30/154 - loss 0.19267853 - samples/sec: 539.31\n", "2020-09-17 09:02:48,101 epoch 112 - iter 45/154 - loss 0.18103294 - samples/sec: 521.27\n", "2020-09-17 09:02:49,042 epoch 112 - iter 60/154 - loss 0.18451293 - samples/sec: 530.74\n", "2020-09-17 09:02:50,008 epoch 112 - iter 75/154 - loss 0.18484828 - samples/sec: 518.43\n", "2020-09-17 09:02:50,957 epoch 112 - iter 90/154 - loss 0.18284381 - samples/sec: 529.42\n", "2020-09-17 09:02:51,918 epoch 112 - iter 105/154 - loss 0.18590387 - samples/sec: 518.13\n", "2020-09-17 09:02:52,895 epoch 112 - iter 120/154 - loss 0.18572209 - samples/sec: 510.85\n", "2020-09-17 09:02:53,838 epoch 112 - iter 135/154 - loss 0.18512123 - samples/sec: 530.20\n", "2020-09-17 09:02:54,809 epoch 112 - iter 150/154 - loss 0.18547342 - samples/sec: 516.36\n", "2020-09-17 09:02:54,991 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:02:54,992 EPOCH 112 done: loss 0.1842 - lr 0.0002\n", "2020-09-17 09:02:55,788 DEV : loss 0.3532208800315857 - score 0.8917\n", "2020-09-17 09:02:55,802 BAD EPOCHS (no improvement): 4\n", "2020-09-17 09:02:55,803 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:02:55,873 epoch 113 - iter 0/154 - loss 0.08220699 - samples/sec: 7058.57\n", "2020-09-17 09:02:56,851 epoch 113 - iter 15/154 - loss 0.19432080 - samples/sec: 510.08\n", "2020-09-17 09:02:57,811 epoch 113 - iter 30/154 - loss 0.19584573 - samples/sec: 526.75\n", "2020-09-17 09:02:58,741 epoch 113 - iter 45/154 - loss 0.20053910 - samples/sec: 536.53\n", "2020-09-17 09:02:59,749 epoch 113 - iter 60/154 - loss 0.19701828 - samples/sec: 493.85\n", "2020-09-17 09:03:00,705 epoch 113 - iter 75/154 - loss 0.19683385 - samples/sec: 526.86\n", "2020-09-17 09:03:01,679 epoch 113 - iter 90/154 - loss 0.19811802 - samples/sec: 509.17\n", "2020-09-17 09:03:02,594 epoch 113 - iter 105/154 - loss 0.20016069 - samples/sec: 549.23\n", "2020-09-17 09:03:03,536 epoch 113 - iter 120/154 - loss 0.20088703 - samples/sec: 530.25\n", "2020-09-17 09:03:04,511 epoch 113 - iter 135/154 - loss 0.20212986 - samples/sec: 516.91\n", "2020-09-17 09:03:05,419 epoch 113 - iter 150/154 - loss 0.20262971 - samples/sec: 550.65\n", "2020-09-17 09:03:05,604 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:03:05,606 EPOCH 113 done: loss 0.2023 - lr 0.0002\n", "2020-09-17 09:03:06,433 DEV : loss 0.3534841239452362 - score 0.8917\n", "2020-09-17 09:03:06,452 BAD EPOCHS (no improvement): 5\n", "2020-09-17 09:03:06,454 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:03:06,534 epoch 114 - iter 0/154 - loss 0.05012216 - samples/sec: 6264.01\n", "2020-09-17 09:03:07,472 epoch 114 - iter 15/154 - loss 0.18781554 - samples/sec: 529.50\n", "2020-09-17 09:03:08,457 epoch 114 - iter 30/154 - loss 0.19996087 - samples/sec: 505.93\n", "2020-09-17 09:03:09,409 epoch 114 - iter 45/154 - loss 0.20597989 - samples/sec: 524.57\n", "2020-09-17 09:03:10,347 epoch 114 - iter 60/154 - loss 0.19506772 - samples/sec: 532.58\n", "2020-09-17 09:03:11,404 epoch 114 - iter 75/154 - loss 0.20868460 - samples/sec: 469.56\n", "2020-09-17 09:03:12,344 epoch 114 - iter 90/154 - loss 0.20749844 - samples/sec: 527.82\n", "2020-09-17 09:03:13,336 epoch 114 - iter 105/154 - loss 0.20449429 - samples/sec: 501.92\n", "2020-09-17 09:03:14,263 epoch 114 - iter 120/154 - loss 0.20572401 - samples/sec: 541.42\n", "2020-09-17 09:03:15,224 epoch 114 - iter 135/154 - loss 0.20465263 - samples/sec: 522.02\n", "2020-09-17 09:03:16,238 epoch 114 - iter 150/154 - loss 0.19974582 - samples/sec: 497.04\n", "2020-09-17 09:03:16,417 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:03:16,418 EPOCH 114 done: loss 0.1992 - lr 0.0002\n", "2020-09-17 09:03:17,206 DEV : loss 0.35383978486061096 - score 0.8917\n", "Epoch 113: reducing learning rate of group 0 to 9.7656e-05.\n", "2020-09-17 09:03:17,222 BAD EPOCHS (no improvement): 6\n", "2020-09-17 09:03:17,224 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:03:17,225 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:03:17,226 learning rate too small - quitting training!\n", "2020-09-17 09:03:17,227 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:03:21,785 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 09:03:21,786 Testing using best model ...\n", "2020-09-17 09:03:21,787 loading file resources\\taggers\\trec\\best-model.pt\n", "2020-09-17 09:03:23,515 0.916\t0.916\t0.916\n", "2020-09-17 09:03:23,516 \n", "MICRO_AVG: acc 0.845 - f1-score 0.916\n", "MACRO_AVG: acc 0.8411 - f1-score 0.9117500000000001\n", "ABBR tp: 7 - fp: 0 - fn: 2 - tn: 491 - precision: 1.0000 - recall: 0.7778 - accuracy: 0.7778 - f1-score: 0.8750\n", "DESC tp: 138 - fp: 29 - fn: 0 - tn: 333 - precision: 0.8263 - recall: 1.0000 - accuracy: 0.8263 - f1-score: 0.9049\n", "ENTY tp: 70 - fp: 4 - fn: 24 - tn: 402 - precision: 0.9459 - recall: 0.7447 - accuracy: 0.7143 - f1-score: 0.8333\n", "HUM tp: 61 - fp: 0 - fn: 4 - tn: 435 - precision: 1.0000 - recall: 0.9385 - accuracy: 0.9385 - f1-score: 0.9683\n", "LOC tp: 76 - fp: 5 - fn: 5 - tn: 414 - precision: 0.9383 - recall: 0.9383 - accuracy: 0.8837 - f1-score: 0.9383\n", "NUM tp: 106 - fp: 4 - fn: 7 - tn: 383 - precision: 0.9636 - recall: 0.9381 - accuracy: 0.9060 - f1-score: 0.9507\n", "2020-09-17 09:03:23,517 ----------------------------------------------------------------------------------------------------\n" ] }, { "data": { "text/plain": [ "{'test_score': 0.916,\n", " 'dev_score_history': [0.411,\n", " 0.3743,\n", " 0.4294,\n", " 0.4147,\n", " 0.5817,\n", " 0.5358,\n", " 0.6422,\n", " 0.6349,\n", " 0.7193,\n", " 0.7046,\n", " 0.6312,\n", " 0.7394,\n", " 0.7394,\n", " 0.7358,\n", " 0.7284,\n", " 0.7706,\n", " 0.7963,\n", " 0.8,\n", " 0.7725,\n", " 0.778,\n", " 0.7945,\n", " 0.789,\n", " 0.7523,\n", " 0.8312,\n", " 0.8257,\n", " 0.8404,\n", " 0.8367,\n", " 0.8294,\n", " 0.8092,\n", " 0.8202,\n", " 0.844,\n", " 0.8385,\n", " 0.822,\n", " 0.8294,\n", " 0.8587,\n", " 0.8606,\n", " 0.8385,\n", " 0.8202,\n", " 0.8606,\n", " 0.8532,\n", " 0.8697,\n", " 0.8422,\n", " 0.8569,\n", " 0.855,\n", " 0.8679,\n", " 0.8734,\n", " 0.8569,\n", " 0.8606,\n", " 0.8881,\n", " 0.8716,\n", " 0.8477,\n", " 0.8771,\n", " 0.8679,\n", " 0.8789,\n", " 0.8789,\n", " 0.8789,\n", " 0.8752,\n", " 0.8807,\n", " 0.8936,\n", " 0.9009,\n", " 0.8789,\n", " 0.8807,\n", " 0.8624,\n", " 0.8899,\n", " 0.8716,\n", " 0.8862,\n", " 0.8936,\n", " 0.8826,\n", " 0.8862,\n", " 0.8899,\n", " 0.8881,\n", " 0.8899,\n", " 0.8936,\n", " 0.8972,\n", " 0.8991,\n", " 0.8881,\n", " 0.8881,\n", " 0.8972,\n", " 0.8972,\n", " 0.8954,\n", " 0.8954,\n", " 0.8991,\n", " 0.8936,\n", " 0.8954,\n", " 0.8954,\n", " 0.8991,\n", " 0.8936,\n", " 0.8954,\n", " 0.8899,\n", " 0.8917,\n", " 0.8936,\n", " 0.8936,\n", " 0.8936,\n", " 0.8936,\n", " 0.8954,\n", " 0.8954,\n", " 0.8936,\n", " 0.8917,\n", " 0.8954,\n", " 0.8954,\n", " 0.8954,\n", " 0.8954,\n", " 0.8954,\n", " 0.8954,\n", " 0.8954,\n", " 0.8954,\n", " 0.8954,\n", " 0.8936,\n", " 0.8936,\n", " 0.8936,\n", " 0.8936,\n", " 0.8917,\n", " 0.8917,\n", " 0.8917],\n", " 'train_loss_history': [1.623803038875778,\n", " 1.4751895635158985,\n", " 1.3539273088628596,\n", " 1.2665504050719274,\n", " 1.2022556999287048,\n", " 1.0989332574528534,\n", " 1.0324746076162759,\n", " 0.9453888276180664,\n", " 0.9073464963343236,\n", " 0.8448862450850474,\n", " 0.7943861803838185,\n", " 0.7624232318107184,\n", " 0.7177381469057752,\n", " 0.7222172072181454,\n", " 0.6909760513475963,\n", " 0.6556571362080512,\n", " 0.6340047879652544,\n", " 0.6206931077040635,\n", " 0.5978651851802678,\n", " 0.5878555914991862,\n", " 0.5742910655287953,\n", " 0.5721298178488557,\n", " 0.5485162321816791,\n", " 0.5256959537600542,\n", " 0.5364196108920234,\n", " 0.5102886271167111,\n", " 0.48673206200073293,\n", " 0.4934533980082382,\n", " 0.47849951461925133,\n", " 0.47886634201972517,\n", " 0.46175198610146323,\n", " 0.4444880872577816,\n", " 0.45063225741123225,\n", " 0.4433224099603566,\n", " 0.4416732205586,\n", " 0.4219861655653297,\n", " 0.4241042013292189,\n", " 0.4040676319076643,\n", " 0.39172548800706863,\n", " 0.40660234801955036,\n", " 0.38799728816012286,\n", " 0.3930157378025643,\n", " 0.3847577641052859,\n", " 0.3731525679687401,\n", " 0.36748001022965876,\n", " 0.3576956857237723,\n", " 0.3491114801787711,\n", " 0.3628535376160176,\n", " 0.3672764946694498,\n", " 0.33301084624095395,\n", " 0.36200482112827237,\n", " 0.3340135342985779,\n", " 0.34431070157072763,\n", " 0.3334019233944354,\n", " 0.3070976599663883,\n", " 0.2860608868204154,\n", " 0.2603152832621104,\n", " 0.2840379754734504,\n", " 0.27133475906856647,\n", " 0.28250507686238785,\n", " 0.2679124508875531,\n", " 0.2624280698984474,\n", " 0.2733986359815319,\n", " 0.24834942778983674,\n", " 0.255558418472866,\n", " 0.23961261352645113,\n", " 0.2411714465683931,\n", " 0.23452102155847984,\n", " 0.22794407538392328,\n", " 0.22713523749987802,\n", " 0.2204862692236126,\n", " 0.2238566107177115,\n", " 0.21400024949923738,\n", " 0.21099147321535394,\n", " 0.2263214501080575,\n", " 0.21111334146714056,\n", " 0.21006998661663626,\n", " 0.21434478805243196,\n", " 0.21075206046754663,\n", " 0.2074475312097506,\n", " 0.212246003330915,\n", " 0.21485535452118168,\n", " 0.20711339468305762,\n", " 0.20082600295011485,\n", " 0.20050999072271508,\n", " 0.21893818057202674,\n", " 0.19497514071015568,\n", " 0.2017280255722535,\n", " 0.20025141251745163,\n", " 0.19892099241472103,\n", " 0.20901773753878358,\n", " 0.2117263821812419,\n", " 0.20360777756900755,\n", " 0.19037547949459646,\n", " 0.2157724822399678,\n", " 0.2063891390072448,\n", " 0.20160789512001076,\n", " 0.1995632725876647,\n", " 0.19750643139342208,\n", " 0.19465236504888767,\n", " 0.20239055921117982,\n", " 0.20646288937755994,\n", " 0.2002591782479317,\n", " 0.1905817040665583,\n", " 0.19992739189561312,\n", " 0.20700094286449156,\n", " 0.21215771622471996,\n", " 0.21677971481309308,\n", " 0.208018726802298,\n", " 0.20411140846644904,\n", " 0.20789395161456875,\n", " 0.1842138704106018,\n", " 0.20233926065749935,\n", " 0.19919203807200705],\n", " 'dev_loss_history': [tensor(1.4586, device='cuda:0'),\n", " tensor(1.5128, device='cuda:0'),\n", " tensor(1.4478, device='cuda:0'),\n", " tensor(1.4595, device='cuda:0'),\n", " tensor(1.0863, device='cuda:0'),\n", " tensor(1.1175, device='cuda:0'),\n", " tensor(0.9614, device='cuda:0'),\n", " tensor(0.9537, device='cuda:0'),\n", " tensor(0.7254, device='cuda:0'),\n", " tensor(0.7877, device='cuda:0'),\n", " tensor(1.0041, device='cuda:0'),\n", " tensor(0.7418, device='cuda:0'),\n", " tensor(0.6499, device='cuda:0'),\n", " tensor(0.6645, device='cuda:0'),\n", " tensor(0.6555, device='cuda:0'),\n", " tensor(0.5636, device='cuda:0'),\n", " tensor(0.5325, device='cuda:0'),\n", " tensor(0.5046, device='cuda:0'),\n", " tensor(0.5927, device='cuda:0'),\n", " tensor(0.6165, device='cuda:0'),\n", " tensor(0.5154, device='cuda:0'),\n", " tensor(0.6012, device='cuda:0'),\n", " tensor(0.6499, device='cuda:0'),\n", " tensor(0.4620, device='cuda:0'),\n", " tensor(0.4626, device='cuda:0'),\n", " tensor(0.4160, device='cuda:0'),\n", " tensor(0.4338, device='cuda:0'),\n", " tensor(0.4328, device='cuda:0'),\n", " tensor(0.5461, device='cuda:0'),\n", " tensor(0.4950, device='cuda:0'),\n", " tensor(0.4721, device='cuda:0'),\n", " tensor(0.4731, device='cuda:0'),\n", " tensor(0.4689, device='cuda:0'),\n", " tensor(0.4542, device='cuda:0'),\n", " tensor(0.3903, device='cuda:0'),\n", " tensor(0.3705, device='cuda:0'),\n", " tensor(0.4270, device='cuda:0'),\n", " tensor(0.4972, device='cuda:0'),\n", " tensor(0.3936, device='cuda:0'),\n", " tensor(0.4120, device='cuda:0'),\n", " tensor(0.3772, device='cuda:0'),\n", " tensor(0.4321, device='cuda:0'),\n", " tensor(0.4324, device='cuda:0'),\n", " tensor(0.3916, device='cuda:0'),\n", " tensor(0.3788, device='cuda:0'),\n", " tensor(0.3910, device='cuda:0'),\n", " tensor(0.4490, device='cuda:0'),\n", " tensor(0.3893, device='cuda:0'),\n", " tensor(0.3643, device='cuda:0'),\n", " tensor(0.3749, device='cuda:0'),\n", " tensor(0.4560, device='cuda:0'),\n", " tensor(0.3762, device='cuda:0'),\n", " tensor(0.3900, device='cuda:0'),\n", " tensor(0.3526, device='cuda:0'),\n", " tensor(0.3829, device='cuda:0'),\n", " tensor(0.3736, device='cuda:0'),\n", " tensor(0.3716, device='cuda:0'),\n", " tensor(0.3527, device='cuda:0'),\n", " tensor(0.3730, device='cuda:0'),\n", " tensor(0.3573, device='cuda:0'),\n", " tensor(0.3652, device='cuda:0'),\n", " tensor(0.3708, device='cuda:0'),\n", " tensor(0.4011, device='cuda:0'),\n", " tensor(0.3385, device='cuda:0'),\n", " tensor(0.3943, device='cuda:0'),\n", " tensor(0.3736, device='cuda:0'),\n", " tensor(0.3528, device='cuda:0'),\n", " tensor(0.3614, device='cuda:0'),\n", " tensor(0.3607, device='cuda:0'),\n", " tensor(0.3521, device='cuda:0'),\n", " tensor(0.3857, device='cuda:0'),\n", " tensor(0.3715, device='cuda:0'),\n", " tensor(0.3536, device='cuda:0'),\n", " tensor(0.3510, device='cuda:0'),\n", " tensor(0.3394, device='cuda:0'),\n", " tensor(0.3567, device='cuda:0'),\n", " tensor(0.3694, device='cuda:0'),\n", " tensor(0.3583, device='cuda:0'),\n", " tensor(0.3438, device='cuda:0'),\n", " tensor(0.3514, device='cuda:0'),\n", " tensor(0.3523, device='cuda:0'),\n", " tensor(0.3489, device='cuda:0'),\n", " tensor(0.3494, device='cuda:0'),\n", " tensor(0.3445, device='cuda:0'),\n", " tensor(0.3483, device='cuda:0'),\n", " tensor(0.3464, device='cuda:0'),\n", " tensor(0.3506, device='cuda:0'),\n", " tensor(0.3492, device='cuda:0'),\n", " tensor(0.3497, device='cuda:0'),\n", " tensor(0.3539, device='cuda:0'),\n", " tensor(0.3520, device='cuda:0'),\n", " tensor(0.3534, device='cuda:0'),\n", " tensor(0.3529, device='cuda:0'),\n", " tensor(0.3520, device='cuda:0'),\n", " tensor(0.3530, device='cuda:0'),\n", " tensor(0.3509, device='cuda:0'),\n", " tensor(0.3533, device='cuda:0'),\n", " tensor(0.3548, device='cuda:0'),\n", " tensor(0.3538, device='cuda:0'),\n", " tensor(0.3541, device='cuda:0'),\n", " tensor(0.3530, device='cuda:0'),\n", " tensor(0.3524, device='cuda:0'),\n", " tensor(0.3531, device='cuda:0'),\n", " tensor(0.3523, device='cuda:0'),\n", " tensor(0.3521, device='cuda:0'),\n", " tensor(0.3529, device='cuda:0'),\n", " tensor(0.3523, device='cuda:0'),\n", " tensor(0.3524, device='cuda:0'),\n", " tensor(0.3533, device='cuda:0'),\n", " tensor(0.3529, device='cuda:0'),\n", " tensor(0.3529, device='cuda:0'),\n", " tensor(0.3532, device='cuda:0'),\n", " tensor(0.3535, device='cuda:0'),\n", " tensor(0.3538, device='cuda:0')]}" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 7. start the training\n", "trainer.train('resources/taggers/trec',\n", " learning_rate=0.1,\n", " mini_batch_size=32,\n", " anneal_factor=0.5,\n", " patience=5,\n", " max_epochs=150)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T16:08:27.918133Z", "start_time": "2020-09-17T16:08:26.786657Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 09:08:26,787 loading file resources/taggers/trec/final-model.pt\n", "[HUM (0.9996542930603027)]\n" ] } ], "source": [ "classifier = TextClassifier.load('resources/taggers/trec/final-model.pt')\n", "\n", "# create example sentence\n", "from flair.data import Sentence\n", "\n", "sentence = Sentence('Who built the Eiffel Tower ?')\n", "\n", "# predict class and print\n", "classifier.predict(sentence)\n", "\n", "print(sentence.labels)" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "## prepare data files\n", "\n", "Malignant data spread between several sets with different size for different DGA algorithms. We need stratification for these algorithms between train, dev, and test samples. So we split each DGA set into train, dev, and test and ONLY after that compound samples of all DGA algos into the result train, dev, and test sample sets.\n", "\n", "DNs split by pseudo-words by wordninja package. Using pseudo-words inplace of DN we can use the models with word embeddings.\n", "We prepare two data sets: one with samples split by pseudo-words and the second without such spliting. Then we experiment with both data sets.\n", "\n", "Samples compound of a lablel and a sample text (DN in form of pseudo-words), where label is in the FastText format (as \"__label__<label> <sample text>\")." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2020-09-18T03:43:38.344391Z", "start_time": "2020-09-18T03:43:38.178180Z" }, "hidden": true }, "outputs": [ { "data": { "text/plain": [ "'my shop if y . biz . com'" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import wordninja\n", "\n", "def split(txt):\n", " return ' . '.join(' '.join(wordninja.split(t)) for t in txt.split('.'))\n", "\n", "split('myshopify.biz.com')" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "ExecuteTime": { "end_time": "2020-09-18T03:03:31.522756Z", "start_time": "2020-09-18T03:03:31.513717Z" }, "hidden": true }, "outputs": [], "source": [ "# benign data\n", "\n", "def read_data(file_name):\n", " ret = open(file_name, encoding='utf-8').read().splitlines()\n", " print(f'Load {len(ret):,} from file \"{file_name}\"')\n", " return ret\n", " \n", "def remove_TLD(dns):\n", " return [el.split('.')[0] for el in dns]\n", "\n", "\n", "def add_label(in_file, out_file, label, split_by_words=True):\n", " lines = read_data(in_file)\n", " if split_by_words: lines = [split(el) for el in lines]\n", " lines = [f'__label__{label} {el}\\n' for el in lines]\n", " with open(out_file, 'w') as f:\n", " f.writelines(lines)\n", " print(f'Saved {len(lines):,} into \"{out_file}\"')\n", "\n", "in_file = 'data/Alex_top-0.5M.txt'\n", "label = 'binign'\n", "split_by_words=True\n", "out_file = f'{in_file[:-4]}.{label}.{\"split_by_words.\" if split_by_words else \"\"}txt'\n", "add_label(in_file, out_file, 'binign', split_by_words=split_by_words)\n" ] }, { "cell_type": "code", "execution_count": 101, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T20:54:46.055628Z", "start_time": "2020-09-17T20:54:40.898458Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "malignant files: 31\n", " Load 1,000 from file \"domain_generation_algorithms\\banjori\\example_domains.txt\"\n", " Get: [0:800]\n", " Load 2,160 from file \"domain_generation_algorithms\\bazarbackdoor\\example_domains.txt\"\n", " Get: [0:1,728]\n", " Load 256 from file \"domain_generation_algorithms\\chinad\\example_domains.txt\"\n", " Get: [0:204]\n", " Load 40 from file \"domain_generation_algorithms\\corebot\\example_domains.txt\"\n", " Get: [0:32]\n", " Load 30 from file \"domain_generation_algorithms\\dircrypt\\example_domains.txt\"\n", " Get: [0:24]\n", " Load 5 from file \"domain_generation_algorithms\\dnschanger\\example_domains.txt\"\n", " Get: [0:4]\n", " Load 12 from file \"domain_generation_algorithms\\gozi\\example_domains.txt\"\n", " Get: [0:9]\n", " Load 8 from file \"domain_generation_algorithms\\locky\\example_domains.txt\"\n", " Get: [0:6]\n", " Load 2,500 from file \"domain_generation_algorithms\\monerodownloader\\example_domains.txt\"\n", " Get: [0:2,000]\n", " Load 99 from file \"domain_generation_algorithms\\mydoom\\example_domains.txt\"\n", " Get: [0:79]\n", " Load 2,048 from file \"domain_generation_algorithms\\necurs\\example_domains.txt\"\n", " Get: [0:1,638]\n", " Load 1,000 from file \"domain_generation_algorithms\\newgoz\\example_domains.txt\"\n", " Get: [0:800]\n", " Load 128 from file \"domain_generation_algorithms\\nymaim\\example_domains.txt\"\n", " Get: [0:102]\n", " Load 704 from file \"domain_generation_algorithms\\nymaim2\\example_domains.txt\"\n", " Get: [0:563]\n", " Load 24 from file \"domain_generation_algorithms\\padcrypt\\example_domains.txt\"\n", " Get: [0:19]\n", " Load 306 from file \"domain_generation_algorithms\\pitou\\example_domains.txt\"\n", " Get: [0:244]\n", " Load 85 from file \"domain_generation_algorithms\\pizd\\example_domains.txt\"\n", " Get: [0:68]\n", " Load 100 from file \"domain_generation_algorithms\\proslikefan\\example_domains.txt\"\n", " Get: [0:80]\n", " Load 1,350 from file \"domain_generation_algorithms\\pushdo\\example_domains.txt\"\n", " Get: [0:1,080]\n", " Load 200 from file \"domain_generation_algorithms\\qadars\\example_domains.txt\"\n", " Get: [0:160]\n", " Load 5,000 from file \"domain_generation_algorithms\\qakbot\\example_domains.txt\"\n", " Get: [0:4,000]\n", " Load 100 from file \"domain_generation_algorithms\\ramnit\\example_domains.txt\"\n", " Get: [0:80]\n", " Load 100 from file \"domain_generation_algorithms\\reconyc\\example_domains.txt\"\n", " Get: [0:80]\n", " Load 2,001 from file \"domain_generation_algorithms\\shiotob\\example_domains.txt\"\n", " Get: [0:1,600]\n", " Load 1,000 from file \"domain_generation_algorithms\\simda\\example_domains.txt\"\n", " Get: [0:800]\n", " Load 40 from file \"domain_generation_algorithms\\sisron\\example_domains.txt\"\n", " Get: [0:32]\n", " Load 64 from file \"domain_generation_algorithms\\symmi\\example_domains.txt\"\n", " Get: [0:51]\n", " Load 1,746 from file \"domain_generation_algorithms\\tempedreve\\example_domains.txt\"\n", " Get: [0:1,396]\n", " Load 401 from file \"domain_generation_algorithms\\tinba\\example_domains.txt\"\n", " Get: [0:320]\n", " Load 120 from file \"domain_generation_algorithms\\unnamed_downloader\\example_domains.txt\"\n", " Get: [0:96]\n", " Load 30 from file \"domain_generation_algorithms\\unnamed_javascript_dga\\example_domains.txt\"\n", " Get: [0:24]\n", "Saved 18,119 into \"data/malignant.split_by_words.train.txt\"\n", " Load 1,000 from file \"domain_generation_algorithms\\banjori\\example_domains.txt\"\n", " Get: [800:900]\n", " Load 2,160 from file \"domain_generation_algorithms\\bazarbackdoor\\example_domains.txt\"\n", " Get: [1,728:1,944]\n", " Load 256 from file \"domain_generation_algorithms\\chinad\\example_domains.txt\"\n", " Get: [204:230]\n", " Load 40 from file \"domain_generation_algorithms\\corebot\\example_domains.txt\"\n", " Get: [32:36]\n", " Load 30 from file \"domain_generation_algorithms\\dircrypt\\example_domains.txt\"\n", " Get: [24:27]\n", " Load 5 from file \"domain_generation_algorithms\\dnschanger\\example_domains.txt\"\n", " Get: [4:4]\n", " Load 12 from file \"domain_generation_algorithms\\gozi\\example_domains.txt\"\n", " Get: [9:10]\n", " Load 8 from file \"domain_generation_algorithms\\locky\\example_domains.txt\"\n", " Get: [6:7]\n", " Load 2,500 from file \"domain_generation_algorithms\\monerodownloader\\example_domains.txt\"\n", " Get: [2,000:2,250]\n", " Load 99 from file \"domain_generation_algorithms\\mydoom\\example_domains.txt\"\n", " Get: [79:89]\n", " Load 2,048 from file \"domain_generation_algorithms\\necurs\\example_domains.txt\"\n", " Get: [1,638:1,843]\n", " Load 1,000 from file \"domain_generation_algorithms\\newgoz\\example_domains.txt\"\n", " Get: [800:900]\n", " Load 128 from file \"domain_generation_algorithms\\nymaim\\example_domains.txt\"\n", " Get: [102:115]\n", " Load 704 from file \"domain_generation_algorithms\\nymaim2\\example_domains.txt\"\n", " Get: [563:633]\n", " Load 24 from file \"domain_generation_algorithms\\padcrypt\\example_domains.txt\"\n", " Get: [19:21]\n", " Load 306 from file \"domain_generation_algorithms\\pitou\\example_domains.txt\"\n", " Get: [244:275]\n", " Load 85 from file \"domain_generation_algorithms\\pizd\\example_domains.txt\"\n", " Get: [68:76]\n", " Load 100 from file \"domain_generation_algorithms\\proslikefan\\example_domains.txt\"\n", " Get: [80:90]\n", " Load 1,350 from file \"domain_generation_algorithms\\pushdo\\example_domains.txt\"\n", " Get: [1,080:1,215]\n", " Load 200 from file \"domain_generation_algorithms\\qadars\\example_domains.txt\"\n", " Get: [160:180]\n", " Load 5,000 from file \"domain_generation_algorithms\\qakbot\\example_domains.txt\"\n", " Get: [4,000:4,500]\n", " Load 100 from file \"domain_generation_algorithms\\ramnit\\example_domains.txt\"\n", " Get: [80:90]\n", " Load 100 from file \"domain_generation_algorithms\\reconyc\\example_domains.txt\"\n", " Get: [80:90]\n", " Load 2,001 from file \"domain_generation_algorithms\\shiotob\\example_domains.txt\"\n", " Get: [1,600:1,800]\n", " Load 1,000 from file \"domain_generation_algorithms\\simda\\example_domains.txt\"\n", " Get: [800:900]\n", " Load 40 from file \"domain_generation_algorithms\\sisron\\example_domains.txt\"\n", " Get: [32:36]\n", " Load 64 from file \"domain_generation_algorithms\\symmi\\example_domains.txt\"\n", " Get: [51:57]\n", " Load 1,746 from file \"domain_generation_algorithms\\tempedreve\\example_domains.txt\"\n", " Get: [1,396:1,571]\n", " Load 401 from file \"domain_generation_algorithms\\tinba\\example_domains.txt\"\n", " Get: [320:360]\n", " Load 120 from file \"domain_generation_algorithms\\unnamed_downloader\\example_domains.txt\"\n", " Get: [96:108]\n", " Load 30 from file \"domain_generation_algorithms\\unnamed_javascript_dga\\example_domains.txt\"\n", " Get: [24:27]\n", "Saved 2,265 into \"data/malignant.split_by_words.dev.txt\"\n", " Load 1,000 from file \"domain_generation_algorithms\\banjori\\example_domains.txt\"\n", " Get: [900:1,000]\n", " Load 2,160 from file \"domain_generation_algorithms\\bazarbackdoor\\example_domains.txt\"\n", " Get: [1,944:2,160]\n", " Load 256 from file \"domain_generation_algorithms\\chinad\\example_domains.txt\"\n", " Get: [230:256]\n", " Load 40 from file \"domain_generation_algorithms\\corebot\\example_domains.txt\"\n", " Get: [36:40]\n", " Load 30 from file \"domain_generation_algorithms\\dircrypt\\example_domains.txt\"\n", " Get: [27:30]\n", " Load 5 from file \"domain_generation_algorithms\\dnschanger\\example_domains.txt\"\n", " Get: [4:5]\n", " Load 12 from file \"domain_generation_algorithms\\gozi\\example_domains.txt\"\n", " Get: [10:12]\n", " Load 8 from file \"domain_generation_algorithms\\locky\\example_domains.txt\"\n", " Get: [7:8]\n", " Load 2,500 from file \"domain_generation_algorithms\\monerodownloader\\example_domains.txt\"\n", " Get: [2,250:2,500]\n", " Load 99 from file \"domain_generation_algorithms\\mydoom\\example_domains.txt\"\n", " Get: [89:99]\n", " Load 2,048 from file \"domain_generation_algorithms\\necurs\\example_domains.txt\"\n", " Get: [1,843:2,048]\n", " Load 1,000 from file \"domain_generation_algorithms\\newgoz\\example_domains.txt\"\n", " Get: [900:1,000]\n", " Load 128 from file \"domain_generation_algorithms\\nymaim\\example_domains.txt\"\n", " Get: [115:128]\n", " Load 704 from file \"domain_generation_algorithms\\nymaim2\\example_domains.txt\"\n", " Get: [633:704]\n", " Load 24 from file \"domain_generation_algorithms\\padcrypt\\example_domains.txt\"\n", " Get: [21:24]\n", " Load 306 from file \"domain_generation_algorithms\\pitou\\example_domains.txt\"\n", " Get: [275:306]\n", " Load 85 from file \"domain_generation_algorithms\\pizd\\example_domains.txt\"\n", " Get: [76:85]\n", " Load 100 from file \"domain_generation_algorithms\\proslikefan\\example_domains.txt\"\n", " Get: [90:100]\n", " Load 1,350 from file \"domain_generation_algorithms\\pushdo\\example_domains.txt\"\n", " Get: [1,215:1,350]\n", " Load 200 from file \"domain_generation_algorithms\\qadars\\example_domains.txt\"\n", " Get: [180:200]\n", " Load 5,000 from file \"domain_generation_algorithms\\qakbot\\example_domains.txt\"\n", " Get: [4,500:5,000]\n", " Load 100 from file \"domain_generation_algorithms\\ramnit\\example_domains.txt\"\n", " Get: [90:100]\n", " Load 100 from file \"domain_generation_algorithms\\reconyc\\example_domains.txt\"\n", " Get: [90:100]\n", " Load 2,001 from file \"domain_generation_algorithms\\shiotob\\example_domains.txt\"\n", " Get: [1,800:2,001]\n", " Load 1,000 from file \"domain_generation_algorithms\\simda\\example_domains.txt\"\n", " Get: [900:1,000]\n", " Load 40 from file \"domain_generation_algorithms\\sisron\\example_domains.txt\"\n", " Get: [36:40]\n", " Load 64 from file \"domain_generation_algorithms\\symmi\\example_domains.txt\"\n", " Get: [57:64]\n", " Load 1,746 from file \"domain_generation_algorithms\\tempedreve\\example_domains.txt\"\n", " Get: [1,571:1,746]\n", " Load 401 from file \"domain_generation_algorithms\\tinba\\example_domains.txt\"\n", " Get: [360:401]\n", " Load 120 from file \"domain_generation_algorithms\\unnamed_downloader\\example_domains.txt\"\n", " Get: [108:120]\n", " Load 30 from file \"domain_generation_algorithms\\unnamed_javascript_dga\\example_domains.txt\"\n", " Get: [27:30]\n", "Saved 2,273 into \"data/malignant.split_by_words.test.txt\"\n", " Load 1,000 from file \"domain_generation_algorithms\\banjori\\example_domains.txt\"\n", " Get: [0:800]\n", " Load 2,160 from file \"domain_generation_algorithms\\bazarbackdoor\\example_domains.txt\"\n", " Get: [0:1,728]\n", " Load 256 from file \"domain_generation_algorithms\\chinad\\example_domains.txt\"\n", " Get: [0:204]\n", " Load 40 from file \"domain_generation_algorithms\\corebot\\example_domains.txt\"\n", " Get: [0:32]\n", " Load 30 from file \"domain_generation_algorithms\\dircrypt\\example_domains.txt\"\n", " Get: [0:24]\n", " Load 5 from file \"domain_generation_algorithms\\dnschanger\\example_domains.txt\"\n", " Get: [0:4]\n", " Load 12 from file \"domain_generation_algorithms\\gozi\\example_domains.txt\"\n", " Get: [0:9]\n", " Load 8 from file \"domain_generation_algorithms\\locky\\example_domains.txt\"\n", " Get: [0:6]\n", " Load 2,500 from file \"domain_generation_algorithms\\monerodownloader\\example_domains.txt\"\n", " Get: [0:2,000]\n", " Load 99 from file \"domain_generation_algorithms\\mydoom\\example_domains.txt\"\n", " Get: [0:79]\n", " Load 2,048 from file \"domain_generation_algorithms\\necurs\\example_domains.txt\"\n", " Get: [0:1,638]\n", " Load 1,000 from file \"domain_generation_algorithms\\newgoz\\example_domains.txt\"\n", " Get: [0:800]\n", " Load 128 from file \"domain_generation_algorithms\\nymaim\\example_domains.txt\"\n", " Get: [0:102]\n", " Load 704 from file \"domain_generation_algorithms\\nymaim2\\example_domains.txt\"\n", " Get: [0:563]\n", " Load 24 from file \"domain_generation_algorithms\\padcrypt\\example_domains.txt\"\n", " Get: [0:19]\n", " Load 306 from file \"domain_generation_algorithms\\pitou\\example_domains.txt\"\n", " Get: [0:244]\n", " Load 85 from file \"domain_generation_algorithms\\pizd\\example_domains.txt\"\n", " Get: [0:68]\n", " Load 100 from file \"domain_generation_algorithms\\proslikefan\\example_domains.txt\"\n", " Get: [0:80]\n", " Load 1,350 from file \"domain_generation_algorithms\\pushdo\\example_domains.txt\"\n", " Get: [0:1,080]\n", " Load 200 from file \"domain_generation_algorithms\\qadars\\example_domains.txt\"\n", " Get: [0:160]\n", " Load 5,000 from file \"domain_generation_algorithms\\qakbot\\example_domains.txt\"\n", " Get: [0:4,000]\n", " Load 100 from file \"domain_generation_algorithms\\ramnit\\example_domains.txt\"\n", " Get: [0:80]\n", " Load 100 from file \"domain_generation_algorithms\\reconyc\\example_domains.txt\"\n", " Get: [0:80]\n", " Load 2,001 from file \"domain_generation_algorithms\\shiotob\\example_domains.txt\"\n", " Get: [0:1,600]\n", " Load 1,000 from file \"domain_generation_algorithms\\simda\\example_domains.txt\"\n", " Get: [0:800]\n", " Load 40 from file \"domain_generation_algorithms\\sisron\\example_domains.txt\"\n", " Get: [0:32]\n", " Load 64 from file \"domain_generation_algorithms\\symmi\\example_domains.txt\"\n", " Get: [0:51]\n", " Load 1,746 from file \"domain_generation_algorithms\\tempedreve\\example_domains.txt\"\n", " Get: [0:1,396]\n", " Load 401 from file \"domain_generation_algorithms\\tinba\\example_domains.txt\"\n", " Get: [0:320]\n", " Load 120 from file \"domain_generation_algorithms\\unnamed_downloader\\example_domains.txt\"\n", " Get: [0:96]\n", " Load 30 from file \"domain_generation_algorithms\\unnamed_javascript_dga\\example_domains.txt\"\n", " Get: [0:24]\n", "Saved 18,119 into \"data/malignant.train.txt\"\n", " Load 1,000 from file \"domain_generation_algorithms\\banjori\\example_domains.txt\"\n", " Get: [800:900]\n", " Load 2,160 from file \"domain_generation_algorithms\\bazarbackdoor\\example_domains.txt\"\n", " Get: [1,728:1,944]\n", " Load 256 from file \"domain_generation_algorithms\\chinad\\example_domains.txt\"\n", " Get: [204:230]\n", " Load 40 from file \"domain_generation_algorithms\\corebot\\example_domains.txt\"\n", " Get: [32:36]\n", " Load 30 from file \"domain_generation_algorithms\\dircrypt\\example_domains.txt\"\n", " Get: [24:27]\n", " Load 5 from file \"domain_generation_algorithms\\dnschanger\\example_domains.txt\"\n", " Get: [4:4]\n", " Load 12 from file \"domain_generation_algorithms\\gozi\\example_domains.txt\"\n", " Get: [9:10]\n", " Load 8 from file \"domain_generation_algorithms\\locky\\example_domains.txt\"\n", " Get: [6:7]\n", " Load 2,500 from file \"domain_generation_algorithms\\monerodownloader\\example_domains.txt\"\n", " Get: [2,000:2,250]\n", " Load 99 from file \"domain_generation_algorithms\\mydoom\\example_domains.txt\"\n", " Get: [79:89]\n", " Load 2,048 from file \"domain_generation_algorithms\\necurs\\example_domains.txt\"\n", " Get: [1,638:1,843]\n", " Load 1,000 from file \"domain_generation_algorithms\\newgoz\\example_domains.txt\"\n", " Get: [800:900]\n", " Load 128 from file \"domain_generation_algorithms\\nymaim\\example_domains.txt\"\n", " Get: [102:115]\n", " Load 704 from file \"domain_generation_algorithms\\nymaim2\\example_domains.txt\"\n", " Get: [563:633]\n", " Load 24 from file \"domain_generation_algorithms\\padcrypt\\example_domains.txt\"\n", " Get: [19:21]\n", " Load 306 from file \"domain_generation_algorithms\\pitou\\example_domains.txt\"\n", " Get: [244:275]\n", " Load 85 from file \"domain_generation_algorithms\\pizd\\example_domains.txt\"\n", " Get: [68:76]\n", " Load 100 from file \"domain_generation_algorithms\\proslikefan\\example_domains.txt\"\n", " Get: [80:90]\n", " Load 1,350 from file \"domain_generation_algorithms\\pushdo\\example_domains.txt\"\n", " Get: [1,080:1,215]\n", " Load 200 from file \"domain_generation_algorithms\\qadars\\example_domains.txt\"\n", " Get: [160:180]\n", " Load 5,000 from file \"domain_generation_algorithms\\qakbot\\example_domains.txt\"\n", " Get: [4,000:4,500]\n", " Load 100 from file \"domain_generation_algorithms\\ramnit\\example_domains.txt\"\n", " Get: [80:90]\n", " Load 100 from file \"domain_generation_algorithms\\reconyc\\example_domains.txt\"\n", " Get: [80:90]\n", " Load 2,001 from file \"domain_generation_algorithms\\shiotob\\example_domains.txt\"\n", " Get: [1,600:1,800]\n", " Load 1,000 from file \"domain_generation_algorithms\\simda\\example_domains.txt\"\n", " Get: [800:900]\n", " Load 40 from file \"domain_generation_algorithms\\sisron\\example_domains.txt\"\n", " Get: [32:36]\n", " Load 64 from file \"domain_generation_algorithms\\symmi\\example_domains.txt\"\n", " Get: [51:57]\n", " Load 1,746 from file \"domain_generation_algorithms\\tempedreve\\example_domains.txt\"\n", " Get: [1,396:1,571]\n", " Load 401 from file \"domain_generation_algorithms\\tinba\\example_domains.txt\"\n", " Get: [320:360]\n", " Load 120 from file \"domain_generation_algorithms\\unnamed_downloader\\example_domains.txt\"\n", " Get: [96:108]\n", " Load 30 from file \"domain_generation_algorithms\\unnamed_javascript_dga\\example_domains.txt\"\n", " Get: [24:27]\n", "Saved 2,265 into \"data/malignant.dev.txt\"\n", " Load 1,000 from file \"domain_generation_algorithms\\banjori\\example_domains.txt\"\n", " Get: [900:1,000]\n", " Load 2,160 from file \"domain_generation_algorithms\\bazarbackdoor\\example_domains.txt\"\n", " Get: [1,944:2,160]\n", " Load 256 from file \"domain_generation_algorithms\\chinad\\example_domains.txt\"\n", " Get: [230:256]\n", " Load 40 from file \"domain_generation_algorithms\\corebot\\example_domains.txt\"\n", " Get: [36:40]\n", " Load 30 from file \"domain_generation_algorithms\\dircrypt\\example_domains.txt\"\n", " Get: [27:30]\n", " Load 5 from file \"domain_generation_algorithms\\dnschanger\\example_domains.txt\"\n", " Get: [4:5]\n", " Load 12 from file \"domain_generation_algorithms\\gozi\\example_domains.txt\"\n", " Get: [10:12]\n", " Load 8 from file \"domain_generation_algorithms\\locky\\example_domains.txt\"\n", " Get: [7:8]\n", " Load 2,500 from file \"domain_generation_algorithms\\monerodownloader\\example_domains.txt\"\n", " Get: [2,250:2,500]\n", " Load 99 from file \"domain_generation_algorithms\\mydoom\\example_domains.txt\"\n", " Get: [89:99]\n", " Load 2,048 from file \"domain_generation_algorithms\\necurs\\example_domains.txt\"\n", " Get: [1,843:2,048]\n", " Load 1,000 from file \"domain_generation_algorithms\\newgoz\\example_domains.txt\"\n", " Get: [900:1,000]\n", " Load 128 from file \"domain_generation_algorithms\\nymaim\\example_domains.txt\"\n", " Get: [115:128]\n", " Load 704 from file \"domain_generation_algorithms\\nymaim2\\example_domains.txt\"\n", " Get: [633:704]\n", " Load 24 from file \"domain_generation_algorithms\\padcrypt\\example_domains.txt\"\n", " Get: [21:24]\n", " Load 306 from file \"domain_generation_algorithms\\pitou\\example_domains.txt\"\n", " Get: [275:306]\n", " Load 85 from file \"domain_generation_algorithms\\pizd\\example_domains.txt\"\n", " Get: [76:85]\n", " Load 100 from file \"domain_generation_algorithms\\proslikefan\\example_domains.txt\"\n", " Get: [90:100]\n", " Load 1,350 from file \"domain_generation_algorithms\\pushdo\\example_domains.txt\"\n", " Get: [1,215:1,350]\n", " Load 200 from file \"domain_generation_algorithms\\qadars\\example_domains.txt\"\n", " Get: [180:200]\n", " Load 5,000 from file \"domain_generation_algorithms\\qakbot\\example_domains.txt\"\n", " Get: [4,500:5,000]\n", " Load 100 from file \"domain_generation_algorithms\\ramnit\\example_domains.txt\"\n", " Get: [90:100]\n", " Load 100 from file \"domain_generation_algorithms\\reconyc\\example_domains.txt\"\n", " Get: [90:100]\n", " Load 2,001 from file \"domain_generation_algorithms\\shiotob\\example_domains.txt\"\n", " Get: [1,800:2,001]\n", " Load 1,000 from file \"domain_generation_algorithms\\simda\\example_domains.txt\"\n", " Get: [900:1,000]\n", " Load 40 from file \"domain_generation_algorithms\\sisron\\example_domains.txt\"\n", " Get: [36:40]\n", " Load 64 from file \"domain_generation_algorithms\\symmi\\example_domains.txt\"\n", " Get: [57:64]\n", " Load 1,746 from file \"domain_generation_algorithms\\tempedreve\\example_domains.txt\"\n", " Get: [1,571:1,746]\n", " Load 401 from file \"domain_generation_algorithms\\tinba\\example_domains.txt\"\n", " Get: [360:401]\n", " Load 120 from file \"domain_generation_algorithms\\unnamed_downloader\\example_domains.txt\"\n", " Get: [108:120]\n", " Load 30 from file \"domain_generation_algorithms\\unnamed_javascript_dga\\example_domains.txt\"\n", " Get: [27:30]\n", "Saved 2,273 into \"data/malignant.test.txt\"\n" ] } ], "source": [ "# malitious data\n", "#\n", "\n", "def read_data(file_name, start, end):\n", " ret = open(file_name, encoding='utf-8').read().splitlines()\n", " print(f' Load {len(ret):,} from file \"{file_name}\"')\n", " st, en = int(start*len(ret)), int(end*len(ret))\n", " print(f' Get: [{st:,}:{en:,}]')\n", " return ret[st:en]\n", "\n", "def read(f, start, end):\n", " ret = []\n", " if type(f) == list:\n", " for ff in f:\n", " ret += read_data(ff, start=start, end=end)\n", " elif type(f) == str:\n", " ret = read_data(f, start=start, end=end)\n", " return ret\n", " \n", "def add_label(in_file, out_file, label, split_by_words=True, start=0, end=0.8):\n", " lines = read(in_file, start=start, end=end)\n", " if split_by_words: lines = [split(el) for el in lines]\n", " lines = [f'__label__{label} {el}\\n' for el in lines]\n", " with open(out_file, 'w') as f:\n", " f.writelines(lines)\n", " print(f'Saved {len(lines):,} into \"{out_file}\"')\n", "\n", "dr = 'domain_generation_algorithms/*/example_domains.txt'\n", "\n", "import glob\n", "\n", "in_file = list(glob.iglob(dr, recursive=True))\n", "\n", "print('malignant files:', len(in_file), )\n", "\n", "for split_by_words in [True, False]:\n", " for label, start, end in [('train', 0, 0.8), ('dev', 0.8, 0.9), ('test', 0.9, 1)]:\n", " out_file = f'data/malignant.{\"split_by_words.\" if split_by_words else \"\"}{label}.txt'\n", " add_label(in_file, out_file, label, split_by_words=split_by_words, start=start, end=end)\n", "\n", "# Saved 18,119 into \"data/malignant.train.txt\"\n", "\n", "# Saved 2,265 into \"data/malignant.dev.txt\"\n", "\n", "# Saved 2,273 into \"data/malignant.test.txt\"" ] }, { "cell_type": "code", "execution_count": 68, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T17:42:48.668221Z", "start_time": "2020-09-17T17:42:10.107468Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Load 525,215 from file \"data/Alex_top-0.5M.txt\"\n", " Get: [0:420,172]\n", "Saved 420,172 into \"data/Alex_top-0.5M.binign.split_by_words.train.txt\"\n" ] } ], "source": [ "in_file = 'data/Alex_top-0.5M.txt'\n", "label = 'benign'\n", "split_by_words=True\n", "out_file = f'{in_file[:-4]}.{label}.{\"split_by_words.\" if split_by_words else \"\"}train.txt'\n", "add_label(in_file, out_file, label, split_by_words=split_by_words, start=0, end=0.8)" ] }, { "cell_type": "code", "execution_count": 71, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T19:18:34.883155Z", "start_time": "2020-09-17T19:18:34.091865Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved \"data/data.dev.txt\"\n", "Saved \"data/data.test.txt\"\n", "Saved \"data/data.train.txt\"\n", "Saved \"data/data.split_by_words.dev.txt\"\n", "Saved \"data/data.split_by_words.test.txt\"\n", "Saved \"data/data.split_by_words.train.txt\"\n" ] } ], "source": [ "# compound\n", "def compound_files(in_files, out_file):\n", " with open(out_file, 'w') as outfile:\n", " for fname in in_files:\n", " with open(fname) as infile:\n", " outfile.write(infile.read())\n", " print(f'Saved \"{out_file}\"')\n", " \n", "vars = ['dev', 'test', 'train', 'split_by_words.dev', 'split_by_words.test', 'split_by_words.train']\n", "_ = [compound_files([f'data/{el}.{var}.txt' for el in ['Alex_top-0.5M.benign', 'malignant']], f'data/data.{var}.txt') for var in vars]\n" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "## Prepare Corpuses" ] }, { "cell_type": "code", "execution_count": 102, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T21:06:07.377076Z", "start_time": "2020-09-17T21:05:55.288531Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 14:05:55,290 Reading data from data\n", "2020-09-17 14:05:55,292 Train: data\\data.train.txt\n", "2020-09-17 14:05:55,293 Dev: data\\data.dev.txt\n", "2020-09-17 14:05:55,295 Test: data\\data.test.txt\n" ] } ], "source": [ "from flair.data import Corpus\n", "from flair.datasets import ClassificationCorpus\n", "\n", "corpus_no_split: Corpus = ClassificationCorpus('data',\n", " test_file='data.test.txt',\n", " dev_file='data.dev.txt',\n", " train_file='data.train.txt')" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2020-09-18T04:23:32.194390Z", "start_time": "2020-09-18T04:23:09.321567Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 21:23:21,636 Reading data from data\n", "2020-09-17 21:23:21,639 Train: data\\data.split_by_words.train.txt\n", "2020-09-17 21:23:21,640 Dev: data\\data.split_by_words.dev.txt\n", "2020-09-17 21:23:21,642 Test: data\\data.split_by_words.test.txt\n" ] } ], "source": [ "from flair.data import Corpus\n", "from flair.datasets import ClassificationCorpus\n", "\n", "corpus_word_split: Corpus = ClassificationCorpus('data',\n", " test_file='data.split_by_words.test.txt',\n", " dev_file='data.split_by_words.dev.txt',\n", " train_file='data.split_by_words.train.txt')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Train models" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2020-09-18T04:23:37.448030Z", "start_time": "2020-09-18T04:23:37.414995Z" } }, "outputs": [], "source": [ "from flair.data import Corpus\n", "from flair.datasets import TREC_6\n", "from flair.embeddings import WordEmbeddings, FlairEmbeddings, DocumentRNNEmbeddings\n", "from flair.models import TextClassifier\n", "from flair.trainers import ModelTrainer" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### GloVe / no_split" ] }, { "cell_type": "code", "execution_count": 103, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T21:47:32.731841Z", "start_time": "2020-09-17T21:44:26.880976Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 14:44:26,883 Computing label dictionary. Progress:\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "100%|████████████████████████████████████████████████████████████████████████| 422936/422936 [02:02<00:00, 3460.80it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 14:47:32,717 [b'benign', b'malignant']\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "# 2. create the label dictionary\n", "label_dict = corpus_no_split.make_label_dictionary()" ] }, { "cell_type": "code", "execution_count": 77, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T20:00:44.495449Z", "start_time": "2020-09-17T20:00:44.486452Z" }, "hidden": true }, "outputs": [], "source": [ "# len(label_dict), type(label_dict), dir(label_dict)" ] }, { "cell_type": "code", "execution_count": 104, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T21:47:48.868431Z", "start_time": "2020-09-17T21:47:44.498524Z" }, "hidden": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "I0917 14:47:44.510528 11616 utils.py:422] loading Word2VecKeyedVectors object from C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim\n", "I0917 14:47:48.508447 11616 utils.py:461] loading vectors from C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim.vectors.npy with mmap=None\n", "I0917 14:47:48.596435 11616 utils.py:494] setting ignored attribute vectors_norm to None\n", "I0917 14:47:48.598442 11616 utils.py:428] loaded C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim\n" ] } ], "source": [ "word_embeddings = [WordEmbeddings('glove')]\n", "\n", "document_embeddings = DocumentRNNEmbeddings(word_embeddings, hidden_size=256)\n", "classifier = TextClassifier(document_embeddings, label_dictionary=label_dict)\n", "trainer = ModelTrainer(classifier, corpus_no_split)" ] }, { "cell_type": "code", "execution_count": 105, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T22:24:17.376845Z", "start_time": "2020-09-17T21:48:48.168604Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 14:48:48,172 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 14:48:48,174 Model: \"TextClassifier(\n", " (document_embeddings): DocumentRNNEmbeddings(\n", " (embeddings): StackedEmbeddings(\n", " (list_embedding_0): WordEmbeddings('glove')\n", " )\n", " (word_reprojection_map): Linear(in_features=100, out_features=100, bias=True)\n", " (rnn): GRU(100, 256, batch_first=True)\n", " (dropout): Dropout(p=0.5, inplace=False)\n", " )\n", " (decoder): Linear(in_features=256, out_features=2, bias=True)\n", " (loss_function): CrossEntropyLoss()\n", ")\"\n", "2020-09-17 14:48:48,177 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 14:48:48,179 Corpus: \"Corpus: 422936 train + 52867 dev + 52868 test sentences\"\n", "2020-09-17 14:48:48,181 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 14:48:48,183 Parameters:\n", "2020-09-17 14:48:48,185 - learning_rate: \"0.1\"\n", "2020-09-17 14:48:48,186 - mini_batch_size: \"256\"\n", "2020-09-17 14:48:48,188 - patience: \"3\"\n", "2020-09-17 14:48:48,190 - anneal_factor: \"0.5\"\n", "2020-09-17 14:48:48,192 - max_epochs: \"30\"\n", "2020-09-17 14:48:48,194 - shuffle: \"True\"\n", "2020-09-17 14:48:48,196 - train_with_dev: \"False\"\n", "2020-09-17 14:48:48,198 - batch_growth_annealing: \"False\"\n", "2020-09-17 14:48:48,200 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 14:48:48,201 Model training base path: \"models\\glove.no_split\"\n", "2020-09-17 14:48:48,202 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 14:48:48,203 Device: cuda:0\n", "2020-09-17 14:48:48,205 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 14:48:48,207 Embeddings storage mode: cpu\n", "2020-09-17 14:48:48,211 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 14:49:36,153 epoch 1 - iter 0/1653 - loss 0.69388503 - samples/sec: 128393.80\n", "2020-09-17 14:50:15,929 epoch 1 - iter 165/1653 - loss 0.08651164 - samples/sec: 1126.08\n", "2020-09-17 14:50:54,664 epoch 1 - iter 330/1653 - loss 0.06435409 - samples/sec: 1167.26\n", "2020-09-17 14:51:33,434 epoch 1 - iter 495/1653 - loss 0.05646344 - samples/sec: 1166.51\n", "2020-09-17 14:52:10,631 epoch 1 - iter 660/1653 - loss 0.05194066 - samples/sec: 1187.18\n", "2020-09-17 14:52:48,744 epoch 1 - iter 825/1653 - loss 0.04919084 - samples/sec: 1186.55\n", "2020-09-17 14:53:27,445 epoch 1 - iter 990/1653 - loss 0.04732453 - samples/sec: 1168.66\n", "2020-09-17 14:54:06,328 epoch 1 - iter 1155/1653 - loss 0.04645340 - samples/sec: 1162.49\n", "2020-09-17 14:54:43,202 epoch 1 - iter 1320/1653 - loss 0.04545171 - samples/sec: 1197.48\n", "2020-09-17 14:55:21,533 epoch 1 - iter 1485/1653 - loss 0.04474270 - samples/sec: 1181.75\n", "2020-09-17 14:56:00,142 epoch 1 - iter 1650/1653 - loss 0.04432238 - samples/sec: 1174.35\n", "2020-09-17 14:56:01,393 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 14:56:01,394 EPOCH 1 done: loss 0.0444 - lr 0.1000\n", "2020-09-17 14:57:31,944 DEV : loss 0.0511484332382679 - score 0.9935\n", "2020-09-17 14:57:55,029 BAD EPOCHS (no improvement): 0\n", "2020-09-17 14:58:02,371 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 14:58:42,942 epoch 2 - iter 0/1653 - loss 0.02709874 - samples/sec: 179740.57\n", "2020-09-17 14:59:16,022 epoch 2 - iter 165/1653 - loss 0.03845978 - samples/sec: 1295.01\n", "2020-09-17 14:59:50,570 epoch 2 - iter 330/1653 - loss 0.03891962 - samples/sec: 1309.00\n", "2020-09-17 15:00:25,253 epoch 2 - iter 495/1653 - loss 0.03933847 - samples/sec: 1266.20\n", "2020-09-17 15:00:59,358 epoch 2 - iter 660/1653 - loss 0.03953121 - samples/sec: 1325.80\n", "2020-09-17 15:01:33,742 epoch 2 - iter 825/1653 - loss 0.03977078 - samples/sec: 1316.79\n", "2020-09-17 15:02:07,335 epoch 2 - iter 990/1653 - loss 0.03973265 - samples/sec: 1341.63\n", "2020-09-17 15:02:41,530 epoch 2 - iter 1155/1653 - loss 0.03947790 - samples/sec: 1317.62\n", "2020-09-17 15:03:16,204 epoch 2 - iter 1320/1653 - loss 0.03945117 - samples/sec: 1300.75\n", "2020-09-17 15:03:49,646 epoch 2 - iter 1485/1653 - loss 0.03959553 - samples/sec: 1314.02\n", "2020-09-17 15:04:23,484 epoch 2 - iter 1650/1653 - loss 0.03975336 - samples/sec: 1335.19\n", "2020-09-17 15:04:24,578 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:04:24,579 EPOCH 2 done: loss 0.0397 - lr 0.1000\n", "2020-09-17 15:05:51,608 DEV : loss 0.05118423327803612 - score 0.9935\n", "2020-09-17 15:06:15,122 BAD EPOCHS (no improvement): 1\n", "2020-09-17 15:06:20,191 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:07:00,344 epoch 3 - iter 0/1653 - loss 0.02774704 - samples/sec: 147423.40\n", "2020-09-17 15:07:34,306 epoch 3 - iter 165/1653 - loss 0.03882953 - samples/sec: 1324.07\n", "2020-09-17 15:08:08,412 epoch 3 - iter 330/1653 - loss 0.03822373 - samples/sec: 1292.68\n", "2020-09-17 15:08:42,109 epoch 3 - iter 495/1653 - loss 0.03814894 - samples/sec: 1339.67\n", "2020-09-17 15:09:16,169 epoch 3 - iter 660/1653 - loss 0.03895379 - samples/sec: 1328.91\n", "2020-09-17 15:09:50,112 epoch 3 - iter 825/1653 - loss 0.03879977 - samples/sec: 1329.80\n", "2020-09-17 15:10:22,653 epoch 3 - iter 990/1653 - loss 0.03867084 - samples/sec: 1353.54\n", "2020-09-17 15:10:56,275 epoch 3 - iter 1155/1653 - loss 0.03904117 - samples/sec: 1343.59\n", "2020-09-17 15:11:30,477 epoch 3 - iter 1320/1653 - loss 0.03935906 - samples/sec: 1318.72\n", "2020-09-17 15:12:04,681 epoch 3 - iter 1485/1653 - loss 0.03931618 - samples/sec: 1324.61\n", "2020-09-17 15:12:38,752 epoch 3 - iter 1650/1653 - loss 0.03971090 - samples/sec: 1322.50\n", "2020-09-17 15:12:39,850 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:12:39,852 EPOCH 3 done: loss 0.0397 - lr 0.1000\n", "2020-09-17 15:14:05,519 DEV : loss 0.05117087811231613 - score 0.9935\n", "2020-09-17 15:14:29,358 BAD EPOCHS (no improvement): 2\n", "2020-09-17 15:14:34,167 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:15:14,417 epoch 4 - iter 0/1653 - loss 0.06731937 - samples/sec: 189416.39\n", "2020-09-17 15:15:47,173 epoch 4 - iter 165/1653 - loss 0.03893054 - samples/sec: 1343.17\n", "2020-09-17 15:16:18,561 epoch 4 - iter 330/1653 - loss 0.03808983 - samples/sec: 1446.22\n", "2020-09-17 15:16:50,150 epoch 4 - iter 495/1653 - loss 0.03887507 - samples/sec: 1431.33\n", "2020-09-17 15:17:21,311 epoch 4 - iter 660/1653 - loss 0.03973759 - samples/sec: 1454.74\n", "2020-09-17 15:17:50,701 epoch 4 - iter 825/1653 - loss 0.03959222 - samples/sec: 1501.68\n", "2020-09-17 15:18:20,614 epoch 4 - iter 990/1653 - loss 0.03953747 - samples/sec: 1509.51\n", "2020-09-17 15:18:50,904 epoch 4 - iter 1155/1653 - loss 0.03934484 - samples/sec: 1495.35\n", "2020-09-17 15:19:21,653 epoch 4 - iter 1320/1653 - loss 0.03946581 - samples/sec: 1473.24\n", "2020-09-17 15:19:51,603 epoch 4 - iter 1485/1653 - loss 0.03973620 - samples/sec: 1475.42\n", "2020-09-17 15:20:22,624 epoch 4 - iter 1650/1653 - loss 0.03963985 - samples/sec: 1455.37\n", "2020-09-17 15:20:23,779 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:20:23,781 EPOCH 4 done: loss 0.0396 - lr 0.1000\n", "2020-09-17 15:21:43,731 DEV : loss 0.051296789199113846 - score 0.9935\n", "2020-09-17 15:22:05,375 BAD EPOCHS (no improvement): 3\n", "2020-09-17 15:22:10,220 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:22:47,060 epoch 5 - iter 0/1653 - loss 0.06693242 - samples/sec: 190272.62\n", "2020-09-17 15:23:00,494 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:23:00,496 Exiting from training early.\n", "2020-09-17 15:23:00,499 Saving model ...\n", "2020-09-17 15:23:05,153 Done.\n", "2020-09-17 15:23:05,154 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:23:05,157 Testing using best model ...\n", "2020-09-17 15:23:05,197 loading file models\\glove.no_split\\best-model.pt\n", "2020-09-17 15:24:17,229 0.9935\t0.9935\t0.9935\n", "2020-09-17 15:24:17,230 \n", "MICRO_AVG: acc 0.987 - f1-score 0.9935\n", "MACRO_AVG: acc 0.4968 - f1-score 0.49835\n", "benign tp: 52522 - fp: 346 - fn: 0 - tn: 0 - precision: 0.9935 - recall: 1.0000 - accuracy: 0.9935 - f1-score: 0.9967\n", "malignant tp: 0 - fp: 0 - fn: 346 - tn: 52522 - precision: 0.0000 - recall: 0.0000 - accuracy: 0.0000 - f1-score: 0.0000\n", "2020-09-17 15:24:17,231 ----------------------------------------------------------------------------------------------------\n" ] }, { "data": { "text/plain": [ "{'test_score': 0.9935,\n", " 'dev_score_history': [0.9935, 0.9935, 0.9935, 0.9935],\n", " 'train_loss_history': [0.04442076490379541,\n", " 0.03973632867696497,\n", " 0.039671257708426756,\n", " 0.0396350922786416],\n", " 'dev_loss_history': [tensor(0.0511, device='cuda:0'),\n", " tensor(0.0512, device='cuda:0'),\n", " tensor(0.0512, device='cuda:0'),\n", " tensor(0.0513, device='cuda:0')]}" ] }, "execution_count": 105, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trainer.train('models/glove.no_split',\n", " learning_rate=0.1,\n", " mini_batch_size=256, # 32,\n", " anneal_factor=0.5,\n", " patience=3,\n", " max_epochs=30)\n", "\n", "# 2020-09-17 15:23:05,197 loading file models\\glove.no_split\\best-model.pt\n", "# 2020-09-17 15:24:17,229 0.9935\t0.9935\t0.9935\n", "# 2020-09-17 15:24:17,230 \n", "# MICRO_AVG: acc 0.987 - f1-score 0.9935\n", "# MACRO_AVG: acc 0.4968 - f1-score 0.49835\n", "# benign tp: 52522 - fp: 346 - fn: 0 - tn: 0 - precision: 0.9935 - recall: 1.0000 - accuracy: 0.9935 - f1-score: 0.9967\n", "# malignant tp: 0 - fp: 0 - fn: 346 - tn: 52522 - precision: 0.0000 - recall: 0.0000 - accuracy: 0.0000 - f1-score: 0.0000\n" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true }, "source": [ "### GloVe / word_split" ] }, { "cell_type": "code", "execution_count": 106, "metadata": { "ExecuteTime": { "end_time": "2020-09-17T22:37:51.286611Z", "start_time": "2020-09-17T22:31:13.551693Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created the directory models/glove.word_split \n", "2020-09-17 15:31:13,744 Computing label dictionary. Progress:\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "100%|████████████████████████████████████████████████████████████████████████| 422936/422936 [02:02<00:00, 3454.79it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 15:34:06,947 [b'benign', b'malignant']\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n", "I0917 15:34:06.951880 11616 utils.py:422] loading Word2VecKeyedVectors object from C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim\n", "I0917 15:34:08.144447 11616 utils.py:461] loading vectors from C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim.vectors.npy with mmap=None\n", "I0917 15:34:08.233437 11616 utils.py:494] setting ignored attribute vectors_norm to None\n", "I0917 15:34:08.234394 11616 utils.py:428] loaded C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 15:34:08,343 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:34:08,345 Model: \"TextClassifier(\n", " (document_embeddings): DocumentRNNEmbeddings(\n", " (embeddings): StackedEmbeddings(\n", " (list_embedding_0): WordEmbeddings('glove')\n", " )\n", " (word_reprojection_map): Linear(in_features=100, out_features=100, bias=True)\n", " (rnn): GRU(100, 256, batch_first=True)\n", " (dropout): Dropout(p=0.5, inplace=False)\n", " )\n", " (decoder): Linear(in_features=256, out_features=2, bias=True)\n", " (loss_function): CrossEntropyLoss()\n", ")\"\n", "2020-09-17 15:34:08,346 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:34:08,347 Corpus: \"Corpus: 422936 train + 52867 dev + 52868 test sentences\"\n", "2020-09-17 15:34:08,348 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:34:08,350 Parameters:\n", "2020-09-17 15:34:08,351 - learning_rate: \"0.1\"\n", "2020-09-17 15:34:08,352 - mini_batch_size: \"256\"\n", "2020-09-17 15:34:08,353 - patience: \"3\"\n", "2020-09-17 15:34:08,355 - anneal_factor: \"0.5\"\n", "2020-09-17 15:34:08,356 - max_epochs: \"30\"\n", "2020-09-17 15:34:08,357 - shuffle: \"True\"\n", "2020-09-17 15:34:08,359 - train_with_dev: \"False\"\n", "2020-09-17 15:34:08,361 - batch_growth_annealing: \"False\"\n", "2020-09-17 15:34:08,362 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:34:08,363 Model training base path: \"models\\glove.word_split\"\n", "2020-09-17 15:34:08,364 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:34:08,366 Device: cuda:0\n", "2020-09-17 15:34:08,366 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:34:08,367 Embeddings storage mode: cpu\n", "2020-09-17 15:34:08,371 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:34:48,021 epoch 1 - iter 0/1653 - loss 0.59104705 - samples/sec: 50693.60\n", "2020-09-17 15:35:44,035 epoch 1 - iter 165/1653 - loss 0.03837221 - samples/sec: 850.76\n", "2020-09-17 15:36:33,208 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:36:33,209 Exiting from training early.\n", "2020-09-17 15:36:33,210 Saving model ...\n", "2020-09-17 15:36:37,719 Done.\n", "2020-09-17 15:36:37,722 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 15:36:37,726 Testing using best model ...\n", "2020-09-17 15:37:51,270 0.9973\t0.9973\t0.9973\n", "2020-09-17 15:37:51,272 \n", "MICRO_AVG: acc 0.9947 - f1-score 0.9973\n", "MACRO_AVG: acc 0.7949 - f1-score 0.8713500000000001\n", "benign tp: 52522 - fp: 141 - fn: 0 - tn: 205 - precision: 0.9973 - recall: 1.0000 - accuracy: 0.9973 - f1-score: 0.9986\n", "malignant tp: 205 - fp: 0 - fn: 141 - tn: 52522 - precision: 1.0000 - recall: 0.5925 - accuracy: 0.5925 - f1-score: 0.7441\n", "2020-09-17 15:37:51,274 ----------------------------------------------------------------------------------------------------\n" ] }, { "data": { "text/plain": [ "{'test_score': 0.9973,\n", " 'dev_score_history': [],\n", " 'train_loss_history': [],\n", " 'dev_loss_history': []}" ] }, "execution_count": 106, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_name = 'glove'\n", "model_var = 'word_split'\n", "corpus = corpus_word_split\n", "\n", "import os\n", "\n", "model_dir = f'models/{model_name}.{model_var}'\n", "try:\n", " os.mkdir(model_dir)\n", "except OSError:\n", " print (f\"Creation of the directory '{model_dir}' failed.\")\n", "else:\n", " print (f\"Created the directory '{model_dir}'\")\n", "\n", "label_dict = corpus.make_label_dictionary()\n", "word_embeddings = [WordEmbeddings(model_name)]\n", "document_embeddings = DocumentRNNEmbeddings(word_embeddings, hidden_size=256)\n", "classifier = TextClassifier(document_embeddings, label_dictionary=label_dict)\n", "trainer = ModelTrainer(classifier, corpus)\n", "\n", "trainer.train(f'models/{model_name}.{model_var}',\n", " learning_rate=0.1,\n", " mini_batch_size=256, # 32,\n", " anneal_factor=0.5,\n", " patience=3,\n", " max_epochs=30)\n", "# MICRO_AVG: acc 0.9947 - f1-score 0.9973\n", "# MACRO_AVG: acc 0.7949 - f1-score 0.8713500000000001\n", "# benign tp: 52522 - fp: 141 - fn: 0 - tn: 205 - precision: 0.9973 - recall: 1.0000 - accuracy: 0.9973 - f1-score: 0.9986\n", "# malignant tp: 205 - fp: 0 - fn: 141 - tn: 52522 - precision: 1.0000 - recall: 0.5925 - accuracy: 0.5925 - f1-score: 0.7441\n", "# 20" ] }, { "cell_type": "code", "execution_count": 107, "metadata": { "ExecuteTime": { "end_time": "2020-09-18T02:19:07.799282Z", "start_time": "2020-09-17T22:59:57.534785Z" }, "hidden": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created the directory 'models/glove.word_split_0.1_32'\n", "2020-09-17 15:59:57,686 Computing label dictionary. Progress:\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "100%|████████████████████████████████████████████████████████████████████████| 422936/422936 [01:49<00:00, 3879.39it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 16:02:37,984 [b'benign', b'malignant']\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n", "I0917 16:02:37.986963 11616 utils.py:422] loading Word2VecKeyedVectors object from C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim\n", "I0917 16:02:39.216480 11616 utils.py:461] loading vectors from C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim.vectors.npy with mmap=None\n", "I0917 16:02:39.284472 11616 utils.py:494] setting ignored attribute vectors_norm to None\n", "I0917 16:02:39.285509 11616 utils.py:428] loaded C:\\Users\\leo_g\\.flair\\embeddings\\glove.gensim\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 16:02:39,303 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:02:39,305 Model: \"TextClassifier(\n", " (document_embeddings): DocumentRNNEmbeddings(\n", " (embeddings): StackedEmbeddings(\n", " (list_embedding_0): WordEmbeddings('glove')\n", " )\n", " (word_reprojection_map): Linear(in_features=100, out_features=100, bias=True)\n", " (rnn): GRU(100, 256, batch_first=True)\n", " (dropout): Dropout(p=0.5, inplace=False)\n", " )\n", " (decoder): Linear(in_features=256, out_features=2, bias=True)\n", " (loss_function): CrossEntropyLoss()\n", ")\"\n", "2020-09-17 16:02:39,306 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:02:39,308 Corpus: \"Corpus: 422936 train + 52867 dev + 52868 test sentences\"\n", "2020-09-17 16:02:39,310 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:02:39,311 Parameters:\n", "2020-09-17 16:02:39,312 - learning_rate: \"0.1\"\n", "2020-09-17 16:02:39,314 - mini_batch_size: \"32\"\n", "2020-09-17 16:02:39,315 - patience: \"3\"\n", "2020-09-17 16:02:39,316 - anneal_factor: \"0.5\"\n", "2020-09-17 16:02:39,317 - max_epochs: \"30\"\n", "2020-09-17 16:02:39,318 - shuffle: \"True\"\n", "2020-09-17 16:02:39,319 - train_with_dev: \"False\"\n", "2020-09-17 16:02:39,321 - batch_growth_annealing: \"False\"\n", "2020-09-17 16:02:39,322 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:02:39,323 Model training base path: \"models\\glove.word_split_0.1_32\"\n", "2020-09-17 16:02:39,325 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:02:39,326 Device: cuda:0\n", "2020-09-17 16:02:39,328 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:02:39,329 Embeddings storage mode: cpu\n", "2020-09-17 16:02:39,333 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:03:16,248 epoch 1 - iter 0/13217 - loss 0.65148681 - samples/sec: 612646.10\n", "2020-09-17 16:04:21,020 epoch 1 - iter 1321/13217 - loss 0.02143074 - samples/sec: 703.52\n", "2020-09-17 16:05:27,157 epoch 1 - iter 2642/13217 - loss 0.01829847 - samples/sec: 699.90\n", "2020-09-17 16:06:32,250 epoch 1 - iter 3963/13217 - loss 0.01450372 - samples/sec: 693.00\n", "2020-09-17 16:07:37,059 epoch 1 - iter 5284/13217 - loss 0.01213865 - samples/sec: 687.21\n", "2020-09-17 16:08:40,767 epoch 1 - iter 6605/13217 - loss 0.01044630 - samples/sec: 717.94\n", "2020-09-17 16:09:46,138 epoch 1 - iter 7926/13217 - loss 0.00926823 - samples/sec: 691.00\n", "2020-09-17 16:10:49,523 epoch 1 - iter 9247/13217 - loss 0.00840319 - samples/sec: 712.97\n", "2020-09-17 16:11:53,588 epoch 1 - iter 10568/13217 - loss 0.00771009 - samples/sec: 715.12\n", "2020-09-17 16:12:57,914 epoch 1 - iter 11889/13217 - loss 0.00717964 - samples/sec: 710.71\n", "2020-09-17 16:14:01,497 epoch 1 - iter 13210/13217 - loss 0.00670514 - samples/sec: 720.37\n", "2020-09-17 16:14:02,538 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:14:02,539 EPOCH 1 done: loss 0.0067 - lr 0.1000\n", "2020-09-17 16:15:54,563 DEV : loss 0.001983748283237219 - score 0.9995\n", "2020-09-17 16:16:19,794 BAD EPOCHS (no improvement): 0\n", "2020-09-17 16:16:24,827 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:17:01,905 epoch 2 - iter 0/13217 - loss 0.00001842 - samples/sec: 621672.50\n", "2020-09-17 16:18:05,268 epoch 2 - iter 1321/13217 - loss 0.00167200 - samples/sec: 720.59\n", "2020-09-17 16:19:08,323 epoch 2 - iter 2642/13217 - loss 0.00169976 - samples/sec: 724.11\n", "2020-09-17 16:20:11,218 epoch 2 - iter 3963/13217 - loss 0.00181361 - samples/sec: 727.21\n", "2020-09-17 16:21:16,829 epoch 2 - iter 5284/13217 - loss 0.00196471 - samples/sec: 688.71\n", "2020-09-17 16:22:23,891 epoch 2 - iter 6605/13217 - loss 0.00200321 - samples/sec: 673.55\n", "2020-09-17 16:23:28,083 epoch 2 - iter 7926/13217 - loss 0.00198566 - samples/sec: 703.62\n", "2020-09-17 16:24:31,990 epoch 2 - iter 9247/13217 - loss 0.00192942 - samples/sec: 709.07\n", "2020-09-17 16:25:34,692 epoch 2 - iter 10568/13217 - loss 0.00198261 - samples/sec: 720.19\n", "2020-09-17 16:26:38,197 epoch 2 - iter 11889/13217 - loss 0.00191374 - samples/sec: 719.92\n", "2020-09-17 16:27:42,290 epoch 2 - iter 13210/13217 - loss 0.00187714 - samples/sec: 714.01\n", "2020-09-17 16:27:43,357 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:27:43,358 EPOCH 2 done: loss 0.0019 - lr 0.1000\n", "2020-09-17 16:29:28,787 DEV : loss 0.0007546731503680348 - score 0.9998\n", "2020-09-17 16:29:52,139 BAD EPOCHS (no improvement): 0\n", "2020-09-17 16:29:57,178 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:30:33,775 epoch 3 - iter 0/13217 - loss 0.00001745 - samples/sec: 703368.52\n", "2020-09-17 16:31:37,246 epoch 3 - iter 1321/13217 - loss 0.00152487 - samples/sec: 700.40\n", "2020-09-17 16:32:41,092 epoch 3 - iter 2642/13217 - loss 0.00137410 - samples/sec: 705.65\n", "2020-09-17 16:33:43,323 epoch 3 - iter 3963/13217 - loss 0.00143205 - samples/sec: 724.32\n", "2020-09-17 16:34:46,758 epoch 3 - iter 5284/13217 - loss 0.00160484 - samples/sec: 711.72\n", "2020-09-17 16:35:51,277 epoch 3 - iter 6605/13217 - loss 0.00159391 - samples/sec: 681.73\n", "2020-09-17 16:36:53,881 epoch 3 - iter 7926/13217 - loss 0.00154109 - samples/sec: 719.91\n", "2020-09-17 16:37:58,160 epoch 3 - iter 9247/13217 - loss 0.00149967 - samples/sec: 701.76\n", "2020-09-17 16:39:01,961 epoch 3 - iter 10568/13217 - loss 0.00145412 - samples/sec: 689.06\n", "2020-09-17 16:40:06,163 epoch 3 - iter 11889/13217 - loss 0.00141854 - samples/sec: 701.42\n", "2020-09-17 16:41:08,791 epoch 3 - iter 13210/13217 - loss 0.00142902 - samples/sec: 730.81\n", "2020-09-17 16:41:09,868 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:41:09,870 EPOCH 3 done: loss 0.0014 - lr 0.1000\n", "2020-09-17 16:42:55,711 DEV : loss 0.0009889831999316812 - score 0.9997\n", "2020-09-17 16:43:18,507 BAD EPOCHS (no improvement): 1\n", "2020-09-17 16:43:18,509 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:43:53,945 epoch 4 - iter 0/13217 - loss 0.06294222 - samples/sec: 741646.07\n", "2020-09-17 16:44:55,889 epoch 4 - iter 1321/13217 - loss 0.00212140 - samples/sec: 728.76\n", "2020-09-17 16:45:58,822 epoch 4 - iter 2642/13217 - loss 0.00172214 - samples/sec: 717.55\n", "2020-09-17 16:47:01,410 epoch 4 - iter 3963/13217 - loss 0.00155593 - samples/sec: 713.05\n", "2020-09-17 16:48:04,763 epoch 4 - iter 5284/13217 - loss 0.00148084 - samples/sec: 722.40\n", "2020-09-17 16:49:11,122 epoch 4 - iter 6605/13217 - loss 0.00139651 - samples/sec: 680.34\n", "2020-09-17 16:50:19,015 epoch 4 - iter 7926/13217 - loss 0.00138601 - samples/sec: 676.17\n", "2020-09-17 16:51:22,369 epoch 4 - iter 9247/13217 - loss 0.00140159 - samples/sec: 708.62\n", "2020-09-17 16:52:25,858 epoch 4 - iter 10568/13217 - loss 0.00135113 - samples/sec: 706.04\n", "2020-09-17 16:53:31,400 epoch 4 - iter 11889/13217 - loss 0.00133608 - samples/sec: 693.47\n", "2020-09-17 16:54:35,024 epoch 4 - iter 13210/13217 - loss 0.00128189 - samples/sec: 713.85\n", "2020-09-17 16:54:36,104 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:54:36,106 EPOCH 4 done: loss 0.0013 - lr 0.1000\n", "2020-09-17 16:56:28,489 DEV : loss 0.0009011110523715615 - score 0.9998\n", "2020-09-17 16:56:51,851 BAD EPOCHS (no improvement): 2\n", "2020-09-17 16:56:56,646 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 16:57:32,960 epoch 5 - iter 0/13217 - loss 0.00001501 - samples/sec: 704562.00\n", "2020-09-17 16:58:37,417 epoch 5 - iter 1321/13217 - loss 0.00071459 - samples/sec: 704.45\n", "2020-09-17 16:59:41,029 epoch 5 - iter 2642/13217 - loss 0.00108001 - samples/sec: 714.56\n", "2020-09-17 17:00:45,127 epoch 5 - iter 3963/13217 - loss 0.00107526 - samples/sec: 716.24\n", "2020-09-17 17:01:49,192 epoch 5 - iter 5284/13217 - loss 0.00109712 - samples/sec: 710.53\n", "2020-09-17 17:02:53,486 epoch 5 - iter 6605/13217 - loss 0.00119771 - samples/sec: 715.39\n", "2020-09-17 17:03:56,394 epoch 5 - iter 7926/13217 - loss 0.00124413 - samples/sec: 722.50\n", "2020-09-17 17:05:00,330 epoch 5 - iter 9247/13217 - loss 0.00116674 - samples/sec: 709.42\n", "2020-09-17 17:06:03,429 epoch 5 - iter 10568/13217 - loss 0.00115461 - samples/sec: 721.75\n", "2020-09-17 17:07:07,310 epoch 5 - iter 11889/13217 - loss 0.00112642 - samples/sec: 701.50\n", "2020-09-17 17:08:12,959 epoch 5 - iter 13210/13217 - loss 0.00116030 - samples/sec: 698.27\n", "2020-09-17 17:08:14,009 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 17:08:14,011 EPOCH 5 done: loss 0.0012 - lr 0.1000\n", "2020-09-17 17:10:03,072 DEV : loss 0.0006250182632356882 - score 0.9998\n", "2020-09-17 17:10:26,699 BAD EPOCHS (no improvement): 3\n", "2020-09-17 17:10:31,861 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 17:11:08,996 epoch 6 - iter 0/13217 - loss 0.00000176 - samples/sec: 704559.20\n", "2020-09-17 17:12:13,408 epoch 6 - iter 1321/13217 - loss 0.00091146 - samples/sec: 711.40\n", "2020-09-17 17:13:18,116 epoch 6 - iter 2642/13217 - loss 0.00087117 - samples/sec: 701.09\n", "2020-09-17 17:14:22,687 epoch 6 - iter 3963/13217 - loss 0.00081208 - samples/sec: 710.96\n", "2020-09-17 17:15:26,211 epoch 6 - iter 5284/13217 - loss 0.00090058 - samples/sec: 715.26\n", "2020-09-17 17:16:29,287 epoch 6 - iter 6605/13217 - loss 0.00089744 - samples/sec: 718.03\n", "2020-09-17 17:17:33,826 epoch 6 - iter 7926/13217 - loss 0.00093396 - samples/sec: 701.95\n", "2020-09-17 17:18:37,474 epoch 6 - iter 9247/13217 - loss 0.00089984 - samples/sec: 713.70\n", "2020-09-17 17:19:41,026 epoch 6 - iter 10568/13217 - loss 0.00093344 - samples/sec: 715.10\n", "2020-09-17 17:20:43,824 epoch 6 - iter 11889/13217 - loss 0.00091009 - samples/sec: 724.39\n", "2020-09-17 17:21:47,979 epoch 6 - iter 13210/13217 - loss 0.00092920 - samples/sec: 708.41\n", "2020-09-17 17:21:49,475 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 17:21:49,476 EPOCH 6 done: loss 0.0009 - lr 0.1000\n", "2020-09-17 17:23:38,189 DEV : loss 0.0004954988835379481 - score 0.9998\n", "Epoch 5: reducing learning rate of group 0 to 5.0000e-02.\n", "2020-09-17 17:24:01,825 BAD EPOCHS (no improvement): 4\n", "2020-09-17 17:24:08,855 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 17:24:45,630 epoch 7 - iter 0/13217 - loss 0.00000322 - samples/sec: 704553.60\n", "2020-09-17 17:25:49,729 epoch 7 - iter 1321/13217 - loss 0.00089106 - samples/sec: 712.68\n", "2020-09-17 17:26:54,492 epoch 7 - iter 2642/13217 - loss 0.00086925 - samples/sec: 705.96\n", "2020-09-17 17:27:59,660 epoch 7 - iter 3963/13217 - loss 0.00077463 - samples/sec: 701.42\n", "2020-09-17 17:29:04,049 epoch 7 - iter 5284/13217 - loss 0.00089802 - samples/sec: 701.92\n", "2020-09-17 17:30:13,671 epoch 7 - iter 6605/13217 - loss 0.00090462 - samples/sec: 649.18\n", "2020-09-17 17:31:24,745 epoch 7 - iter 7926/13217 - loss 0.00085079 - samples/sec: 635.67\n", "2020-09-17 17:32:41,295 epoch 7 - iter 9247/13217 - loss 0.00082527 - samples/sec: 598.37\n", "2020-09-17 17:33:56,378 epoch 7 - iter 10568/13217 - loss 0.00085088 - samples/sec: 602.41\n", "2020-09-17 17:35:13,064 epoch 7 - iter 11889/13217 - loss 0.00082188 - samples/sec: 598.52\n", "2020-09-17 17:36:31,179 epoch 7 - iter 13210/13217 - loss 0.00082095 - samples/sec: 587.65\n", "2020-09-17 17:36:32,722 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 17:36:32,725 EPOCH 7 done: loss 0.0008 - lr 0.0500\n", "2020-09-17 17:38:47,766 DEV : loss 0.0004934129538014531 - score 0.9998\n", "2020-09-17 17:39:16,238 BAD EPOCHS (no improvement): 1\n", "2020-09-17 17:39:22,009 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 17:40:07,743 epoch 8 - iter 0/13217 - loss 0.00001816 - samples/sec: 640491.94\n", "2020-09-17 17:41:13,203 epoch 8 - iter 1321/13217 - loss 0.00072492 - samples/sec: 698.49\n", "2020-09-17 17:42:15,113 epoch 8 - iter 2642/13217 - loss 0.00071857 - samples/sec: 736.85\n", "2020-09-17 17:43:17,100 epoch 8 - iter 3963/13217 - loss 0.00070769 - samples/sec: 736.24\n", "2020-09-17 17:44:19,633 epoch 8 - iter 5284/13217 - loss 0.00069033 - samples/sec: 733.01\n", "2020-09-17 17:45:21,982 epoch 8 - iter 6605/13217 - loss 0.00065237 - samples/sec: 723.92\n", "2020-09-17 17:46:24,392 epoch 8 - iter 7926/13217 - loss 0.00068676 - samples/sec: 731.28\n", "2020-09-17 17:47:29,668 epoch 8 - iter 9247/13217 - loss 0.00070600 - samples/sec: 701.02\n", "2020-09-17 17:48:32,941 epoch 8 - iter 10568/13217 - loss 0.00070774 - samples/sec: 721.29\n", "2020-09-17 17:49:38,541 epoch 8 - iter 11889/13217 - loss 0.00073834 - samples/sec: 697.21\n", "2020-09-17 17:50:43,043 epoch 8 - iter 13210/13217 - loss 0.00072599 - samples/sec: 700.09\n", "2020-09-17 17:50:45,556 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 17:50:45,558 EPOCH 8 done: loss 0.0007 - lr 0.0500\n", "2020-09-17 17:52:31,078 DEV : loss 0.000518464541528374 - score 0.9998\n", "2020-09-17 17:52:53,715 BAD EPOCHS (no improvement): 2\n", "2020-09-17 17:52:58,331 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 17:53:35,893 epoch 9 - iter 0/13217 - loss 0.00000280 - samples/sec: 681800.81\n", "2020-09-17 17:54:40,360 epoch 9 - iter 1321/13217 - loss 0.00053752 - samples/sec: 711.24\n", "2020-09-17 17:55:44,220 epoch 9 - iter 2642/13217 - loss 0.00055666 - samples/sec: 708.48\n", "2020-09-17 17:56:49,121 epoch 9 - iter 3963/13217 - loss 0.00056536 - samples/sec: 707.48\n", "2020-09-17 17:57:51,591 epoch 9 - iter 5284/13217 - loss 0.00064781 - samples/sec: 724.58\n", "2020-09-17 17:58:54,946 epoch 9 - iter 6605/13217 - loss 0.00066503 - samples/sec: 714.97\n", "2020-09-17 17:59:59,131 epoch 9 - iter 7926/13217 - loss 0.00062696 - samples/sec: 714.57\n", "2020-09-17 18:01:03,209 epoch 9 - iter 9247/13217 - loss 0.00062627 - samples/sec: 707.09\n", "2020-09-17 18:02:06,753 epoch 9 - iter 10568/13217 - loss 0.00061224 - samples/sec: 720.24\n", "2020-09-17 18:03:08,811 epoch 9 - iter 11889/13217 - loss 0.00061774 - samples/sec: 721.60\n", "2020-09-17 18:04:10,771 epoch 9 - iter 13210/13217 - loss 0.00062220 - samples/sec: 730.80\n", "2020-09-17 18:04:11,785 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 18:04:11,948 EPOCH 9 done: loss 0.0006 - lr 0.0500\n", "2020-09-17 18:05:56,720 DEV : loss 0.00046400047722272575 - score 0.9998\n", "2020-09-17 18:06:19,653 BAD EPOCHS (no improvement): 3\n", "2020-09-17 18:06:24,785 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 18:06:59,914 epoch 10 - iter 0/13217 - loss 0.00000693 - samples/sec: 681829.65\n", "2020-09-17 18:08:01,805 epoch 10 - iter 1321/13217 - loss 0.00036199 - samples/sec: 717.82\n", "2020-09-17 18:09:03,560 epoch 10 - iter 2642/13217 - loss 0.00058025 - samples/sec: 737.93\n", "2020-09-17 18:10:05,546 epoch 10 - iter 3963/13217 - loss 0.00081828 - samples/sec: 735.65\n", "2020-09-17 18:11:07,529 epoch 10 - iter 5284/13217 - loss 0.00080180 - samples/sec: 726.78\n", "2020-09-17 18:12:09,807 epoch 10 - iter 6605/13217 - loss 0.00087584 - samples/sec: 731.39\n", "2020-09-17 18:13:11,888 epoch 10 - iter 7926/13217 - loss 0.00084962 - samples/sec: 732.69\n", "2020-09-17 18:14:14,196 epoch 10 - iter 9247/13217 - loss 0.00078623 - samples/sec: 722.14\n", "2020-09-17 18:15:17,225 epoch 10 - iter 10568/13217 - loss 0.00077308 - samples/sec: 722.68\n", "2020-09-17 18:16:18,905 epoch 10 - iter 11889/13217 - loss 0.00074222 - samples/sec: 723.74\n", "2020-09-17 18:17:19,675 epoch 10 - iter 13210/13217 - loss 0.00073419 - samples/sec: 751.40\n", "2020-09-17 18:17:20,618 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 18:17:20,620 EPOCH 10 done: loss 0.0007 - lr 0.0500\n", "2020-09-17 18:19:00,219 DEV : loss 0.00045348730054683983 - score 0.9998\n", "Epoch 9: reducing learning rate of group 0 to 2.5000e-02.\n", "2020-09-17 18:19:21,740 BAD EPOCHS (no improvement): 4\n", "2020-09-17 18:19:25,873 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 18:19:58,631 epoch 11 - iter 0/13217 - loss 0.00002681 - samples/sec: 812967.09\n", "2020-09-17 18:20:58,485 epoch 11 - iter 1321/13217 - loss 0.00066178 - samples/sec: 758.56\n", "2020-09-17 18:21:58,721 epoch 11 - iter 2642/13217 - loss 0.00069827 - samples/sec: 754.66\n", "2020-09-17 18:22:58,852 epoch 11 - iter 3963/13217 - loss 0.00064905 - samples/sec: 748.66\n", "2020-09-17 18:23:58,641 epoch 11 - iter 5284/13217 - loss 0.00061713 - samples/sec: 743.08\n", "2020-09-17 18:24:58,994 epoch 11 - iter 6605/13217 - loss 0.00061299 - samples/sec: 753.66\n", "2020-09-17 18:25:58,980 epoch 11 - iter 7926/13217 - loss 0.00058855 - samples/sec: 740.87\n", "2020-09-17 18:26:58,778 epoch 11 - iter 9247/13217 - loss 0.00058982 - samples/sec: 761.88\n", "2020-09-17 18:27:58,999 epoch 11 - iter 10568/13217 - loss 0.00060913 - samples/sec: 757.18\n", "2020-09-17 18:28:59,406 epoch 11 - iter 11889/13217 - loss 0.00060015 - samples/sec: 745.92\n", "2020-09-17 18:29:59,266 epoch 11 - iter 13210/13217 - loss 0.00059535 - samples/sec: 752.76\n", "2020-09-17 18:30:00,235 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 18:30:01,578 EPOCH 11 done: loss 0.0006 - lr 0.0250\n", "2020-09-17 18:31:41,077 DEV : loss 0.00044427913962863386 - score 0.9998\n", "2020-09-17 18:32:02,406 BAD EPOCHS (no improvement): 1\n", "2020-09-17 18:32:06,602 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 18:32:39,431 epoch 12 - iter 0/13217 - loss 0.00000679 - samples/sec: 812970.82\n", "2020-09-17 18:33:39,144 epoch 12 - iter 1321/13217 - loss 0.00055620 - samples/sec: 756.94\n", "2020-09-17 18:34:39,042 epoch 12 - iter 2642/13217 - loss 0.00063030 - samples/sec: 755.70\n", "2020-09-17 18:35:39,027 epoch 12 - iter 3963/13217 - loss 0.00075583 - samples/sec: 746.12\n", "2020-09-17 18:36:39,686 epoch 12 - iter 5284/13217 - loss 0.00071614 - samples/sec: 745.98\n", "2020-09-17 18:37:39,366 epoch 12 - iter 6605/13217 - loss 0.00069230 - samples/sec: 759.08\n", "2020-09-17 18:38:38,953 epoch 12 - iter 7926/13217 - loss 0.00061878 - samples/sec: 750.27\n", "2020-09-17 18:39:38,624 epoch 12 - iter 9247/13217 - loss 0.00062715 - samples/sec: 758.18\n", "2020-09-17 18:40:39,235 epoch 12 - iter 10568/13217 - loss 0.00064604 - samples/sec: 745.70\n", "2020-09-17 18:41:38,792 epoch 12 - iter 11889/13217 - loss 0.00064063 - samples/sec: 759.89\n", "2020-09-17 18:42:38,415 epoch 12 - iter 13210/13217 - loss 0.00062219 - samples/sec: 750.87\n", "2020-09-17 18:42:39,366 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 18:42:39,367 EPOCH 12 done: loss 0.0006 - lr 0.0250\n", "2020-09-17 18:44:19,259 DEV : loss 0.000488361984025687 - score 0.9998\n", "2020-09-17 18:44:40,711 BAD EPOCHS (no improvement): 2\n", "2020-09-17 18:44:45,089 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 18:45:18,156 epoch 13 - iter 0/13217 - loss 0.00000286 - samples/sec: 671053.08\n", "2020-09-17 18:46:18,358 epoch 13 - iter 1321/13217 - loss 0.00071845 - samples/sec: 749.22\n", "2020-09-17 18:47:17,996 epoch 13 - iter 2642/13217 - loss 0.00057776 - samples/sec: 758.33\n", "2020-09-17 18:48:17,699 epoch 13 - iter 3963/13217 - loss 0.00061696 - samples/sec: 759.01\n", "2020-09-17 18:49:17,929 epoch 13 - iter 5284/13217 - loss 0.00066540 - samples/sec: 761.57\n", "2020-09-17 18:50:17,534 epoch 13 - iter 6605/13217 - loss 0.00065236 - samples/sec: 759.77\n", "2020-09-17 18:51:17,181 epoch 13 - iter 7926/13217 - loss 0.00060414 - samples/sec: 742.39\n", "2020-09-17 18:52:17,335 epoch 13 - iter 9247/13217 - loss 0.00059723 - samples/sec: 752.87\n", "2020-09-17 18:53:16,725 epoch 13 - iter 10568/13217 - loss 0.00060264 - samples/sec: 753.94\n", "2020-09-17 18:54:16,183 epoch 13 - iter 11889/13217 - loss 0.00058259 - samples/sec: 761.86\n", "2020-09-17 18:55:15,960 epoch 13 - iter 13210/13217 - loss 0.00059086 - samples/sec: 749.53\n", "2020-09-17 18:55:17,000 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 18:55:17,001 EPOCH 13 done: loss 0.0006 - lr 0.0250\n", "2020-09-17 18:56:55,497 DEV : loss 0.0006003173184581101 - score 0.9998\n", "2020-09-17 18:57:17,955 BAD EPOCHS (no improvement): 3\n", "2020-09-17 18:57:22,804 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 18:57:55,360 epoch 14 - iter 0/13217 - loss 0.00000189 - samples/sec: 768660.85\n", "2020-09-17 18:58:54,831 epoch 14 - iter 1321/13217 - loss 0.00067411 - samples/sec: 754.52\n", "2020-09-17 18:59:54,195 epoch 14 - iter 2642/13217 - loss 0.00057056 - samples/sec: 756.04\n", "2020-09-17 19:00:53,966 epoch 14 - iter 3963/13217 - loss 0.00058276 - samples/sec: 759.73\n", "2020-09-17 19:01:53,819 epoch 14 - iter 5284/13217 - loss 0.00060312 - samples/sec: 741.09\n", "2020-09-17 19:02:53,828 epoch 14 - iter 6605/13217 - loss 0.00060565 - samples/sec: 757.60\n", "2020-09-17 19:03:54,273 epoch 14 - iter 7926/13217 - loss 0.00065312 - samples/sec: 758.64\n", "2020-09-17 19:04:53,758 epoch 14 - iter 9247/13217 - loss 0.00062427 - samples/sec: 762.74\n", "2020-09-17 19:05:53,618 epoch 14 - iter 10568/13217 - loss 0.00066188 - samples/sec: 749.75\n", "2020-09-17 19:06:53,232 epoch 14 - iter 11889/13217 - loss 0.00062903 - samples/sec: 757.20\n", "2020-09-17 19:07:53,345 epoch 14 - iter 13210/13217 - loss 0.00062997 - samples/sec: 758.68\n", "2020-09-17 19:07:54,298 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 19:07:54,598 EPOCH 14 done: loss 0.0006 - lr 0.0250\n", "2020-09-17 19:09:34,243 DEV : loss 0.0005741727072745562 - score 0.9998\n", "Epoch 13: reducing learning rate of group 0 to 1.2500e-02.\n", "2020-09-17 19:09:58,117 BAD EPOCHS (no improvement): 4\n", "2020-09-17 19:10:02,621 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 19:10:38,388 epoch 15 - iter 0/13217 - loss 0.00002065 - samples/sec: 548957.11\n", "2020-09-17 19:11:44,101 epoch 15 - iter 1321/13217 - loss 0.00049880 - samples/sec: 669.97\n", "2020-09-17 19:12:47,609 epoch 15 - iter 2642/13217 - loss 0.00051558 - samples/sec: 710.81\n", "2020-09-17 19:13:49,819 epoch 15 - iter 3963/13217 - loss 0.00050111 - samples/sec: 732.15\n", "2020-09-17 19:14:52,715 epoch 15 - iter 5284/13217 - loss 0.00050706 - samples/sec: 716.51\n", "2020-09-17 19:15:55,525 epoch 15 - iter 6605/13217 - loss 0.00047694 - samples/sec: 718.97\n", "2020-09-17 19:16:57,705 epoch 15 - iter 7926/13217 - loss 0.00047132 - samples/sec: 732.82\n", "2020-09-17 19:17:47,551 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 19:17:47,553 Exiting from training early.\n", "2020-09-17 19:17:47,554 Saving model ...\n", "2020-09-17 19:17:51,736 Done.\n", "2020-09-17 19:17:51,738 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 19:17:51,739 Testing using best model ...\n", "2020-09-17 19:17:51,743 loading file models\\glove.word_split_0.1_32\\best-model.pt\n", "2020-09-17 19:19:07,756 0.9998\t0.9998\t0.9998\n", "2020-09-17 19:19:07,757 \n", "MICRO_AVG: acc 0.9996 - f1-score 0.9998\n", "MACRO_AVG: acc 0.9844 - f1-score 0.9921\n", "benign tp: 52513 - fp: 2 - fn: 9 - tn: 344 - precision: 1.0000 - recall: 0.9998 - accuracy: 0.9998 - f1-score: 0.9999\n", "malignant tp: 344 - fp: 9 - fn: 2 - tn: 52513 - precision: 0.9745 - recall: 0.9942 - accuracy: 0.9690 - f1-score: 0.9843\n", "2020-09-17 19:19:07,758 ----------------------------------------------------------------------------------------------------\n" ] }, { "data": { "text/plain": [ "{'test_score': 0.9998,\n", " 'dev_score_history': [0.9995,\n", " 0.9998,\n", " 0.9997,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998],\n", " 'train_loss_history': [0.006704615072667729,\n", " 0.0018764245461755271,\n", " 0.001428390409741658,\n", " 0.001281433457158759,\n", " 0.001160885160813556,\n", " 0.0009288391432520336,\n", " 0.0008205886772528867,\n", " 0.0007256656202444912,\n", " 0.0006219205977933742,\n", " 0.0007342968501146098,\n", " 0.0005951481113686868,\n", " 0.0006219105104267317,\n", " 0.0005906871863142453,\n", " 0.0006297557100612004],\n", " 'dev_loss_history': [tensor(0.0020, device='cuda:0'),\n", " tensor(0.0008, device='cuda:0'),\n", " tensor(0.0010, device='cuda:0'),\n", " tensor(0.0009, device='cuda:0'),\n", " tensor(0.0006, device='cuda:0'),\n", " tensor(0.0005, device='cuda:0'),\n", " tensor(0.0005, device='cuda:0'),\n", " tensor(0.0005, device='cuda:0'),\n", " tensor(0.0005, device='cuda:0'),\n", " tensor(0.0005, device='cuda:0'),\n", " tensor(0.0004, device='cuda:0'),\n", " tensor(0.0005, device='cuda:0'),\n", " tensor(0.0006, device='cuda:0'),\n", " tensor(0.0006, device='cuda:0')]}" ] }, "execution_count": 107, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_name = 'glove'\n", "corpus = corpus_word_split\n", "lr = 0.1\n", "batch_size = 32\n", "model_var = f'word_split_{lr}_{batch_size}'\n", "\n", "import os\n", "\n", "model_dir = f'models/{model_name}.{model_var}'\n", "try:\n", " os.mkdir(model_dir)\n", "except OSError:\n", " print (f\"Creation of the directory '{model_dir}' failed.\")\n", "else:\n", " print (f\"Created the directory '{model_dir}'\")\n", "\n", "label_dict = corpus.make_label_dictionary()\n", "word_embeddings = [WordEmbeddings(model_name)]\n", "document_embeddings = DocumentRNNEmbeddings(word_embeddings, hidden_size=256)\n", "classifier = TextClassifier(document_embeddings, label_dictionary=label_dict)\n", "trainer = ModelTrainer(classifier, corpus)\n", "\n", "trainer.train(model_dir,\n", " learning_rate=lr,\n", " mini_batch_size=batch_size, # 32,\n", " anneal_factor=0.5,\n", " patience=3,\n", " max_epochs=30)\n", "\n", "# models\\glove.word_split_0.5_256\\best-model.pt\n", "# MICRO_AVG: acc 0.9947 - f1-score 0.9973\n", "# MACRO_AVG: acc 0.7949 - f1-score 0.8713500000000001\n", "# benign tp: 52522 - fp: 141 - fn: 0 - tn: 205 - precision: 0.9973 - recall: 1.0000 - accuracy: 0.9973 - f1-score: 0.9986\n", "# malignant tp: 205 - fp: 0 - fn: 141 - tn: 52522 - precision: 1.0000 - recall: 0.5925 - accuracy: 0.5925 - f1-score: 0.7441\n", "\n", "\n", "# ******************** The BEST ***************************************************************\n", "# ... training not finished\n", "# 2020-09-17 19:17:51,743 loading file models\\glove.word_split_0.1_32\\best-model.pt\n", "# 2020-09-17 19:19:07,756 0.9998\t0.9998\t0.9998\n", "# MICRO_AVG: acc 0.9996 - f1-score 0.9998\n", "# MACRO_AVG: acc 0.9844 - f1-score 0.9921\n", "# benign tp: 52513 - fp: 2 - fn: 9 - tn: 344 - precision: 1.0000 - recall: 0.9998 - accuracy: 0.9998 - f1-score: 0.9999\n", "# malignant tp: 344 - fp: 9 - fn: 2 - tn: 52513 - precision: 0.9745 - recall: 0.9942 - accuracy: 0.9690 - f1-score: 0.9843\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### greedy conclusion\n", "The GloVe experiments show that splitting DNs to words works much better. So, we will use only data sets with word-splitting in the next experiments." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### en (FastText) / word_split\n", "[CLASSIC_WORD_EMBEDDINGS](https://github.com/flairNLP/flair/blob/master/resources/docs/embeddings/CLASSIC_WORD_EMBEDDINGS.md)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2020-09-18T03:56:35.524460Z", "start_time": "2020-09-18T03:48:50.520107Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 20:48:51,533 https://flair.informatik.hu-berlin.de/resources/embeddings/token/en-fasttext-news-300d-1M.vectors.npy not found in cache, downloading to D:\\Temp\\tmp8nn7o21t\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████████████████████████████████████████████████████████| 1200000128/1200000128 [07:17<00:00, 2743514.45B/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 20:56:09,623 copying D:\\Temp\\tmp8nn7o21t to cache at C:\\Users\\leo_g\\.flair\\embeddings\\en-fasttext-news-300d-1M.vectors.npy\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 20:56:11,760 removing temp file D:\\Temp\\tmp8nn7o21t\n", "2020-09-17 20:56:12,478 https://flair.informatik.hu-berlin.de/resources/embeddings/token/en-fasttext-news-300d-1M not found in cache, downloading to D:\\Temp\\tmpbf28rx8w\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████████████████████████████████████████████████████████████| 54600983/54600983 [00:19<00:00, 2804971.30B/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 20:56:32,619 copying D:\\Temp\\tmpbf28rx8w to cache at C:\\Users\\leo_g\\.flair\\embeddings\\en-fasttext-news-300d-1M\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 20:56:32,711 removing temp file D:\\Temp\\tmpbf28rx8w\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "I0917 20:56:32.719858 11228 utils.py:422] loading Word2VecKeyedVectors object from C:\\Users\\leo_g\\.flair\\embeddings\\en-fasttext-news-300d-1M\n", "I0917 20:56:35.019429 11228 utils.py:461] loading vectors from C:\\Users\\leo_g\\.flair\\embeddings\\en-fasttext-news-300d-1M.vectors.npy with mmap=None\n", "I0917 20:56:35.518429 11228 utils.py:494] setting ignored attribute vectors_norm to None\n", "I0917 20:56:35.519431 11228 utils.py:428] loaded C:\\Users\\leo_g\\.flair\\embeddings\\en-fasttext-news-300d-1M\n" ] } ], "source": [ "# emb = WordEmbeddings('en')" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2020-09-18T09:27:52.413550Z", "start_time": "2020-09-18T04:24:03.841482Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Creation of the directory 'models/en.word_split_0.1_32' failed.\n", "2020-09-17 21:24:03,844 Computing label dictionary. Progress:\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "100%|████████████████████████████████████████████████████████████████████████| 475804/475804 [01:54<00:00, 4172.25it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 21:27:03,865 [b'benign', b'malignant']\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n", "I0917 21:27:03.870359 7756 utils.py:422] loading Word2VecKeyedVectors object from C:\\Users\\leo_g\\.flair\\embeddings\\en-fasttext-news-300d-1M\n", "I0917 21:27:06.158385 7756 utils.py:461] loading vectors from C:\\Users\\leo_g\\.flair\\embeddings\\en-fasttext-news-300d-1M.vectors.npy with mmap=None\n", "I0917 21:27:08.727932 7756 utils.py:494] setting ignored attribute vectors_norm to None\n", "I0917 21:27:08.728893 7756 utils.py:428] loaded C:\\Users\\leo_g\\.flair\\embeddings\\en-fasttext-news-300d-1M\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2020-09-17 21:27:13,074 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 21:27:13,076 Model: \"TextClassifier(\n", " (document_embeddings): DocumentRNNEmbeddings(\n", " (embeddings): StackedEmbeddings(\n", " (list_embedding_0): WordEmbeddings('en')\n", " )\n", " (word_reprojection_map): Linear(in_features=300, out_features=300, bias=True)\n", " (rnn): GRU(300, 256, batch_first=True)\n", " (dropout): Dropout(p=0.5, inplace=False)\n", " )\n", " (decoder): Linear(in_features=256, out_features=2, bias=True)\n", " (loss_function): CrossEntropyLoss()\n", " (beta): 1.0\n", " (weights): None\n", " (weight_tensor) None\n", ")\"\n", "2020-09-17 21:27:13,076 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 21:27:13,078 Corpus: \"Corpus: 422936 train + 52867 dev + 52868 test sentences\"\n", "2020-09-17 21:27:13,079 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 21:27:13,081 Parameters:\n", "2020-09-17 21:27:13,081 - learning_rate: \"0.1\"\n", "2020-09-17 21:27:13,082 - mini_batch_size: \"32\"\n", "2020-09-17 21:27:13,084 - patience: \"3\"\n", "2020-09-17 21:27:13,085 - anneal_factor: \"0.5\"\n", "2020-09-17 21:27:13,086 - max_epochs: \"30\"\n", "2020-09-17 21:27:13,087 - shuffle: \"True\"\n", "2020-09-17 21:27:13,087 - train_with_dev: \"False\"\n", "2020-09-17 21:27:13,089 - batch_growth_annealing: \"False\"\n", "2020-09-17 21:27:13,090 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 21:27:13,090 Model training base path: \"models\\en.word_split_0.1_32\"\n", "2020-09-17 21:27:13,093 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 21:27:13,094 Device: cuda:0\n", "2020-09-17 21:27:13,095 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 21:27:13,096 Embeddings storage mode: cpu\n", "2020-09-17 21:27:13,098 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 21:28:44,065 epoch 1 - iter 1321/13217 - loss 0.02749483 - samples/sec: 983.40 - lr: 0.100000\n", "2020-09-17 21:29:28,420 epoch 1 - iter 2642/13217 - loss 0.02243038 - samples/sec: 1027.57 - lr: 0.100000\n", "2020-09-17 21:30:13,668 epoch 1 - iter 3963/13217 - loss 0.01833959 - samples/sec: 1040.18 - lr: 0.100000\n", "2020-09-17 21:30:58,126 epoch 1 - iter 5284/13217 - loss 0.01478741 - samples/sec: 1040.63 - lr: 0.100000\n", "2020-09-17 21:31:43,170 epoch 1 - iter 6605/13217 - loss 0.01231753 - samples/sec: 1054.21 - lr: 0.100000\n", "2020-09-17 21:32:26,684 epoch 1 - iter 7926/13217 - loss 0.01053886 - samples/sec: 1067.22 - lr: 0.100000\n", "2020-09-17 21:33:13,083 epoch 1 - iter 9247/13217 - loss 0.00927905 - samples/sec: 1024.75 - lr: 0.100000\n", "2020-09-17 21:33:58,596 epoch 1 - iter 10568/13217 - loss 0.00834516 - samples/sec: 1022.44 - lr: 0.100000\n", "2020-09-17 21:34:42,560 epoch 1 - iter 11889/13217 - loss 0.00759126 - samples/sec: 1030.51 - lr: 0.100000\n", "2020-09-17 21:35:27,889 epoch 1 - iter 13210/13217 - loss 0.00700785 - samples/sec: 1045.66 - lr: 0.100000\n", "2020-09-17 21:35:29,144 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 21:35:29,146 EPOCH 1 done: loss 0.0070 - lr 0.1000000\n", "2020-09-17 21:37:20,745 DEV : loss 0.0009424208546988666 - score 0.9997\n", "2020-09-17 21:37:31,036 BAD EPOCHS (no improvement): 0\n", "saving best model\n", "2020-09-17 21:38:59,523 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 21:41:46,896 epoch 2 - iter 1321/13217 - loss 0.00199193 - samples/sec: 398.37 - lr: 0.100000\n", "2020-09-17 21:42:38,658 epoch 2 - iter 2642/13217 - loss 0.00193292 - samples/sec: 890.71 - lr: 0.100000\n", "2020-09-17 21:43:29,330 epoch 2 - iter 3963/13217 - loss 0.00173339 - samples/sec: 914.45 - lr: 0.100000\n", "2020-09-17 21:44:15,314 epoch 2 - iter 5284/13217 - loss 0.00165898 - samples/sec: 1010.92 - lr: 0.100000\n", "2020-09-17 21:45:00,741 epoch 2 - iter 6605/13217 - loss 0.00155553 - samples/sec: 1020.19 - lr: 0.100000\n", "2020-09-17 21:45:44,788 epoch 2 - iter 7926/13217 - loss 0.00151862 - samples/sec: 1057.58 - lr: 0.100000\n", "2020-09-17 21:46:29,563 epoch 2 - iter 9247/13217 - loss 0.00146917 - samples/sec: 1066.70 - lr: 0.100000\n", "2020-09-17 21:47:11,926 epoch 2 - iter 10568/13217 - loss 0.00146954 - samples/sec: 1102.69 - lr: 0.100000\n", "2020-09-17 21:47:54,454 epoch 2 - iter 11889/13217 - loss 0.00144656 - samples/sec: 1095.06 - lr: 0.100000\n", "2020-09-17 21:48:36,464 epoch 2 - iter 13210/13217 - loss 0.00138725 - samples/sec: 1113.49 - lr: 0.100000\n", "2020-09-17 21:48:37,624 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 21:48:37,625 EPOCH 2 done: loss 0.0014 - lr 0.1000000\n", "2020-09-17 21:50:19,035 DEV : loss 0.0007725355098955333 - score 0.9998\n", "2020-09-17 21:50:29,000 BAD EPOCHS (no improvement): 0\n", "saving best model\n", "2020-09-17 21:51:28,569 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 21:53:10,221 epoch 3 - iter 1321/13217 - loss 0.00096874 - samples/sec: 861.80 - lr: 0.100000\n", "2020-09-17 21:53:53,698 epoch 3 - iter 2642/13217 - loss 0.00087977 - samples/sec: 1072.30 - lr: 0.100000\n", "2020-09-17 21:54:36,320 epoch 3 - iter 3963/13217 - loss 0.00096622 - samples/sec: 1095.03 - lr: 0.100000\n", "2020-09-17 21:55:18,915 epoch 3 - iter 5284/13217 - loss 0.00087260 - samples/sec: 1094.18 - lr: 0.100000\n", "2020-09-17 21:56:02,657 epoch 3 - iter 6605/13217 - loss 0.00091062 - samples/sec: 1063.38 - lr: 0.100000\n", "2020-09-17 21:56:45,057 epoch 3 - iter 7926/13217 - loss 0.00108953 - samples/sec: 1101.72 - lr: 0.100000\n", "2020-09-17 21:57:27,374 epoch 3 - iter 9247/13217 - loss 0.00107605 - samples/sec: 1105.00 - lr: 0.100000\n", "2020-09-17 21:58:10,026 epoch 3 - iter 10568/13217 - loss 0.00104900 - samples/sec: 1095.87 - lr: 0.100000\n", "2020-09-17 21:58:52,383 epoch 3 - iter 11889/13217 - loss 0.00107801 - samples/sec: 1100.63 - lr: 0.100000\n", "2020-09-17 21:59:34,676 epoch 3 - iter 13210/13217 - loss 0.00104315 - samples/sec: 1102.81 - lr: 0.100000\n", "2020-09-17 21:59:35,909 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 21:59:35,910 EPOCH 3 done: loss 0.0010 - lr 0.1000000\n", "2020-09-17 22:01:19,820 DEV : loss 0.000515893567353487 - score 0.9998\n", "2020-09-17 22:01:29,649 BAD EPOCHS (no improvement): 0\n", "saving best model\n", "2020-09-17 22:01:50,409 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 22:03:14,045 epoch 4 - iter 1321/13217 - loss 0.00080507 - samples/sec: 1114.63 - lr: 0.100000\n", "2020-09-17 22:03:57,032 epoch 4 - iter 2642/13217 - loss 0.00109913 - samples/sec: 1084.44 - lr: 0.100000\n", "2020-09-17 22:04:39,431 epoch 4 - iter 3963/13217 - loss 0.00102957 - samples/sec: 1101.28 - lr: 0.100000\n", "2020-09-17 22:05:22,614 epoch 4 - iter 5284/13217 - loss 0.00099505 - samples/sec: 1080.86 - lr: 0.100000\n", "2020-09-17 22:06:05,321 epoch 4 - iter 6605/13217 - loss 0.00106298 - samples/sec: 1092.58 - lr: 0.100000\n", "2020-09-17 22:06:49,106 epoch 4 - iter 7926/13217 - loss 0.00103351 - samples/sec: 1088.37 - lr: 0.100000\n", "2020-09-17 22:07:32,399 epoch 4 - iter 9247/13217 - loss 0.00100500 - samples/sec: 1078.15 - lr: 0.100000\n", "2020-09-17 22:08:15,130 epoch 4 - iter 10568/13217 - loss 0.00104720 - samples/sec: 1094.07 - lr: 0.100000\n", "2020-09-17 22:08:57,271 epoch 4 - iter 11889/13217 - loss 0.00100514 - samples/sec: 1108.48 - lr: 0.100000\n", "2020-09-17 22:09:39,276 epoch 4 - iter 13210/13217 - loss 0.00101502 - samples/sec: 1084.70 - lr: 0.100000\n", "2020-09-17 22:09:40,476 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 22:09:40,496 EPOCH 4 done: loss 0.0010 - lr 0.1000000\n", "2020-09-17 22:11:22,182 DEV : loss 0.0013346497435122728 - score 0.9997\n", "2020-09-17 22:11:31,547 BAD EPOCHS (no improvement): 1\n", "2020-09-17 22:11:31,549 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 22:12:54,828 epoch 5 - iter 1321/13217 - loss 0.00062756 - samples/sec: 1103.59 - lr: 0.100000\n", "2020-09-17 22:13:38,130 epoch 5 - iter 2642/13217 - loss 0.00075971 - samples/sec: 1074.83 - lr: 0.100000\n", "2020-09-17 22:14:20,329 epoch 5 - iter 3963/13217 - loss 0.00091977 - samples/sec: 1103.07 - lr: 0.100000\n", "2020-09-17 22:15:03,194 epoch 5 - iter 5284/13217 - loss 0.00095851 - samples/sec: 1085.33 - lr: 0.100000\n", "2020-09-17 22:15:46,222 epoch 5 - iter 6605/13217 - loss 0.00097530 - samples/sec: 1086.41 - lr: 0.100000\n", "2020-09-17 22:16:28,123 epoch 5 - iter 7926/13217 - loss 0.00095412 - samples/sec: 1110.44 - lr: 0.100000\n", "2020-09-17 22:17:10,307 epoch 5 - iter 9247/13217 - loss 0.00099935 - samples/sec: 1104.07 - lr: 0.100000\n", "2020-09-17 22:17:52,639 epoch 5 - iter 10568/13217 - loss 0.00103127 - samples/sec: 1099.68 - lr: 0.100000\n", "2020-09-17 22:18:34,395 epoch 5 - iter 11889/13217 - loss 0.00100628 - samples/sec: 1114.95 - lr: 0.100000\n", "2020-09-17 22:19:16,582 epoch 5 - iter 13210/13217 - loss 0.00099259 - samples/sec: 1104.27 - lr: 0.100000\n", "2020-09-17 22:19:18,089 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 22:19:18,090 EPOCH 5 done: loss 0.0010 - lr 0.1000000\n", "2020-09-17 22:21:03,462 DEV : loss 0.0005379541544243693 - score 0.9998\n", "2020-09-17 22:21:13,394 BAD EPOCHS (no improvement): 2\n", "2020-09-17 22:21:13,395 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 22:22:39,931 epoch 6 - iter 1321/13217 - loss 0.00175980 - samples/sec: 1070.65 - lr: 0.100000\n", "2020-09-17 22:23:24,830 epoch 6 - iter 2642/13217 - loss 0.00128146 - samples/sec: 1046.75 - lr: 0.100000\n", "2020-09-17 22:24:09,527 epoch 6 - iter 3963/13217 - loss 0.00104917 - samples/sec: 1053.57 - lr: 0.100000\n", "2020-09-17 22:24:52,115 epoch 6 - iter 5284/13217 - loss 0.00102266 - samples/sec: 1081.82 - lr: 0.100000\n", "2020-09-17 22:25:35,776 epoch 6 - iter 6605/13217 - loss 0.00103956 - samples/sec: 1057.63 - lr: 0.100000\n", "2020-09-17 22:26:21,057 epoch 6 - iter 7926/13217 - loss 0.00099516 - samples/sec: 1042.80 - lr: 0.100000\n", "2020-09-17 22:27:04,223 epoch 6 - iter 9247/13217 - loss 0.00106780 - samples/sec: 1042.44 - lr: 0.100000\n", "2020-09-17 22:27:50,444 epoch 6 - iter 10568/13217 - loss 0.00105846 - samples/sec: 994.88 - lr: 0.100000\n", "2020-09-17 22:28:33,582 epoch 6 - iter 11889/13217 - loss 0.00111116 - samples/sec: 1069.00 - lr: 0.100000\n", "2020-09-17 22:29:16,301 epoch 6 - iter 13210/13217 - loss 0.00105933 - samples/sec: 1079.79 - lr: 0.100000\n", "2020-09-17 22:29:17,606 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 22:29:17,608 EPOCH 6 done: loss 0.0011 - lr 0.1000000\n", "2020-09-17 22:31:09,847 DEV : loss 0.000892516050953418 - score 0.9998\n", "2020-09-17 22:31:20,384 BAD EPOCHS (no improvement): 3\n", "2020-09-17 22:31:20,385 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 22:32:47,053 epoch 7 - iter 1321/13217 - loss 0.00114820 - samples/sec: 1036.53 - lr: 0.100000\n", "2020-09-17 22:33:33,565 epoch 7 - iter 2642/13217 - loss 0.00131240 - samples/sec: 1004.34 - lr: 0.100000\n", "2020-09-17 22:34:18,649 epoch 7 - iter 3963/13217 - loss 0.00109407 - samples/sec: 1016.19 - lr: 0.100000\n", "2020-09-17 22:35:02,849 epoch 7 - iter 5284/13217 - loss 0.00097479 - samples/sec: 1039.22 - lr: 0.100000\n", "2020-09-17 22:35:48,071 epoch 7 - iter 6605/13217 - loss 0.00100282 - samples/sec: 1013.77 - lr: 0.100000\n", "2020-09-17 22:36:32,863 epoch 7 - iter 7926/13217 - loss 0.00098928 - samples/sec: 1048.48 - lr: 0.100000\n", "2020-09-17 22:37:16,852 epoch 7 - iter 9247/13217 - loss 0.00096584 - samples/sec: 1019.59 - lr: 0.100000\n", "2020-09-17 22:38:01,399 epoch 7 - iter 10568/13217 - loss 0.00088957 - samples/sec: 1057.19 - lr: 0.100000\n", "2020-09-17 22:38:44,205 epoch 7 - iter 11889/13217 - loss 0.00087897 - samples/sec: 1097.60 - lr: 0.100000\n", "2020-09-17 22:39:31,541 epoch 7 - iter 13210/13217 - loss 0.00086255 - samples/sec: 969.12 - lr: 0.100000\n", "2020-09-17 22:39:32,941 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 22:39:32,943 EPOCH 7 done: loss 0.0009 - lr 0.1000000\n", "2020-09-17 22:41:48,341 DEV : loss 0.0008064075373113155 - score 0.9998\n", "Epoch 7: reducing learning rate of group 0 to 5.0000e-02.\n", "2020-09-17 22:41:59,136 BAD EPOCHS (no improvement): 4\n", "2020-09-17 22:41:59,137 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 22:43:30,728 epoch 8 - iter 1321/13217 - loss 0.00062236 - samples/sec: 1011.96 - lr: 0.050000\n", "2020-09-17 22:44:16,122 epoch 8 - iter 2642/13217 - loss 0.00059675 - samples/sec: 1005.54 - lr: 0.050000\n", "2020-09-17 22:45:02,065 epoch 8 - iter 3963/13217 - loss 0.00060280 - samples/sec: 1002.83 - lr: 0.050000\n", "2020-09-17 22:45:48,922 epoch 8 - iter 5284/13217 - loss 0.00073052 - samples/sec: 986.79 - lr: 0.050000\n", "2020-09-17 22:46:36,123 epoch 8 - iter 6605/13217 - loss 0.00084951 - samples/sec: 1000.21 - lr: 0.050000\n", "2020-09-17 22:47:26,739 epoch 8 - iter 7926/13217 - loss 0.00082870 - samples/sec: 914.78 - lr: 0.050000\n", "2020-09-17 22:48:12,400 epoch 8 - iter 9247/13217 - loss 0.00080816 - samples/sec: 1011.96 - lr: 0.050000\n", "2020-09-17 22:49:01,164 epoch 8 - iter 10568/13217 - loss 0.00082938 - samples/sec: 945.46 - lr: 0.050000\n", "2020-09-17 22:49:48,386 epoch 8 - iter 11889/13217 - loss 0.00084065 - samples/sec: 978.20 - lr: 0.050000\n", "2020-09-17 22:50:34,986 epoch 8 - iter 13210/13217 - loss 0.00080793 - samples/sec: 991.85 - lr: 0.050000\n", "2020-09-17 22:50:36,130 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 22:50:36,132 EPOCH 8 done: loss 0.0008 - lr 0.0500000\n", "2020-09-17 22:52:27,110 DEV : loss 0.0005345970275811851 - score 0.9998\n", "2020-09-17 22:52:36,952 BAD EPOCHS (no improvement): 1\n", "2020-09-17 22:52:36,953 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 22:54:05,502 epoch 9 - iter 1321/13217 - loss 0.00071195 - samples/sec: 1017.12 - lr: 0.050000\n", "2020-09-17 22:54:50,926 epoch 9 - iter 2642/13217 - loss 0.00064299 - samples/sec: 1041.63 - lr: 0.050000\n", "2020-09-17 22:55:33,705 epoch 9 - iter 3963/13217 - loss 0.00069158 - samples/sec: 1084.08 - lr: 0.050000\n", "2020-09-17 22:56:18,589 epoch 9 - iter 5284/13217 - loss 0.00070995 - samples/sec: 1031.23 - lr: 0.050000\n", "2020-09-17 22:57:01,354 epoch 9 - iter 6605/13217 - loss 0.00068931 - samples/sec: 1082.05 - lr: 0.050000\n", "2020-09-17 22:57:48,414 epoch 9 - iter 7926/13217 - loss 0.00071717 - samples/sec: 1004.75 - lr: 0.050000\n", "2020-09-17 22:58:35,578 epoch 9 - iter 9247/13217 - loss 0.00068423 - samples/sec: 984.61 - lr: 0.050000\n", "2020-09-17 22:59:18,940 epoch 9 - iter 10568/13217 - loss 0.00068221 - samples/sec: 1066.66 - lr: 0.050000\n", "2020-09-17 23:00:04,861 epoch 9 - iter 11889/13217 - loss 0.00071908 - samples/sec: 1005.66 - lr: 0.050000\n", "2020-09-17 23:00:50,641 epoch 9 - iter 13210/13217 - loss 0.00073145 - samples/sec: 1010.13 - lr: 0.050000\n", "2020-09-17 23:00:51,780 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:00:51,781 EPOCH 9 done: loss 0.0007 - lr 0.0500000\n", "2020-09-17 23:02:37,250 DEV : loss 0.0005398467765189707 - score 0.9998\n", "2020-09-17 23:02:47,407 BAD EPOCHS (no improvement): 2\n", "2020-09-17 23:02:47,409 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:04:15,315 epoch 10 - iter 1321/13217 - loss 0.00062731 - samples/sec: 1010.56 - lr: 0.050000\n", "2020-09-17 23:04:59,841 epoch 10 - iter 2642/13217 - loss 0.00054062 - samples/sec: 1029.96 - lr: 0.050000\n", "2020-09-17 23:05:43,718 epoch 10 - iter 3963/13217 - loss 0.00056302 - samples/sec: 1049.17 - lr: 0.050000\n", "2020-09-17 23:06:29,156 epoch 10 - iter 5284/13217 - loss 0.00053000 - samples/sec: 1034.33 - lr: 0.050000\n", "2020-09-17 23:07:14,013 epoch 10 - iter 6605/13217 - loss 0.00051255 - samples/sec: 1023.16 - lr: 0.050000\n", "2020-09-17 23:07:57,890 epoch 10 - iter 7926/13217 - loss 0.00057478 - samples/sec: 1050.42 - lr: 0.050000\n", "2020-09-17 23:08:42,381 epoch 10 - iter 9247/13217 - loss 0.00058789 - samples/sec: 1060.00 - lr: 0.050000\n", "2020-09-17 23:09:29,604 epoch 10 - iter 10568/13217 - loss 0.00068931 - samples/sec: 996.91 - lr: 0.050000\n", "2020-09-17 23:10:13,025 epoch 10 - iter 11889/13217 - loss 0.00069817 - samples/sec: 1060.66 - lr: 0.050000\n", "2020-09-17 23:10:54,865 epoch 10 - iter 13210/13217 - loss 0.00070031 - samples/sec: 1107.11 - lr: 0.050000\n", "2020-09-17 23:10:56,346 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:10:56,348 EPOCH 10 done: loss 0.0007 - lr 0.0500000\n", "2020-09-17 23:12:37,809 DEV : loss 0.0004569814773276448 - score 0.9998\n", "2020-09-17 23:12:48,056 BAD EPOCHS (no improvement): 0\n", "saving best model\n", "2020-09-17 23:13:36,712 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:15:19,223 epoch 11 - iter 1321/13217 - loss 0.00104728 - samples/sec: 808.25 - lr: 0.050000\n", "2020-09-17 23:16:02,256 epoch 11 - iter 2642/13217 - loss 0.00090430 - samples/sec: 1052.94 - lr: 0.050000\n", "2020-09-17 23:16:44,175 epoch 11 - iter 3963/13217 - loss 0.00088196 - samples/sec: 1109.74 - lr: 0.050000\n", "2020-09-17 23:17:26,454 epoch 11 - iter 5284/13217 - loss 0.00077961 - samples/sec: 1101.63 - lr: 0.050000\n", "2020-09-17 23:18:08,392 epoch 11 - iter 6605/13217 - loss 0.00077794 - samples/sec: 1107.68 - lr: 0.050000\n", "2020-09-17 23:18:50,254 epoch 11 - iter 7926/13217 - loss 0.00077593 - samples/sec: 1109.59 - lr: 0.050000\n", "2020-09-17 23:19:31,914 epoch 11 - iter 9247/13217 - loss 0.00077653 - samples/sec: 1116.32 - lr: 0.050000\n", "2020-09-17 23:20:13,673 epoch 11 - iter 10568/13217 - loss 0.00071789 - samples/sec: 1117.21 - lr: 0.050000\n", "2020-09-17 23:20:55,315 epoch 11 - iter 11889/13217 - loss 0.00071327 - samples/sec: 1118.07 - lr: 0.050000\n", "2020-09-17 23:21:37,415 epoch 11 - iter 13210/13217 - loss 0.00071910 - samples/sec: 1104.69 - lr: 0.050000\n", "2020-09-17 23:21:38,670 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:21:38,671 EPOCH 11 done: loss 0.0007 - lr 0.0500000\n", "2020-09-17 23:23:22,360 DEV : loss 0.0005163941532373428 - score 0.9998\n", "2020-09-17 23:23:31,783 BAD EPOCHS (no improvement): 1\n", "2020-09-17 23:23:31,857 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:24:54,457 epoch 12 - iter 1321/13217 - loss 0.00058690 - samples/sec: 1104.99 - lr: 0.050000\n", "2020-09-17 23:25:37,450 epoch 12 - iter 2642/13217 - loss 0.00073232 - samples/sec: 1066.05 - lr: 0.050000\n", "2020-09-17 23:26:18,413 epoch 12 - iter 3963/13217 - loss 0.00078745 - samples/sec: 1124.25 - lr: 0.050000\n", "2020-09-17 23:27:00,223 epoch 12 - iter 5284/13217 - loss 0.00082269 - samples/sec: 1098.02 - lr: 0.050000\n", "2020-09-17 23:27:41,221 epoch 12 - iter 6605/13217 - loss 0.00080245 - samples/sec: 1096.04 - lr: 0.050000\n", "2020-09-17 23:28:22,265 epoch 12 - iter 7926/13217 - loss 0.00083855 - samples/sec: 1121.78 - lr: 0.050000\n", "2020-09-17 23:29:04,358 epoch 12 - iter 9247/13217 - loss 0.00081311 - samples/sec: 1119.61 - lr: 0.050000\n", "2020-09-17 23:29:45,460 epoch 12 - iter 10568/13217 - loss 0.00078404 - samples/sec: 1122.95 - lr: 0.050000\n", "2020-09-17 23:30:27,504 epoch 12 - iter 11889/13217 - loss 0.00075962 - samples/sec: 1119.37 - lr: 0.050000\n", "2020-09-17 23:31:08,363 epoch 12 - iter 13210/13217 - loss 0.00074903 - samples/sec: 1126.96 - lr: 0.050000\n", "2020-09-17 23:31:09,495 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:31:09,496 EPOCH 12 done: loss 0.0007 - lr 0.0500000\n", "2020-09-17 23:32:48,851 DEV : loss 0.000496959313750267 - score 0.9998\n", "2020-09-17 23:32:58,229 BAD EPOCHS (no improvement): 2\n", "2020-09-17 23:32:58,231 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:34:18,540 epoch 13 - iter 1321/13217 - loss 0.00092109 - samples/sec: 1116.12 - lr: 0.050000\n", "2020-09-17 23:34:59,808 epoch 13 - iter 2642/13217 - loss 0.00074178 - samples/sec: 1120.27 - lr: 0.050000\n", "2020-09-17 23:35:41,316 epoch 13 - iter 3963/13217 - loss 0.00079842 - samples/sec: 1110.44 - lr: 0.050000\n", "2020-09-17 23:36:22,568 epoch 13 - iter 5284/13217 - loss 0.00080866 - samples/sec: 1121.36 - lr: 0.050000\n", "2020-09-17 23:37:04,615 epoch 13 - iter 6605/13217 - loss 0.00082617 - samples/sec: 1123.77 - lr: 0.050000\n", "2020-09-17 23:37:45,712 epoch 13 - iter 7926/13217 - loss 0.00082122 - samples/sec: 1125.93 - lr: 0.050000\n", "2020-09-17 23:38:26,976 epoch 13 - iter 9247/13217 - loss 0.00087026 - samples/sec: 1125.61 - lr: 0.050000\n", "2020-09-17 23:39:08,156 epoch 13 - iter 10568/13217 - loss 0.00081592 - samples/sec: 1123.34 - lr: 0.050000\n", "2020-09-17 23:39:49,401 epoch 13 - iter 11889/13217 - loss 0.00079744 - samples/sec: 1126.80 - lr: 0.050000\n", "2020-09-17 23:40:30,874 epoch 13 - iter 13210/13217 - loss 0.00078226 - samples/sec: 1116.87 - lr: 0.050000\n", "2020-09-17 23:40:32,096 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:40:32,097 EPOCH 13 done: loss 0.0008 - lr 0.0500000\n", "2020-09-17 23:42:11,240 DEV : loss 0.0004584364651236683 - score 0.9998\n", "2020-09-17 23:42:20,532 BAD EPOCHS (no improvement): 3\n", "2020-09-17 23:42:20,534 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:43:40,501 epoch 14 - iter 1321/13217 - loss 0.00053675 - samples/sec: 1126.42 - lr: 0.050000\n", "2020-09-17 23:44:21,457 epoch 14 - iter 2642/13217 - loss 0.00060776 - samples/sec: 1125.60 - lr: 0.050000\n", "2020-09-17 23:45:03,966 epoch 14 - iter 3963/13217 - loss 0.00061385 - samples/sec: 1111.71 - lr: 0.050000\n", "2020-09-17 23:45:45,387 epoch 14 - iter 5284/13217 - loss 0.00057113 - samples/sec: 1088.11 - lr: 0.050000\n", "2020-09-17 23:46:26,819 epoch 14 - iter 6605/13217 - loss 0.00069374 - samples/sec: 1113.34 - lr: 0.050000\n", "2020-09-17 23:47:08,505 epoch 14 - iter 7926/13217 - loss 0.00071601 - samples/sec: 1133.57 - lr: 0.050000\n", "2020-09-17 23:47:49,406 epoch 14 - iter 9247/13217 - loss 0.00068669 - samples/sec: 1129.93 - lr: 0.050000\n", "2020-09-17 23:48:30,567 epoch 14 - iter 10568/13217 - loss 0.00066670 - samples/sec: 1121.66 - lr: 0.050000\n", "2020-09-17 23:49:12,498 epoch 14 - iter 11889/13217 - loss 0.00071801 - samples/sec: 1123.23 - lr: 0.050000\n", "2020-09-17 23:49:53,662 epoch 14 - iter 13210/13217 - loss 0.00069615 - samples/sec: 1093.79 - lr: 0.050000\n", "2020-09-17 23:49:55,215 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:49:55,217 EPOCH 14 done: loss 0.0007 - lr 0.0500000\n", "2020-09-17 23:51:33,860 DEV : loss 0.00045535029494203627 - score 0.9998\n", "2020-09-17 23:51:43,216 BAD EPOCHS (no improvement): 0\n", "saving best model\n", "2020-09-17 23:52:13,877 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:53:39,381 epoch 15 - iter 1321/13217 - loss 0.00081524 - samples/sec: 1125.90 - lr: 0.050000\n", "2020-09-17 23:54:20,857 epoch 15 - iter 2642/13217 - loss 0.00081375 - samples/sec: 1134.39 - lr: 0.050000\n", "2020-09-17 23:55:01,944 epoch 15 - iter 3963/13217 - loss 0.00074301 - samples/sec: 1123.38 - lr: 0.050000\n", "2020-09-17 23:55:42,761 epoch 15 - iter 5284/13217 - loss 0.00070735 - samples/sec: 1129.88 - lr: 0.050000\n", "2020-09-17 23:56:24,484 epoch 15 - iter 6605/13217 - loss 0.00071017 - samples/sec: 1131.26 - lr: 0.050000\n", "2020-09-17 23:57:05,452 epoch 15 - iter 7926/13217 - loss 0.00067085 - samples/sec: 1125.74 - lr: 0.050000\n", "2020-09-17 23:57:47,030 epoch 15 - iter 9247/13217 - loss 0.00070789 - samples/sec: 1131.30 - lr: 0.050000\n", "2020-09-17 23:58:27,782 epoch 15 - iter 10568/13217 - loss 0.00070416 - samples/sec: 1131.28 - lr: 0.050000\n", "2020-09-17 23:59:08,457 epoch 15 - iter 11889/13217 - loss 0.00068933 - samples/sec: 1132.36 - lr: 0.050000\n", "2020-09-17 23:59:50,424 epoch 15 - iter 13210/13217 - loss 0.00065182 - samples/sec: 1122.44 - lr: 0.050000\n", "2020-09-17 23:59:51,670 ----------------------------------------------------------------------------------------------------\n", "2020-09-17 23:59:51,672 EPOCH 15 done: loss 0.0007 - lr 0.0500000\n", "2020-09-18 00:01:34,291 DEV : loss 0.0006950036040507257 - score 0.9998\n", "2020-09-18 00:01:44,060 BAD EPOCHS (no improvement): 1\n", "2020-09-18 00:01:44,129 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 00:03:06,628 epoch 16 - iter 1321/13217 - loss 0.00054618 - samples/sec: 1113.32 - lr: 0.050000\n", "2020-09-18 00:03:48,159 epoch 16 - iter 2642/13217 - loss 0.00048265 - samples/sec: 1093.09 - lr: 0.050000\n", "2020-09-18 00:04:29,906 epoch 16 - iter 3963/13217 - loss 0.00046876 - samples/sec: 1116.56 - lr: 0.050000\n", "2020-09-18 00:05:11,642 epoch 16 - iter 5284/13217 - loss 0.00048654 - samples/sec: 1114.03 - lr: 0.050000\n", "2020-09-18 00:05:53,121 epoch 16 - iter 6605/13217 - loss 0.00046446 - samples/sec: 1096.79 - lr: 0.050000\n", "2020-09-18 00:06:33,798 epoch 16 - iter 7926/13217 - loss 0.00056249 - samples/sec: 1118.52 - lr: 0.050000\n", "2020-09-18 00:07:15,206 epoch 16 - iter 9247/13217 - loss 0.00063623 - samples/sec: 1097.11 - lr: 0.050000\n", "2020-09-18 00:07:56,748 epoch 16 - iter 10568/13217 - loss 0.00067024 - samples/sec: 1122.61 - lr: 0.050000\n", "2020-09-18 00:08:38,385 epoch 16 - iter 11889/13217 - loss 0.00066958 - samples/sec: 1121.44 - lr: 0.050000\n", "2020-09-18 00:09:19,693 epoch 16 - iter 13210/13217 - loss 0.00069373 - samples/sec: 1101.83 - lr: 0.050000\n", "2020-09-18 00:09:20,758 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 00:09:20,760 EPOCH 16 done: loss 0.0007 - lr 0.0500000\n", "2020-09-18 00:10:58,935 DEV : loss 0.000414311361964792 - score 0.9998\n", "2020-09-18 00:11:08,267 BAD EPOCHS (no improvement): 0\n", "saving best model\n", "2020-09-18 00:11:30,961 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 00:12:54,605 epoch 17 - iter 1321/13217 - loss 0.00069737 - samples/sec: 1107.25 - lr: 0.050000\n", "2020-09-18 00:13:36,053 epoch 17 - iter 2642/13217 - loss 0.00081176 - samples/sec: 1124.29 - lr: 0.050000\n", "2020-09-18 00:14:17,914 epoch 17 - iter 3963/13217 - loss 0.00071901 - samples/sec: 1091.52 - lr: 0.050000\n", "2020-09-18 00:14:58,882 epoch 17 - iter 5284/13217 - loss 0.00069263 - samples/sec: 1089.86 - lr: 0.050000\n", "2020-09-18 00:15:40,944 epoch 17 - iter 6605/13217 - loss 0.00061132 - samples/sec: 1115.23 - lr: 0.050000\n", "2020-09-18 00:16:22,907 epoch 17 - iter 7926/13217 - loss 0.00062491 - samples/sec: 1117.29 - lr: 0.050000\n", "2020-09-18 00:17:03,785 epoch 17 - iter 9247/13217 - loss 0.00064098 - samples/sec: 1092.55 - lr: 0.050000\n", "2020-09-18 00:17:45,812 epoch 17 - iter 10568/13217 - loss 0.00061365 - samples/sec: 1114.66 - lr: 0.050000\n", "2020-09-18 00:18:26,813 epoch 17 - iter 11889/13217 - loss 0.00060507 - samples/sec: 1117.71 - lr: 0.050000\n", "2020-09-18 00:19:08,452 epoch 17 - iter 13210/13217 - loss 0.00059522 - samples/sec: 1100.44 - lr: 0.050000\n", "2020-09-18 00:19:09,498 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 00:19:09,499 EPOCH 17 done: loss 0.0006 - lr 0.0500000\n", "2020-09-18 00:20:48,948 DEV : loss 0.00039736932376399636 - score 0.9999\n", "2020-09-18 00:20:58,187 BAD EPOCHS (no improvement): 0\n", "saving best model\n", "2020-09-18 00:21:17,872 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 00:22:42,840 epoch 18 - iter 1321/13217 - loss 0.00055192 - samples/sec: 1111.90 - lr: 0.050000\n", "2020-09-18 00:23:24,756 epoch 18 - iter 2642/13217 - loss 0.00046943 - samples/sec: 1108.80 - lr: 0.050000\n", "2020-09-18 00:24:06,570 epoch 18 - iter 3963/13217 - loss 0.00057278 - samples/sec: 1092.09 - lr: 0.050000\n", "2020-09-18 00:24:48,483 epoch 18 - iter 5284/13217 - loss 0.00063458 - samples/sec: 1112.45 - lr: 0.050000\n", "2020-09-18 00:25:31,052 epoch 18 - iter 6605/13217 - loss 0.00059057 - samples/sec: 1072.14 - lr: 0.050000\n", "2020-09-18 00:26:12,930 epoch 18 - iter 7926/13217 - loss 0.00056107 - samples/sec: 1114.68 - lr: 0.050000\n", "2020-09-18 00:26:53,668 epoch 18 - iter 9247/13217 - loss 0.00059366 - samples/sec: 1123.19 - lr: 0.050000\n", "2020-09-18 00:27:35,529 epoch 18 - iter 10568/13217 - loss 0.00063478 - samples/sec: 1088.78 - lr: 0.050000\n", "2020-09-18 00:28:17,317 epoch 18 - iter 11889/13217 - loss 0.00064515 - samples/sec: 1118.67 - lr: 0.050000\n", "2020-09-18 00:28:59,118 epoch 18 - iter 13210/13217 - loss 0.00065530 - samples/sec: 1091.16 - lr: 0.050000\n", "2020-09-18 00:29:00,180 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 00:29:00,182 EPOCH 18 done: loss 0.0007 - lr 0.0500000\n", "2020-09-18 00:30:40,843 DEV : loss 0.0005417444044724107 - score 0.9998\n", "2020-09-18 00:30:50,237 BAD EPOCHS (no improvement): 1\n", "2020-09-18 00:30:50,333 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 00:32:11,466 epoch 19 - iter 1321/13217 - loss 0.00061922 - samples/sec: 1133.07 - lr: 0.050000\n", "2020-09-18 00:32:52,203 epoch 19 - iter 2642/13217 - loss 0.00061772 - samples/sec: 1140.09 - lr: 0.050000\n", "2020-09-18 00:33:33,312 epoch 19 - iter 3963/13217 - loss 0.00068926 - samples/sec: 1131.97 - lr: 0.050000\n", "2020-09-18 00:34:14,461 epoch 19 - iter 5284/13217 - loss 0.00060730 - samples/sec: 1127.66 - lr: 0.050000\n", "2020-09-18 00:34:55,488 epoch 19 - iter 6605/13217 - loss 0.00063248 - samples/sec: 1131.98 - lr: 0.050000\n", "2020-09-18 00:35:36,804 epoch 19 - iter 7926/13217 - loss 0.00066299 - samples/sec: 1122.92 - lr: 0.050000\n", "2020-09-18 00:36:18,798 epoch 19 - iter 9247/13217 - loss 0.00067791 - samples/sec: 1132.60 - lr: 0.050000\n", "2020-09-18 00:36:59,983 epoch 19 - iter 10568/13217 - loss 0.00068468 - samples/sec: 1125.64 - lr: 0.050000\n", "2020-09-18 00:37:41,188 epoch 19 - iter 11889/13217 - loss 0.00066572 - samples/sec: 1127.55 - lr: 0.050000\n", "2020-09-18 00:38:22,116 epoch 19 - iter 13210/13217 - loss 0.00065572 - samples/sec: 1137.67 - lr: 0.050000\n", "2020-09-18 00:38:23,225 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 00:38:23,227 EPOCH 19 done: loss 0.0007 - lr 0.0500000\n", "2020-09-18 00:40:01,963 DEV : loss 0.0004346835776232183 - score 0.9998\n", "2020-09-18 00:40:11,295 BAD EPOCHS (no improvement): 2\n", "2020-09-18 00:40:11,297 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 00:41:31,617 epoch 20 - iter 1321/13217 - loss 0.00054619 - samples/sec: 1126.62 - lr: 0.050000\n", "2020-09-18 00:42:13,487 epoch 20 - iter 2642/13217 - loss 0.00056691 - samples/sec: 1122.83 - lr: 0.050000\n", "2020-09-18 00:42:54,337 epoch 20 - iter 3963/13217 - loss 0.00058844 - samples/sec: 1126.80 - lr: 0.050000\n", "2020-09-18 00:43:36,054 epoch 20 - iter 5284/13217 - loss 0.00056151 - samples/sec: 1099.76 - lr: 0.050000\n", "2020-09-18 00:44:16,932 epoch 20 - iter 6605/13217 - loss 0.00053835 - samples/sec: 1098.46 - lr: 0.050000\n", "2020-09-18 00:44:59,074 epoch 20 - iter 7926/13217 - loss 0.00054889 - samples/sec: 1116.63 - lr: 0.050000\n", "2020-09-18 00:45:39,977 epoch 20 - iter 9247/13217 - loss 0.00057376 - samples/sec: 1125.68 - lr: 0.050000\n", "2020-09-18 00:46:21,679 epoch 20 - iter 10568/13217 - loss 0.00059314 - samples/sec: 1131.53 - lr: 0.050000\n", "2020-09-18 00:47:02,670 epoch 20 - iter 11889/13217 - loss 0.00061495 - samples/sec: 1123.22 - lr: 0.050000\n", "2020-09-18 00:47:44,698 epoch 20 - iter 13210/13217 - loss 0.00058933 - samples/sec: 1121.13 - lr: 0.050000\n", "2020-09-18 00:47:45,892 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 00:47:45,893 EPOCH 20 done: loss 0.0006 - lr 0.0500000\n", "2020-09-18 00:49:24,751 DEV : loss 0.000669807312078774 - score 0.9998\n", "2020-09-18 00:49:34,060 BAD EPOCHS (no improvement): 3\n", "2020-09-18 00:49:34,062 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 00:50:54,332 epoch 21 - iter 1321/13217 - loss 0.00061406 - samples/sec: 1104.68 - lr: 0.050000\n", "2020-09-18 00:51:35,043 epoch 21 - iter 2642/13217 - loss 0.00052486 - samples/sec: 1127.01 - lr: 0.050000\n", "2020-09-18 00:52:16,694 epoch 21 - iter 3963/13217 - loss 0.00062521 - samples/sec: 1124.11 - lr: 0.050000\n", "2020-09-18 00:52:57,507 epoch 21 - iter 5284/13217 - loss 0.00065708 - samples/sec: 1120.91 - lr: 0.050000\n", "2020-09-18 00:53:39,625 epoch 21 - iter 6605/13217 - loss 0.00061899 - samples/sec: 1086.54 - lr: 0.050000\n", "2020-09-18 00:54:20,433 epoch 21 - iter 7926/13217 - loss 0.00061459 - samples/sec: 1094.60 - lr: 0.050000\n", "2020-09-18 00:55:02,420 epoch 21 - iter 9247/13217 - loss 0.00067146 - samples/sec: 1113.43 - lr: 0.050000\n", "2020-09-18 00:55:43,288 epoch 21 - iter 10568/13217 - loss 0.00066082 - samples/sec: 1119.36 - lr: 0.050000\n", "2020-09-18 00:56:25,058 epoch 21 - iter 11889/13217 - loss 0.00061890 - samples/sec: 1122.43 - lr: 0.050000\n", "2020-09-18 00:57:07,032 epoch 21 - iter 13210/13217 - loss 0.00059348 - samples/sec: 1114.12 - lr: 0.050000\n", "2020-09-18 00:57:08,043 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 00:57:08,044 EPOCH 21 done: loss 0.0006 - lr 0.0500000\n", "2020-09-18 00:58:45,913 DEV : loss 0.000368967856047675 - score 0.9999\n", "2020-09-18 00:58:55,489 BAD EPOCHS (no improvement): 0\n", "saving best model\n", "2020-09-18 00:59:19,467 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:00:47,652 epoch 22 - iter 1321/13217 - loss 0.00044193 - samples/sec: 1111.80 - lr: 0.050000\n", "2020-09-18 01:01:29,336 epoch 22 - iter 2642/13217 - loss 0.00076676 - samples/sec: 1122.70 - lr: 0.050000\n", "2020-09-18 01:02:10,446 epoch 22 - iter 3963/13217 - loss 0.00064941 - samples/sec: 1114.95 - lr: 0.050000\n", "2020-09-18 01:02:52,214 epoch 22 - iter 5284/13217 - loss 0.00058470 - samples/sec: 1124.63 - lr: 0.050000\n", "2020-09-18 01:03:33,171 epoch 22 - iter 6605/13217 - loss 0.00055028 - samples/sec: 1117.48 - lr: 0.050000\n", "2020-09-18 01:04:14,503 epoch 22 - iter 7926/13217 - loss 0.00054697 - samples/sec: 1108.53 - lr: 0.050000\n", "2020-09-18 01:04:56,767 epoch 22 - iter 9247/13217 - loss 0.00056764 - samples/sec: 1109.39 - lr: 0.050000\n", "2020-09-18 01:05:37,740 epoch 22 - iter 10568/13217 - loss 0.00056117 - samples/sec: 1118.30 - lr: 0.050000\n", "2020-09-18 01:06:19,879 epoch 22 - iter 11889/13217 - loss 0.00054372 - samples/sec: 1116.42 - lr: 0.050000\n", "2020-09-18 01:07:00,971 epoch 22 - iter 13210/13217 - loss 0.00053481 - samples/sec: 1089.81 - lr: 0.050000\n", "2020-09-18 01:07:01,988 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:07:01,989 EPOCH 22 done: loss 0.0005 - lr 0.0500000\n", "2020-09-18 01:08:41,069 DEV : loss 0.00034007462090812624 - score 0.9998\n", "2020-09-18 01:08:50,363 BAD EPOCHS (no improvement): 1\n", "2020-09-18 01:08:50,461 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:10:11,286 epoch 23 - iter 1321/13217 - loss 0.00045428 - samples/sec: 1118.61 - lr: 0.050000\n", "2020-09-18 01:10:52,251 epoch 23 - iter 2642/13217 - loss 0.00072321 - samples/sec: 1119.53 - lr: 0.050000\n", "2020-09-18 01:11:33,914 epoch 23 - iter 3963/13217 - loss 0.00078153 - samples/sec: 1124.52 - lr: 0.050000\n", "2020-09-18 01:12:15,503 epoch 23 - iter 5284/13217 - loss 0.00073427 - samples/sec: 1128.85 - lr: 0.050000\n", "2020-09-18 01:12:56,301 epoch 23 - iter 6605/13217 - loss 0.00071744 - samples/sec: 1125.23 - lr: 0.050000\n", "2020-09-18 01:13:38,281 epoch 23 - iter 7926/13217 - loss 0.00063450 - samples/sec: 1063.92 - lr: 0.050000\n", "2020-09-18 01:14:19,844 epoch 23 - iter 9247/13217 - loss 0.00061790 - samples/sec: 1128.76 - lr: 0.050000\n", "2020-09-18 01:15:01,001 epoch 23 - iter 10568/13217 - loss 0.00060377 - samples/sec: 1112.94 - lr: 0.050000\n", "2020-09-18 01:15:42,782 epoch 23 - iter 11889/13217 - loss 0.00061352 - samples/sec: 1122.41 - lr: 0.050000\n", "2020-09-18 01:16:24,599 epoch 23 - iter 13210/13217 - loss 0.00060697 - samples/sec: 1094.34 - lr: 0.050000\n", "2020-09-18 01:16:25,664 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:16:25,666 EPOCH 23 done: loss 0.0006 - lr 0.0500000\n", "2020-09-18 01:18:03,311 DEV : loss 0.0003522391489241272 - score 0.9999\n", "2020-09-18 01:18:12,642 BAD EPOCHS (no improvement): 0\n", "saving best model\n", "2020-09-18 01:18:36,987 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:20:04,294 epoch 24 - iter 1321/13217 - loss 0.00033728 - samples/sec: 1070.17 - lr: 0.050000\n", "2020-09-18 01:20:45,845 epoch 24 - iter 2642/13217 - loss 0.00051913 - samples/sec: 1091.86 - lr: 0.050000\n", "2020-09-18 01:21:28,127 epoch 24 - iter 3963/13217 - loss 0.00052048 - samples/sec: 1085.79 - lr: 0.050000\n", "2020-09-18 01:22:09,484 epoch 24 - iter 5284/13217 - loss 0.00051398 - samples/sec: 1107.99 - lr: 0.050000\n", "2020-09-18 01:22:51,538 epoch 24 - iter 6605/13217 - loss 0.00050334 - samples/sec: 1115.76 - lr: 0.050000\n", "2020-09-18 01:23:32,712 epoch 24 - iter 7926/13217 - loss 0.00052259 - samples/sec: 1115.29 - lr: 0.050000\n", "2020-09-18 01:24:14,765 epoch 24 - iter 9247/13217 - loss 0.00051943 - samples/sec: 1088.95 - lr: 0.050000\n", "2020-09-18 01:24:56,004 epoch 24 - iter 10568/13217 - loss 0.00052383 - samples/sec: 1112.43 - lr: 0.050000\n", "2020-09-18 01:25:38,400 epoch 24 - iter 11889/13217 - loss 0.00050500 - samples/sec: 1106.66 - lr: 0.050000\n", "2020-09-18 01:26:19,453 epoch 24 - iter 13210/13217 - loss 0.00051652 - samples/sec: 1117.56 - lr: 0.050000\n", "2020-09-18 01:26:20,487 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:26:20,488 EPOCH 24 done: loss 0.0005 - lr 0.0500000\n", "2020-09-18 01:27:59,589 DEV : loss 0.0003301367105450481 - score 0.9999\n", "2020-09-18 01:28:08,799 BAD EPOCHS (no improvement): 0\n", "saving best model\n", "2020-09-18 01:28:30,138 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:29:54,715 epoch 25 - iter 1321/13217 - loss 0.00052541 - samples/sec: 1093.07 - lr: 0.050000\n", "2020-09-18 01:30:35,952 epoch 25 - iter 2642/13217 - loss 0.00059338 - samples/sec: 1127.99 - lr: 0.050000\n", "2020-09-18 01:31:17,547 epoch 25 - iter 3963/13217 - loss 0.00061089 - samples/sec: 1116.61 - lr: 0.050000\n", "2020-09-18 01:31:58,698 epoch 25 - iter 5284/13217 - loss 0.00057554 - samples/sec: 1127.46 - lr: 0.050000\n", "2020-09-18 01:32:39,846 epoch 25 - iter 6605/13217 - loss 0.00063377 - samples/sec: 1128.34 - lr: 0.050000\n", "2020-09-18 01:33:21,343 epoch 25 - iter 7926/13217 - loss 0.00061069 - samples/sec: 1120.35 - lr: 0.050000\n", "2020-09-18 01:34:02,577 epoch 25 - iter 9247/13217 - loss 0.00055732 - samples/sec: 1125.91 - lr: 0.050000\n", "2020-09-18 01:34:43,813 epoch 25 - iter 10568/13217 - loss 0.00057570 - samples/sec: 1100.18 - lr: 0.050000\n", "2020-09-18 01:35:26,006 epoch 25 - iter 11889/13217 - loss 0.00056456 - samples/sec: 1103.04 - lr: 0.050000\n", "2020-09-18 01:36:07,126 epoch 25 - iter 13210/13217 - loss 0.00055012 - samples/sec: 1129.47 - lr: 0.050000\n", "2020-09-18 01:36:08,194 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:36:08,196 EPOCH 25 done: loss 0.0005 - lr 0.0500000\n", "2020-09-18 01:37:46,993 DEV : loss 0.0005273263668641448 - score 0.9998\n", "2020-09-18 01:37:56,321 BAD EPOCHS (no improvement): 1\n", "2020-09-18 01:37:56,418 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:39:16,836 epoch 26 - iter 1321/13217 - loss 0.00038367 - samples/sec: 1122.42 - lr: 0.050000\n", "2020-09-18 01:39:58,825 epoch 26 - iter 2642/13217 - loss 0.00047249 - samples/sec: 1123.00 - lr: 0.050000\n", "2020-09-18 01:40:39,964 epoch 26 - iter 3963/13217 - loss 0.00040075 - samples/sec: 1120.28 - lr: 0.050000\n", "2020-09-18 01:41:21,827 epoch 26 - iter 5284/13217 - loss 0.00043398 - samples/sec: 1124.40 - lr: 0.050000\n", "2020-09-18 01:42:03,048 epoch 26 - iter 6605/13217 - loss 0.00045870 - samples/sec: 1116.79 - lr: 0.050000\n", "2020-09-18 01:42:44,195 epoch 26 - iter 7926/13217 - loss 0.00046333 - samples/sec: 1121.58 - lr: 0.050000\n", "2020-09-18 01:43:26,204 epoch 26 - iter 9247/13217 - loss 0.00042562 - samples/sec: 1125.18 - lr: 0.050000\n", "2020-09-18 01:44:07,366 epoch 26 - iter 10568/13217 - loss 0.00044791 - samples/sec: 1119.99 - lr: 0.050000\n", "2020-09-18 01:44:49,533 epoch 26 - iter 11889/13217 - loss 0.00050263 - samples/sec: 1120.27 - lr: 0.050000\n", "2020-09-18 01:45:31,037 epoch 26 - iter 13210/13217 - loss 0.00048086 - samples/sec: 1112.06 - lr: 0.050000\n", "2020-09-18 01:45:32,241 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:45:32,243 EPOCH 26 done: loss 0.0005 - lr 0.0500000\n", "2020-09-18 01:47:10,691 DEV : loss 0.0003236537449993193 - score 0.9999\n", "2020-09-18 01:47:20,018 BAD EPOCHS (no improvement): 0\n", "saving best model\n", "2020-09-18 01:47:39,140 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:49:03,510 epoch 27 - iter 1321/13217 - loss 0.00077307 - samples/sec: 1118.38 - lr: 0.050000\n", "2020-09-18 01:49:45,198 epoch 27 - iter 2642/13217 - loss 0.00060027 - samples/sec: 1089.18 - lr: 0.050000\n", "2020-09-18 01:50:27,464 epoch 27 - iter 3963/13217 - loss 0.00075467 - samples/sec: 1108.30 - lr: 0.050000\n", "2020-09-18 01:51:09,126 epoch 27 - iter 5284/13217 - loss 0.00071988 - samples/sec: 1094.30 - lr: 0.050000\n", "2020-09-18 01:51:49,882 epoch 27 - iter 6605/13217 - loss 0.00067537 - samples/sec: 1120.54 - lr: 0.050000\n", "2020-09-18 01:52:31,652 epoch 27 - iter 7926/13217 - loss 0.00061337 - samples/sec: 1119.02 - lr: 0.050000\n", "2020-09-18 01:53:13,326 epoch 27 - iter 9247/13217 - loss 0.00056807 - samples/sec: 1095.47 - lr: 0.050000\n", "2020-09-18 01:53:54,853 epoch 27 - iter 10568/13217 - loss 0.00056620 - samples/sec: 1097.48 - lr: 0.050000\n", "2020-09-18 01:54:35,717 epoch 27 - iter 11889/13217 - loss 0.00057231 - samples/sec: 1118.16 - lr: 0.050000\n", "2020-09-18 01:55:18,160 epoch 27 - iter 13210/13217 - loss 0.00056693 - samples/sec: 1073.98 - lr: 0.050000\n", "2020-09-18 01:55:23,042 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:55:23,045 EPOCH 27 done: loss 0.0006 - lr 0.0500000\n", "2020-09-18 01:57:06,266 DEV : loss 0.00032912680762819946 - score 0.9999\n", "2020-09-18 01:57:15,553 BAD EPOCHS (no improvement): 1\n", "2020-09-18 01:57:15,643 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 01:58:36,980 epoch 28 - iter 1321/13217 - loss 0.00036654 - samples/sec: 1127.75 - lr: 0.050000\n", "2020-09-18 01:59:18,402 epoch 28 - iter 2642/13217 - loss 0.00031160 - samples/sec: 1124.45 - lr: 0.050000\n", "2020-09-18 01:59:59,136 epoch 28 - iter 3963/13217 - loss 0.00032681 - samples/sec: 1128.68 - lr: 0.050000\n", "2020-09-18 02:00:40,777 epoch 28 - iter 5284/13217 - loss 0.00039054 - samples/sec: 1128.67 - lr: 0.050000\n", "2020-09-18 02:01:21,722 epoch 28 - iter 6605/13217 - loss 0.00045015 - samples/sec: 1121.37 - lr: 0.050000\n", "2020-09-18 02:02:03,471 epoch 28 - iter 7926/13217 - loss 0.00042001 - samples/sec: 1126.17 - lr: 0.050000\n", "2020-09-18 02:02:44,215 epoch 28 - iter 9247/13217 - loss 0.00044134 - samples/sec: 1131.42 - lr: 0.050000\n", "2020-09-18 02:03:25,703 epoch 28 - iter 10568/13217 - loss 0.00045340 - samples/sec: 1104.05 - lr: 0.050000\n", "2020-09-18 02:04:07,691 epoch 28 - iter 11889/13217 - loss 0.00046093 - samples/sec: 1120.15 - lr: 0.050000\n", "2020-09-18 02:04:48,700 epoch 28 - iter 13210/13217 - loss 0.00046915 - samples/sec: 1123.74 - lr: 0.050000\n", "2020-09-18 02:04:49,929 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 02:04:49,930 EPOCH 28 done: loss 0.0005 - lr 0.0500000\n", "2020-09-18 02:06:28,999 DEV : loss 0.0003881795855704695 - score 0.9999\n", "2020-09-18 02:06:38,296 BAD EPOCHS (no improvement): 2\n", "2020-09-18 02:06:38,298 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 02:07:59,599 epoch 29 - iter 1321/13217 - loss 0.00034590 - samples/sec: 1092.89 - lr: 0.050000\n", "2020-09-18 02:08:41,131 epoch 29 - iter 2642/13217 - loss 0.00035025 - samples/sec: 1121.52 - lr: 0.050000\n", "2020-09-18 02:09:21,905 epoch 29 - iter 3963/13217 - loss 0.00044862 - samples/sec: 1122.43 - lr: 0.050000\n", "2020-09-18 02:10:03,536 epoch 29 - iter 5284/13217 - loss 0.00046454 - samples/sec: 1123.05 - lr: 0.050000\n", "2020-09-18 02:10:45,227 epoch 29 - iter 6605/13217 - loss 0.00045521 - samples/sec: 1120.33 - lr: 0.050000\n", "2020-09-18 02:11:26,827 epoch 29 - iter 7926/13217 - loss 0.00045473 - samples/sec: 1122.82 - lr: 0.050000\n", "2020-09-18 02:12:07,640 epoch 29 - iter 9247/13217 - loss 0.00044949 - samples/sec: 1119.20 - lr: 0.050000\n", "2020-09-18 02:12:49,432 epoch 29 - iter 10568/13217 - loss 0.00046502 - samples/sec: 1117.45 - lr: 0.050000\n", "2020-09-18 02:13:31,075 epoch 29 - iter 11889/13217 - loss 0.00048714 - samples/sec: 1095.38 - lr: 0.050000\n", "2020-09-18 02:14:12,677 epoch 29 - iter 13210/13217 - loss 0.00049239 - samples/sec: 1070.36 - lr: 0.050000\n", "2020-09-18 02:14:13,691 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 02:14:13,692 EPOCH 29 done: loss 0.0005 - lr 0.0500000\n", "2020-09-18 02:15:52,245 DEV : loss 0.0003162016218993813 - score 0.9999\n", "2020-09-18 02:16:01,576 BAD EPOCHS (no improvement): 0\n", "saving best model\n", "2020-09-18 02:16:22,308 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 02:17:46,639 epoch 30 - iter 1321/13217 - loss 0.00043100 - samples/sec: 1102.60 - lr: 0.050000\n", "2020-09-18 02:18:27,865 epoch 30 - iter 2642/13217 - loss 0.00035493 - samples/sec: 1115.36 - lr: 0.050000\n", "2020-09-18 02:19:09,969 epoch 30 - iter 3963/13217 - loss 0.00035672 - samples/sec: 1113.07 - lr: 0.050000\n", "2020-09-18 02:19:51,037 epoch 30 - iter 5284/13217 - loss 0.00042555 - samples/sec: 1092.16 - lr: 0.050000\n", "2020-09-18 02:20:33,384 epoch 30 - iter 6605/13217 - loss 0.00041532 - samples/sec: 1109.89 - lr: 0.050000\n", "2020-09-18 02:21:14,632 epoch 30 - iter 7926/13217 - loss 0.00038346 - samples/sec: 1112.60 - lr: 0.050000\n", "2020-09-18 02:21:55,677 epoch 30 - iter 9247/13217 - loss 0.00038193 - samples/sec: 1117.85 - lr: 0.050000\n", "2020-09-18 02:22:37,722 epoch 30 - iter 10568/13217 - loss 0.00038348 - samples/sec: 1120.14 - lr: 0.050000\n", "2020-09-18 02:23:18,848 epoch 30 - iter 11889/13217 - loss 0.00043016 - samples/sec: 1115.98 - lr: 0.050000\n", "2020-09-18 02:24:01,240 epoch 30 - iter 13210/13217 - loss 0.00041636 - samples/sec: 1080.81 - lr: 0.050000\n", "2020-09-18 02:24:02,303 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 02:24:02,304 EPOCH 30 done: loss 0.0004 - lr 0.0500000\n", "2020-09-18 02:25:41,012 DEV : loss 0.0003216553886886686 - score 0.9999\n", "2020-09-18 02:25:50,309 BAD EPOCHS (no improvement): 1\n", "2020-09-18 02:26:11,143 ----------------------------------------------------------------------------------------------------\n", "2020-09-18 02:26:11,145 Testing using best model ...\n", "2020-09-18 02:26:11,258 loading file models\\en.word_split_0.1_32\\best-model.pt\n", "2020-09-18 02:27:51,902 \t0.9999\n", "2020-09-18 02:27:52,074 \n", "Results:\n", "- F-score (micro) 0.9999\n", "- F-score (macro) 0.9956\n", "- Accuracy 0.9999\n", "\n", "By class:\n", " precision recall f1-score support\n", "\n", " benign 0.9999 1.0000 0.9999 52522\n", " malignant 0.9942 0.9884 0.9913 346\n", "\n", " micro avg 0.9999 0.9999 0.9999 52868\n", " macro avg 0.9971 0.9942 0.9956 52868\n", "weighted avg 0.9999 0.9999 0.9999 52868\n", " samples avg 0.9999 0.9999 0.9999 52868\n", "\n", "2020-09-18 02:27:52,076 ----------------------------------------------------------------------------------------------------\n" ] }, { "data": { "text/plain": [ "{'test_score': 0.9999,\n", " 'dev_score_history': [0.9997,\n", " 0.9998,\n", " 0.9998,\n", " 0.9997,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9999,\n", " 0.9998,\n", " 0.9998,\n", " 0.9998,\n", " 0.9999,\n", " 0.9998,\n", " 0.9999,\n", " 0.9999,\n", " 0.9998,\n", " 0.9999,\n", " 0.9999,\n", " 0.9999,\n", " 0.9999,\n", " 0.9999],\n", " 'train_loss_history': [0.007005688341932229,\n", " 0.0013865314515874637,\n", " 0.0010426083367080436,\n", " 0.0010144814260991692,\n", " 0.0009920659119428336,\n", " 0.0010587797197936935,\n", " 0.0008621925794808552,\n", " 0.0008075173992434479,\n", " 0.000731073996334196,\n", " 0.0007000347918036083,\n", " 0.0007187440749044018,\n", " 0.000748671631027228,\n", " 0.000783193307360384,\n", " 0.0006957890033008677,\n", " 0.0006514761315378964,\n", " 0.0006951858777576511,\n", " 0.0005949204746242241,\n", " 0.0006549592268721019,\n", " 0.0006606985747614043,\n", " 0.0005890207203470428,\n", " 0.0005931733206512098,\n", " 0.0005345358036626473,\n", " 0.0006067134716228813,\n", " 0.000516379041056304,\n", " 0.0005498284672124913,\n", " 0.00048060644434001716,\n", " 0.0005666411932743337,\n", " 0.00046890764830589835,\n", " 0.0004921441088793227,\n", " 0.00041614394101802864],\n", " 'dev_loss_history': [0.0009424208546988666,\n", " 0.0007725355098955333,\n", " 0.000515893567353487,\n", " 0.0013346497435122728,\n", " 0.0005379541544243693,\n", " 0.000892516050953418,\n", " 0.0008064075373113155,\n", " 0.0005345970275811851,\n", " 0.0005398467765189707,\n", " 0.0004569814773276448,\n", " 0.0005163941532373428,\n", " 0.000496959313750267,\n", " 0.0004584364651236683,\n", " 0.00045535029494203627,\n", " 0.0006950036040507257,\n", " 0.000414311361964792,\n", " 0.00039736932376399636,\n", " 0.0005417444044724107,\n", " 0.0004346835776232183,\n", " 0.000669807312078774,\n", " 0.000368967856047675,\n", " 0.00034007462090812624,\n", " 0.0003522391489241272,\n", " 0.0003301367105450481,\n", " 0.0005273263668641448,\n", " 0.0003236537449993193,\n", " 0.00032912680762819946,\n", " 0.0003881795855704695,\n", " 0.0003162016218993813,\n", " 0.0003216553886886686]}" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import os\n", "\n", "# 'en' (or 'en-news' or 'news')\tEnglish\tFastText embeddings over news and wikipedia data\n", "\n", "model_name = 'en'\n", "\n", "corpus = corpus_word_split\n", "lr = 0.1\n", "batch_size = 32\n", "model_var = f'word_split_{lr}_{batch_size}'\n", "\n", "\n", "model_dir = f'models/{model_name}.{model_var}'\n", "try:\n", " os.mkdir(model_dir)\n", "except OSError:\n", " print (f\"Creation of the directory '{model_dir}' failed.\")\n", "else:\n", " print (f\"Created the directory '{model_dir}'\")\n", "\n", "label_dict = corpus.make_label_dictionary()\n", "word_embeddings = [WordEmbeddings(model_name)]\n", "document_embeddings = DocumentRNNEmbeddings(word_embeddings, hidden_size=256)\n", "classifier = TextClassifier(document_embeddings, label_dictionary=label_dict)\n", "trainer = ModelTrainer(classifier, corpus)\n", "\n", "trainer.train(model_dir,\n", " learning_rate=lr,\n", " mini_batch_size=batch_size, # 32,\n", " anneal_factor=0.5,\n", " patience=3,\n", " max_epochs=30)\n", "\n", "# 2020-09-18 02:26:11,258 loading file models\\en.word_split_0.1_32\\best-model.pt\n", "# 2020-09-18 02:27:51,902 \t0.9999\n", "# 2020-09-18 02:27:52,074 \n", "# Results:\n", "# - F-score (micro) 0.9999\n", "# - F-score (macro) 0.9956\n", "# - Accuracy 0.9999\n", "\n", "# By class:\n", "# precision recall f1-score support\n", "\n", "# benign 0.9999 1.0000 0.9999 52522\n", "# malignant 0.9942 0.9884 0.9913 346\n", "\n", "# micro avg 0.9999 0.9999 0.9999 52868\n", "# macro avg 0.9971 0.9942 0.9956 52868\n", "# weighted avg 0.9999 0.9999 0.9999 52868\n", "# samples avg 0.9999 0.9999 0.9999 52868\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Further Development and Research (OUTDATED)\n", "1. Additional Data Sources:\n", " 1. Feedback data from the production systems: Predicted TPs and TNs. Can we discover simple heuristics/statistics?\n", "1. The complexity of the existed DGAs and neural networks. Is there a correlation between them? Can we estimate the complexity of the DGA in terms of a number of parameters (or any other NN complexity measurements)?\n", "1. Can we group the DGA by algo groups? If YES, can we train different NNs for different DGA groups and use an ensemble of the models?" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2020-09-23T21:50:54.469707Z", "start_time": "2020-09-23T21:50:54.461708Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[5, -2, 4, 'Z', 'X', 9, '+', '+'] 18 27\n", "[1, 'X'] 1 2\n", "[1, 2, '+', 'Z'] 0 3\n" ] } ], "source": [ "def totalScore(num, blocks):\n", " # WRITE YOUR CODE HERE\n", " if not blocks: \n", " return 0 \n", " score, prev_score, prev_prev_score = 0, 0, 0\n", " for el in blocks:\n", " if type(el) == int:\n", " score, prev_score, prev_prev_score = prev_score + el, el, prev_score\n", " elif el == 'X':\n", " score, prev_score, prev_prev_score = prev_score*2, prev_score*2, prev_score \n", " elif el == '+':\n", " score, prev_score, prev_prev_score = prev_score + prev_prev_score, prev_score + prev_prev_score, prev_score\n", " elif and el == 'Z':\n", " score, prev_score, prev_prev_score = \n", " return score\n", " \n", "\n", "\n", "tests = [\n", " ([], 0)\n", " ([5,-2, 4], 7),\n", " ([1], 1),\n", " (['X'], 0),\n", " ([1, 'X'], 2),\n", " ([5,-2, 4], 4),([1, 2, '+', 'Z'], 3),\n", " ([5,-2,4,'Z','X',9, '+', '+'], 27),\n", "]\n", "\n", "_ = [print(t, totalScore(len(t), t), res) for t, res in tests if totalScore(len(t), t) != res]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# \"character as a feature\" approach \n", "\n", "(like in the DNS-Tunnel detector)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Additional changes:\n", "* URL parsing with the `tldextract`\n", "* Additional benign domains from the DNS-Tunnel project\n", "\n", "Steps:\n", "* prepare data\n", " - benign \n", " - malitious\n", " \n", "* preprocess data\n", "* train" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prepare data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### benign data\n", "\n", "Data were prepared in the `researches/notebooks/domain_data.ipynb`, `Preprocess data/benign` section.\n", "\n", ">Saved 28,709,889 into ../data/benign_domains/external/benign_domains.external.all.11M.csv" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from pathlib import Path\n", "\n", "bening_file = \"data/benign/benign_domains.external.all.11M.csv\"\n", "Path(bening_file).exists()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### malignant data" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(True, True)" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "time: 5.29 ms (started: 2022-09-19 11:10:44 -07:00)\n" ] } ], "source": [ "from pathlib import Path\n", "\n", "dga_synthetic_file = \"data/domain_generation_algorithms/malignant_domains.dga_generated.43_algos.4M.csv\"\n", "dga_file = 'data/Adversarial-DGA-Datasets/Adversarial-DGA-Datasets.41K.csv'\n", "Path(dga_synthetic_file).exists(), Path(dga_file).exists()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Concatenate all with labels" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Loaded 11,937,220 data/benign/benign_domains.external.all.11M.csv, columns: ['domain']\n", " Domain extraction...\n", " Added the label: bening\n", " Concatenated 11,937,220 -> 11,937,220\n", " Deduplicated domains: 7,749,605\n", " Loaded 4,524,427 data/domain_generation_algorithms/malignant_domains.dga_generated.43_algos.4M.csv, columns: ['domain']\n", " Added the label: malignant\n", " Concatenated 4,524,427 -> 12,274,032\n", " Deduplicated domains: 12,272,225\n", " Loaded 41,584 data/Adversarial-DGA-Datasets/Adversarial-DGA-Datasets.41K.csv, columns: ['domain']\n", " Domain extraction...\n", " Added the label: malignant\n", " Concatenated 41,584 -> 12,313,809\n", " Deduplicated domains: 12,310,512\n", "========================================\n", "Shuffled: 12,310,512\n", "X_train: 9,848,409, y_train: 9,848,409, samples: konzrdsk - malignant\n", "X_dev: 1,231,051, y_dev: 1,231,051, samples: rwnhwhrwha - malignant\n", "X_test: 1,231,052, y_test: 1,231,052, samples: justinmoorhouse - bening\n", "Saved 9,848,409 data/training_data/x_train.csv, name: domain\n", "Saved 9,848,409 data/training_data/y_train.csv, name: label\n", "Saved 1,231,051 data/training_data/x_dev.csv, name: domain\n", "Saved 1,231,051 data/training_data/y_dev.csv, name: label\n", "Saved 1,231,052 data/training_data/x_test.csv, name: domain\n", "Saved 1,231,052 data/training_data/y_test.csv, name: label\n", "time: 1min 24s (started: 2022-09-21 16:34:00 -07:00)\n" ] } ], "source": [ "from pathlib import Path\n", "import pandas as pd\n", "import tldextract\n", "from sklearn.model_selection import train_test_split\n", "\n", "\n", "def concatenate_all(files, out_dir, nrows=100):\n", " df_res = pd.DataFrame()\n", " for el in files:\n", " df = pd.read_csv(el[\"file\"], nrows=nrows)\n", " print(f\" Loaded {df.shape[0]:,} {el['file']}, columns: {list(df.columns)}\")\n", " \n", " if not el['is_domain_extracted']:\n", " print(f\" Domain extraction...\") \n", " df[\"domain\"] = df[\"domain\"].apply(lambda d: tldextract.extract(d).domain)\n", " df['label'] = el['label']\n", " print(f\" Added the label: {el['label']}\")\n", " \n", " df_res = pd.concat([df_res, df])\n", " print(f\" Concatenated {df.shape[0]:,} -> {df_res.shape[0]:,}\")\n", " \n", " df_res = df_res.drop_duplicates(subset='domain') # Preference to the first samples (to bening)!!!\n", " print(f\" Deduplicated domains: {df_res.shape[0]:,}\") \n", "\n", " df = df_res\n", " df = shuffle(df).reset_index(drop=True)\n", " print('='*40)\n", " print(f\"Shuffled: {df.shape[0]:,}\")\n", " \n", " out_file = f\"{out_dir}/data.csv\"\n", " df.to_csv(out_file, index=False)\n", " print(f\"Saved {df.shape[0]:,} {out_file}\")\n", "# X_train, X_tmp, y_train, y_tmp = train_test_split(\n", "# df['domain'], df['label'], test_size=0.20, random_state=42, \n", "# stratify=df['label'], shuffle=True)\n", "# X_dev, X_test, y_dev, y_test = train_test_split(\n", "# X_tmp, y_tmp, test_size=0.50, random_state=42, \n", "# stratify=y_tmp, shuffle=True)\n", " \n", "# print(f\"X_train: {len(X_train):,}, y_train: {len(y_train):,}, samples: {X_train.iloc[0]} - {y_train.iloc[0]}\")\n", "# print(f\"X_dev: {len(X_dev):,}, y_dev: {len(y_dev):,}, samples: {X_dev.iloc[0]} - {y_dev.iloc[0]}\")\n", "# print(f\"X_test: {len(X_test):,}, y_test: {len(y_test):,}, samples: {X_test.iloc[0]} - {y_test.iloc[0]}\")\n", " \n", "# for name, x, y in zip(['train', 'dev', 'test'], [X_train, X_dev, X_test], [y_train, y_dev, y_test]):\n", "# out_file = f\"{out_dir}/x_{name}.csv\"\n", "# x.to_csv(out_file, index=False)\n", "# print(f\"Saved {len(x):,} {out_file}, name: {x.name}\")\n", " \n", "# out_file = f\"{out_dir}/y_{name}.csv\"\n", "# y.to_csv(out_file, index=False)\n", "# print(f\"Saved {len(y):,} {out_file}, name: {y.name}\")\n", " return df\n", "\n", "# all files have a single \"domain\" column:\n", "# it has the prefix part ('.com')\n", "files = [\n", " {\"label\": \"bening\", \n", " \"file\": \"data/benign/benign_domains.external.all.11M.csv\", \n", " \"is_domain_extracted\": False\n", " },\n", " {\"label\": \"malignant\", \n", " \"file\": \"data/domain_generation_algorithms/malignant_domains.dga_generated.43_algos.4M.csv\", \n", " \"is_domain_extracted\": True\n", " },\n", " {\"label\": \"malignant\", \n", " \"file\": \"data/Adversarial-DGA-Datasets/Adversarial-DGA-Datasets.41K.csv\", \n", " \"is_domain_extracted\": False},\n", "]\n", "\n", "out_dir = 'data/training_data'\n", "df = concatenate_all(files, out_dir, nrows=None)\n" ] }, { "cell_type": "code", "execution_count": 84, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved 1,000 data/training_data/bytes_features/data.csv\n", "time: 8.85 ms (started: 2022-09-23 08:48:05 -07:00)\n" ] } ], "source": [ "out_file = f\"{out_dir}/data.csv\"\n", "df.to_csv(out_file, index=False)\n", "print(f\"Saved {df.shape[0]:,} {out_file}\")" ] }, { "cell_type": "code", "execution_count": 98, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "time: 2.62 s (started: 2022-09-23 09:26:27 -07:00)\n" ] } ], "source": [ "# what is the domain string length distribution?\n", "df_2 = df.copy(deep=True)\n", "df_2['len'] = df_2['domain'].str.len()" ] }, { "cell_type": "code", "execution_count": 117, "metadata": {}, "outputs": [ { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>domain</th>\n", " <th>label</th>\n", " <th>len</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>706</th>\n", " <td>3mf</td>\n", " <td>bening</td>\n", " <td>3.0</td>\n", " </tr>\n", " <tr>\n", " <th>1477</th>\n", " <td>jr7</td>\n", " <td>bening</td>\n", " <td>3.0</td>\n", " </tr>\n", " <tr>\n", " <th>3460</th>\n", " <td>a4g</td>\n", " <td>bening</td>\n", " <td>3.0</td>\n", " </tr>\n", " <tr>\n", " <th>5198</th>\n", " <td>mwi</td>\n", " <td>bening</td>\n", " <td>3.0</td>\n", " </tr>\n", " <tr>\n", " <th>5858</th>\n", " <td>3sz</td>\n", " <td>bening</td>\n", " <td>3.0</td>\n", " </tr>\n", " <tr>\n", " <th>...</th>\n", " <td>...</td>\n", " <td>...</td>\n", " <td>...</td>\n", " </tr>\n", " <tr>\n", " <th>12308406</th>\n", " <td>l-o</td>\n", " <td>bening</td>\n", " <td>3.0</td>\n", " </tr>\n", " <tr>\n", " <th>12308564</th>\n", " <td>bjm</td>\n", " <td>bening</td>\n", " <td>3.0</td>\n", " </tr>\n", " <tr>\n", " <th>12309336</th>\n", " <td>2hz</td>\n", " <td>bening</td>\n", " <td>3.0</td>\n", " </tr>\n", " <tr>\n", " <th>12309891</th>\n", " <td>tia</td>\n", " <td>bening</td>\n", " <td>3.0</td>\n", " </tr>\n", " <tr>\n", " <th>12310270</th>\n", " <td>do5</td>\n", " <td>bening</td>\n", " <td>3.0</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "<p>25649 rows × 3 columns</p>\n", "</div>" ], "text/plain": [ " domain label len\n", "706 3mf bening 3.0\n", "1477 jr7 bening 3.0\n", "3460 a4g bening 3.0\n", "5198 mwi bening 3.0\n", "5858 3sz bening 3.0\n", "... ... ... ...\n", "12308406 l-o bening 3.0\n", "12308564 bjm bening 3.0\n", "12309336 2hz bening 3.0\n", "12309891 tia bening 3.0\n", "12310270 do5 bening 3.0\n", "\n", "[25649 rows x 3 columns]" ] }, "execution_count": 117, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "time: 34.5 ms (started: 2022-09-23 09:50:14 -07:00)\n" ] } ], "source": [ "df_2[(df_2[\"len\"] < 4)]" ] }, { "cell_type": "code", "execution_count": 115, "metadata": {}, "outputs": [ { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>domain</th>\n", " <th>label</th>\n", " <th>len</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>5833227</th>\n", " <td>ddwul</td>\n", " <td>malignant</td>\n", " <td>5.0</td>\n", " </tr>\n", " <tr>\n", " <th>6934420</th>\n", " <td>lyran</td>\n", " <td>malignant</td>\n", " <td>5.0</td>\n", " </tr>\n", " <tr>\n", " <th>7030327</th>\n", " <td>gaton</td>\n", " <td>malignant</td>\n", " <td>5.0</td>\n", " </tr>\n", " <tr>\n", " <th>11719763</th>\n", " <td>mamet</td>\n", " <td>malignant</td>\n", " <td>5.0</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " domain label len\n", "5833227 ddwul malignant 5.0\n", "6934420 lyran malignant 5.0\n", "7030327 gaton malignant 5.0\n", "11719763 mamet malignant 5.0" ] }, "execution_count": 115, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "time: 414 ms (started: 2022-09-23 09:49:06 -07:00)\n" ] } ], "source": [ "df_2[(df_2[\"len\"] < 6) & (df_2[\"label\"] == \"malignant\")]" ] }, { "cell_type": "code", "execution_count": 120, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "time: 109 ms (started: 2022-09-23 10:00:27 -07:00)\n" ] } ], "source": [ "df_lens = df_2['len'].value_counts()" ] }, { "cell_type": "code", "execution_count": 127, "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "25.0 46169\n", "29.0 28235\n", "26.0 25370\n", "27.0 25243\n", "3.0 24404\n", "31.0 22386\n", "37.0 20707\n", "38.0 20681\n", "30.0 20274\n", "39.0 18118\n", "36.0 14799\n", "32.0 14038\n", "40.0 12101\n", "35.0 11187\n", "33.0 10939\n", "34.0 8690\n", "41.0 6344\n", "42.0 2768\n", "2.0 1208\n", "43.0 966\n", "44.0 334\n", "45.0 105\n", "46.0 55\n", "1.0 37\n", "49.0 14\n", "56.0 13\n", "48.0 9\n", "50.0 9\n", "47.0 9\n", "60.0 7\n", "54.0 7\n", "63.0 6\n", "61.0 6\n", "62.0 6\n", "58.0 4\n", "55.0 3\n", "51.0 3\n", "53.0 3\n", "57.0 2\n", "59.0 1\n", "Name: len, dtype: int64" ] }, "execution_count": 127, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "time: 8.56 ms (started: 2022-09-23 10:02:27 -07:00)\n" ] } ], "source": [ "df_lens[df_lens < 50000]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# conclusion:\n", " ## the domains with len < 26 keep most of the samples." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## old model, tokens based" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### preprocessing" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved 300 data/training_data/old_features/data.csv\n", "time: 90.8 ms (started: 2022-09-21 17:08:14 -07:00)\n" ] } ], "source": [ "from transformers import DistilBertTokenizer\n", "from collections import Counter\n", "\n", "def extract_features(s: str, max_length: int = 14, tokenizer=None) -> List[int]:\n", " \"\"\"\n", " It is the same transformation that was applied on the training samples.\n", " If we change it, we need to change it on the training samples and retrain a model!\n", " Transform a string into a sequence of tokens; then into the lengths of the tokens;\n", " then into the counters of lengths from 1 to 14.\n", " Tokens with lengths > 14 counted as 14 length. It is super rare when a token has a length > 14.\n", " Example:\n", " returns: [0, 3, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n", " - here a string transformed into 5 tokens; 3 tokens with length of 2 and 2 tokens - length of 3.\n", " Additional internal transformations:\n", " * [optional] replace all digits to '9' digit (it didn't show an improvement in the model performance).\n", " * remove the '##' markers because we don't care if the token is on the word border or not\n", " * remove the service tokens ([CLS], [SEP]) and the '.' token\n", "\n", " Parameters\n", " ----------\n", " s - a string to be parsed\n", " replace_digits - do we need to replace all digits to '9' digit?\n", " max_length - an upper limit of a token length\n", "\n", " Returns\n", " -------\n", " a list of the counters of the token lengths\n", "\n", " \"\"\"\n", " # if replace_digits:\n", " # s = s.translate(self._digit_replacer)\n", " d = dict(\n", " Counter(\n", " [\n", " len(t.replace(\"##\", \"\"))\n", " for t in tokenizer.convert_ids_to_tokens(tokenizer(s)[\"input_ids\"])\n", " if t not in [\"[CLS]\", \"[SEP]\", \".\"]\n", " ]\n", " )\n", " )\n", " features = [d[i] if i in d else 0 for i in range(1, 19)]\n", " features_cut = features[: max_length - 1] + [sum(features[max_length:])]\n", " return features_cut\n", "\n", "def prepare_old_features(df, nrows=None):\n", " tokenizer = DistilBertTokenizer.from_pretrained(\"tokenizer/\")\n", " df = df[:nrows].copy(deep=True)\n", " df['features'] = df['domain'].apply(lambda d: extract_features(d, max_length=14, tokenizer=tokenizer))\n", " df['y'] = df['label'].apply(lambda d: 0 if d == 'bening' else 1)\n", " out_file = \"data/training_data/old_features/data.csv\"\n", " df.to_csv(out_file, index=False)\n", " print(f\"Saved {df.shape[0]:,} {out_file}\") \n", " return df\n", "\n", "\n", "out_dir = 'data/training_data/old_features/'\n" ] }, { "cell_type": "code", "execution_count": 63, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved 12,310,512 data/training_data/old_features/data.csv\n", "time: 19min 7s (started: 2022-09-21 17:41:33 -07:00)\n" ] } ], "source": [ "nrows = None\n", "df_res = prepare_old_features(df, nrows=nrows)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### training" ] }, { "cell_type": "code", "execution_count": 64, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "train_size=9848409, dev_size=11079460, test_size=12310512\n" ] }, { "data": { "text/plain": [ "<catboost.core.CatBoostClassifier at 0x7f7794b2dca0>" ] }, "execution_count": 64, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "time: 9min 39s (started: 2022-09-21 18:00:41 -07:00)\n" ] } ], "source": [ "model = CatBoostClassifier(loss_function='MultiClass') # Logloss, MultiClass\n", "\n", "size = df_res.shape[0]\n", "train_size, dev_size, test_size = int(size*0.8), int(size*0.9), size\n", "print(f\"{train_size=}, {dev_size=}, {test_size=}\")\n", "X_train, y_train = np.array(list(df_res.loc[0:train_size, 'features'])), df_res.loc[0:train_size, 'y'] \n", "X_dev, y_dev = np.array(list(df_res.loc[train_size:dev_size, 'features'])), df_res.loc[train_size:dev_size, 'y']\n", "X_test, y_test, domain_test = np.array(list(df_res.loc[dev_size:, 'features'])), df_res.loc[dev_size:, 'y'], df_res.loc[size*2:size*3, 'domain']\n", "\n", "model.fit(X_train, y_train, logging_level='Silent', eval_set=(X_dev, y_dev))\n" ] }, { "cell_type": "code", "execution_count": 65, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "time: 2.37 s (started: 2022-09-21 18:11:26 -07:00)\n" ] } ], "source": [ "y_pred = model.predict_proba(X_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Results" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "time: 1.27 ms (started: 2022-09-26 12:26:22 -07:00)\n" ] } ], "source": [ "def results(y_true, y_scores, y_pred=None):\n", " if not y_pred:\n", " y_pred = [1 if sc > 0.5 else 0 for sc in y_scores]\n", " support = len(y_true)\n", " print(f\"support: {support:}\")\n", " print(f\"average_precision_score: {average_precision_score(y_true, y_scores):.3}\")\n", " roc_auc_score_val = roc_auc_score(y_true, y_scores)\n", " print(f\"roc_auc_score: {roc_auc_score_val:.3}\")\n", " print(f\"classification_report: \\n{classification_report(y_true, y_pred)}\")" ] }, { "cell_type": "code", "execution_count": 66, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "support: 1231052\n", "average_precision_score: 0.942\n", "roc_auc_score: 0.959\n", "classification_report: \n", " precision recall f1-score support\n", "\n", " 0 0.91 0.95 0.93 775348\n", " 1 0.90 0.84 0.87 455704\n", "\n", " accuracy 0.91 1231052\n", " macro avg 0.90 0.89 0.90 1231052\n", "weighted avg 0.91 0.91 0.90 1231052\n", "\n", "time: 1.83 s (started: 2022-09-21 18:11:35 -07:00)\n" ] } ], "source": [ "y_scores = [y[1] for y in y_pred]\n", "\n", "results(y_test, y_scores)\n", "\n", "# support: 1_231_052\n", "# average_precision_score: 0.942\n", "# roc_auc_score: 0.959\n", "# classification_report: \n", "# precision recall f1-score support\n", "\n", "# 0 0.91 0.95 0.93 775348\n", "# 1 0.90 0.84 0.87 455704\n", "\n", "# accuracy 0.91 1231052\n", "# macro avg 0.90 0.89 0.90 1231052\n", "# weighted avg 0.91 0.91 0.90 1231052\n", "\n", "# support: 10000\n", "# average_precision_score: 0.942\n", "# roc_auc_score: 0.957\n", "# classification_report: \n", "# precision recall f1-score support\n", "\n", "# 0 0.90 0.94 0.92 6222\n", "# 1 0.90 0.83 0.87 3778\n", "\n", "# accuracy 0.90 10000\n", "# macro avg 0.90 0.89 0.89 10000\n", "# weighted avg 0.90 0.90 0.90 10000\n", "\n", "# support: 100\n", "# average_precision_score: 0.881\n", "# roc_auc_score: 0.888\n", "# classification_report: \n", "# precision recall f1-score support\n", "\n", "# 0 0.91 0.91 0.91 67\n", "# 1 0.82 0.82 0.82 33\n", "\n", "# accuracy 0.88 100\n", "# macro avg 0.86 0.86 0.86 100\n", "# weighted avg 0.88 0.88 0.88 100\n", "\n" ] }, { "cell_type": "code", "execution_count": 70, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved models/catboost.0.942.tokens.model\n", "time: 335 ms (started: 2022-09-21 18:19:18 -07:00)\n" ] } ], "source": [ "y_scores = [y[1] for y in y_pred]\n", "aps = average_precision_score(y_test, y_scores)\n", "out_file_name = f'models/catboost.{aps:.3}.tokens.model'\n", "\n", "model.save_model(out_file_name)\n", "print(f'Saved {out_file_name}')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## model \"bytes as a feature\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### preprocessing" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loaded (12310512, 2) data/training_data/data.csv\n", "df.shape=(12310512, 2) max_len=32 nrows=None \n", "Start with 12,310,512 samples\n", "After dropna 12,310,509 samples\n", "Saved 12,310,509 data/training_data/bytes_features/data.csv\n", "time: 1min 6s (started: 2022-09-23 11:33:02 -07:00)\n" ] } ], "source": [ "def extract_features(s: str, pad_char: str = \"=\", max_len=20) -> str:\n", " \"\"\"\n", " It converts a string into the features.\n", " Features are the bytes decoded from the string.\n", " Features are formatted as the '99 111 108 111 ...' string.\n", " Features is the fixed-size array.\n", " The feature array is taken from the middle of the string if the string is longer than the feature_num;\n", " the string is padded with a pad char if it is shorter than the feature_num.\n", " Parameters\n", " ----------\n", " s - a subdomain name, like 'player' in 'player.my-gaming.com'\n", " Returns\n", " -------\n", " s - a feature array in form of the string '99 111 108 111 ...'.\n", " \"\"\"\n", " # encode to the byte array:\n", " if isinstance(s, bytes):\n", " s = s.decode()\n", " elif not isinstance(s, (str, bytes)):\n", " raise ValueError(s, \"Only str or bytearray type can be processed by the _extract_features().\")\n", "\n", " s_len = len(s)\n", " # cut or pad a string\n", " if s_len < max_len:\n", " s = s + pad_char * (max_len - s_len)\n", " elif s_len > max_len:\n", " delta = (s_len - max_len) // 2\n", " s = s[delta : (max_len + delta)]\n", "\n", " return [ord(b) for b in s]\n", "\n", "def prepare_features(df, out_dir=None, max_len=20, nrows=None):\n", " print(f\"{df.shape=} {max_len=} {nrows=} \")\n", " df = df[:nrows].copy(deep=True)\n", " print(f\"Start with {df.shape[0]:,} samples\")\n", " df = df.dropna()\n", " print(f\"After dropna {df.shape[0]:,} samples\")\n", " df['features'] = df['domain'].apply(lambda d: extract_features(d, max_len=max_len))\n", " df['y'] = df['label'].apply(lambda d: 0 if d == 'bening' else 1)\n", " out_file = f\"{out_dir}/data.csv\"\n", " df.to_csv(out_file, index=False)\n", " print(f\"Saved {df.shape[0]:,} {out_file}\") \n", " return df\n", "\n", "\n", "df = pd.read_csv('data/training_data/data.csv')\n", "print(f\"Loaded {df.shape} data/training_data/data.csv\")\n", "nrows = None\n", "max_len = 32\n", "out_dir = 'data/training_data/bytes_features'\n", "df_res = prepare_features(df, out_dir=out_dir, max_len=max_len, nrows=nrows)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### training" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "9848407, 11079458, 12310509\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a22c1acbd62a46e28ed72179893659e9", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "<catboost.core.CatBoostClassifier at 0x7f0eaa606a90>" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "time: 13min 6s (started: 2022-09-23 11:34:43 -07:00)\n" ] } ], "source": [ "size = df_res.shape[0]\n", "train_size, dev_size, test_size = int(size*0.8), int(size*0.9), size\n", "print(f\"{train_size:.=}, {dev_size:.=}, {test_size:.=}\")\n", "X_train, y_train = np.array(list(df_res.loc[0:train_size, 'features'])), df_res.loc[0:train_size, 'y'] \n", "X_dev, y_dev = np.array(list(df_res.loc[train_size:dev_size, 'features'])), df_res.loc[train_size:dev_size, 'y']\n", "X_test, y_test, domain_test = np.array(list(df_res.loc[dev_size:, 'features'])), df_res.loc[dev_size:, 'y'], df_res.loc[size*2:size*3, 'domain']\n", "\n", "model = CatBoostClassifier(loss_function='MultiClass') # Logloss, MultiClass\n", "\n", "model.fit(X_train, y_train, eval_set=(X_dev, y_dev), verbose=False,\n", " plot=True)\n", "\n", "# train_size=799999, dev_size=899999, test_size=999999\n", "# time: 53.3 s (started: 2022-09-23 08:57:16 -07:00)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Results" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "support: 1231054\n", "average_precision_score: 0.962\n", "roc_auc_score: 0.972\n", "classification_report: \n", " precision recall f1-score support\n", "\n", " 0 0.92 0.97 0.94 775350\n", " 1 0.94 0.86 0.90 455704\n", "\n", " accuracy 0.93 1231054\n", " macro avg 0.93 0.91 0.92 1231054\n", "weighted avg 0.93 0.93 0.93 1231054\n", "\n", "time: 6.63 s (started: 2022-09-23 11:47:50 -07:00)\n" ] } ], "source": [ "y_pred = model.predict_proba(X_test)\n", "y_scores = [y[1] for y in y_pred]\n", "\n", "results(y_test, y_scores, y_pred=None)\n", "\n", "# support: 1231054 (len = 26, iterations = 1400)\n", "# average_precision_score: 0.964\n", "# roc_auc_score: 0.973\n", "# classification_report: \n", "# precision recall f1-score support\n", "# 0 0.92 0.97 0.95 775350\n", "# 1 0.95 0.86 0.90 455704\n", "# accuracy 0.93 1231054\n", "# macro avg 0.93 0.92 0.92 1231054\n", "# weighted avg 0.93 0.93 0.93 1231054\n", "\n", "# support: 1231054 (len = 26)\n", "# average_precision_score: 0.961\n", "# roc_auc_score: 0.971\n", "# classification_report: \n", "# precision recall f1-score support\n", "# 0 0.92 0.97 0.94 775350\n", "# 1 0.94 0.86 0.90 455704\n", "# accuracy 0.93 1231054\n", "# macro avg 0.93 0.91 0.92 1231054\n", "# weighted avg 0.93 0.93 0.93 1231054\n", "\n", "# support: 1_231_054 \n", "# average_precision_score: 0.958\n", "# roc_auc_score: 0.968\n", "# classification_report: \n", "# precision recall f1-score support\n", "# 0 0.91 0.97 0.94 775350\n", "# 1 0.94 0.84 0.89 455704\n", "# accuracy 0.92 1231054\n", "# macro avg 0.93 0.91 0.92 1231054\n", "# weighted avg 0.92 0.92 0.92 1231054\n", "\n", "# support: 100001\n", "# average_precision_score: 0.957\n", "# roc_auc_score: 0.966\n", "# classification_report: \n", "# precision recall f1-score support\n", "# 0 0.91 0.97 0.94 62847\n", "# 1 0.94 0.84 0.89 37154\n", "# accuracy 0.92 100001\n", "# macro avg 0.93 0.90 0.91 100001\n", "# weighted avg 0.92 0.92 0.92 100001\n", "\n", "# support: 10000\n", "# average_precision_score: 0.946\n", "# roc_auc_score: 0.958\n", "# classification_report: \n", "# precision recall f1-score support\n", "# 0 0.89 0.97 0.93 6222\n", "# 1 0.94 0.81 0.87 3778\n", "# accuracy 0.91 10000\n", "# macro avg 0.92 0.89 0.90 10000\n", "# weighted avg 0.91 0.91 0.91 10000\n", "# time: 77.5 ms (started: 2022-09-23 08:52:08 -07:00)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved models/catboost.0.962.32_bytes.model\n", "time: 239 ms (started: 2022-09-23 11:47:57 -07:00)\n" ] } ], "source": [ "aps = average_precision_score(y_test, y_scores)\n", "out_file_name = f'models/catboost.{aps:.3}.{max_len}_bytes.model'\n", "\n", "model.save_model(out_file_name)\n", "print(f'Saved {out_file_name}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Ensemble: token-based and bytes-based" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### preprocessing" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loaded (12310512, 2) data/training_data/data.csv\n", "df.shape=(12310512, 2) max_len=26 nrows=None \n", "Start with 12,310,512 samples\n", "After dropna 12,310,509 samples\n" ] }, { "ename": "KeyboardInterrupt", "evalue": "", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", "Input \u001b[0;32mIn [3]\u001b[0m, in \u001b[0;36m<cell line: 79>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 77\u001b[0m max_len \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m26\u001b[39m\n\u001b[1;32m 78\u001b[0m out_dir \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mdata/training_data/ensemble_features\u001b[39m\u001b[38;5;124m'\u001b[39m\n\u001b[0;32m---> 79\u001b[0m df_res \u001b[38;5;241m=\u001b[39m \u001b[43mprepare_features\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdf\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mout_dir\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mout_dir\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmax_len\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_len\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnrows\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mnrows\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msave\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m)\u001b[49m\n", "Input \u001b[0;32mIn [3]\u001b[0m, in \u001b[0;36mprepare_features\u001b[0;34m(df, out_dir, max_len, nrows, save)\u001b[0m\n\u001b[1;32m 59\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mAfter dropna \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mdf\u001b[38;5;241m.\u001b[39mshape[\u001b[38;5;241m0\u001b[39m]\u001b[38;5;132;01m:\u001b[39;00m\u001b[38;5;124m,\u001b[39m\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m samples\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 60\u001b[0m tokenizer \u001b[38;5;241m=\u001b[39m DistilBertTokenizer\u001b[38;5;241m.\u001b[39mfrom_pretrained(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodels/tokenizer/\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m---> 61\u001b[0m df[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mfeatures\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m \u001b[43mdf\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mdomain\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m]\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mapply\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43;01mlambda\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43md\u001b[49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mextract_features\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 62\u001b[0m \u001b[43m \u001b[49m\u001b[43md\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmax_len\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_len\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 63\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;66;43;03m# max_token_num=max_token_num,\u001b[39;49;00m\n\u001b[1;32m 64\u001b[0m \u001b[43m \u001b[49m\u001b[43mtokenizer\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtokenizer\u001b[49m\n\u001b[1;32m 65\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 66\u001b[0m df[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124my\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m df[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mlabel\u001b[39m\u001b[38;5;124m'\u001b[39m]\u001b[38;5;241m.\u001b[39mapply(\u001b[38;5;28;01mlambda\u001b[39;00m d: \u001b[38;5;241m0\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m d \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mbening\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;241m1\u001b[39m)\n\u001b[1;32m 67\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m save:\n", "File \u001b[0;32m~/.local/lib/python3.8/site-packages/pandas/core/series.py:4433\u001b[0m, in \u001b[0;36mSeries.apply\u001b[0;34m(self, func, convert_dtype, args, **kwargs)\u001b[0m\n\u001b[1;32m 4323\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mapply\u001b[39m(\n\u001b[1;32m 4324\u001b[0m \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m 4325\u001b[0m func: AggFuncType,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 4328\u001b[0m \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs,\n\u001b[1;32m 4329\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m DataFrame \u001b[38;5;241m|\u001b[39m Series:\n\u001b[1;32m 4330\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 4331\u001b[0m \u001b[38;5;124;03m Invoke function on values of Series.\u001b[39;00m\n\u001b[1;32m 4332\u001b[0m \n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 4431\u001b[0m \u001b[38;5;124;03m dtype: float64\u001b[39;00m\n\u001b[1;32m 4432\u001b[0m \u001b[38;5;124;03m \"\"\"\u001b[39;00m\n\u001b[0;32m-> 4433\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mSeriesApply\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mfunc\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mconvert_dtype\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mapply\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/.local/lib/python3.8/site-packages/pandas/core/apply.py:1082\u001b[0m, in \u001b[0;36mSeriesApply.apply\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1078\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mf, \u001b[38;5;28mstr\u001b[39m):\n\u001b[1;32m 1079\u001b[0m \u001b[38;5;66;03m# if we are a string, try to dispatch\u001b[39;00m\n\u001b[1;32m 1080\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mapply_str()\n\u001b[0;32m-> 1082\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mapply_standard\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/.local/lib/python3.8/site-packages/pandas/core/apply.py:1137\u001b[0m, in \u001b[0;36mSeriesApply.apply_standard\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1131\u001b[0m values \u001b[38;5;241m=\u001b[39m obj\u001b[38;5;241m.\u001b[39mastype(\u001b[38;5;28mobject\u001b[39m)\u001b[38;5;241m.\u001b[39m_values\n\u001b[1;32m 1132\u001b[0m \u001b[38;5;66;03m# error: Argument 2 to \"map_infer\" has incompatible type\u001b[39;00m\n\u001b[1;32m 1133\u001b[0m \u001b[38;5;66;03m# \"Union[Callable[..., Any], str, List[Union[Callable[..., Any], str]],\u001b[39;00m\n\u001b[1;32m 1134\u001b[0m \u001b[38;5;66;03m# Dict[Hashable, Union[Union[Callable[..., Any], str],\u001b[39;00m\n\u001b[1;32m 1135\u001b[0m \u001b[38;5;66;03m# List[Union[Callable[..., Any], str]]]]]\"; expected\u001b[39;00m\n\u001b[1;32m 1136\u001b[0m \u001b[38;5;66;03m# \"Callable[[Any], Any]\"\u001b[39;00m\n\u001b[0;32m-> 1137\u001b[0m mapped \u001b[38;5;241m=\u001b[39m \u001b[43mlib\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmap_infer\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 1138\u001b[0m \u001b[43m \u001b[49m\u001b[43mvalues\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1139\u001b[0m \u001b[43m \u001b[49m\u001b[43mf\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;66;43;03m# type: ignore[arg-type]\u001b[39;49;00m\n\u001b[1;32m 1140\u001b[0m \u001b[43m \u001b[49m\u001b[43mconvert\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mconvert_dtype\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1141\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1143\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(mapped) \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(mapped[\u001b[38;5;241m0\u001b[39m], ABCSeries):\n\u001b[1;32m 1144\u001b[0m \u001b[38;5;66;03m# GH#43986 Need to do list(mapped) in order to get treated as nested\u001b[39;00m\n\u001b[1;32m 1145\u001b[0m \u001b[38;5;66;03m# See also GH#25959 regarding EA support\u001b[39;00m\n\u001b[1;32m 1146\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m obj\u001b[38;5;241m.\u001b[39m_constructor_expanddim(\u001b[38;5;28mlist\u001b[39m(mapped), index\u001b[38;5;241m=\u001b[39mobj\u001b[38;5;241m.\u001b[39mindex)\n", "File \u001b[0;32m~/.local/lib/python3.8/site-packages/pandas/_libs/lib.pyx:2870\u001b[0m, in \u001b[0;36mpandas._libs.lib.map_infer\u001b[0;34m()\u001b[0m\n", "Input \u001b[0;32mIn [3]\u001b[0m, in \u001b[0;36mprepare_features.<locals>.<lambda>\u001b[0;34m(d)\u001b[0m\n\u001b[1;32m 59\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mAfter dropna \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mdf\u001b[38;5;241m.\u001b[39mshape[\u001b[38;5;241m0\u001b[39m]\u001b[38;5;132;01m:\u001b[39;00m\u001b[38;5;124m,\u001b[39m\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m samples\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 60\u001b[0m tokenizer \u001b[38;5;241m=\u001b[39m DistilBertTokenizer\u001b[38;5;241m.\u001b[39mfrom_pretrained(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodels/tokenizer/\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m---> 61\u001b[0m df[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mfeatures\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m df[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mdomain\u001b[39m\u001b[38;5;124m'\u001b[39m]\u001b[38;5;241m.\u001b[39mapply(\u001b[38;5;28;01mlambda\u001b[39;00m d: \u001b[43mextract_features\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 62\u001b[0m \u001b[43m \u001b[49m\u001b[43md\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmax_len\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_len\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 63\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;66;43;03m# max_token_num=max_token_num,\u001b[39;49;00m\n\u001b[1;32m 64\u001b[0m \u001b[43m \u001b[49m\u001b[43mtokenizer\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtokenizer\u001b[49m\n\u001b[1;32m 65\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[1;32m 66\u001b[0m df[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124my\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m df[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mlabel\u001b[39m\u001b[38;5;124m'\u001b[39m]\u001b[38;5;241m.\u001b[39mapply(\u001b[38;5;28;01mlambda\u001b[39;00m d: \u001b[38;5;241m0\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m d \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mbening\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;241m1\u001b[39m)\n\u001b[1;32m 67\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m save:\n", "Input \u001b[0;32mIn [3]\u001b[0m, in \u001b[0;36mextract_features\u001b[0;34m(s, pad_char, max_len, max_token_num, tokenizer)\u001b[0m\n\u001b[1;32m 49\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mextract_features\u001b[39m(s: \u001b[38;5;28mstr\u001b[39m, pad_char: \u001b[38;5;28mstr\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m=\u001b[39m\u001b[38;5;124m\"\u001b[39m, max_len\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m20\u001b[39m, max_token_num: \u001b[38;5;28mint\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m14\u001b[39m, tokenizer\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m):\n\u001b[0;32m---> 50\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m (\u001b[43mextract_old_features\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmax_length\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_token_num\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtokenizer\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtokenizer\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 51\u001b[0m \u001b[38;5;241m+\u001b[39m extract_new_features(s, max_len\u001b[38;5;241m=\u001b[39mmax_len))\n", "Input \u001b[0;32mIn [3]\u001b[0m, in \u001b[0;36mextract_old_features\u001b[0;34m(s, max_length, tokenizer)\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mextract_old_features\u001b[39m(s: \u001b[38;5;28mstr\u001b[39m, max_length: \u001b[38;5;28mint\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m14\u001b[39m, tokenizer\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m List[\u001b[38;5;28mint\u001b[39m]:\n\u001b[1;32m 8\u001b[0m d \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mdict\u001b[39m(\n\u001b[1;32m 9\u001b[0m Counter(\n\u001b[1;32m 10\u001b[0m [\n\u001b[1;32m 11\u001b[0m \u001b[38;5;28mlen\u001b[39m(t\u001b[38;5;241m.\u001b[39mreplace(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m##\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m\"\u001b[39m))\n\u001b[0;32m---> 12\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m t \u001b[38;5;129;01min\u001b[39;00m tokenizer\u001b[38;5;241m.\u001b[39mconvert_ids_to_tokens(\u001b[43mtokenizer\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m)\u001b[49m[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124minput_ids\u001b[39m\u001b[38;5;124m\"\u001b[39m])\n\u001b[1;32m 13\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m t \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;129;01min\u001b[39;00m [\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m[CLS]\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m[SEP]\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m.\u001b[39m\u001b[38;5;124m\"\u001b[39m]\n\u001b[1;32m 14\u001b[0m ]\n\u001b[1;32m 15\u001b[0m )\n\u001b[1;32m 16\u001b[0m )\n\u001b[1;32m 17\u001b[0m features \u001b[38;5;241m=\u001b[39m [d[i] \u001b[38;5;28;01mif\u001b[39;00m i \u001b[38;5;129;01min\u001b[39;00m d \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;241m0\u001b[39m \u001b[38;5;28;01mfor\u001b[39;00m i \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mrange\u001b[39m(\u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m19\u001b[39m)]\n\u001b[1;32m 18\u001b[0m features_cut \u001b[38;5;241m=\u001b[39m features[: max_length \u001b[38;5;241m-\u001b[39m \u001b[38;5;241m1\u001b[39m] \u001b[38;5;241m+\u001b[39m [\u001b[38;5;28msum\u001b[39m(features[max_length:])]\n", "File \u001b[0;32m~/.local/lib/python3.8/site-packages/transformers/tokenization_utils_base.py:2515\u001b[0m, in \u001b[0;36mPreTrainedTokenizerBase.__call__\u001b[0;34m(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)\u001b[0m\n\u001b[1;32m 2495\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mbatch_encode_plus(\n\u001b[1;32m 2496\u001b[0m batch_text_or_text_pairs\u001b[38;5;241m=\u001b[39mbatch_text_or_text_pairs,\n\u001b[1;32m 2497\u001b[0m add_special_tokens\u001b[38;5;241m=\u001b[39madd_special_tokens,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 2512\u001b[0m \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs,\n\u001b[1;32m 2513\u001b[0m )\n\u001b[1;32m 2514\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 2515\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mencode_plus\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 2516\u001b[0m \u001b[43m \u001b[49m\u001b[43mtext\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtext\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2517\u001b[0m \u001b[43m \u001b[49m\u001b[43mtext_pair\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtext_pair\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2518\u001b[0m \u001b[43m \u001b[49m\u001b[43madd_special_tokens\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43madd_special_tokens\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2519\u001b[0m \u001b[43m \u001b[49m\u001b[43mpadding\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mpadding\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2520\u001b[0m \u001b[43m \u001b[49m\u001b[43mtruncation\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtruncation\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2521\u001b[0m \u001b[43m \u001b[49m\u001b[43mmax_length\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_length\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2522\u001b[0m \u001b[43m \u001b[49m\u001b[43mstride\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstride\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2523\u001b[0m \u001b[43m \u001b[49m\u001b[43mis_split_into_words\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mis_split_into_words\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2524\u001b[0m \u001b[43m \u001b[49m\u001b[43mpad_to_multiple_of\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mpad_to_multiple_of\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2525\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_tensors\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_tensors\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2526\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_token_type_ids\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_token_type_ids\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2527\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_attention_mask\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_attention_mask\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2528\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_overflowing_tokens\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_overflowing_tokens\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2529\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_special_tokens_mask\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_special_tokens_mask\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2530\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_offsets_mapping\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_offsets_mapping\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2531\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_length\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_length\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2532\u001b[0m \u001b[43m \u001b[49m\u001b[43mverbose\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mverbose\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2533\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2534\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/.local/lib/python3.8/site-packages/transformers/tokenization_utils_base.py:2588\u001b[0m, in \u001b[0;36mPreTrainedTokenizerBase.encode_plus\u001b[0;34m(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)\u001b[0m\n\u001b[1;32m 2578\u001b[0m \u001b[38;5;66;03m# Backward compatibility for 'truncation_strategy', 'pad_to_max_length'\u001b[39;00m\n\u001b[1;32m 2579\u001b[0m padding_strategy, truncation_strategy, max_length, kwargs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_get_padding_truncation_strategies(\n\u001b[1;32m 2580\u001b[0m padding\u001b[38;5;241m=\u001b[39mpadding,\n\u001b[1;32m 2581\u001b[0m truncation\u001b[38;5;241m=\u001b[39mtruncation,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 2585\u001b[0m \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs,\n\u001b[1;32m 2586\u001b[0m )\n\u001b[0;32m-> 2588\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_encode_plus\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 2589\u001b[0m \u001b[43m \u001b[49m\u001b[43mtext\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtext\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2590\u001b[0m \u001b[43m \u001b[49m\u001b[43mtext_pair\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtext_pair\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2591\u001b[0m \u001b[43m \u001b[49m\u001b[43madd_special_tokens\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43madd_special_tokens\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2592\u001b[0m \u001b[43m \u001b[49m\u001b[43mpadding_strategy\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mpadding_strategy\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2593\u001b[0m \u001b[43m \u001b[49m\u001b[43mtruncation_strategy\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtruncation_strategy\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2594\u001b[0m \u001b[43m \u001b[49m\u001b[43mmax_length\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_length\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2595\u001b[0m \u001b[43m \u001b[49m\u001b[43mstride\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstride\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2596\u001b[0m \u001b[43m \u001b[49m\u001b[43mis_split_into_words\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mis_split_into_words\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2597\u001b[0m \u001b[43m \u001b[49m\u001b[43mpad_to_multiple_of\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mpad_to_multiple_of\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2598\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_tensors\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_tensors\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2599\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_token_type_ids\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_token_type_ids\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2600\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_attention_mask\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_attention_mask\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2601\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_overflowing_tokens\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_overflowing_tokens\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2602\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_special_tokens_mask\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_special_tokens_mask\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2603\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_offsets_mapping\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_offsets_mapping\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2604\u001b[0m \u001b[43m \u001b[49m\u001b[43mreturn_length\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreturn_length\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2605\u001b[0m \u001b[43m \u001b[49m\u001b[43mverbose\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mverbose\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2606\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2607\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/.local/lib/python3.8/site-packages/transformers/tokenization_utils.py:649\u001b[0m, in \u001b[0;36mPreTrainedTokenizer._encode_plus\u001b[0;34m(self, text, text_pair, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)\u001b[0m\n\u001b[1;32m 640\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m return_offsets_mapping:\n\u001b[1;32m 641\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mNotImplementedError\u001b[39;00m(\n\u001b[1;32m 642\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mreturn_offset_mapping is not available when using Python tokenizers. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 643\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mTo use this feature, change your tokenizer to one deriving from \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 646\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhttps://github.com/huggingface/transformers/pull/2674\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 647\u001b[0m )\n\u001b[0;32m--> 649\u001b[0m first_ids \u001b[38;5;241m=\u001b[39m \u001b[43mget_input_ids\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtext\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 650\u001b[0m second_ids \u001b[38;5;241m=\u001b[39m get_input_ids(text_pair) \u001b[38;5;28;01mif\u001b[39;00m text_pair \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 652\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mprepare_for_model(\n\u001b[1;32m 653\u001b[0m first_ids,\n\u001b[1;32m 654\u001b[0m pair_ids\u001b[38;5;241m=\u001b[39msecond_ids,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 668\u001b[0m verbose\u001b[38;5;241m=\u001b[39mverbose,\n\u001b[1;32m 669\u001b[0m )\n", "File \u001b[0;32m~/.local/lib/python3.8/site-packages/transformers/tokenization_utils.py:616\u001b[0m, in \u001b[0;36mPreTrainedTokenizer._encode_plus.<locals>.get_input_ids\u001b[0;34m(text)\u001b[0m\n\u001b[1;32m 614\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mget_input_ids\u001b[39m(text):\n\u001b[1;32m 615\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(text, \u001b[38;5;28mstr\u001b[39m):\n\u001b[0;32m--> 616\u001b[0m tokens \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtokenize\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtext\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 617\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mconvert_tokens_to_ids(tokens)\n\u001b[1;32m 618\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(text, (\u001b[38;5;28mlist\u001b[39m, \u001b[38;5;28mtuple\u001b[39m)) \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(text) \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m0\u001b[39m \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(text[\u001b[38;5;241m0\u001b[39m], \u001b[38;5;28mstr\u001b[39m):\n", "File \u001b[0;32m~/.local/lib/python3.8/site-packages/transformers/tokenization_utils.py:499\u001b[0m, in \u001b[0;36mPreTrainedTokenizer.tokenize\u001b[0;34m(self, text, **kwargs)\u001b[0m\n\u001b[1;32m 482\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 483\u001b[0m \u001b[38;5;124;03mConverts a string in a sequence of tokens, using the tokenizer.\u001b[39;00m\n\u001b[1;32m 484\u001b[0m \n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 495\u001b[0m \u001b[38;5;124;03m `List[str]`: The list of tokens.\u001b[39;00m\n\u001b[1;32m 496\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 497\u001b[0m \u001b[38;5;66;03m# Simple mapping string => AddedToken for special tokens with specific tokenization behaviors\u001b[39;00m\n\u001b[1;32m 498\u001b[0m all_special_tokens_extended \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mdict\u001b[39m(\n\u001b[0;32m--> 499\u001b[0m (\u001b[38;5;28mstr\u001b[39m(t), t) \u001b[38;5;28;01mfor\u001b[39;00m t \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mall_special_tokens_extended\u001b[49m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(t, AddedToken)\n\u001b[1;32m 500\u001b[0m )\n\u001b[1;32m 502\u001b[0m text, kwargs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mprepare_for_tokenization(text, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m 504\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m kwargs:\n", "File \u001b[0;32m~/.local/lib/python3.8/site-packages/transformers/tokenization_utils_base.py:1241\u001b[0m, in \u001b[0;36mSpecialTokensMixin.all_special_tokens_extended\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1232\u001b[0m \u001b[38;5;129m@property\u001b[39m\n\u001b[1;32m 1233\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mall_special_tokens_extended\u001b[39m(\u001b[38;5;28mself\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m List[Union[\u001b[38;5;28mstr\u001b[39m, AddedToken]]:\n\u001b[1;32m 1234\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 1235\u001b[0m \u001b[38;5;124;03m `List[Union[str, tokenizers.AddedToken]]`: All the special tokens (`'<unk>'`, `'<cls>'`, etc.) mapped to class\u001b[39;00m\n\u001b[1;32m 1236\u001b[0m \u001b[38;5;124;03m attributes.\u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 1239\u001b[0m \u001b[38;5;124;03m special tokens are tokenized.\u001b[39;00m\n\u001b[1;32m 1240\u001b[0m \u001b[38;5;124;03m \"\"\"\u001b[39;00m\n\u001b[0;32m-> 1241\u001b[0m all_toks \u001b[38;5;241m=\u001b[39m []\n\u001b[1;32m 1242\u001b[0m set_attr \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mspecial_tokens_map_extended\n\u001b[1;32m 1243\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m attr_value \u001b[38;5;129;01min\u001b[39;00m set_attr\u001b[38;5;241m.\u001b[39mvalues():\n", "\u001b[0;31mKeyboardInterrupt\u001b[0m: " ] }, { "name": "stdout", "output_type": "stream", "text": [ "time: 12min 39s (started: 2022-09-26 11:22:20 -07:00)\n" ] } ], "source": [ "# Run it once when prepare a data file. Otherwise, be aware it is not fast!\n", "\n", "from transformers import DistilBertTokenizer\n", "from collections import Counter\n", "\n", "def extract_old_features(s: str, max_length: int = 14, tokenizer=None) -> List[int]:\n", "\n", " d = dict(\n", " Counter(\n", " [\n", " len(t.replace(\"##\", \"\"))\n", " for t in tokenizer.convert_ids_to_tokens(tokenizer(s)[\"input_ids\"])\n", " if t not in [\"[CLS]\", \"[SEP]\", \".\"]\n", " ]\n", " )\n", " )\n", " features = [d[i] if i in d else 0 for i in range(1, 19)]\n", " features_cut = features[: max_length - 1] + [sum(features[max_length:])]\n", " return features_cut\n", "\n", "# def prepare_old_features(df, nrows=None):\n", "# tokenizer = DistilBertTokenizer.from_pretrained(\"tokenizer/\")\n", "# df = df[:nrows].copy(deep=True)\n", "# df['features'] = df['domain'].apply(lambda d: extract_features(d, max_length=14, tokenizer=tokenizer))\n", "# df['y'] = df['label'].apply(lambda d: 0 if d == 'bening' else 1)\n", "# out_file = \"data/training_data/old_features/data.csv\"\n", "# df.to_csv(out_file, index=False)\n", "# print(f\"Saved {df.shape[0]:,} {out_file}\") \n", "# return df\n", "\n", "\n", "def extract_new_features(s: str, pad_char: str = \"=\", max_len=20) -> List[int]:\n", " # encode to the byte array:\n", " if isinstance(s, bytes):\n", " s = s.decode()\n", " elif not isinstance(s, (str, bytes)):\n", " raise ValueError(s, \"Only str or bytearray type can be processed by the _extract_features().\")\n", "\n", " s_len = len(s)\n", " # cut or pad a string\n", " if s_len < max_len:\n", " s = s + pad_char * (max_len - s_len)\n", " elif s_len > max_len:\n", " delta = (s_len - max_len) // 2\n", " s = s[delta : (max_len + delta)]\n", "\n", " return [ord(b) for b in s]\n", "\n", "def extract_features(s: str, pad_char: str = \"=\", max_len=20, max_token_num: int = 14, tokenizer=None):\n", " return (extract_old_features(s, max_length=max_token_num, tokenizer=tokenizer)\n", " + extract_new_features(s, max_len=max_len))\n", " \n", "\n", "def prepare_features(df, out_dir=None, max_len=20, nrows=None, save=False):\n", " print(f\"{df.shape=} {max_len=} {nrows=} \")\n", " df = df[:nrows].copy(deep=True)\n", " print(f\"Start with {df.shape[0]:,} samples\")\n", " df = df.dropna()\n", " print(f\"After dropna {df.shape[0]:,} samples\")\n", " tokenizer = DistilBertTokenizer.from_pretrained(\"models/tokenizer/\")\n", " df['features'] = df['domain'].apply(lambda d: extract_features(\n", " d, max_len=max_len,\n", " # max_token_num=max_token_num,\n", " tokenizer=tokenizer\n", " ))\n", " df['y'] = df['label'].apply(lambda d: 0 if d == 'bening' else 1)\n", " if save:\n", " out_file = f\"{out_dir}/data.csv\"\n", " df.to_csv(out_file, index=False)\n", " print(f\"Saved {df.shape[0]:,} {out_file}\") \n", " print(f\"Result {df.shape[0]:,}\")\n", " return df\n", "\n", "\n", "df = pd.read_csv('data/training_data/data.csv')\n", "print(f\"Loaded {df.shape} data/training_data/data.csv\")\n", "nrows = None\n", "max_len = 26\n", "out_dir = 'data/training_data/ensemble_features'\n", "df_res = prepare_features(df, out_dir=out_dir, max_len=max_len, nrows=nrows, save=False)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### training" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "time: 1.12 ms (started: 2022-09-26 11:37:08 -07:00)\n" ] } ], "source": [ "import numpy as np\n", "\n", "def cast_to_array(s):\n", " return np.array([int(el) for el in s[1:-1].split(\", \")])\n", "\n", "# s = \"[0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 121, 97, 114, 100, 122, 101, 110, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61]\"\n", "# cast_to_array(s)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((12310509, 4),\n", " domain label features y\n", " 0 yardzen bening [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 121... 0)" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "time: 1min 13s (started: 2022-09-26 11:37:11 -07:00)\n" ] } ], "source": [ "# Run it OLY if you didn't get the df_res in the \"preprocessing\" before!\n", "# To debug it decrease nrows!. You save time on training on a small dataset:\n", "in_file = \"data/training_data/ensemble_features/data.csv\"\n", "nrows = None\n", "df_res = pd.read_csv(in_file, nrows=nrows)\n", "df_res['features'] = df_res['features'].apply(cast_to_array)\n", "df_res.shape, df_res[:1]" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "train: 9,848,407, dev: 11,079,458, test: 12,310,509\n", "Start training...\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "07aa7a00ee1d4115ad63216406409c5a", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "<catboost.core.CatBoostClassifier at 0x7fd81043cb20>" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "time: 14min 7s (started: 2022-09-26 11:40:42 -07:00)\n" ] } ], "source": [ "size = df_res.shape[0]\n", "train_size, dev_size, test_size = int(size*0.8), int(size*0.9), size\n", "print(f\"train: {train_size:,}, dev: {dev_size:,}, test: {test_size:,}\")\n", "X_train, y_train = np.array(list(df_res.loc[0:train_size, 'features'])), df_res.loc[0:train_size, 'y'] \n", "X_dev, y_dev = np.array(list(df_res.loc[train_size:dev_size, 'features'])), df_res.loc[train_size:dev_size, 'y']\n", "# X_test, y_test, domain_test = np.array(list(df_res.loc[dev_size:, 'features'])), df_res.loc[dev_size:, 'y'], df_res.loc[size*2:size*3, 'domain']\n", "\n", "print(\"Start training...\")\n", "model = CatBoostClassifier(loss_function='MultiClass') # Logloss, MultiClass\n", "\n", "model.fit(X_train, y_train, eval_set=(X_dev, y_dev), verbose=False,\n", " plot=True)\n", "\n", "# train_size=799999, dev_size=899999, test_size=999999\n", "# time: 53.3 s (started: 2022-09-23 08:57:16 -07:00)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Results" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "support: 1231051\n", "average_precision_score: 0.977\n", "roc_auc_score: 0.985\n", "classification_report: \n", " precision recall f1-score support\n", "\n", " 0 0.95 0.97 0.96 775348\n", " 1 0.94 0.91 0.93 455703\n", "\n", " accuracy 0.95 1231051\n", " macro avg 0.95 0.94 0.94 1231051\n", "weighted avg 0.95 0.95 0.95 1231051\n", "\n", "time: 8.03 s (started: 2022-09-26 12:26:28 -07:00)\n" ] } ], "source": [ "X_test, y_test = np.array(list(df_res.loc[dev_size:, 'features'])), df_res.loc[dev_size:, 'y'] # , df_res.loc[size*2:size*3, 'domain']\n", "\n", "y_pred = model.predict_proba(X_test)\n", "y_scores = [y[1] for y in y_pred]\n", "\n", "results(y_test, y_scores, y_pred=None)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# support: 1231051\n", "# average_precision_score: 0.977\n", "# roc_auc_score: 0.985\n", "# classification_report: \n", "# precision recall f1-score support\n", "# 0 0.95 0.97 0.96 775348\n", "# 1 0.94 0.91 0.93 455703\n", "# accuracy 0.95 1231051\n", "# macro avg 0.95 0.94 0.94 1231051\n", "# weighted avg 0.95 0.95 0.95 1231051\n", "\n", "# support: 100000\n", "# average_precision_score: 0.976\n", "# roc_auc_score: 0.983\n", "# classification_report: \n", "# precision recall f1-score support\n", "# 0 0.94 0.97 0.96 62848\n", "# 1 0.94 0.90 0.92 37152\n", "# accuracy 0.94 100000\n", "# macro avg 0.94 0.94 0.94 100000\n", "# weighted avg 0.94 0.94 0.94 100000\n", "\n", "# support: 10000\n", "# average_precision_score: 0.971\n", "# roc_auc_score: 0.979\n", "# classification_report: \n", "# precision recall f1-score support\n", "# 0 0.93 0.96 0.95 6222\n", "# 1 0.94 0.89 0.91 3778\n", "# accuracy 0.94 10000\n", "# macro avg 0.94 0.93 0.93 10000\n", "# weighted avg 0.94 0.94 0.93 10000" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved models/catboost.0.977.26_ensemble.model\n", "time: 286 ms (started: 2022-09-26 12:28:09 -07:00)\n" ] } ], "source": [ "max_len = 26\n", "\n", "aps = average_precision_score(y_test, y_scores)\n", "out_file_name = f'models/catboost.{aps:.3}.{max_len}_ensemble.model'\n", "\n", "model.save_model(out_file_name)\n", "print(f'Saved {out_file_name}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Additional analysis" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### feature importance" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "(40,\n", " [1457,\n", " 1293,\n", " 311,\n", " 373,\n", " 215,\n", " 112,\n", " 75,\n", " 42,\n", " 33,\n", " 19,\n", " 11,\n", " 7,\n", " 3,\n", " 0,\n", " 816,\n", " 383,\n", " 328,\n", " 461,\n", " 351,\n", " 592,\n", " 507,\n", " 433,\n", " 433,\n", " 291,\n", " 285,\n", " 263,\n", " 156,\n", " 82,\n", " 50,\n", " 113,\n", " 122,\n", " 54,\n", " 10,\n", " 3,\n", " 20,\n", " 7,\n", " 5,\n", " 10,\n", " 20,\n", " 237],\n", " 9983)" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "time: 25 ms (started: 2022-09-26 15:16:28 -07:00)\n" ] } ], "source": [ "model_file = \"models/catboost.0.977.26_ensemble.model\"\n", "\n", "model = CatBoostClassifier().load_model(model_file)\n", "len(model.get_feature_importance()), [int(f*100) for f in model.get_feature_importance()], sum([int(f*100) for f in model.get_feature_importance()])" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }