seq2seq_intent_detection_slot_filling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型介绍\n",
    "\n",
    "![](https://github.com/applenob/RNN-for-Joint-NLU/raw/master/res/arc.png)\n",
    "\n",
    "形式化表达整理：\n",
    "\n",
    "- 输入序列：$x = (x_1,...x_T)$\n",
    "- 输出序列：$y = (y_1,...y_T)$，长度和$x$相同。\n",
    "- Encoder：时刻i，\n",
    "- 隐藏状态：$h_i = [fh_i, bh_i]$，前向状态+后向状态。\n",
    "- Decoder：时刻i，\n",
    "- 状态：$s_i$，$s_i = f(s_{i-1}, y_{i-1}, h_i, c_i)$\n",
    "- 其中，context向量：$c_i$，$c_i = \\sum^{T}_{j=1}\\alpha_{i,j}h_j$\n",
    "- attention参数：$\\alpha_{i,j} = \\frac{exp(e_{i,j})}{\\sum^T_{k=1}exp(e_{i,k})}$\n",
    "- $e_{i,k} = g(s_{i-1}, h_k)$\n",
    "- $g$是一个小型神经网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from data import *\n",
    "from my_metrics import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* http://www.isca-speech.org/archive/Interspeech_2016/pdfs/1352.PDF\n",
    "* https://arxiv.org/pdf/1409.0473.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data = open(\"dataset/atis-2.train.w-intent.iob\", \"r\").readlines()\n",
    "test_data = open(\"dataset/atis-2.dev.w-intent.iob\", \"r\").readlines()\n",
    "train_data_ed = data_pipeline(train_data)\n",
    "test_data_ed = data_pipeline(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(['i', 'want', 'to', 'fly', 'from', 'baltimore', 'to', 'dallas', 'round', 'trip', '<EOS>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>'], ['O', 'O', 'O', 'O', 'O', 'B-fromloc.city_name', 'O', 'B-toloc.city_name', 'B-round_trip', 'I-round_trip', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>'], 'atis_flight')\n"
     ]
    }
   ],
   "source": [
    "print(train_data_ed[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每行的训练数据是：[加padding后的输入，长度，加padding的标注，intent]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "word2index, index2word, slot2index, index2slot, intent2index, index2intent = \\\n",
    "        get_info_from_training_data(train_data_ed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "index_train = to_index(train_data_ed, word2index, slot2index, intent2index)\n",
    "index_test = to_index(test_data_ed, word2index, slot2index, intent2index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[88, 684, 157, 475, 721, 691, 157, 435, 312, 78, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 10, [2, 2, 2, 2, 2, 114, 2, 24, 34, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 16]\n"
     ]
    }
   ],
   "source": [
    "print(index_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_steps = 50\n",
    "embedding_size = 64\n",
    "hidden_size = 100\n",
    "n_layers = 2\n",
    "batch_size = 16\n",
    "vocab_size = 871\n",
    "slot_size = 122\n",
    "intent_size = 22\n",
    "epoch_num = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling\n",
    "\n",
    "模型实现。\n",
    "\n",
    "### Tensorflow的动态rnn\n",
    "\n",
    "`tf.nn.rnn creates an unrolled graph for a fixed RNN length. That means, if you call tf.nn.rnn with inputs having 200 time steps you are creating a static graph with 200 RNN steps. First, graph creation is slow. Second, you’re unable to pass in longer sequences (> 200) than you’ve originally specified.tf.nn.dynamic_rnn solves this. It uses a tf.While loop to dynamically construct the graph when it is executed. That means graph creation is faster and you can feed batches of variable size.`\n",
    "\n",
    "摘自[Whats the difference between tensorflow dynamic_rnn and rnn?](https://stackoverflow.com/questions/39734146/whats-the-difference-between-tensorflow-dynamic-rnn-and-rnn)。也就是说，静态的rnn必须提前将图展开，在执行的时候，图是固定的，并且最大长度有限制。而动态rnn可以在执行的时候，将图循环地的复用。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder_inputs = tf.placeholder(tf.int32, [input_steps, batch_size],\n",
    "                                             name='encoder_inputs')\n",
    "# 每句输入的实际长度，除了padding\n",
    "encoder_inputs_actual_length = tf.placeholder(tf.int32, [batch_size],\n",
    "                                                   name='encoder_inputs_actual_length')\n",
    "decoder_targets = tf.placeholder(tf.int32, [batch_size, input_steps],\n",
    "                                      name='decoder_targets')\n",
    "intent_targets = tf.placeholder(tf.int32, [batch_size],\n",
    "                                     name='intent_targets')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "embeddings = tf.Variable(tf.random_uniform([vocab_size, embedding_size],\n",
    "                                                        -0.1, 0.1), dtype=tf.float32, name=\"embedding\")\n",
    "\n",
    "encoder_inputs_embedded = tf.nn.embedding_lookup(embeddings, encoder_inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'embedding_lookup:0' shape=(50, 16, 64) dtype=float32>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoder_inputs_embedded"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.contrib.rnn import LSTMCell, LSTMStateTuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 使用单个LSTM cell\n",
    "encoder_f_cell = LSTMCell(hidden_size)\n",
    "encoder_b_cell = LSTMCell(hidden_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 下面四个变量的尺寸：T*B*D，T*B*D，B*D，B*D\n",
    "(encoder_fw_outputs, encoder_bw_outputs), (encoder_fw_final_state, encoder_bw_final_state) = \\\n",
    "    tf.nn.bidirectional_dynamic_rnn(cell_fw=encoder_f_cell,\n",
    "                                    cell_bw=encoder_b_cell,\n",
    "                                    inputs=encoder_inputs_embedded,\n",
    "                                    sequence_length=encoder_inputs_actual_length,\n",
    "                                    dtype=tf.float32, time_major=True)\n",
    "encoder_outputs = tf.concat((encoder_fw_outputs, encoder_bw_outputs), 2)\n",
    "\n",
    "encoder_final_state_c = tf.concat(\n",
    "    (encoder_fw_final_state.c, encoder_bw_final_state.c), 1)\n",
    "\n",
    "encoder_final_state_h = tf.concat(\n",
    "    (encoder_fw_final_state.h, encoder_bw_final_state.h), 1)\n",
    "\n",
    "encoder_final_state = LSTMStateTuple(\n",
    "    c=encoder_final_state_c,\n",
    "    h=encoder_final_state_h\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoder_outputs:  Tensor(\"concat:0\", shape=(50, 16, 200), dtype=float32)\n",
      "encoder_outputs[0]:  Tensor(\"strided_slice:0\", shape=(16, 200), dtype=float32)\n",
      "encoder_final_state_c:  Tensor(\"concat_1:0\", shape=(16, 200), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(\"encoder_outputs: \", encoder_outputs)\n",
    "print(\"encoder_outputs[0]: \", encoder_outputs[0])\n",
    "print(\"encoder_final_state_c: \", encoder_final_state_c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "decoder_lengths = encoder_inputs_actual_length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "slot_W = tf.Variable(tf.random_uniform([hidden_size * 2, slot_size], -1, 1),\n",
    "                             dtype=tf.float32, name=\"slot_W\")\n",
    "slot_b = tf.Variable(tf.zeros([slot_size]), dtype=tf.float32, name=\"slot_b\")\n",
    "intent_W = tf.Variable(tf.random_uniform([hidden_size * 2, intent_size], -0.1, 0.1),\n",
    "                               dtype=tf.float32, name=\"intent_W\")\n",
    "intent_b = tf.Variable(tf.zeros([intent_size]), dtype=tf.float32, name=\"intent_b\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 求intent\n",
    "intent_logits = tf.add(tf.matmul(encoder_final_state_h, intent_W), intent_b)\n",
    "intent = tf.argmax(intent_logits, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sos_time_slice = tf.ones([batch_size], dtype=tf.int32, name='SOS') * 2\n",
    "sos_step_embedded = tf.nn.embedding_lookup(embeddings, sos_time_slice)\n",
    "pad_step_embedded = tf.zeros([batch_size, hidden_size * 2 + embedding_size],\n",
    "                             dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 开始Hack\n",
    "\n",
    "像上面Encoder使用的那样，标准的`tf.nn.dynamic_rnn`需要提前将所有的输入都提前包装到一个tensor里传过去。\n",
    "\n",
    "当Decoder需要使用上一个时间节点的输出时，这就不可能提前包装好。即标准的动态rnn相当于：$s_i = f(s_{i-1}, x_i)$；但如果这个函数的参数需要扩充，比如我们做的：$s_i = f(s_{i-1}, y_{i-1}, h_i, c_i)$。\n",
    "\n",
    "于是我们需要Hack：使用`tf.contrib.seq2seq.CustomHelper`，传入三个函数：\n",
    "\n",
    "- `initial_fn()`：第一个时间点的输入。\n",
    "- `sample_fn()`：如何从logit到确定的某个固定的类别id。\n",
    "- `next_inputs_fn()`：确定一般的时间点的输入。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def initial_fn():\n",
    "    initial_elements_finished = (0 >= decoder_lengths)  # all False at the initial step\n",
    "    initial_input = tf.concat((sos_step_embedded, encoder_outputs[0]), 1)\n",
    "    return initial_elements_finished, initial_input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sample_fn(time, outputs, state):\n",
    "    # 选择logit最大的下标作为sample\n",
    "    prediction_id = tf.to_int32(tf.argmax(outputs, axis=1))\n",
    "    return prediction_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def next_inputs_fn(time, outputs, state, sample_ids):\n",
    "    # 上一个时间节点上的输出类别，获取embedding再作为下一个时间节点的输入\n",
    "    pred_embedding = tf.nn.embedding_lookup(embeddings, sample_ids)\n",
    "    # 输入是h_i+o_{i-1}+c_i\n",
    "    next_input = tf.concat((pred_embedding, encoder_outputs[time]), 1)\n",
    "    elements_finished = (time >= decoder_lengths)  # this operation produces boolean tensor of [batch_size]\n",
    "    all_finished = tf.reduce_all(elements_finished)  # -> boolean scalar\n",
    "    next_inputs = tf.cond(all_finished, lambda: pad_step_embedded, lambda: next_input)\n",
    "    next_state = state\n",
    "    return elements_finished, next_inputs, next_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 定义自己的helper\n",
    "my_helper = tf.contrib.seq2seq.CustomHelper(initial_fn, sample_fn, next_inputs_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def decode(helper, scope, reuse=None):\n",
    "    with tf.variable_scope(scope, reuse=reuse):\n",
    "        memory = tf.transpose(encoder_outputs, [1, 0, 2])\n",
    "        attention_mechanism = tf.contrib.seq2seq.BahdanauAttention(\n",
    "            num_units=hidden_size, memory=memory,\n",
    "            memory_sequence_length=encoder_inputs_actual_length)\n",
    "        cell = tf.contrib.rnn.LSTMCell(num_units=hidden_size * 2)\n",
    "        attn_cell = tf.contrib.seq2seq.AttentionWrapper(\n",
    "            cell, attention_mechanism, attention_layer_size=hidden_size)\n",
    "        out_cell = tf.contrib.rnn.OutputProjectionWrapper(\n",
    "            attn_cell, slot_size, reuse=reuse\n",
    "        )\n",
    "        decoder = tf.contrib.seq2seq.BasicDecoder(\n",
    "            cell=out_cell, helper=helper,\n",
    "            initial_state=out_cell.zero_state(\n",
    "                dtype=tf.float32, batch_size=batch_size))\n",
    "        # initial_state=encoder_final_state)\n",
    "        final_outputs, final_state, final_sequence_lengths = tf.contrib.seq2seq.dynamic_decode(\n",
    "            decoder=decoder, output_time_major=True,\n",
    "            impute_finished=True, maximum_iterations=input_steps\n",
    "        )\n",
    "        return final_outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "outputs:  BasicDecoderOutput(rnn_output=<tf.Tensor 'decode/decoder/TensorArrayStack/TensorArrayGatherV3:0' shape=(?, 16, 122) dtype=float32>, sample_id=<tf.Tensor 'decode/decoder/TensorArrayStack_1/TensorArrayGatherV3:0' shape=(?, 16) dtype=int32>)\n",
      "outputs.rnn_output:  Tensor(\"decode/decoder/TensorArrayStack/TensorArrayGatherV3:0\", shape=(?, 16, 122), dtype=float32)\n",
      "outputs.sample_id:  Tensor(\"decode/decoder/TensorArrayStack_1/TensorArrayGatherV3:0\", shape=(?, 16), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "outputs = decode(my_helper, 'decode')\n",
    "print(\"outputs: \", outputs)\n",
    "print(\"outputs.rnn_output: \", outputs.rnn_output)\n",
    "print(\"outputs.sample_id: \", outputs.sample_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意这里的输出的第一维依然是T，但已经不是之前定义的最大的50，而是当前batch的长度最大值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "decoder_targets_true_length:  Tensor(\"strided_slice_1:0\", shape=(?, 16), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "decoder_prediction = outputs.sample_id\n",
    "decoder_max_steps, decoder_batch_size, decoder_dim = tf.unstack(tf.shape(outputs.rnn_output))\n",
    "decoder_targets_time_majored = tf.transpose(decoder_targets, [1, 0])\n",
    "decoder_targets_true_length = decoder_targets_time_majored[:decoder_max_steps]\n",
    "print(\"decoder_targets_true_length: \", decoder_targets_true_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 定义mask，使padding不计入loss计算\n",
    "mask = tf.to_float(tf.not_equal(decoder_targets_true_length, 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 定义slot标注的损失\n",
    "loss_slot = tf.contrib.seq2seq.sequence_loss(\n",
    "    outputs.rnn_output, decoder_targets_true_length, weights=mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 定义intent分类的损失\n",
    "cross_entropy = tf.nn.softmax_cross_entropy_with_logits(\n",
    "    labels=tf.one_hot(intent_targets, depth=intent_size, dtype=tf.float32),\n",
    "    logits=intent_logits)\n",
    "loss_intent = tf.reduce_mean(cross_entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss = loss_slot + loss_intent\n",
    "optimizer = tf.train.AdamOptimizer(name=\"a_optimizer\")\n",
    "grads, vars = zip(*optimizer.compute_gradients(loss))\n",
    "gradients, _ = tf.clip_by_global_norm(grads, 5)  # clip gradients\n",
    "train_op = optimizer.apply_gradients(zip(grads, vars))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def step(sess, mode, trarin_batch):\n",
    "    \"\"\" perform each batch\"\"\"\n",
    "    if mode not in ['train', 'test']:\n",
    "        print >> sys.stderr, 'mode is not supported'\n",
    "        sys.exit(1)\n",
    "    unziped = list(zip(*trarin_batch))\n",
    "    if mode == 'train':\n",
    "        output_feeds = [train_op, loss, decoder_prediction,\n",
    "                        intent]\n",
    "        feed_dict = {encoder_inputs: np.transpose(unziped[0], [1, 0]),\n",
    "                     encoder_inputs_actual_length: unziped[1],\n",
    "                     decoder_targets: unziped[2],\n",
    "                     intent_targets: unziped[3]}\n",
    "    if mode in ['test']:\n",
    "        output_feeds = [decoder_prediction, intent]\n",
    "        feed_dict = {encoder_inputs: np.transpose(unziped[0], [1, 0]),\n",
    "                     encoder_inputs_actual_length: unziped[1]}\n",
    "\n",
    "    results = sess.run(output_feeds, feed_dict=feed_dict)\n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess = tf.Session()\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average train loss at epoch 0, step 0: 7.889627\n",
      "Average train loss at epoch 0, step 30: 4.827731\n",
      "Average train loss at epoch 0, step 60: 2.778382\n",
      "Average train loss at epoch 0, step 90: 2.676947\n",
      "Average train loss at epoch 0, step 120: 2.607563\n",
      "Average train loss at epoch 0, step 150: 2.514908\n",
      "Average train loss at epoch 0, step 180: 2.233277\n",
      "Average train loss at epoch 0, step 210: 2.042282\n",
      "Average train loss at epoch 0, step 240: 1.881606\n",
      "Average train loss at epoch 0, step 270: 1.732901\n",
      "[Epoch 0] Average train loss: 2.579982285431209\n",
      "Input Sentence        :  ['how', 'many', 'flights', 'does', 'each', 'airline', 'have', 'with', 'booking', 'class', '<UNK>', '<EOS>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Slot Truth            :  ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-fare_basis_code', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Slot Prediction       :  ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Intent Truth          :  atis_quantity\n",
      "Intent Prediction     :  atis_ground_service\n",
      "slot accuracy: 0.7548076923076923, intent accuracy: 0.625\n",
      "slot accuracy: 0.7754010695187166, intent accuracy: 0.75\n",
      "slot accuracy: 0.7409326424870466, intent accuracy: 0.9375\n",
      "slot accuracy: 0.7650602409638554, intent accuracy: 0.75\n",
      "slot accuracy: 0.8028169014084507, intent accuracy: 0.6875\n",
      "slot accuracy: 0.7452229299363057, intent accuracy: 0.75\n",
      "slot accuracy: 0.7808988764044944, intent accuracy: 0.75\n",
      "slot accuracy: 0.7696629213483146, intent accuracy: 0.75\n",
      "slot accuracy: 0.7314285714285714, intent accuracy: 0.9375\n",
      "slot accuracy: 0.7894736842105263, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8045977011494253, intent accuracy: 0.875\n",
      "slot accuracy: 0.7922705314009661, intent accuracy: 0.8125\n",
      "slot accuracy: 0.7243589743589743, intent accuracy: 0.8125\n",
      "slot accuracy: 0.7773109243697479, intent accuracy: 0.9375\n",
      "slot accuracy: 0.6951219512195121, intent accuracy: 0.8125\n",
      "slot accuracy: 0.7733990147783252, intent accuracy: 0.875\n",
      "slot accuracy: 0.7386934673366834, intent accuracy: 0.875\n",
      "slot accuracy: 0.7417582417582418, intent accuracy: 0.875\n",
      "slot accuracy: 0.7606382978723404, intent accuracy: 0.6875\n",
      "slot accuracy: 0.8032786885245902, intent accuracy: 0.75\n",
      "slot accuracy: 0.7643312101910829, intent accuracy: 0.875\n",
      "slot accuracy: 0.7287234042553191, intent accuracy: 0.875\n",
      "slot accuracy: 0.6911764705882353, intent accuracy: 0.875\n",
      "slot accuracy: 0.7468354430379747, intent accuracy: 0.8125\n",
      "slot accuracy: 0.8012422360248447, intent accuracy: 0.75\n",
      "slot accuracy: 0.7532467532467533, intent accuracy: 0.875\n",
      "slot accuracy: 0.735, intent accuracy: 0.875\n",
      "slot accuracy: 0.7110091743119266, intent accuracy: 0.8125\n",
      "slot accuracy: 0.7277486910994765, intent accuracy: 0.8125\n",
      "slot accuracy: 0.8445945945945946, intent accuracy: 0.8125\n",
      "slot accuracy: 0.8260869565217391, intent accuracy: 0.75\n",
      "F1 score for epoch 0: 0.7604240282685513\n",
      "Average train loss at epoch 1, step 0: 1.872046\n",
      "Average train loss at epoch 1, step 30: 1.535891\n",
      "Average train loss at epoch 1, step 60: 1.545106\n",
      "Average train loss at epoch 1, step 90: 1.415383\n",
      "Average train loss at epoch 1, step 120: 1.357902\n",
      "Average train loss at epoch 1, step 150: 1.303606\n",
      "Average train loss at epoch 1, step 180: 1.191032\n",
      "Average train loss at epoch 1, step 210: 1.262403\n",
      "Average train loss at epoch 1, step 240: 1.166604\n",
      "Average train loss at epoch 1, step 270: 1.122978\n",
      "[Epoch 1] Average train loss: 1.319355525423549\n",
      "Input Sentence        :  ['is', 'there', 'a', 'flight', 'in', 'the', 'afternoon', 'from', 'philadelphia', 'that', 'arrives', 'in', 'the', 'evening', 'in', 'denver', '<EOS>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Slot Truth            :  ['O', 'O', 'O', 'O', 'O', 'O', 'B-depart_time.period_of_day', 'O', 'B-fromloc.city_name', 'O', 'O', 'O', 'O', 'B-arrive_time.period_of_day', 'O', 'B-toloc.city_name', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Slot Prediction       :  ['O', 'O', 'O', 'O', 'O', 'B-toloc.city_name', 'O', 'O', 'B-toloc.city_name', 'O', 'O', 'O', 'B-stoploc.city_name', 'O', 'O', 'O', 'O', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Intent Truth          :  atis_flight\n",
      "Intent Prediction     :  atis_flight\n",
      "slot accuracy: 0.8041237113402062, intent accuracy: 0.8125\n",
      "slot accuracy: 0.7544642857142857, intent accuracy: 0.9375\n",
      "slot accuracy: 0.7292817679558011, intent accuracy: 0.875\n",
      "slot accuracy: 0.8, intent accuracy: 0.9375\n",
      "slot accuracy: 0.7891891891891892, intent accuracy: 0.875\n",
      "slot accuracy: 0.7828571428571428, intent accuracy: 0.9375\n",
      "slot accuracy: 0.7670454545454546, intent accuracy: 0.75\n",
      "slot accuracy: 0.8267326732673267, intent accuracy: 0.75\n",
      "slot accuracy: 0.8729281767955801, intent accuracy: 0.75\n",
      "slot accuracy: 0.8156424581005587, intent accuracy: 0.875\n",
      "slot accuracy: 0.8277777777777777, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8071065989847716, intent accuracy: 0.75\n",
      "slot accuracy: 0.7590361445783133, intent accuracy: 0.8125\n",
      "slot accuracy: 0.84, intent accuracy: 0.8125\n",
      "slot accuracy: 0.8176100628930818, intent accuracy: 0.9375\n",
      "slot accuracy: 0.785, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8647058823529412, intent accuracy: 0.875\n",
      "slot accuracy: 0.7921348314606742, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8727272727272727, intent accuracy: 0.875\n",
      "slot accuracy: 0.8115942028985508, intent accuracy: 0.875\n",
      "slot accuracy: 0.8831168831168831, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8758169934640523, intent accuracy: 0.8125\n",
      "slot accuracy: 0.7857142857142857, intent accuracy: 0.6875\n",
      "slot accuracy: 0.8038277511961722, intent accuracy: 0.875\n",
      "slot accuracy: 0.7823834196891192, intent accuracy: 0.875\n",
      "slot accuracy: 0.815028901734104, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8536585365853658, intent accuracy: 0.875\n",
      "slot accuracy: 0.8449197860962567, intent accuracy: 0.875\n",
      "slot accuracy: 0.8157894736842105, intent accuracy: 0.8125\n",
      "slot accuracy: 0.8218390804597702, intent accuracy: 1.0\n",
      "slot accuracy: 0.8362573099415205, intent accuracy: 0.875\n",
      "F1 score for epoch 1: 0.8131072248719308\n",
      "Average train loss at epoch 2, step 0: 1.483146\n",
      "Average train loss at epoch 2, step 30: 0.970039\n",
      "Average train loss at epoch 2, step 60: 0.904584\n",
      "Average train loss at epoch 2, step 90: 0.819873\n",
      "Average train loss at epoch 2, step 120: 0.877163\n",
      "Average train loss at epoch 2, step 150: 0.894660\n",
      "Average train loss at epoch 2, step 180: 0.784309\n",
      "Average train loss at epoch 2, step 210: 0.833733\n",
      "Average train loss at epoch 2, step 240: 0.773044\n",
      "Average train loss at epoch 2, step 270: 0.732293\n",
      "[Epoch 2] Average train loss: 0.8375709517241379\n",
      "Input Sentence        :  ['how', 'many', 'first', 'class', 'flights', 'does', 'delta', 'airlines', 'have', '<EOS>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Slot Truth            :  ['O', 'O', 'B-class_type', 'I-class_type', 'O', 'O', 'B-airline_name', 'I-airline_name', 'O', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Slot Prediction       :  ['O', 'O', 'B-class_type', 'I-class_type', 'O', 'O', 'B-airline_name', 'O', 'O', 'B-airline_name', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Intent Truth          :  atis_quantity\n",
      "Intent Prediction     :  atis_quantity\n",
      "slot accuracy: 0.8895705521472392, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8827586206896552, intent accuracy: 0.875\n",
      "slot accuracy: 0.8769230769230769, intent accuracy: 0.875\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "slot accuracy: 0.8784530386740331, intent accuracy: 1.0\n",
      "slot accuracy: 0.8831168831168831, intent accuracy: 0.875\n",
      "slot accuracy: 0.8333333333333334, intent accuracy: 0.75\n",
      "slot accuracy: 0.7752293577981652, intent accuracy: 0.75\n",
      "slot accuracy: 0.9263157894736842, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8771929824561403, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8777777777777778, intent accuracy: 0.9375\n",
      "slot accuracy: 0.892018779342723, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8813559322033898, intent accuracy: 1.0\n",
      "slot accuracy: 0.8846153846153846, intent accuracy: 1.0\n",
      "slot accuracy: 0.9030303030303031, intent accuracy: 0.875\n",
      "slot accuracy: 0.8697916666666666, intent accuracy: 0.875\n",
      "slot accuracy: 0.875, intent accuracy: 0.875\n",
      "slot accuracy: 0.8238341968911918, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9419354838709677, intent accuracy: 1.0\n",
      "slot accuracy: 0.8631578947368421, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8723404255319149, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8433734939759037, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9139784946236559, intent accuracy: 0.8125\n",
      "slot accuracy: 0.8628571428571429, intent accuracy: 0.875\n",
      "slot accuracy: 0.9113924050632911, intent accuracy: 0.75\n",
      "slot accuracy: 0.824468085106383, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8303030303030303, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8265895953757225, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8589743589743589, intent accuracy: 1.0\n",
      "slot accuracy: 0.8888888888888888, intent accuracy: 1.0\n",
      "slot accuracy: 0.8888888888888888, intent accuracy: 0.9375\n",
      "slot accuracy: 0.879746835443038, intent accuracy: 1.0\n",
      "F1 score for epoch 2: 0.8709448679312178\n",
      "Average train loss at epoch 3, step 0: 0.452591\n",
      "Average train loss at epoch 3, step 30: 0.623310\n",
      "Average train loss at epoch 3, step 60: 0.591932\n",
      "Average train loss at epoch 3, step 90: 0.586563\n",
      "Average train loss at epoch 3, step 120: 0.460913\n",
      "Average train loss at epoch 3, step 150: 0.529364\n",
      "Average train loss at epoch 3, step 180: 0.568908\n",
      "Average train loss at epoch 3, step 210: 0.550679\n",
      "Average train loss at epoch 3, step 240: 0.557029\n",
      "Average train loss at epoch 3, step 270: 0.506767\n",
      "[Epoch 3] Average train loss: 0.5504333976135459\n",
      "Input Sentence        :  ['could', 'i', 'have', 'a', 'list', 'of', 'flights', 'in', 'first', 'class', 'on', 'monday', 'from', 'san', 'francisco', 'to', 'pittsburgh', 'starting', 'at', 'noon', 'and', '<UNK>', '<EOS>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Slot Truth            :  ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-class_type', 'I-class_type', 'O', 'B-depart_date.day_name', 'O', 'B-fromloc.city_name', 'I-fromloc.city_name', 'O', 'B-toloc.city_name', 'O', 'O', 'B-depart_time.time', 'O', 'B-depart_time.time_relative', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Slot Prediction       :  ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-depart_time.time', 'B-depart_date.day_number', 'O', 'B-toloc.city_name', 'O', 'B-fromloc.city_name', 'I-fromloc.city_name', 'O', 'B-toloc.city_name', 'O', 'O', 'B-depart_time.time_relative', 'O', 'O', 'B-depart_time.period_of_day', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Intent Truth          :  atis_flight\n",
      "Intent Prediction     :  atis_flight\n",
      "slot accuracy: 0.8647058823529412, intent accuracy: 1.0\n",
      "slot accuracy: 0.88, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9425287356321839, intent accuracy: 0.875\n",
      "slot accuracy: 0.9452054794520548, intent accuracy: 1.0\n",
      "slot accuracy: 0.8984771573604061, intent accuracy: 0.8125\n",
      "slot accuracy: 0.9685534591194969, intent accuracy: 1.0\n",
      "slot accuracy: 0.8775510204081632, intent accuracy: 0.9375\n",
      "slot accuracy: 0.907103825136612, intent accuracy: 1.0\n",
      "slot accuracy: 0.9323671497584541, intent accuracy: 1.0\n",
      "slot accuracy: 0.9292929292929293, intent accuracy: 0.75\n",
      "slot accuracy: 0.8324022346368715, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9024390243902439, intent accuracy: 0.875\n",
      "slot accuracy: 0.8497109826589595, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9333333333333333, intent accuracy: 1.0\n",
      "slot accuracy: 0.8870056497175142, intent accuracy: 1.0\n",
      "slot accuracy: 0.9151515151515152, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9104477611940298, intent accuracy: 0.875\n",
      "slot accuracy: 0.9011627906976745, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9392265193370166, intent accuracy: 1.0\n",
      "slot accuracy: 0.898876404494382, intent accuracy: 0.875\n",
      "slot accuracy: 0.8994974874371859, intent accuracy: 1.0\n",
      "slot accuracy: 0.8988095238095238, intent accuracy: 1.0\n",
      "slot accuracy: 0.9430379746835443, intent accuracy: 1.0\n",
      "slot accuracy: 0.9139784946236559, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8888888888888888, intent accuracy: 1.0\n",
      "slot accuracy: 0.9213483146067416, intent accuracy: 0.875\n",
      "slot accuracy: 0.9010416666666666, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8719211822660099, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8686868686868687, intent accuracy: 0.875\n",
      "slot accuracy: 0.9383561643835616, intent accuracy: 0.9375\n",
      "slot accuracy: 0.8899521531100478, intent accuracy: 1.0\n",
      "F1 score for epoch 3: 0.9036315323294952\n",
      "Average train loss at epoch 4, step 0: 0.174314\n",
      "Average train loss at epoch 4, step 30: 0.409511\n",
      "Average train loss at epoch 4, step 60: 0.379571\n",
      "Average train loss at epoch 4, step 90: 0.458410\n",
      "Average train loss at epoch 4, step 120: 0.351397\n",
      "Average train loss at epoch 4, step 150: 0.416896\n",
      "Average train loss at epoch 4, step 180: 0.383937\n",
      "Average train loss at epoch 4, step 210: 0.400827\n",
      "Average train loss at epoch 4, step 240: 0.380572\n",
      "Average train loss at epoch 4, step 270: 0.344981\n",
      "[Epoch 4] Average train loss: 0.38854543325294305\n",
      "Input Sentence        :  [\"i'd\", 'like', 'a', 'cheap', 'flight', 'from', 'dallas', 'to', 'baltimore', 'on', 'january', 'first', '<EOS>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Slot Truth            :  ['O', 'O', 'O', 'B-flight_mod', 'O', 'O', 'B-fromloc.city_name', 'O', 'B-toloc.city_name', 'O', 'B-depart_date.month_name', 'B-depart_date.day_number', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Slot Prediction       :  ['O', 'O', 'O', 'B-cost_relative', 'O', 'O', 'B-fromloc.city_name', 'O', 'B-toloc.city_name', 'O', 'B-depart_date.month_name', 'B-depart_date.day_number', 'B-depart_date.year', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']\n",
      "Intent Truth          :  atis_flight\n",
      "Intent Prediction     :  atis_flight\n",
      "slot accuracy: 0.9294117647058824, intent accuracy: 0.875\n",
      "slot accuracy: 0.9444444444444444, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9106145251396648, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9590643274853801, intent accuracy: 0.875\n",
      "slot accuracy: 0.8556701030927835, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9285714285714286, intent accuracy: 1.0\n",
      "slot accuracy: 0.9166666666666666, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9269662921348315, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9053254437869822, intent accuracy: 1.0\n",
      "slot accuracy: 0.9235294117647059, intent accuracy: 0.875\n",
      "slot accuracy: 0.918918918918919, intent accuracy: 0.875\n",
      "slot accuracy: 0.8980582524271845, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9333333333333333, intent accuracy: 1.0\n",
      "slot accuracy: 0.9281767955801105, intent accuracy: 1.0\n",
      "slot accuracy: 0.9723756906077348, intent accuracy: 0.9375\n",
      "slot accuracy: 0.927536231884058, intent accuracy: 1.0\n",
      "slot accuracy: 0.9215686274509803, intent accuracy: 0.875\n",
      "slot accuracy: 0.9518072289156626, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9040404040404041, intent accuracy: 0.9375\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "slot accuracy: 0.9368932038834952, intent accuracy: 1.0\n",
      "slot accuracy: 0.9, intent accuracy: 0.9375\n",
      "slot accuracy: 0.9152542372881356, intent accuracy: 0.9375\n",
      "slot accuracy: 0.925, intent accuracy: 1.0\n",
      "slot accuracy: 0.9040404040404041, intent accuracy: 1.0\n",
      "slot accuracy: 0.9009433962264151, intent accuracy: 1.0\n",
      "slot accuracy: 0.9602272727272727, intent accuracy: 1.0\n",
      "slot accuracy: 0.9433962264150944, intent accuracy: 1.0\n",
      "slot accuracy: 0.8975609756097561, intent accuracy: 1.0\n",
      "slot accuracy: 0.9144385026737968, intent accuracy: 1.0\n",
      "slot accuracy: 0.9548022598870056, intent accuracy: 1.0\n",
      "slot accuracy: 0.9426751592356688, intent accuracy: 1.0\n",
      "F1 score for epoch 4: 0.9233899504600142\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(epoch_num):\n",
    "    mean_loss = 0.0\n",
    "    train_loss = 0.0\n",
    "    for i, batch in enumerate(getBatch(batch_size, index_train)):\n",
    "        # 执行一个batch的训练\n",
    "        _, loss_v, decoder_prediction_v, intent_v = step(sess, \"train\", batch)\n",
    "        mean_loss += loss_v\n",
    "        train_loss += loss_v\n",
    "        if i % 30 == 0:\n",
    "            if i > 0:\n",
    "                mean_loss = mean_loss / 30.0\n",
    "            print('Average train loss at epoch %d, step %d: %f' % (epoch, i, mean_loss))\n",
    "            mean_loss = 0\n",
    "    train_loss /= (i + 1)\n",
    "    print(\"[Epoch {}] Average train loss: {}\".format(epoch, train_loss))\n",
    "\n",
    "    # 每训一个epoch，测试一次\n",
    "    pred_slots = []\n",
    "    for j, batch in enumerate(getBatch(batch_size, index_test)):\n",
    "        decoder_prediction_v, intent_v = step(sess, \"test\", batch)\n",
    "        decoder_prediction_v = np.transpose(decoder_prediction_v, [1, 0])\n",
    "        if j == 0:\n",
    "            index = random.choice(range(len(batch)))\n",
    "            print(\"Input Sentence        : \", index_seq2word(batch[index][0], index2word))\n",
    "            print(\"Slot Truth            : \", index_seq2slot(batch[index][2], index2slot))\n",
    "            print(\"Slot Prediction       : \", index_seq2slot(decoder_prediction_v[index], index2slot))\n",
    "            print(\"Intent Truth          : \", index2intent[batch[index][3]])\n",
    "            print(\"Intent Prediction     : \", index2intent[intent_v[index]])\n",
    "        slot_pred_length = list(np.shape(decoder_prediction_v))[1]\n",
    "        pred_padded = np.lib.pad(decoder_prediction_v, ((0, 0), (0, input_steps-slot_pred_length)),\n",
    "                                 mode=\"constant\", constant_values=0)\n",
    "        pred_slots.append(pred_padded)\n",
    "        true_slot = np.array((list(zip(*batch))[2]))\n",
    "        true_length = np.array((list(zip(*batch))[1]))\n",
    "        true_slot = true_slot[:, :slot_pred_length]\n",
    "        slot_acc = accuracy_score(true_slot, decoder_prediction_v, true_length)\n",
    "        intent_acc = accuracy_score(list(zip(*batch))[3], intent_v)\n",
    "        print(\"slot accuracy: {}, intent accuracy: {}\".format(slot_acc, intent_acc))\n",
    "    pred_slots_a = np.vstack(pred_slots)\n",
    "    true_slots_a = np.array(list(zip(*index_test))[2])[:pred_slots_a.shape[0]]\n",
    "    print(\"F1 score for epoch {}: {}\".format(epoch, f1_for_sequence_batch(true_slots_a, pred_slots_a)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:py35]",
   "language": "python",
   "name": "conda-env-py35-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}