Hybrid Generative Model for learning Phrase Representative human like conversation -- Chat_Agent

Asian University Machine Learning Camp - Jeju 2018

Overview

Basically,many emergency services,service company and ecommerce companies require helplines and people. But the catch here is… Emergency disaster /natural disaster requires people to not only assure them help but also to guide them, pacify them, and at the same time provide back-up to the police or doctors with their current locations. Now can all the police/doctors/people be available all the days for 24hrs???No! It’s not feasible, right? hence, I came up with this solution.How about an AI assistant like google or Alexa. An AI which could exhibit emotions and can response to consult as well as console people with provided service specific answers along with basic knowledge and intelligence of a human. Now this should pass the turning test of AI i.e. we all know an AI is the best if it can pass the turning test, that is a human will interact with it not knowing that he is interacting with an AI and gets convinced that he is speaking or chatting to a human.

Table of Contents

• Overview
• Introduction
  • Chat Agent
  • Practical Demand meets solution
• Motivation
• Goal
• Prerequisite
• Installation
• Graphical Representation
• Create Chat Agent
• Getting Dataset
• Chat Data Structure
• Chat Buffer and Insert Logic
• Building Database
• Training the model
• Interacting with Chat Agent

Introduction

Chat Agent?

Design of codes through machine learning’s NLP AND ARTIFICIAL INTELLIGENCE THAT COULD BE A GENERIC INFORMATION RETRIVAL TO SPECIFIC INFORMATION

Level 1 : generic information retrieval fused with domain specific

Level 2 : generic information retrieval fused with guided information direction

Level 3 : generic information retrieval master artificial intelligence.

Practical Demand meets solution

Basically, how many emergency services, commercial services, service company and ecommerce companies require helplines and people??? Can they work 24hrs??? Let me ask you another question… How many first aid services, emergency services, helplines are there??? There are many, right!! But the catch here is… Emergency disaster /natural disaster requires people to not only assure them help but also to guide them, pacify them, and at the same time provide back-up to the police or doctors with their current locations. Now can all the police/doctors/people be available all the days for 24hrs??? No! It’s not feasible, right? hence, I came up with this solution.

I am not talking about AI dominating world or dominating even jobs of people. But how about an AI assistant like google or Alexa. An AI which could exhibit emotions and can response to consult as well as console people with provided service specific answers along with basic knowledge and intelligence of a human. Now this should pass the turning test of AI i.e. we all know an AI is the best if it can pass the turing test, that is a human will interact with it not knowing that he is interacting with an AI and gets convinced that he is speaking or chatting to a human.

Motivation

Research on data handling and sql and retrieval 21st century – bigdata and nosql – basically information retrieval – putting data into structured, semi-structured and unstructured data World is shifting to new CONCEPTS - a new technology of search and retrieval style of working.

Goal

Generic information Retrieval Master Artificial Intelligence.

Prerequisite

1. python 3.6
2. pip version 9

Installation

1. tqdm
2. colorama
3. regex
4. Python-Levenshtein
5. requests

downloading can be done from the requirements file uploaded using the command pip install -r requirements.txt

Getting Dataset

Dataset can be downloaded from https://www.reddit.com/r/bigquery/comments/3cej2b/17_billion_reddit_comments_loaded_on_bigquery/?st=j9xmvats&sh=5843d18e

Graphical Representation

Step 1: WHAT DO WE DO when we have some data for ingestion???�we leverage nlp to make our own dataset for "the end better dataset to train"

Step 2: we use nlp concepts in step by step order for better Results And prediction corpus building

Step 3: we focus ON THE FOLLOWING to build a chatbot

Step 4:Lexical analysis to break down THE SENTENCE to WORDS AND keep only meaningful And parse it to next…..

Step 4b: syntactic analysis is to make sense of what the words mean, then sort and order them to make sense for everyone, as well as help the machine learning to be sensible

Step 5:our model is not only step by step, but a fuse of mixing in enumeration style, thus we get more efficient result

Step 6: what parameters do we use ???We combine into next phase of concepts, entities and keywords

Step 7: forming our final search, like to be analyzed in neural networks training

Step 8: Finally to make sense, not only to us but also, help in fuse corpus building in advance form in starting so... training of understanding will be better.. once nlp completes and pass to neural networks

Step 9: the neural networks forms a style of question and answer on its own...as we make sense to its understanding

Step 9b: Look!! how a sentence breaks down to understand and reply

Step 9c:Using our style of break down to further answer close to human

Step 9d:When passed to neural networks to train.. it self scores all the answer And relevant answers it streams down to pull the right answer based on score

Step 10:Final Result

Create Chat Agent

Chat Data Structure

There is alot of unwanted information in the data we downloaded. So it must be cleaned. Thus, we perform data preprocessing
https://github.com/YangShyrMing/Chat_Agent/blob/master/t.PNG

Chat Buffer and Insert Logic

Building Database

Training the model

We will get a set of question and answers in the form of 2 text files as text.to and text.from

Interacting with Chat Agent

References

https://medium.com/@erikhallstrm/hello-world-tensorflow-649b15aed18c

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://karpathy.github.io/2015/05/21/rnn-effectiveness/