README

=====================================================================
                                                                       
                              ======                                   
                              README                                   
                              ======                                   
                                                                       
                          KEA as XML-RPC service 
                          4 August 2010

	KeaXML-RPC Service applicable to KEA 5.0 (http://www.nzdl.org/Kea/)

    Commissioned through vworker, coded by Ali Afshar (it.zvision@yahoo.com)

    Released under a BSD license.
                                                                       
=====================================================================

Contents:
---------

1. Installation 

2. Getting started

   - Running the server
   - Java Client Example
   - Python Client Example

3. History 

4. Compilation

5. How it is done


----------------------------------------------------------------------

NOTE:
-----

The service is packed and delivered as a patch

It contains two modules:

1) patch\main\KEAServer.java
   This is the server module. 

2) patch\filters\KEAFilter.java
	 This is a copy of the module kea\filters\KEAFilter.java from KEA 5.0 with only one difference 
	 in method, see lines 1339-1341: 
	 
	 private FastVector convertInstance(Instance instance, boolean training)
	 		....	
	 
	 		//double prob = probs[1]; 		//Commented out
	 		//If numeric class is used change to:
			double prob = probs[0];		//Uncommented
			....
			
		

The Service is developed in Java 5.0, on Eclipse under Win XP. 
It has been also tested on Windows XP and Windows 2000.  


----------------------------------------------------------------------

1. Installation:
----------------
Assumes kea-5.0_full is installed.

a) Download the KeaXML-RPC Service. 
	
b) Create a folder called patch in kea-5.0_full home directory (KEAHOME)
	 
		e.g. E:\programs\kea-5.0_full\patch
		
c) Unzip the KeaXML-RPC and copy the kea_server.jar into %KEAHOME%\patch\kea_server.jar

d) set KEAXMLRPC=%KEAHOME%\patch\kea_server.jar

e) Add %KEAXMLRPC% to your CLASSPATH environment variable.

		i.e. set CLASSPATH=%KEAXMLRPC%;%CLASSPATH%;  		Note KEAXMLRPC must come first 		

f) Download:
    http://ftp.us.xemacs.org/pub/mirrors/maven2/xmlrpc/xmlrpc/1.2-b1/xmlrpc-1.2-b1.jar
and add it also to your classpath.
	
----------------------------------------------------------------------

2. Getting started:
-------------------
Assume xmlrpc-1.2-b1.jar is included in your classpath

* Running the server

Type at command prompt:
		java patch.main.KEAServer <model> [port]   # defult port 8000 is used if not provided
		
		e.g. java patch.main.KEAServer path\to\my_model 8090
		
		If xmlrpc-1.2-b1.jar is not included then the command would be:
			
					java -cp %classpath%;path\to\xmlrpc-1.2-b1.jar; patch.main.KEAServer <model> [port]
					
					e.g. java -cp %classpath%;E:\tools\xmlrpc-1.2-b1\xmlrpc-1.2-b1.jar; patch.main.KEAServer C:\Temp\my_model 8090

* You can also use the Python server wrapper (port 8001) on top of the
Java server. If the Java KEA service throws an exception, the wrapper
returns an empty list of keywords. This appears to happen on weird
Unicode characters, in 0.3% of page text. YMMV.

    ./python-server-wrapper.py

Note that Python server wrapper doesn't seem to work under Pytohn 2.6. It does work under 2.4.

					

* Java Client Example
//////////////////////////////////////////////////////

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.util.Vector;

import org.apache.xmlrpc.XmlRpcClient;


public class KEAClient {
	 	    	
	 public static void main(String[] args) {
	     try {	        	
	        XmlRpcClient server = new XmlRpcClient("127.0.0.1", 8090);
	        System.out.println("Input document: "+args[0]);
	        Vector<String> v = new Vector<String>();	            
	        v.add(readFile(args[0])); 
	        Vector<String> keywords = (Vector<String>) server.execute("kea.extractKeyphrases", v);
	        System.out.println("\nmessage recieved: " + Arrays.toString(keywords.toArray()));
	     } catch (Exception e) {
	     		e.printStackTrace();
	        System.out.println(e.getMessage());
	     }
	 }
	 
	 
	 public static String readFile(String filename ) {		 
		 File f = new File(filename);
			StringBuilder contents = new StringBuilder();
			try {			
				BufferedReader input = new BufferedReader(new FileReader(f));
				try {
					String line = null; 
					while ((line = input.readLine()) != null) {
						contents.append(line);
						contents.append(System.getProperty("line.separator"));
					}
				} finally {
					input.close();
				}
			} catch (IOException ex) {
				ex.printStackTrace();
			}			
			return contents.toString();
		}
}

How to execute the client:
--------------------------
Assume xmlrpc-1.2-b1.jar is included in the classpath.
 
Compile and then run the client by typing at command prompt: 

	java KEAClient path\to\document\document.txt
		e.g.
		java  KEAClient E:\Temp\kea-5.0_full\my_testing\winrock_wi10ce.txt
	
	If xmlrpc-1.2-b1.jar is not included in your classpath:
	
		java -cp ./;E:\programs\tools\xmlrpc-1.2-b1\xmlrpc-1.2-b1.jar;	KEAClient E:\Temp\kea-5.0_full\my_testing\winrock_wi10ce.txt

/////////////////////////////////////////////////////


* Python Client Example, 

/////////////////////////////////////////////////////

import xmlrpclib

s = xmlrpclib.ServerProxy('http://127.0.0.1:8090')
txt = open("E:\\Temp\\iirr_ii02re.txt").read()
keywords = s.kea.extractKeyphrases(txt)
print keywords 

/////////////////////////////////////////////////////

Start python and import client.
  
----------------------------------------------------------------------

3. Further enhancement:
-----------------------
- eliminate hard coded init parameters, 
- give the clients the option of choosing different number of keywords and
- test it with the latest XML-RPC library.

-----------------------------------------------------------------------

4. Compilation:
---------------

javac patch/main/KEAServer.java

jar cvf0 kea_server.jar patch/main/KEAServer.class 


-----------------------------------------------------------------------

5. How it is done:
------------------

The service to be exposed is specified as follows: Given a (.txt) document and model, key phrases needed to be generated.

KEA 5.0 provides a function which with some modifications realizes this requirement. KEA extracts key phrases from the (.txt) documents in a given directory and puts these key phrases back into the same directory (corresponding files with extension “.key” are generated). 
Where and how KEA does this extraction was to be identified, after some investigation it turned out that the module kea.main.KEAKeyphraseExtractor.java is the starting point for this service. 

Assumptions:
	Training has taken place and a model is generated. 
	

Steps:

- Create a class in which the XML-RPC service is to be started, in this case it is called patch.main.KEAServer.java. 

- Identify the relevant methods from KEAKeyphraseExtractor and copy them into the patch.main.KEAServer. 
	
	Two methods are identified:
	public void loadModel() throws Exception
	public void extractKeyphrases(Hashtable stems) throws Exception 
	
	

- Start with initializations and modifications
	First we need to do some initializations including document language, encoding, vocabulary etc. 
The next task is loading the model, the model is the only input parameter (the format is drive:\path\to\modelName) to kea\filters\KEAFilter.java. 

	There is an instance of kea.main.KEAKeyphraseExtractor to handle the initialization parameters and an instance of to patch.filters.KEAFilter to read/hold the model.

	The constructor of the KEAServer.java is where we handle these two issues. 

	public KEAServer(String model) {			
		setOptions();
		try {
			loadModel(model);
			m_KEAFilter.setDebug(true);			
			myLogger.info("Loaded model " +model);
		} catch (Exception e) {			
			myLogger.log(Level.SEVERE, "Failed loading model {0} , exception: {1}", new Object[]{model, e.toString()});
		}
	}
	
	In the modified method loadModel(String model) we read the model into an instance of patch.filters.KEAFilter. 
	This filter converts the incoming data into data appropriate for keyword classification, that is it creates a 
	set of weka.core.Instance instances for every document. 
	The only change we need to make here is at lines 1339 and 1341, (see kea\filters\KEAFilter.java)

			1339: //double prob = probs[1];		//Commented
			1340: // If numeric class is used change to:
			1341: double prob = probs[0];			//Uncommented


- Now we are ready to instantiate the XML-RPC service and prepare for incoming requests. 
	The main method is where we do this, see patch\main\KEAServer.java 

		…			
		try {
			System.out.println("Starting server on port: " + port);
			patch.main.KEAServer kea = new patch.main.KEAServer ( model );
			org.apache.xmlrpc.WebServer server = new org.apache.xmlrpc.WebServer ( port );			
			//server.setParanoid(true);
			server.acceptClient(localhost);		// add hosts here
			server.addHandler("kea", kea);
			server.start();
			System.out.println("Server started ");
		} catch (Exception e) {
			System.err.println(e.getMessage());
			e.printStackTrace();
			System.exit(0);
		}


- The other important method in kea.main.KEAKeyphraseExtractor was: 
 		public void extractKeyphrases(Hashtable stems)      (see line 69 in kea\main\KEAKeyphraseExtractor.java)

	We need to make modification so that the method returns the key phrases given a string (representing a .txt document). 

	The corresponding method in class patch.main.KEAServer would look like:

 	private List<weka.core.Instance> myExtractKeyphrases ( String document )
 
	That is, given a string "document" a list of weka.core.Instance is returned. 
	
	Next, in the method  
		public String[] extractKeyphrases ( String document ) 
	
	the list of Instances are parsed and a list of keywords is delivered to the client.