information_extraction

An information extraction task with NLTK, Stanford NER and Spacy.
Only Python3 is supported.

Project Structures

data

contains data and labeled data

jars

stanford jar and model files

dep

dependency module

eval

evaluation module

ner

ner module

utils

utilities

NER task

In this repo, there are four ways to do NER tasks.

• nltk (nltk required)
  python main_ner.py data/task nltk

• nltk enhancement with combining continuous named entities.
  python main_ner.py data/task enltk

• spacy (spacy required) Download spacy model first.
  python -m spacy download en
python main_ner.py data/task spacy

• Stanford NER (Java 8 required)
  export STANFORD_MODE="jars/english.all.3class.distsim.crf.ser.gz"
export STANFORD_JAR="jars/stanford-corenlp-3.9.1.jar"
python main_ner.py data/task stanford

NER evaluation

There is one dataset to evaluate the performance of ners.

• Evaluate nltk
  python main_eval.py data/conll2003.txt nltk

• Evaluate spacy
  python main_eval.py data/conll2003.txt spacy

• Evaluate stanford
  export STANFORD_MODE="jars/english.all.3class.distsim.crf.ser.gz"
export STANFORD_JAR="jars/stanford-corenlp-3.9.1.jar"
python main_eval.py data/conll2003.txt stanford

TODO

Dependency analysis, extract verbs related to named entities.