News Crawler

A repo for news crawling. Then combine similar news. Temporarily used python and Scrapy framework.
Used Jieba and Scrapy

Version 1.0 is the current directory
Version 2.0 is in the reconstruction directory

1.0版本的代码参见当前目录
2.0版本的代码参见目录reconstruction

TODO

1. Find a proper Chinese segmentation tool
2. Split a JSON file into small files, every file contains only one piece of news
3. Read some articles about SVM (Did not use SVM, but tfidf and cosin similarity)
4. Try to categorize different news
5. Build another IDF dictionary from web news
6. Categorize those similar passages in sina and netease but not in tencent
7. Make a website display the results and show the comments
8. Improve categorization performance
9. Make the website more beautiful!
10. Reconstruct this project with php

Website

http://news.reetsee.com