RSSMiner

• Rssminer - a simple, intelligent RSS reader
• Live version http://rssminer.net
• Write in Clojure & Javascript & Java, by shen feng

Features

• Build for readability
• Realtime, instant fulltext search
• Realtime, personalized recommendation based on reading history
• Google Chrome plugin to add subscription
• Clean and compact code
• Super fast (In order to be fast, I write the Web server, Database Connection Pool, Chinese segmentation, Template System, Some JS libs from scratch. I save every bit to make it faster. The server can handle thousands of request per seconds(Including fulltext search) with very low latency.

Technologies

Server-side

• Java. Clojure.
• Apache Lucene, use it to do fulltext, realtime, instant search.
• Ring, compojure, easier http.
• http-kit, super fast event driven HTTP server and HTTP client. Especially written for Rssminer.
• dbcp, Simple database connection pool. Especially written for Rssminer
• mmseg, A java implementation of MMSEG. Especially written for Rssminer
• MySQL, data store
• Redis, Message Queue; Per user per feed score Store. Proxy cache.
• Mustache.clj, Mustache for Clojure

Client-side

• jQuery, nicer API
• Underscore, be functional
• sass, fix css
• mustache.js, Mustache for JS

How to run it

Install Instructions

1. Install Dependencies leiningen, JDK7(JDK6 works), Redis, MySQL, sass, rake

2. clone this repo, install dependency

git clone git://github.com/shenfeng/rssminer.git && cd rssminer && lein deps

3. Initialize database, create user, import schema

cd rssminer && ./scripts/admin init-db

4. Run it

# make sure /var/rssminer/index dir exits and is writable. refer command line args for more info
rake run:dev # run server in dev profile, view it: http://127.0.0.1:9090

Run unit test

rake test

rake mysql_dev # replace my.cnf will a dev one, run mysql in /tmp, run it after understand it.

It will create/drop a temp MySQL database for each test. If MySQL's db path is in tmpfs, It will much faster(12s vs 40+s).

Command line args

rake prepare javac && ./scripts/run --help  # rake -T list commands

Usage:

 Switches                 Default                          Desc
 --------                 -------                          ----
 -p, --port               9090                             Port to listen
 --worker                 2                                Http worker thread count
 --fetcher-concurrency    10
 --fetch-size             20                               Bulk fetch size
 --profile                :dev                             dev or prod
 --redis-host             127.0.0.1                        Redis host
 --redis-port             6379                             Redis port
 --static-server          //192.168.1.200                  static server
 --db-url                 jdbc:mysql://localhost/rssminer  MySQL Database url
 --db-user                feng                             MySQL user name
 --db-pass                                                 MySQL password
 --bind-ip                0.0.0.0                          Which ip to bind
 --events-threshold       20                               How many user feed events buffered before recompute again
 --index-path             /var/rssminer/index              Path to store lucene index
 --no-fetcher, --fetcher  false                            Start rss fetcher
 --no-proxy, --proxy      false                            Enable Socks proxy
 --no-help, --help        false                            Print this help

Copyright

Copyright (c) Feng Shen[email protected]. All rights reserved.