zhihu-crawler:

A zhihu user information crawler, which will collect some useful message including username, education, profession, follower and folling count.

Technicial stack

• requests.
• Bloom Filter.
• XPath.
• rq.
• MySQL
• Anti crawler strategy

Setup

git clone https://github.com/cpselvis/zhihu-crawler.git
cd zhihu-crawler
pip install -r requirements.txt

# Run serveral workers and listen task queue.
python worker.py

# Begin task
python scheduler.py

Documentation

Fantastic documentation is available at WIKI.

How to Contribute

1. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.
2. Fork `the repository`_ on GitHub to start making your changes to the master branch (or branch off of it).
3. Write a test which shows that the bug was fixed or that the feature works as expected.
4. Send a pull request and bug the maintainer until it gets merged and published. :) Make sure to add yourself to AUTHORS_.