Distributed - Asynchronous - Easy-to-use
An Easy-to-use and Fast Python Spider Framework
- AirSpider is a high-performance asynchronous crawler framework for developers
- Based on Redis: task distribution, task deduplication, and distributed ☁️
- Python 3.6
- Works on Linux, Windows, macOS
- Quick to Start
- Low Coupling
- High Cohesion
- Easy Expansion
- Orderly Workflow
# For Linux && MacOS
pip3 install airspider
# For Windows
pip3 install airspider
- Introduction：Introduction to AirSpider
- Tutorials： Quick to program a Spider By AirSpider
- Plugins：Program extensions for AirSpider
- Item：Define Target in HTML
- Selector：Select Target from HTML
- Request：Request Part
- Response：Response Part
- Middleware：Make Airspider Support to Extensions
- Spider：Entrance for AirSpider
- Complete Plugins of Redis
- Complete Distributed Architecture
AirSpider🕷️ is still under Developing
Feel free to open issues💬 and pull requests
- Report or Fix bugs
- Build Powerful plugins
- Make documentation Better
- Add Examples of Spiders
