Your preferred focused crawler based on Venom. Now in a convenient package to quickly get your projects up to speed!
Bundled in this pack is a tutorial package to help get you up and sprinting. If you don't require a tutorial, you can access a fully functional example here. For more information, read the tutorial guide below.
Check out our main Venom page for more information.
Website | Wiki | API Reference | PreferredAI
venom-tutorial includes a set of tutorial designed to quickly get you from 0 to 100.
There are 7 exercises located in the package ai.preferred.crawler.example.tutorial. Alongside the exercises are a set of test suite that automatically checks your code and provide hints on errors, so you do not have to worry about not knowing whether your code works.
For more information you can visit our Wiki.
- Exercise 1: Creating a crawler with default settings.
- Exercise 2: Creating a fetcher that includes three (3) validators.
- Exercise 3: Creating a session store with PAPER_LIST_KEY.
- Exercise 4: Creating a crawler that uses a specified fetcher and session.
- Exercise 5: Creating a validator that validate a page.
- Exercise 6: Parsing the response from the crawl.
- Exercise 7: Putting it all together.
Easily find out what went wrong by running the tests included with the exercises.
You can run this command in the project folder
mvn test
Or use your IDE to run JUnit tests
