Skip to content

AravindRajeshkanna/arkrawler

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

34 Commits
.gitignore
.gitignore
 
 
README.md
README.md
 
 
crawler.js
crawler.js
 
 
index.js
index.js
 
 
package-lock.json
package-lock.json
 
 
package.json
package.json
 
 
seed.json
seed.json
 
 

Repository files navigation

WebCrawler

Crawls the web with a million seed URLs as a starting point

Initial Setup

  1. Install Node.js
  2. Install Redis
  3. Run 'npm install' command from the cloned repository in terminal
  4. Run 'npm start' to crawl the web

About

Node.js web crawler to crawl the entire web

Topics

nodejs javascript redis crawler algorithms web-crawler process data-structures web-crawling in-memory-database

Resources

Readme
Activity

Stars

3 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Languages