Skip to content

Latest commit

 

History

History

web-crawler

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 

Web crawler

Scraping of websites like HN/Reddit/Lobsters/..

Everything described in this issue.

It should at some interval scrape all the websites + use similar techniques to Google/Yandex to index the web with crawlers.

Combining it with what Perplexity is doing to fine tune GPTs with real time data access.