You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As the title describes, I was wondering if we could implement something like this?
The use case I have is that if a spider is crawling a site with lots of different pages with the crawl logic scattered into the different parts of the code, manually declaring the save_html in the meta might be tedious. I'm proposing of a way that perhaps if the middleware finds a certain pattern in the URL, it will save that HTML.
The text was updated successfully, but these errors were encountered:
BurnzZ
changed the title
Add configuration for asving HTML when URL meets a certain pattern
Add configuration for saving HTML when URL meets a certain pattern
Dec 4, 2019
Hello @povilasb!
As the title describes, I was wondering if we could implement something like this?
The use case I have is that if a spider is crawling a site with lots of different pages with the crawl logic scattered into the different parts of the code, manually declaring the
save_html
in themeta
might be tedious. I'm proposing of a way that perhaps if the middleware finds a certain pattern in the URL, it will save that HTML.For example:
Let me know what you think and I could send a PR.
Cheers!
The text was updated successfully, but these errors were encountered: