Writes content to disk, even for cache hits when "Use CrawlCache" is ON

[utchy]

Suggestion to improve crawling job.

• Skip writing content to reduce I/O write if cache exists.
• Make cache only if status code is not 404.
• Delete crawl queue from database to prevent crawling urls again, which are not found.
• Delete previously generated files under the directories both the crawled and the processed.

FYI

I'm preparing a pull request to fix this issue.

[thecodeassassin]

@utchy even with your changes applied CrawlCache is still not working. Even if I update a single posts it crawls ALL pages

[utchy]

@thecodeassassin
Thanks for your comment. Ah, yes the plugin actually always crawls even with applying my changes, but never writes again. Before the changes, the plugin always crawls and "writes" on disk. So, please check your file modified date under wp2static-crawled-site directory if pages (and related pages) which you updated are only modified. Cheers.