You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Presently, it's a little complicated to reconcile data directories with log files, especially if multiple crawls are started for the same source in rapid succession. This is relevant to Kingfisher Archive.
If we write the spider's job ID to a scrapyd-job.txt in the crawl directory, this can be much easier. A _job kwarg is passed to the spider's initializer by Scrapyd.
The text was updated successfully, but these errors were encountered:
Presently, it's a little complicated to reconcile data directories with log files, especially if multiple crawls are started for the same source in rapid succession. This is relevant to Kingfisher Archive.
If we write the spider's job ID to a scrapyd-job.txt in the crawl directory, this can be much easier. A
_job
kwarg is passed to the spider's initializer by Scrapyd.The text was updated successfully, but these errors were encountered: