Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KingfisherFilesStore: Write scrapyd-job.txt #534

Closed
jpmckinney opened this issue Oct 27, 2020 · 0 comments
Closed

KingfisherFilesStore: Write scrapyd-job.txt #534

jpmckinney opened this issue Oct 27, 2020 · 0 comments
Labels
framework-spiders Relating to common spider functionality
Milestone

Comments

@jpmckinney
Copy link
Member

Presently, it's a little complicated to reconcile data directories with log files, especially if multiple crawls are started for the same source in rapid succession. This is relevant to Kingfisher Archive.

If we write the spider's job ID to a scrapyd-job.txt in the crawl directory, this can be much easier. A _job kwarg is passed to the spider's initializer by Scrapyd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
framework-spiders Relating to common spider functionality
Projects
None yet
Development

No branches or pull requests

2 participants