Check Kingfisher Collect log file

[duncandewhurst]

Related to open-contracting/kingfisher-collect#917 (comment)

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[duncandewhurst]

@jpmckinney anything else I should be using from scrapyloganalyzer here? Currently, I'm just using ScrapyLogFile.logparser.

[jpmckinney]

You probably want to use:

• is_finished()
• is_complete()
• error_rate
• item_counts for 'File' and 'FileError'

Note that error_rate is just the ratio of FileError items to File + FileError items.

I don't think you need to use logparser itself.

[duncandewhurst]

From open-contracting/kingfisher-collect#917 (comment):

Kingfisher Collect (unless there's a bug in a spider) always yields a FileError item if any URL ultimately fails (retries don't yield, but intermediary URLs do). Kingfisher Process stores FileErrors in the DB. So, they should have appeared in the DB.

As such, let's hold fire on this PR until the new version of Kingfisher Process has been deployed, after which we'll be able to get the log URL from the database. Otherwise, I don't think looking at error_rate and item_counts for 'File' and 'FileError' adds anything over the existing step of looking in collection_file.

[jpmckinney]

This PR's process is more robust, but Process should be storing the errors, yes. It's possible that the old version of the spider wasn't yielding FileErrors correctly. The current version does.

[jpmckinney]

Noting that the current file was deleted, but this PR could still be useful. Nevermind, the file is restored.

[jpmckinney]

I reverted the pre-commit commits, to avoid merge conflicts, until this PR is ready.