implement job persistance

[mxdev88]

Hello,

Could someone pls review this PR about #12?

Thank you

[codecov]

Codecov Report

Merging #359 (9f04d35) into master (f916d9b) will increase coverage by 2.65%.
The diff coverage is 96.66%.

❗ Current head 9f04d35 differs from pull request most recent head efbdede. Consider uploading reports for the commit efbdede to get more accurate results

@@            Coverage Diff             @@
##           master     #359      +/-   ##
==========================================
+ Coverage   68.66%   71.32%   +2.65%     
==========================================
  Files          17       18       +1     
  Lines         868      952      +84     
  Branches      104      113       +9     
==========================================
+ Hits          596      679      +83     
  Misses        241      241              
- Partials       31       32       +1     

Impacted Files	Coverage Δ
scrapyd/launcher.py	43.58% <66.66%> (-0.17%)	⬇️
scrapyd/jobstorage.py	95.65% <95.65%> (ø)
scrapyd/app.py	88.13% <100.00%> (+1.34%)	⬆️
scrapyd/interfaces.py	100.00% <100.00%> (ø)
scrapyd/sqlite.py	91.24% <100.00%> (+2.35%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f916d9b...efbdede. Read the comment docs.

[mxdev88]

Anyone to take a first look? @Digenis?

[Digenis]

Hi, @mxdev88,
thank you for your contribution.

I don't have much time to maintain scrapyd nowadays.
I only took a quick look.

The travis error can be ignored, it's #359.
Overall it seems cleanly implemented
and I guess it will not complicate merging the queues.

[mxdev88]

Hi @Digenis,
Great - thanks for the feedback. Let me know if anything else needed prior to merging.

[sluceno]

Not an expert on python but it looks good to me.

I will take this as an example for doing similar thing with the Queue so we can have different backends for queueing jobs

[skykery]

Hope this feature will be merged someday.

[markkdev]

Would be awesome to have this feature

[mxdev88]

@Digenis do you think this one could make 1.3 alpha release mentioned in #389 also?

[Digenis]

I added the 1.3 milestone although I haven't estimated the time to review yet.

I see there is merge conflict but an easy to resolve.

You'll probably want to rebase on master after #398 to get a working CI.

[Digenis]

@mxdev88 you can now rebase on master.
The conflicts are only precautions
because of the proximity of the changed lines.

[mxdev88]

Hi @Digenis should be ok to merge now. thks

[polajenko]

When can this feature be released?

[pawelmhm]

This looks really useful, there are unit tests, can you merge recent master branch here and rebuild? Also: finished_to_keep setting seems not documented, worth adding to docs.

edit:

it is in docs already sorry, https://scrapyd.readthedocs.io/en/latest/config.html#finished-to-keep

[pawelmhm]

default is actually:

scrapyd.jobstorage.MemoryJobStorage

let's put full names here:

scrapyd.jobstorage.MemoryJobStorage
and scrapyd.jobstorage.SqliteJobStorage

[pawelmhm]

I integrated with current master and fixed docs here:#418

[pawelmhm]

related PR merged, commit in master now