Scrapyd w/ docker is not persisting *.db file

[ivandir]

Hi,

I thought the *.db associated with my project was supposed to persist in my local docker volume but it seems scrapyd overrides the project sqlite3 db file. I have jobs that I haven't processed yet and when I restart the scrapyd instance, the jobs disappear (i.e no persistence) due to a new db file being created.

Is this the intended behavior on docker deployments?

[Digenis]

@ivandir,
scrapyd is expected to reuse the sqlite3 files when they already exist.

Did you configure a directory for the db files?
If you don't, scrapyd uses an in-memory database.

By jobs that haven't processed yet, you mean jobs waiting in the pending state, right?
Currently there's no feature to persist running or finished jobs.

By "restart the scrapyd instance" do you mean the whole container?

[santteegt]

Indeed, spider jobs don't persist after restarting the docker container. Here's my config file:

[scrapyd]
eggs_dir          = /var/lib/scrapyd/eggs
logs_dir          = /var/lib/scrapyd/logs
items_dir         = /var/lib/scrapyd/items
dbs_dir           = /var/lib/scrapyd/dbs
jobs_to_keep      = 5
max_proc          = 0
max_proc_per_cpu  = 4
finished_to_keep  = 100
poll_interval     = 5
bind_address      = 0.0.0.0
http_port         = 6800
debug             = off
runner            = scrapyd.runner
application       = scrapyd.app.application
launcher          = scrapyd.launcher.Launcher

[services]
schedule.json     = scrapyd.webservice.Schedule
cancel.json       = scrapyd.webservice.Cancel
addversion.json   = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json  = scrapyd.webservice.ListSpiders
delproject.json   = scrapyd.webservice.DeleteProject
delversion.json   = scrapyd.webservice.DeleteVersion
listjobs.json     = scrapyd.webservice.ListJobs
daemonstatus.json = scrapyd.webservice.DaemonStatus

And an example of how I'm deploying it using docker-compose:

scrapyd:
    build: ./scrapyd
    networks:
      default:
        ipv4_address: 10.255.4.15
    ports:
      - "6800:6800"
    volumes:
      - ./scrapyd_data:/var/lib/scrapyd
    restart: always

A file called <project_name>.db is created in the right directory, however, when restarting the container, all spider jobs disappear from the list

[ivandir]

Any updates on this ticket?

[Digenis]

The "db files" store spider queues.

What exactly disappears?
The queue of spiders not yet ran
or the list of finished spiders?

If it's the first and your docker volume mount point is correctly configured then it's a bug.
If it's the second, it's planned for scrapyd-1.3: #359

[jpmckinney]

This sounds like #359, not about Docker, so closing.