Data imports

[mekarpeles]

This is a followup of #5583 which should be merged first which updates the schema for the import_items table.

Closes first half of #5130

Allows manage-imports.py to add {"data": {...}} records in addition to {"ia_id": "..."}records to the https://openlibrary.org/admin/imports pipeline.

⚠️ Please squash

Testing

tty1

ssh -A ol-home0.us.archive.org
# Monitor the ImportBot container (w/ old code) to make sure it's not
# the one attempting to import records -- we want to manually start another
# `import-all` service from the container directly after we enqueue records
# so we can be confident that our new code is performing the imports:
sudo docker logs -f --tail 100 openlibrary_importbot_1

tty2

# Connect to db
ssh -A ol-db1.us.archive.org
sudo -upostgres psql
# Delete old records you want to re-import, e.g:
delete from import_item where data is NOT NULL;
# Confirm only the records you expect are in the db
select * from import_item where data is NOT NULL;

tty3:

ssh -A ol-home0.us.archive.org
# Connect to the cron container
sudo docker exec -uroot -it openlibrary_cron-jobs_1 bash
# rm the statefile (saving our progress processing & queuing data)
rm  /1/var/tmp/imports/2021-08/Bibliographic/2021-08-04/import.log
# re-run the test.py script to queue up a test batch of imports
python3 /olsystem/bin/bwb_etl/test.py /1/var/tmp/imports/2021-08/Bibliographic/*/bettworldbks*

tty4

ssh -A ol-home0.us.archive.org
# Connect to the ImportBot container
sudo docker exec -it -uroot openlibrary_importbot_1 bash
# Run the continuous import process
scripts/manage-imports.py --config /olsystem/etc/openlibrary.yml import-all

Go back to tty2

# ... Re-run this commend *after* you've enqueued items using test.py
select * from import_item where data is NOT NULL;

Stakeholders

[cdrini]

Mek's en route with the changes. Code lgtm after these. We tested doing the import flow with BWB and looks like it's running smoothly!

Might want to do a test that IA is still running smoothly.

Merge at your discretion.

[mekarpeles]

Tested w/ https://openlibrary.org/admin/imports/add