This Django project runs the Canadian legislative scrapers, displays the status of each scraper, and returns the scraped data as JSON.
Follow the instructions in the Python Quick Start Guide to install Homebrew, Git, PostgreSQL, Python and virtualenv.
mkvirtualenv scrapers_ca_app
git clone [email protected]:opennorth/scrapers_ca_app.git
cd scrapers_ca_app
Set up the submodule and switch it to master:
git submodule init
git submodule update
cd scrapers
git checkout master
cd ..
Install the requirements:
pip install -r requirements.txt
Create a database (dropdb pupa if it already exists):
dropdb pupa
createdb pupa
python manage.py migrate --noinput
pupa dbinit ca
Run all the scrapers:
python manage.py update
Or run specific scrapers:
python manage.py update ca_ab_edmonton ca_ab_grande_prairie_county_no_1
Install the foreman gem:
gem install foreman
Start the web app:
foreman start
heroku apps:create
Add configuration variables (replace REPLACE):
heroku config:set PRODUCTION=1
heroku config:set SSL_VERIFY=1
heroku config:set AWS_ACCESS_KEY_ID=REPLACE
heroku config:set AWS_SECRET_ACCESS_KEY=REPLACE
heroku config:set DJANGO_SECRET_KEY=REPLACE
heroku config:set DATABASE_URL=`heroku config:get REPLACE`
You can generate a secret key in Python:
from django.utils.crypto import get_random_string
get_random_string(50, 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)')You'll need a production tier PostgreSQL database to use PostGIS (replace DATABASE):
heroku addons:add heroku-postgresql:standard-0
heroku pg:wait
heroku pg:promote DATABASE
heroku addons:remove heroku-postgresql:dev
heroku pg:psql
In the PostgreSQL shell, run:
CREATE EXTENSION postgis;
You'll need the geo buildpack for GeoDjango:
heroku buildpacks:set https://github.com/cyberdelia/heroku-geo-buildpack.git
heroku buildpacks:add heroku/python
Setup the database (replace DATABASE):
heroku pg:reset DATABASE
heroku run pupa dbinit ca
heroku run python manage.py migrate --noinput
Add to the Heroku Scheduler:
python manage.py update
python manage.py upload
heroku run python manage.py update my_scraper
heroku run python manage.py loaddivisions ca
Or:
heroku run python manage.py loaddivisions --sync ca
python manage.py check
If a scraper creates duplicates, you may need to:
python manage.py flush MODULE_NAME
Make sure PostgreSQL is running. If you use Homebrew, you can find instructions with:
brew info postgres
This repository is on GitHub: https://github.com/opennorth/scrapers_ca_app, where your contributions, forks, bug reports, feature requests, and feedback are greatly welcomed.
Copyright (c) 2013 Open North Inc., released under the MIT license