Tasks: Data maintenance

Tasks

Refer to the Troubleshooting page if you encounter errors.

Regularly

• Check the status and warnings for scrapers
  • Click once on "Last success" to sort broken scrapers to the top
  • Click on a row's "Exception" to see a backtrace to assess the complexity of the error
  • If a scraper has a warning symbol, you can click it to read the warnings
  • If a scraper has a green plus symbol, then you should add it to Represent via the Django admin
• Check the consistency of the database

Quarterly

Run the maintenance tasks of:

• ocd-division-ids
• scrapers-ca
• represent-canada-data

In scrapers_ca_app:

• Run ./manage.py mappings and ./manage.py checkmappings

Check this master spreadsheet for manual requests. If a cell in the Next boundary column is highlighted in yellow, then its shapefile is out-of-date and must be requested.

Annually

• Purchase postal code centroids and run loadpostcodes
• Check for updates to postal code concordances and run loadpostcodeconcordance

Expected output

Some of the maintenance tasks return output that is expected and requires no action.

invoke definitions

Richmond, NS districts                                       Expected slug to be Richmond districts not Richmond, NS districts
Paroisse de Plessisville districts                           Expected slug to be Plessisville districts not Paroisse de Plessisville districts

We need to disambiguate Richmond, NS from Richmond, QC and Paroisse de Plessisville, QC from Ville de Plessisville, QC.

ocd-division/country:ca/csd:2425213/borough:1                unknown name: check slug and domain manually
ocd-division/country:ca/csd:2425213/borough:2                unknown name: check slug and domain manually
ocd-division/country:ca/csd:2425213/borough:3                unknown name: check slug and domain manually

The identifiers don't yet exist in ocd-division-ids.

Hamilton wards                                               Non-unique values

Hamilton's licence_url is the same as its source_url.

Newfoundland and Labrador electoral districts (2006)         Expected LICENSE.txt to match "all rights reserved" template
Newfoundland and Labrador electoral districts                Expected LICENSE.txt to match "all rights reserved" template
Quebec electoral districts                                   Expected authority to be Her Majesty the Queen in Right of Quebec not Directeur général des élections du Québec

Newfoundland and Labrador and Quebec have non-standard licensing arrangements.

Bonnyville No. 87 wards                                      Expected slug to be Bonnyville No. 87 divisions not Bonnyville No. 87 wards

Bonnyville No. 87 has non-standard boundary names.

invoke urls

302 http://chathamkentopendata.chatham-kent.opendata.arcgis.com/datasets/1f4428dd5d764320b4246d190cfb70cb_0
404 https://www.geosask.ca/
404 https://www.geosask.ca/Portal/jsp/terms_popup.jsp

GeoSask and Chatham Kent's open data catalog no longer exist. Saskatchewan electoral districts are now available from Elections Saskatchewan but without a license, so we continue to attribute GeoSask.