update Saskatchewan

[DrMeers]

Currently failing as per 8cb0f975e5#commitcomment-144230855 but I suspect that's not an issue with people.py?

[jpmckinney]

The addition of:

+ocd-division/country:ca/province:sk/ed:moose_jaw-lake_centre,Moose Jaw—Lake Centre,,1968-04-22,,,,,,,,,,,,1953-06-13
+ocd-division/country:ca/province:sk/ed:moose_jaw-lake_centre-1988,Moose Jaw—Lake Centre,,1997-04-26,,,,,,,,,,,,1988-10-01

in this commit, -- updated in 51d5dc18d99 to be:

+ocd-division/country:ca/province:sk/fed:moose_jaw-lake_centre,Moose Jaw—Lake Centre,,1968-04-22,,,,,,,,,,,,1953-06-13
+ocd-division/country:ca/province:sk/fed:moose_jaw-lake_centre-1988,Moose Jaw—Lake Centre,,1997-04-26,,,,,,,,,,,,1988-10-01

-- seems to be causing:

pupa.exceptions.DuplicateItemError: attempt to import data that would conflict with data already in the import: {'label': 'Moose Jaw—Lake Centre', 'role': 'MLA', 'organization_id': 'ocd-organization/5a1768b5-9d97-4cec-947c-a92abf18f6ac', 'division_id': 'ocd-division/country:ca/province:sk/fed:moose_jaw-lake_centre-1988', 'start_date': '', 'end_date': '', 'extras': {}} (already imported as MLA - Moose Jaw—Lake Centre - Legislative Assembly of Saskatchewan)

Would that be right @jpmckinney ?

I don't think that commit can affect the ca_sk scraper. ca_sk uses the identifiers in https://github.com/opencivicdata/ocd-division-ids/blob/master/identifiers/country-ca/province-sk-electoral_districts.csv which haven't changed in years.

[jpmckinney]

It succeeds on the deployment. Maybe pip install -U -r requirements.txt from the scrapers-ca directory, and also python manage.py loaddivisions --sync ca from the scrapers_ca_app directory.

[DrMeers]

I'm haven't been using scrapers_ca_app, only scrapers-ca within a docker container which seems to have been working in isolation until now

[jpmckinney]

You might have to pupa dbinit --reset ca (or maybe its the init command - check the help message).

[DrMeers]

--reset was failing with django.db.migrations.exceptions.InconsistentMigrationHistory: Migration elections.0001_initial is applied before its dependency core.0001_initial on database 'default' so I just started with a fresh postgres database and ran dbinit again, then tried sk again but got this:

  File "/app/src/pupa/pupa/importers/base.py", line 265, in import_item
    raise DuplicateItemError(data, obj, related.get('sources', []))
pupa.exceptions.DuplicateItemError: attempt to import data that would conflict with data already in the import: {'label': 'Meadow Lake', 'role': 'MLA', 'organization_id': 'ocd-organization/0e9ed7eb-1908-4f19-9b55-abee2453ae93', 'division_id': 'ocd-division/country:ca/province:sk/ed:20', 'start_date': '', 'end_date': '', 'extras': {}} (already imported as MLA - Meadow Lake - Legislative Assembly of Saskatchewan)

[jpmckinney]

I don't know. I can't replicate. http://scrapers.herokuapp.com works.

[DrMeers]

No worries, thanks for the tips regardless!

[jpmckinney]

If you can share the precise steps to replicate, I can maybe look into it