Councilmatic Starter Template

This repo provides starter code and documentation for new Councilmatic instances.

About Councilmatic

The councilmatic family is a set of web apps for keeping tabs on city representatives and their legislative activity.

Councilmatic started as a Code for America project by Mjumbe Poe, who designed the earliest version of a Councilmatic site – Councilmatic Philadelphia. DataMade then implemented Councilmatic in New York City (in partnership with the Participatory Politics Foundation), Chicago (in partnership with the Sunlight foundation), and Los Angeles (in partnership with the Los Angeles County Metropolitan Transportation Authority).

To simplify redeployment of Councilmatic instances, DataMade identified two necessities: (1) Open Civic Data, a data standard for describing people, organizations, events, and bills, and (2) the abstraciton of core functionality into a separate app. And so, DataMade built django-councilmatic – a django app with base functionality, common across all cities, which implements the OCD data standard.

With this template, you can create a Councilmatic site for your own city, built using the django-councilmatic app. Read on!

Getting started

1. Install OS Level dependencies

• Python 3.4+
• PostgreSQL 9.4+

2. Clone this project

git clone https://github.com/datamade/councilmatic-starter-template.git yourcity_councilmatic

3. Make a virtualenv and install dependencies

We recommend using virtualenv and virtualenvwrapper for working in a virtualized development environment. Read how to set up virtualenv.

Once you have virtualenvwrapper set up, run the following commands in your terminal:

cd yourcity_councilmatic
mkvirtualenv councilmatic
pip install -r requirements.txt

Afterwards, whenever you want to use this virtual environment, run:

workon yourcity_councilmatic

4. Set up your database

Initialization

Before you can run the website, you need to create a database and install PostGIS.

createdb yourcity_councilmatic
psql -d yourcity_councilmatic -c "CREATE EXTENSION postgis"

Then, run migrations. (Be sure that you are "working on" the correct virtual environment.)

python manage.py migrate --no-initial-data

Finally, create an admin user. Set a username and password when prompted.

python manage.py createsuperuser

Find legislative data

django-councilmatic leverages, and in some instances, lightly extends the Open Civic Data standard, implemented in Django as python-opencivicdata. The next step toward running a new Councilmatic instance is to locate data about your city and use it to populate these OCD models.

How you do that is up to you. At DataMade, we maintain a series of external municipal scrapers that retrieve data from Legistar-backed sites.

If your city runs a Legistar-backed site, see scrapers-us-municipal for several examples of scrapers that will populate a given database with legislative data in the format required by Councilmatic. These scrapers leverage python-legistar-scraper to scrape Legistar and the pupa framework to shape and import data.

If your city does not run a Legistar-backed site, we still recommend using pupa as a legislative scraping framework. You can add your scrapers to your Councilmatic repository, or compose them elsewhere.

Next, install and create a settings file for pupa –

pip install pupa
touch pupa_settings.py

– and add the following settings:

# Leave these blank
OCD_CITY_COUNCIL_NAME = ''
CITY_COUNCIL_NAME = ''
STATIC_PATH = ''

INSTALLED_APPS = (
    'django.contrib.contenttypes',
    'opencivicdata.core.apps.BaseConfig',
    'opencivicdata.legislative.apps.BaseConfig',
    'pupa',
    'councilmatic_core'
)

# Change this if you called your database something different
DATABASE_URL = 'postgres:///yourcity_councilmatic'

Finally, initialize pupa.

pupa dbinit us
pupa init YOUR_CITY_SCRAPER

Once you've filled out the prompts, you will see a new directory called YOUR_CITY_SCRAPER. Inside, you'll find the scaffolding for the necessary scrapers to populate the database.

Define the scrape methods in each of the resulting scraper classes, then run pupa update YOUR_CITY to import data. If you aren't sure where to start, the City Scrapers project includes many examples of scraping all sorts of government websites.

N.b., if you wish to make any changes to the models, see the django-councilmatic README for information on approaches.

Add metadata

django-councilmatic comes with management commands to add headshots and geography shapes to Person objects and Post objects, respectively.

If you've imported people with headshot URLs, run update_headshots to download the images and store them where Django knows where to find them.

python manage.py update_headshots

Likewise, if you've imported Post objects and you'd like to include geographic boundaries that pertain to them, create a GeoJSON file containing shapes for each division represented in your instance, then run the import_shapes command. See the GeoJSON file in chi-councilmatic for an example of the expected format.

python manage.py import_shapes /path/to/your/shapes.geojson

5. Rename the "city" app

Now that you have the data, set up your instance.

Inside the git repository that you cloned above, you should see a folder called city. Rename this folder to something that makes sense for your project, e.g., "chicago."

mv city yourcity

Now, update the main settings file - councilmatic/settings.py. First, in INSTALLED_APPS, add the name of the folder that you just renamed:

INSTALLED_APPS = (
    ...
    ~'city',~
    'yourcity',
)

Then, change the TIME_ZONE. You can use this list to find the correct formatting.

TIME_ZONE = 'America/Chicago'

6. Update city-specific settings

Look for councilmatic/settings_jurisdiction.py. This settings file tells your Councilmatic instance how to populate different parts of the UI and get fresh data from the OCD database. The following table explains why and how to adjust these values.

Name	Description	Example value	Optional?
OCD_JURISDICTION_IDS	An array of the OCD IDs of jurisdictions covered by your Councilmatic instance. Query the opencivicdata_jurisdictions table for options.	['ocd-jurisdiction/country:us/state:il/place:chicago/government']	No
OCD_CITY_COUNCIL_NAME	This identifies the legislative body you'll be tracking. It should correspond to an Organization in your database. Query the opencivicdata_organization table (or opencivicdata.legislative.Organization model in the ORM) for options.	Chicago City Council	No
CITY_COUNCIL_NAME	Display name for the legislative body you'll be tracking.	Chicago City Council	No
LEGISLATIVE_SESSIONS	A list of years that tells Councilmiatic when a new body is elected and which of these sessions you have data for in your OCD database.	['2007', '2011', '2015', '2019']	No
CITY_NAME	Complete, pretty name of the jurisdiction	New York City	No
CITY_NAME_SHORT	Shortened version of the jurisdiction name	NYC	No
CITY_VOCAB	A dictionary telling Councilmatic how to refer to different entities in the jurisdiction	(See individual values below)	No
CITY_VOCAB['MUNICIPAL_DISTRICT']	The name of the city sections, which could be a "ward", "sector", "district", etc.	Ward	No
CITY_VOCAB['SOURCE']	The name of the entity responsible for the source data that your Councilmatic will be based upon.	Chicago City Clerk	No
CITY_VOCAB['COUNCIL_MEMBER']	The name of a member of the legislative body that your Councilmatic tracks.	Alderman	No
CITY_VOCAB['COUNCIL_MEMBERS']	Plural form of the name of a member of the legislative body your Councilmatic tracks.	Aldermen	No
CITY_VOCAB['EVENTS']	The name of the meetings that members of the legislative body attend.	Meetings	No
APP_NAME	The name of the folder where your jurisdiction specific code lives.	chicago	No
SITE_META	Dictionary containing values stored in meta tags.	(See individual values below)	Yes
SITE_META['site_name']	Used for the og:name, og:title, and twitter:title meta tags	Chicago Councilmatic	Yes
SITE_META['site_desc']	Used for the description, og:description, and twitter:description meta tags	City Council, demystified. Keep tabs on Chicago legislation, aldermen, & meetings.	Yes
SITE_META['site_author']	Used for the author meta tag	DataMade	Yes
SITE_META['site_url']	Used to build absolute URLs for the meta tags.	https://chicago.councilmatic.org	Yes
SITE_META['twitter_site']	Used for twitter:site meta tag.	@councilmatic	Yes
SITE_META['twitter_creator']	Used for twitter:creator meta tag.	@DataMadeCo	Yes

7. Update deployment settings

Look for councilmatic/settings_deployment.py.example. Make a copy of it so that you can customize it for your city:

cp councilmatic/settings_deployment.py.example councilmatic/settings_deployment.py

This file is important! It's where you keep the parts of your Councilmatic that should not end up in version control (e.g., passwords, database connection strings, etc., which will vary based on where the app is running and which you probably won't want other people to know). Most of these variables are generic Django settings, which you can look up in the Django docs; but some of these variables require customization. This table shows only the variables that you need to customize.

Name	Description	Optional?
HAYSTACK_CONNECTIONS	Dictionary of connections used by Haystack. More on that here.	No
FLUSH_KEY	Used by the flush view in django-councilmatic to clear Django's cache.	No
DISQUS_SHORTNAME	Will cause Disqus comment threads to be attached to individual bill pages.	Yes
ANALYTICS_TRACKING_CODE	Google Analytics property ID	Yes
GOOGLE_API_KEY	API key for rendering maps on council members and person detail pages. Learn about creating your own Google API Key.	No
HEADSHOT_PATH	Absolute path to where the data loader will save images of members of the legislative body that are often available through Legistar.	Yes
EXTRA_APPS	List of additional django apps that you want to use with your custom councilmatic	Yes

8. Add a favicon

Get an image (suggested 310x310), and transform it in device-specific favicon images using this site. Then, move the images into <city_app>/static/images/icons.

9. Run Councilmatic locally

Now, you are ready to run your unique version of Councilmatic!

python manage.py runserver

Navigate to http://localhost:8000/.

Set up search with Haystack and Solr

On a Councilmatic site, users can search bills according to given query parameters. To power our searches, we use Django Haystack to connect with Solr, an open source tool, written in Java.

Run Solr

Because it's a bit of a heavy dependency, we recommend running a containerized instance of Solr. See the Solr image in DockerHub for more information and startup instructions. Don't forget to update your Haystack config in councilmatic/settings_deployment.py, if you map a port other than the Solr default (8983) to your container!

If you prefer to run Solr locally, see the Solr documentation for startup instructions.

Haystack commands to know

Haystack provides several management commands that make it easy to change and add data to your search index.

To make your data searchable:

python manage.py rebuild_index

If, during the course of development, you need to make changes to the fields that are indexed in yourcity/search_indexes.py, you need to regenerate the Solr schema:

python manage.py build_solr_schema > solr_scripts/schema.xml

Remove and recreate your Solr container to capture changes in the Solr schema.

Errors/Bugs

If something is not behaving intuitively, it is a bug, and should be reported. Report it here.

Note on Patches/Pull Requests

We welcome your ideas and feedback! Here's how to make a contribution:

• Fork the project.
• Make your feature addition or bug fix.
• Commit - please be descriptive, but concise.
• Send a pull request. Bonus points for well-titled topic branches!

Copyright

Copyright (c) 2019 Participatory Politics Foundation and DataMade. Released under the MIT License.