Script for fetching, parsing and storing election night results

[gordonje]

Here is what I did last time around: https://github.com/gordonje/MO_votes_2014/blob/master/get_data.py

[NathanLawrence]

My suggested structure (from #3, where I had put it before by mistake):

Use a function that takes in an elections results URL as string. Inside that function, parse the XML out into a nice, clean Python dict (object type ducked from var = {}), then push that out wherever you need it with Python's built-in json.dumps(dict) function.

[gordonje]

Other stuff this script needs:

• Adding key/values for FIPs to counties (and any other county-level data, number of voters perhaps)
• Conditional formatting of the results:
  • Each contested statewide constitutional office race (type "State of Missouri") should be its own type
  • Ballot measures "races" will all be under their own type, but instead of including a candidates list, there will be 'yes_votes' and 'no_votes' attributes on each county.
  • Some of the judicial races are similar, allowing for voting yes or no for each candidate. James is a liar
• We should probably check the <ElectionResults LastUpdated=...> before saving the XML to s3 or writing to the database (I think?) If not, then our own 'last_updated' attribute of the election_results table items should probably just be the current date time.
• Batch writing of table items? unnecessary

[NathanLawrence]

@qbscarlie and @DanielLevitt32, any progress on these, namely the FIP pairing? Tempus fugit...

[gordonje]

@NathanLawrence Shoot. I meant to reference e25b701 wherein I added this.

Does this look right to you? I've made the fips an attribute of each associative array in the counties array. Could also make the counties into an associative array, keyed off of FIPS, but I assume that wouldn't help much, and iterating over the counties would not be as straight-forward.

{
  "last_updated": 1415152662,
  "race_type": "State Senate",
  "races": [
    {
      "counties": [
        {
          "candidates": [
            {
              "id": "750074485",
              "name": "Joseph (Joe) Keaveny",
              "party": "Democratic",
              "votes": "0"
            },
            {
              "id": "750074484",
              "name": "Courtney Blunt",
              "party": "Republican",
              "votes": "0"
            }
          ],
          "fips": "189",
          "name": "St. Louis",
          "reporting_precincts": "0",
          "total_precincts": "24"
        },
        {
          "candidates": [
            {
              "id": "750074485",
              "name": "Joseph (Joe) Keaveny",
              "party": "Democratic",
              "votes": "0"
            },
            {
              "id": "750074484",
              "name": "Courtney Blunt",
              "party": "Republican",
              "votes": "0"
            }
          ],
          "fips": "510",
          "name": "St. Louis City",
          "reporting_precincts": "0",
          "total_precincts": "101"
        }
      ],
      "title": "State Senator - District 4"
    },

[NathanLawrence]

I like this layout -- it's exactly what I had in mind.