Create a scraper/data dashboard

[reginafcompton]

Metro admin should be able to monitor the amount of data in Councilmatic and its relation to that of Legistar. The below provides a list of possibilities. (Collated from suggestions given over a period of several months.)

Data report: Possible formats

(1) An auto-generated weekly report sent to the Metro team via email.

(2) An admin dashboard, where Metro staff can generate a "data integrity" report, whenever desired. (N.b., Metro staff already have login credentials for Councilmatic, i.e., for uploading agenda PDFs).

Wish list

• A count of the number of events, bills, etc. in Legistar and Councilmatic.
• The time of the last scrape and import, and whether or not those processes ran without error (e.g., red vs. green status).
• A warning about discrepancies between the number of EventAgendaItems and the number of related Board Reports for a given event. This suggestion emerges from issue Event detail: Related board reports are not showing up #328 (comment)
• A warning about EventItems with duplicate EventItemAgendaSequence numbers. Currently, the scrapers fail, when this happens. Discrepancy between EventItems and PDF version of agenda #337

[shrayshray]

@reginafcompton, this looks good to me. @camachoo anything to add?

[hancush]

We're building a data dashboard! We discussed the things this dashboard should display at our 5/6 priorities meeting. Here's a text summary: https://docs.google.com/document/d/1vVcAHISA3_KSEB13CW74Lsplc9-6efe9BvacDRtbB_M/edit?usp=sharing

@shrayshray @camachoo Let me know if you think of anything else.

We also talked about being able to request that specific bills/events are scraped. This is a great long term goal for dashboard (I'll open a separate issue), but it won't be in the first version. We can, however, prioritize recipes for specific scrapes that will allow us to respond more quickly to these requests (i.e., run them with a button click, instead of writing and running them ad hoc), as part of the first iteration.

Also opening a separate issue for exposing useful metadata to admins on report and event detail pages.

[hancush]

P.s., the work for the dashboard is over in a private repository: https://github.com/datamade/la-metro-dashboard. I'll use this issue as a stand-in for that work in our project board.

[hancush]

The first draft of the dashboard is ready for input: https://la-metro-dashboard.datamade.us/dashboard/. Sent an email with more specific prompts – let's continue discussion there!

[hancush]

Really thrilled to report that the dashboard is in a stable place! FYI, @shrayshray, I'm going to be out of pocket 8/13 through 8/17, but I will be reaching out to you all next week to coordinate test data entry and put the dashboard through the paces before we deploy to production and sunset the old scrapers. 🚀

[shrayshray]

Sounds great, thanks @hancush !

[hancush]

Hey, @shrayshray – Thanks for your patience here. I'm putting together the production deployment this week. Are you available Thursday or Friday to coordinate on some test data entry, specifically adding some fake agendas / other sorts of data entry that happens during support windows? Let me know!

[shrayshray]

@hancush - yes! How about Thursday 8/27 at 11am or 12pm PDT?

[hancush]

12 p.m. Pacific / 2 p.m. Central would be perfect! I'll configure the support scrapes to run during that time period, and update this thread to confirm they're ready for you.

[shrayshray]

@hancush I apologize - would it be possible to do this Monday 8/31 instead? My mistake for not having it on the calendar, but the Regular Board meeting will still be running at noon on Thursday, and after it I'm booked until 5pm PDT.

[hancush]

No worries, @shrayshray! Monday would be great. Same time (12 PDT / 2 CDT)?

[shrayshray]

@hancush Yes, 12 PDT / 2 CDT Monday is good. Should I set up a GoToMeeting or do you just want to use email?

[hancush]

Email should be fine, @shrayshray!