Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scrape missed location change of special board meeting: June 15, 2018 #310

Closed
derekeder opened this issue Jun 20, 2018 · 3 comments
Closed

Comments

@derekeder
Copy link

from @fgregg:

The location for a special board meeting was changed by LA Metro staff at around June 15, 4:42 PM, Chicago time. Within 30 minutes, these changes were not showing up on https://boardagendas.metro.net/event/special-board-meeting-0fe4529b3dfd/

In the Legistar API, the last modified event field was June 13: <EventLastModifiedUtc>2018-06-13T23:52:23.45</EventLastModifiedUtc>

Every 15 minutes, a scraper checks the API for events modified in the last 72 minutes. Since the API said this event had been last changed two days previously, it did not pick up this change to the event. It would have been caught by our nightly, comprehensive scrape.

The ideal place to fix this is the underlying data system from Granicus.

Since we cannot control system, we can mitigate these source of source system data errors by increasingly how aggressively we scrape. Perhaps on Fridays, we can do a comprehensive fast scrape of events every 20 minutes. This mitigation may reduce system performance of the Legistar system.

@derekeder derekeder changed the title scrape missed location change of special board meeting scrape missed location change of special board meeting: June 15, 2018 Jun 20, 2018
@fgregg
Copy link
Collaborator

fgregg commented Jun 22, 2018

This is needs consultation with the client.

@fgregg
Copy link
Collaborator

fgregg commented Jul 17, 2018

Related to #328, we may need to very aggressively scrape bills too.

@reginafcompton
Copy link
Contributor

Handled with: datamade/scrapers-us-municipal#22

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants