Metro scraper did not scrape EventAgendaStatusName

[reginafcompton]

Yesterday Metro canceled two events:
http://webapi.legistar.com/v1/metro/events/1366
http://webapi.legistar.com/v1/metro/events/1367

Neither the DataMade windowed scrape nor the nightly full scrape captured this change.

I manually executed pupa update lametro events window=7, which captured the change. Let's determine what happened.

[reginafcompton]

This occurred again, and the cause of the issue is slightly more apparent. This morning, Metro canceled the following events:

http://webapi.legistar.com/v1/metro/events/1378
http://webapi.legistar.com/v1/metro/events/1379

However, the EventLastModifiedUtc reads as 2018-06-20.... That is, the Legistar API did not adjust this timestamp, when the EventAgendaStatusName changes.

I manually executed pupa update lametro events (which scrapes everything), and it pulled in the status changes.

I've reported a similar issue here: Metro-Records/la-metro-councilmatic#267

Thoughts

1. The Legistar EventLastModifiedUTC does not communicate the way our scrapers expect. As a result, we may need to scrape events more aggressively. Scraping all events, once or twice per hour, seems reasonable. Currently, we scrape all events once per day, except on Fridays, when we scrape all events every 15 minutes. This frequency came about due to this related issue: scrape missed location change of special board meeting: June 15, 2018 Metro-Records/la-metro-councilmatic#310 – (We saw a similar issue with bills)
2. Does Metro have any control over this timestamp in the API? Can they insure that the timestamp changes when they update an event, or does Legistar fully automate this?

[reginafcompton]

We also saw this behavior with the media links. Metro staff added the audio URLs, but the timestamps did not change. Incidentally, our nightly scrape would have grabbed the URLs.