Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metro event scraper did not scrape new media links #256

Open
reginafcompton opened this issue Dec 11, 2018 · 2 comments
Open

Metro event scraper did not scrape new media links #256

reginafcompton opened this issue Dec 11, 2018 · 2 comments

Comments

@reginafcompton
Copy link
Contributor

The Metro staff added media sources to the following events on December 11.

https://ocd.datamade.us/ocd-event/a4341b90-793d-4c17-9563-52018b89a7ba/
https://ocd.datamade.us/ocd-event/4e091b19-b407-4e4a-adc7-be6afb4e36bb/

However, the scraper did not update the events with the expected links, since none of the timestamps in the API were updated (i.e., 'EventLastModifiedUtc', 'EventAgendaLastPublishedUTC', 'EventMinutesLastPublishedUTC'):

http://webapi.legistar.com/v1/metro/events/1389
http://webapi.legistar.com/v1/metro/events/1419

Note: the Metro scraper runs with window=0.5. Running the full events scrape successfully imported the media data.

This issue seems to be in the same class as this one. @hancush - I'd appreciate any input you have on this one!

@hancush
Copy link
Collaborator

hancush commented Dec 11, 2018

@reginafcompton i think you've done a great job capturing this issue and possible remedies here and in #239. did you ever get an answer to your question of whether metro has any control of last updated timestamps?

@reginafcompton
Copy link
Contributor Author

We've done a fair amount of testing for bills, but not events. This would certainly be something to consider with the Metro team.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants