Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a method for scraping specific bills and events #59

Open
hancush opened this issue Dec 4, 2017 · 2 comments
Open

Add a method for scraping specific bills and events #59

hancush opened this issue Dec 4, 2017 · 2 comments

Comments

@hancush
Copy link
Collaborator

hancush commented Dec 4, 2017

It's currently impossible to scrape a single bill without also scraping lots of other bills. Ditto for events. Let's look into a way to update specific bills and events by passing in matter/event IDs, respectively.

@hancush
Copy link
Collaborator Author

hancush commented Dec 4, 2017

Preliminary design

Add a fetch method to the API scraper that accepts the appropriate ID of the bill/event to retrieve.

legistar.bills

class LegistarAPIBillScraper(LegistarAPIScraper) :
    ...
    fetch = partialmethod(endpoint, '/matters/{}')

In the downstream scrapers, break the matter formatting and related information that currently takes place in the scrape method into a separate method or methods (i.e., for bill formatting and action formatting).

Add a keyword argument that defaults to None but can be set to an ID, should the user desire to scrape specific matters/events.

chicago.bills, for example

class ChicagoBillScraper(LegistarAPIBillScraper):
    ...
    def scrape(self, window=3, matter_id=None):
        if matter_id:
            matter = self.fetch(matter_id)
            yield self.format_matter(matter)

        else:
            n_days_ago = datetime.datetime.utcnow() - datetime.timedelta(float(window))
            for matter in self.matters(n_days_ago):
                yield self.format_matter(matter)

@hancush
Copy link
Collaborator Author

hancush commented Mar 31, 2020

Done for bills in #60.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant