0572 spider chi ssa 38 #962
Conversation
…wrote spider parse_start function. WHY: wrote parse_start to extract unstructured date from page
…veloped functionality on a test set
|
|
||
| def _parse_classification(self, item): | ||
| """Parse or generate classification from allowed options.""" | ||
| return NOT_CLASSIFIED |
There was a problem hiding this comment.
This should be COMMISSION for all meetings on this spider
| def _parse_description(self, item): | ||
| """Parse or generate meeting description.""" | ||
| description = "" | ||
| return description |
There was a problem hiding this comment.
It's fine to just return "" instead of setting a variable first
| def _parse_title(self, item): | ||
| """Parse or generate meeting title.""" | ||
| title = "Chamber of Commerce" | ||
| return title |
There was a problem hiding this comment.
Mentioned in _parse_description, but it's fine to just return the string without assigning to a variable first. It's a bit odd for SSAs, but this one should be "Commission". They're technically separate entities managed by a nonprofit
| "name": "Northcenter Chamber of Commerce", | ||
| } | ||
|
|
||
| def _parse_links(self, item): |
There was a problem hiding this comment.
We'll need to parse a mapping of dates to relevant links from the page so that things like meeting minutes can be associated with the meetings listed. Here's an example of that:
city-scrapers/city_scrapers/spiders/chi_il_medical_district.py
Lines 109 to 140 in c6771d5
| return False | ||
|
|
||
| def _parse_location(self, item): | ||
| # Meetings seemingly ocurred at |
There was a problem hiding this comment.
It looks like an abbreviated version of this is in the meeting item, since we don't have examples of another format we can do something like this to return a default location if "4054" is present and raise an exception otherwise:
if "4054" not in item.extract():
raise ValueError("Meeting location has changed")
return {
"address": "4054 N Lincoln Ave, Chicago, IL 60618",
"name": "Northcenter Chamber of Commerce",
}|
|
||
| def _parse_start(self, item): | ||
| """Parse start datetime as a naive datetime object.""" | ||
| date_str = item.extract() |
There was a problem hiding this comment.
It looks like we might be able to simplify this a bit, and we'll also need to handle situations where minutes may be supplied for the time. Haven't tested this, but something like this snippet could work:
item_str = item.extract()
month_day_str = re.search(r"[A-Z][a-z]{2,9} \d{1,2}", item_str).group()
year_str = re.search(r"\d{4}", item_str).group()
if not year_str[:2] == "20":
year_str = str(datetime.today().year) # Default to current year
time_match = re.search(r"\d{1,2}(\:\d\d) [apm\.]{2,4}", item_str) # We want to check for a minutes portion here
time_str = "12 am"
if time_match:
time_str = time_match.group().replace(".", "")
time_fmt = "%I %p"
if ":" in time_str:
time_fmt = "%I:%M %p"
return datetime.strptime(f"{month_day_str} {year_str} {time_str}", f"%B %d %Y {time_str}")
Summary
Issue: #572
Checklist
All checks are run in GitHub Actions. You'll be able to see the results of the checks at the bottom of the pull request page after it's been opened, and you can click on any of the specific checks listed to see the output of each step and debug failures.
Questions
Include any questions you have about what you're working on.