split agendas and minutes

[fgregg]

In the API, eventitems combines both agenda items and items from the minutes. This makes some sense because often these items are the same.

However, agendas and minutes are conceptually different things even if composed, in part, of the same substance and we should distinguish the two.

Since this is pretty big change, I'd like us to test this out locally.

Note that minute items are included in the eventitems even if the minutes are in draft, so right now we sometimes include items that are not intended to be published.

[fgregg]

Before we deploy, we should have a conversation with LA Metro about how this will change what appears on website.

They have three options

1. Always display the agenda
2. If there are finalized minutes available, display the minutes instead of the agenda
3. Give an option to see both agendas and minutes if both are available.

Options 1 and 2 are pretty easy, we can just handle what we are storing as an "agenda" in the scraper. Option 2 is also the closest to what we are currently doing.

Option 3 is a bigger lift. We would need to

1. add a minutes property to the opencivicdata model. Add Minutes to Events docs.opencivicdata.org#106
2. change import_data
3. and make some medium to big sized changes to the councilmatic web app.

cc @hancush @derekeder @reginafcompton

[fgregg]

Actually, looking through their data they have never finalized any minutes. It's pretty clear their preference is to to always only show the agenda.

I think we can proceed. In the future we may add minutes to other councilmatic sites.

[reginafcompton]

An addendum to the previous comment...Metro does finalize and publicize minutes for Board Meetings, although they do so in an attachment to a board report, rather than as EventItems in the Legistar API.
E.g., this board report points to this attachment with meeting minutes, and the Councilmatic site links to it from the event page.

In other words, Metro does make public some minutes, but not in a way that requires us to scrape them as EventItems...at least, not for the time being.

[reginafcompton]

I ran a full scrape and import of events locally.

While this change should not affect how Metro currently scrapes agenda items, I saw a lot of ERROR pupa: cannot resolve pseudo id to Bill: ~{"identifier": xxxxxxxx}: https://gist.github.com/reginafcompton/df4371f7b02228950909c5c12161b8e5

Does this not meet your expectations @fgregg ? I expected to see far fewer of these.

[fgregg]

I'm a little confused by your comment. The number of unresolved bill with this PR should be less than or equal than the number of unresolved bills without this PR. Are you saying that the number of unresolved bills went up?

[reginafcompton]

Oh, right. I just compared: the scraper with and without the PR returns the same number of "cannot resolve" errors. (I was thinking that there should be less of such error "in general," but I do not have any reason to think that, especially with a full scrape and import on my local database.)

So, this PR looks good to me!

[fgregg]

So what is your review?