Board Reports: Filter by Fiscal Year error

[shrayshray]

"Filter by Fiscal Year" displays results outside of the selected fiscal year.

[hancush]

hey, @shrayshray. this looks to be the same issue as #181 (comment).

namely, the filters are working as expected. the legislative session filter applies to date the item was introduced, whereas the highlighted dates refer to the last action taken or, absent actions, the last agenda the item appeared on.

we added the most recent action label to clarify this in #181, however it seems it may still be confusing. would you like us to implement one of the other solutions @reginafcompton proposed? i.e.,

• add a line for legislative session to each search result. (small change)
• add the introduction date to each search result. (large change, involving edits to the scraper and data import)

[shrayshray]

@hancush I reviewed this thread and #181, and I think crossed our wires somewhere a long the line, so I'll try to clarify. Let me know whether this makes sense:

1. The filter on the sidebar should definitely match the date on the report as displayed in the search results.
2. If possible, we should use the Agenda date as the criteria for filter by Legislative Session.

Context:
A. Right now, the "Legislative Session" is determined by the 'MatterIntroDate' given in the Legistar API. The 'MatterIntroDate' comes from the date the file was created. Regina uses File ID 2016-0499 as an example in #181 "For example, Board Report 2016-0499 has a MatterIntroDate of "2016-06-06": http://webapi.legistar.com/v1/metro/matters/3016. Thus, this board report appears in the 7/1/2015 to 6/30/2016 fiscal year (i.e., legislative session) when filtering.

It usually takes at least a few months from the time a report was created until it ends up on an Agenda and in front of the Board. Until it's on an Agenda, it's not public; the date created isn't entirely helpful, from a user standpoint. Most people would be be interested in the Agenda date and Action date.

B. We're not getting entries in Minutes History consistently enough to rely on "Most Recent Action".
Thus, (2.) above -- can we use the Agenda date as the filter criteria?

[hancush]

hey, @shrayshray, happy monday, and thanks a ton for helpful context! i'm going to address your points in reverse order.

2. If possible, we should use the Agenda date as the criteria for filter by Legislative Session.

this definitely makes sense to me. to implement this, we'll need to make a change to the scraper to grab the MatterAgendaDate.

a couple of questions:

• is MatterAgendaDate null for private bills?
• is the MatterAgendaDate the first agenda on which an item appears, i.e., it is set once and does not change, or is it the next agenda on which an items appears, i.e., its value changes if an item appears on more than one agenda?
• do all public bills have a reliable MatterAgendaDate?

1. The filter on the sidebar should definitely match the date on the report as displayed in the search results.

if all public bills have a reliable MatterAgendaDate, we can incorporate that into our logic for determining the last action on an item. this will contribute toward syncing up the filter and the date appearing in the search result.

does it ever happen that items are introduced in one legislative session and carried over into the next?

[shrayshray]

@hancush, Happy Tuesday. :) Some answers for you:

* is MatterAgendaDate null for private bills? No. Agenda Date is typically added while the Report is being drafted, well before it's added to an Agenda and published.

* is the MatterAgendaDate the first agenda on which an item appears, i.e., it is set once and does not change, or is it the next agenda on which an items appears, i.e., its value changes if an item appears on more than one agenda? It is the first Agenda on which an item appears. It does not change. E.g., 2018-0739 was on the Operations Committee Agenda 1/17/19, then on the Regular Board Meeting Agenda 1/24/19 -- the Date remains 1/17/19.

* do all public bills have a reliable MatterAgendaDate? Unfortunately, no. The Agenda Date is a not a required field. It's generally very well managed by users, but we don't have a way to systematically manage it so I can't say it's 100% reliable. We could go back and clean this up to make sure it's reliable if necessary.

does it ever happen that items are introduced in one legislative session and carried over into the next?
This isn't typical, but it's entirely possible.

[hancush]

@shrayshray how big of a lift would it be to go back through and clean up the MatterAgendaDate? we'd like to avoid logic like "use the agenda date if it exists, otherwise use the intro date" to place a bill in a legislative session for a number of reasons, including ease of use and avoiding the potential for introducing duplicates.

[shrayshray]

@hancush time traveling back to February to finally give you an answer! I don't think it would be a huge lift to ensure there is a 'MatterAgendaDate' entered for every report. I doubt it will be much of a lift at all, as users are very good about managing it; it's just that we don't have a way of requiring it. I still think we should go this route right away, and I set aside sometime tomorrow to review and "lift" if needed on the data-entry side.

[jmithani]

@shrayshray
How did evaluating the state of MatterAgendaDate completion go? What kind of lift is required to ensure every bill has one?

[shrayshray]

@hancush Omar is going to discuss making MatterAgendaDate a required field in Legistar with the Board Secretary's office and office of CEO. I reviewed all the reports in the system which are not "Draft" and do not have MatterAgendaDate, and on 9/4 sent a spreadsheet of them to the Board Secretary's office for them to either add agenda date, change the status, or address them in some other way. I'll follow up to see how far they've gotten and when they expect to finish.

[jmithani]

I started a PR that swaps MatterIntroDate with MatterAgendaDate. It can be picked up whenever this issue comes to the forefront again: opencivicdata/scrapers-us-municipal#289

[hancush]

Going to use existing actions_and_agendas code to populate facet. Use SCHEDULED (agenda appearance). If not on any agendas, check action history. Use the latest date, in either case.

[hancush]

@shrayshray This is done and deployed to the staging site, thanks to @fatima3558! Can you take a look at the fiscal year facet and let us know if it looks good to you?

[shrayshray]

@hancush this looks great to me! Thank you @fatima3558 !

[hancush]

Woohoo! Just published, revised facet values should appear with the next full reindex in half an hour.