PHASE 3: Findings
This was originally compiled into a PDF presentation given at the eRegs Notice and Comment team's last demo.
Will commenters break their comments down and use the pilot?
- If commenters are using the pilot, how does it fit into their drafting/submitting process?
- Do commenters want to use the pilot, or something similar, in the future?
- Is anything missing or impeding their ability to use the pilot?
Although we had intended to speak to users both before and after the comment period, due to the general length of the comment drafting process among other things, we spoke to all of these external users towards the end and/or after the comment period closed.
Types of commenters
2 - Hazardous Waste Technologist
4 - State gov commenters
3 - private company or university representative
Breakdown of behaviors
5 - Commented successfully using the pilot
1 - Didn’t comment successfully using the pilot
3 - Weren’t going to comment anyway
First let’s take a look at the raw stats. Of the 31 total comments submitted to EPA, 17 were submitted through the pilot. That is a 54% usage rate.
Analytics also show that some commenters came to check the site out early on in the comment period, but the vast majority of activity came very close to the end of the comment period when most people submitted their comments. [Shown below.]
This chart matches the behaviors we heard through user interviews. Most users said they took a look at the pilot early on in the comment period, but then either printed it out to mark up or created summaries to pass around their team. Once the decision was made (by the larger team) to write a comment, usually one person takes the lead in drafting the comment. That draft goes through various rounds of clearance before the main commenter gets the chance to submit/publish the final comment to EPA. Most of the commenters at that point came back to the pilot to break up their comments and submit.
Generally speaking, we heard good feedback from commenters who used the pilot.
“10/10. It was easy to do.”
“Modern looking. It enticed you to make comments.”
“I think you guys did a good job! We were always intimidated putting our comments into the EPA but this was so easy.”
“I personally think this is [...] the public sharing of the 21st century. You don’t have to come into city hall to make comments, you can comment from wherever you are.”
“Especially being able to comment with the text side by side. They [general public commenters] are going to read it, think about it, and type. That’s a great model for the public.”
Many commenters appreciated the instructions to break down their comments because it organizes the rule into a set number of issues. But, having two different instructions on how to break down comments was confusing (i.e. “Comment headings” in the body of the text versus the links to comment on specific paragraphs or sections in the pilot). Especially since the links in the pilot didn’t match up with the comment headings that were requested in the paper copy.
Lesson learned: Consider choosing one commenting instruction next time.
One consideration to keep in mind is that many of the users wanted to reference their own responses in other responses to link their whole comment together a little more. The comment headings worked great for this because commenters could say “please see our comments in topic 5 above.” The pilot’s citation approach was not as straightforward for this task.
1. We should be more explicit about what the user is looking at.
- Some users didn’t notice the CFR tab or understand what kind of content they were seeing at first glance once they were inside the tab.
- There was a lot of confusion when the user first got to the “preamble introduction” landing page:
- “Since I’m used to seeing it all at once, [the whole rule] I’m feeling like I’m wondering if I’m looking at the right thing.”
- “Don’t see anything that tells me ‘this is the rule!’”
- The tool can be overwhelming at first glance if you don’t know what you’re coming to look at. However, once you spend some time with it, it becomes easier to use. We need to pay more attention to that first interaction with the tool to make onboarding easier.
2. We should clarify the difference between saving a response and submitting a comment.
- The distinction between the two phrases was hard to determine. Many users questioned if saving a response meant their comment was sent to EPA or not.
3. We need to better understand user needs around navigation after saving a response.
- What does the user expect once they’ve saved their comment? How to they get back to their spot in the rule?
4. We shouldn’t automatically double space comments.
- We need to accommodate users who are copying a lot of text into the text box. We heard that copying and pasting 2 pages of a word doc into the textbox, became hard to organize when the pilot defaulted to double spacing the text.
- One user reported seeing “access denied” when looking at images of the formulas in Internet Explorer.
- Spellcheck was acting strange in Internet Explorer
- One user experienced a series of errors in Internet Explorer. While trying to save a comment, the text of the response was not saved (or seemed to disappear), but the header seemed to exist. They were able to submit a list of blank headers to EPA.
- When commenting the section you are commenting on is no longer highlighted in the left-hand navigation. The top section (preamble introduction) is highlighted even if you are commenting on something else.
Is anything missing? Answer: We heard about 4 potential new features that could improve the user’s experience.
1. In-progress drafts and multi-session editing.
- We heard that many commenters were intimidated by trying to compile and submit their comment all in one sitting. (This was corroborated by what the EPA rulewriters heard from commenters.)
- But, some commenters were fine with the experience as is: They liked the ability to submit commentary in chunks in case there wasn’t time to go back and keep commenting on more sections.
2. Ability to tie responses together.
- As mentioned above, users wanted the ability to easily reference other responses in their comment. And, people generally found the print version of the comment headings easy to reference compared to the pilot’s citations.
3. Space for general, overarching comments.
- Many regulations have individual pieces but the sum of the whole is greater than the sum of the pieces. There are overarching parts of the regulation that many users wanted to comment on. Many users followed instructions from EPA to use the “SUMMARY” section of the rule, but that wasn’t an intuitive or explicit place to put overarching comments.
- Additionally, if something is missing from the rule, where do you comment? If there is no paragraph that talks about a specific issue, users have no paragraph to comment on.
4. Organization branding and qualifications.
- We heard a lot of requests for a space that explains who the commenting organization is, why they are writing a comment, and potentially the ability to add some branding so the document looks more official.
- “We would have liked a header. Would satisfy the concern of our leadership to explain who we are and this is an official [organization’s] comment.”
Will commenters break their comments down and use the pilot?
YES!
- If commenters are using the pilot, how does it fit into their drafting/submitting process?
- Mostly at the end.
- Do commenters want to use the pilot, or something similar, in the future?
- Generally, yes.
- Is anything missing or impeding their ability to use the pilot?
- Generally those who tried to use the pilot succeeded in submitting comments. And, now we have feedback, and a list of things to work on for the next iteration of the tool.
Will this new comment format make the agency rulewriters’ work easier?
- Does this new format, make it easier and faster to sort comments?
- Does this new format change the way people write comments for the better?
We spoke to the rule writers at EPA both before the comment period started (or at least before comments started coming in) and after it closed. We wanted to see what they expected from the new pilot and comment format and compare that to what happened once they received comments.
Here is what the rulewriters expected before the comment period opened:
-
Indexing comments as they come in will be a huge asset to rulewriters.
-
If commenters break up their comments as asked, it will help keep their comments on track.
Yes, the comments are easily sortable!
- Comments were easy to organize whether they came through the pilot or using the comment headers.
- The rulewriters were asked to summarize the comments quickly in preparation for a meeting (right before they went on vacation). By using the new comment format, it was easy to summarize the comments quickly.
Yes, asking users to use the comment headings forced them to focus on specific issues.
- Only issue: In having the response to the headings/sections as discrete organizing efforts, you can lose the context of the larger comment and the narrative effect commenters and rulewriters are used to.
- Anonymous comments are hard to work with.
- The rulewriters mentioned that they were not quite sure how to deal with the anonymous comments that they have. It would be nice to be able to solicit more information from the commenters to better understand their perspective. Or determine if different comments were actually submitted by the same group.
- This matches with commenters asking for a spot to better explain themselves and their organization.
Will this new comment format make the agency rulewriters’ work easier?
YES!
- Does this new format, make it easier and faster to sort comments?
- Yes.
- Does this new format change the way people write comments for the better?
- Yes.
Yes, but it’s not perfect. We were able to successfully collect comments during the eManifest comment period for EPA.
Although the tool wasn’t perfect, it was usable. We were able to get something out fast, people were able to use it, and now we have a list of areas to focus on for the next iteration of this tool.
- Phase 1: Generative research
- Phase 2: Testing the tool
-
Phase 3: Success?
- Comment period research plan
- Findings < You are here!