Skip to content
This repository has been archived by the owner on Dec 8, 2017. It is now read-only.

PHASE 1: Findings

Jump to bottom
Jen E edited this page Oct 24, 2016 · 5 revisions

PDF of the presentation at the 3/8 Workshop can be found here: https://github.com/eregs/notice-and-comment/files/312384/EPA.N.C.Workshop.3-8.1.pdf

eRegulations - Notice & Comment Pilot Project

N&C Project Goals

Goals:

  • Find out if this interface will help facilitate the comment process by making it easier and less intimidating for the general public to comment.

  • Build this feature fully-open source so other agencies and organizations can build, modify, and enhance the platform.

  • HYPOTHESIS for COMMENTERS: By presenting the new rule in the context of the changing regulation, we will make it easier for commenters to understand the changes and better explain their views.

  • HYPOTHESIS for AGENCIES: By providing agencies with more context as to what parts of the rule comments relate to, we will make their job of sorting the comments easier and faster.

Research Findings

Internal staff

Who did we talk to?

  • 11 Total
    • 8 EPA
      • 1 Office of Water
      • 2 Office of Chemical Safety and Pollution Prevention
      • 1 eRulemaking
      • 3 Rule writers
      • 1 Docket office
    • 2 CFPB
      • 2 Rule writers in Research, Markets, & Regulation
    • 1 FCC
      • 1 Office of General Council

General learnings

Comments come in many forms.

  • A wide range of comment styles to address
  • Comments can be one paragraph to 100 pages
  • User's knowledge of the process can range from “What is a regulation?” to full legal backgrounds
  • Comment content can range from “This is the worst!” to a detailed cost benefit analysis

Who should comment?

  • Would love to see more comments from individuals
  • Far more people are affected by changes to regulations than those who comment
  • Individual commenters write with a lot of passion from personal experience
  • Getting more individual commenters requires going beyond publishing to the Federal Register
  • Awareness of the comment process is low

Process learnings

Sorting Comments - What do we do with all these comments?

  • It’s hard to sort comments well
    • Contractors are often used but can be expensive and still requires work from internal staff
    • Takes some familiarity with the rule to do a great job
  • Sorting is still a very manual process
    • Algorithms can help pick out form letters
    • Sorting out the comments that are just expressing an opinion is hard
  • Proposals provide a framework to sort by
    • Proposals are often written in a way that separates the different subject areas they’d like to see comments on
  • Large comments can establish buckets
    • Large comments are often broken down into sections that help establish the buckets smaller comments fall in

Responding to Comments - Making sure everyone is heard.

  • Parsing the comments
    • The most time consuming thing is sorting through all the pages and identifying each distinct piece of commentary
    • Form letters grouped together and a response is prepared that addresses all of them
  • Responding
    • If we agree, we have to fix it, if we disagree with a comment we have to explain why
    • We collaborate on responses, sending them back and forth to the people who worked on different sections of the proposal

External users

Who did we talk to?

  • 18 Total
    • 6 Professional commenters
    • 3 Journalist / Analyst
    • 3 Law Librarian / Professors
    • 3 Lawyers commenting for a client
    • 3 General public

Spectrum of users

Findings

Missing context - Finding something to comment on

Professional commenter quote: "I’m supposed to know this is coming."

General public commenter quote: "I just got lucky!"

Types of missing context

  • Larger context
    • Why a regulation? Why are these agencies writing this as opposed to congress? More than one agency has jurisdiction over this issue? How do I figure that out?
    • What tools do I have? What’s in the FR vs the eCFR vs regs.gov?
    • What stage is the rule in? ANPRM, NPRM, NFRM, Etc...
  • Context in terms of the regulation
    • Some people print out the rule and the regulation to compare the changes between them.
    • Need to understand the full regulation in order to understand what’s changing.

Forms of commenting - The act of commenting

  • Commenting in the box is looked at by professional commenters as the shortcut version.

  • The in-site comments are often the ones with less value.

  • Why is a PDF important?

    • Formatting is very important to lawyers. They want this comment to look like a legal document.
      • Footnotes are vital to the validity of the legal argument
    • Use long PDFs to tell a story over many different issues inside the rule.
    • PDFs offer control
      • “We’re lawyers so we cite everything.”
      • “I pride myself in writing very effective comments.”

Very little feedback loop - Post-comment submission

"I want to make sure some human actually reads this and thinks about this for a few seconds."

"No one’s going to read this thing, what are you doing?**

  • Professional commenter

    • “I know I’m going to wait a year to get an answer.”
    • “I understand why it may not be feasible to respond to every comment.”

  • General public commenter

    • Doesn’t know what the checkpoints are in the commenting process.
    • What happened to my comment?
    • Did the agency hear me?
    • Did the agency respond to my comment?

Open Discussion - Should commenters talk amongst themselves?

  • Reply Periods
    • Having a reply period is standard at the FCC
    • Reply periods help narrow the issue
  • Some interest in a back and forth conversation
    • “The only way we can get a back and forth going is if we can get them to meet with us”
    • “Would be interesting if people could have a discussion amongst themselves without us having to mediate the discussion”

Differences in behaviors:

  • Professional commenter
    • Groups of people are writing & approving these comments.
    • Takes time to craft a good comment. ( 10-100+ page documents )
    • Advocacy groups often build their own comment systems to collect on behalf of the public.
  • General public commenter
    • 1 sentence - 1 page comments.
    • Can be commenting through an advocacy group.

"Can't see what everyone else has to say. It is hard to collaborate with other commenters."

"If others are making the same comment, I am more brief: ‘As many commenters have said...’"

  • General consensus
    • Having a back and forth between commenters could change the commenting experience.

Effective commenting - What makes a good comment?

  • Making a better comment
    • Commenters should back up their ideas with examples/data
    • “This could be improved by adding…” is helpful
    • Reviewers love seeing cost information in the comments
    • Commenters should be steered towards commenting on what the proposal actually says.
    • A good commenter has subdivided their letter into the things they are going to be commenting on.
    • Great comments make direct references to sections in the proposal.
    • “A substantive comment is one that says whether or not we should be modifying the rule”
    • “It would be great if people could comment on specific paragraphs”
  • Commenters want to know what they should comment on
  • External view of what makes a good comment:
    • A good comment understands the purpose of the regulation
    • “I understand what you’re trying to do but here’s the problem with how you’re going about that.”
    • “I support this rule change for X, Y, and Z. And I agree.”

"It seems inefficient to be filing written comments when I know EPA is going to cut them up and put them in different buckets. If they would indicate to me how they do that, I would file them in a way that makes it easier for them."

General background

eRegulations Notice & Comment is part of a larger platform. Find out more at https://github.com/18F/eregs-platform/wiki

Clone this wiki locally