Research Plan

Plan from April 2021 for some of our initial research. Note for external readers: Some links are to internal documents that aren't publicly accessible.

Project Overview

Summary

eRegulations is going to be a web application to provide a better user experience for individuals who read, consume and analyze Medicaid and CHIP regulations. It will weave relevant context, such as definitions, cross references, and sub-regulatory guidance into the display of Medicaid and CHIP regulations, as well as provide users the ability to view the history of regulation parts and sections. It is an adaptation of open source software developed by 18F.

Project Management

See Confluence for an up-to-date list of team members.

Anticipated Outcome/Scoping

Modify the eRegulations open source platform to help CMCS, State Medicaid staff, and the general public find, link, and interpret regulations by including additional policy-related context (statutes, rules, subregulatory guidance).

• Implement functionality to allow people to work more efficiently and easily:
  • View the history of a regulation part with a familiar “track changes” style interface
  • View term definitions in a sidebar for each regulation part
  • View contextual guidance for each regulation part, with links to related information and resources
  • For each regulation part, view a summary that helps the reader find what they need, including shortcuts to commonly-referenced sections
• We plan to publish summarized eRegs research results in our open source repository, to benefit other teams that may use eRegs code

Past Work as Relevant to Current Project

The CMCS eRegulations web application is based on the open-source application built by the Consumer Financial Protection Bureau and 18F. While the regulations, contextualized documents, and structure will differ, 18F eRegs will serve as the conceptual and code base for CMCS eRegs. User research will guide usability and shape some of what information is displayed, but all regulation, guidance, and other relevant documents will be pulled from official sources. Key learnings will also be pulled from relevant tools owned by other individual agencies, the Government Printing office, as well as third party tools. For more detailed information please see the comparative analysis.

Research Plan

Research Description and Goals

Research will take a primarily qualitative approach in alignment with Human and User Centered Design as a core methodology. The research process will place an emphasis on interviews, observations, and usability testing; while not expected, surveys may be used when relevant if users are unavailable or to validate the ethnographic research findings among a larger sample size.

Participant Focus

For a more up-to-date list of respondents please see our in progress respondent list.

While CMCS staff are considered our primary users, the potential audience for this project is wide reaching. Given this, we seek to hear from respondents in a wide range of positions throughout CMCS, as well as states, contractors, and other stakeholders. While information gleaned from our primary stakeholders will be given greater consideration, other respondent groups will help us better understand edge cases and will work to broaden and refine the ways users will be able to work with the eRegs tool.

Prioritization Summary

This reflects the importance of these groups to our decision-making, not necessarily sequence of interviews.

Priority	Role/Characteristics	Goal per feature
Primary	People who have expert knowledge of policy and take part in drafting and interpreting it. In the regs at least a few times a week, maybe every day. This includes our team’s domain SME, who has served in related roles at CMCS. Examples: State Officer Team Leads, Policy Group members such as CAHPG or DEHPG, some DSG Leadership, Technical Director who oversees staff who process state submissions.	1-2
Secondary	People at CMCS or states who are in the regs on a regular basis (at a few times a month) as part of doing their work, but usually don’t own the policy. They have varying levels of expertise in policy. Examples: Some DSG leadership, state submission SPA + waiver + APD) adjudicators, Product Owners, State Officers, State representatives (IT developers, state policy analyst), Contractors.	2
Tertiary	People who use regs but are not part of federal or state government. Examples: Journalist, Advocates, Attorneys, FFS Providers, MCOs, Non-profits.	0-1

Rationale

• Our primary research group is grouped around people at CMCS who interpret policy to help others understand, as well as those who write policy. In addition to being the user group we expect to utilize this tool most heavily, understanding how they find salient information will allow the eRegs tool to be more flexible for a wide variety of users.
• The secondary research group consists of people who need to understand regulation so that they can implement it in their work. This includes people at the state level, system teams, contractors, SPA adjudicators, and product owners who are not domain experts. Talking with users in this group will help us understand how information can be shown to expand understanding around implementation.
• The final group includes users who might use the eRegs tool occasionally such as journalists, advocates, attorneys, fee-for-service providers, and non-profit organizations. While they aren’t main users of eRegs, they should provide valuable insight as an edge case in assessing building access to information through context.

Research Methods

Sampling and Recruitment

Respondents will be recruited through a mixture of purposive, convenience, and snowball samples where the initial respondents are CMCS partners. These partners members will be asked for contact information for suggested respondents that they believe will help inform our work. While we will allow the breadth of research to expand from the advised suggestions, suggested respondents will also be measured against pre-established prioritization metrics to meet a theoretical saturation point. If a saturation point hasn’t been met, participating respondents will be asked to provide contact information to potential participants of the needed types, and respondents will continue to be collected in a purposive snowball sample until saturation is met.

Research Types and Tools

Research for the scope of this project will consist both generative and evaluative primary research, while secondary research will be conducted when possible to gather a greater breadth of information as well as assess the applicability of best practices for the given work.

The time burden for most research participants will be approximately 1.25 hours for a session; this time consists of a response to an initial invitation from the team, followed by an interview lasting up to 1 hour, and, when necessary, a follow-up email requesting relevant documentation. While some respondents may be asked to participate in multiple discussions, we prioritize the benefit that can come from a diversity of experience, role, and perspective and will generally seek to recruit new respondents as research continues.

Interviews will take place remotely using Zoom and sessions will be recorded if the user grants permission.

Generative research

For the initial phase of work semi-structured interviews will be conducted to orient the team towards experiences, perspectives, and eventually features that are the most salient for our users. Interview guides will be targeted towards individual respondents but address the following primary goals:

• Understand why they need to use regulations
• Understand how they currently access regulations
• Understand their challenges and begin to define a dream state
• See what is successful in the different tools currently available

These generative interviews will also allow the team to build trust and networks with users that will be crucial as the project moves into evaluative research. While it will not be the only source, information gathered from these interviews will be used to assess which features are most crucial to users and help the team identify the unique value proposition of the eRegs tool. As features and user needs are better understood generative research will shift to a supplemental role.

Analogous Research

Analogous primary research will allow the team to hone the project focus by better understanding how other tools address the challenges and features the team is looking to implement into the eRegs tool. This research will include feature comparisons with other research and annotation tools.

Secondary Research

To reduce the time and burden on both respondents and the project team, secondary research will be utilized insofar as it is relevant. This research will consist of interviews conducted by other teams relevant to regulations, as well as discussion held by CMS collaborators around the eRegs tool and its use. Relevant information will also be gathered from Slack conversations and email relating to the eRegs tool. Secondary research will also include comparative analysis with other regulations tools that currently exist, including analysis of ATF-eRegs and similar tools.

Evaluative Research

While generative research will not completely cease, once the team better understands user needs and feature prioritization, research will shift towards formative and summative evaluation, primarily in the form of usability testing. For these discussions respondents will be asked for feedback on various prototypes, testing various feature hypotheses. For a given session users will be asked to view one or two prototypes and complete specific tasks relevant to the specific feature hypothesis using a think-aloud protocol.

Paperwork Reduction Act (PRA)

While the general process will remain the same, questions will vary between features as well as between users. User testing will seek feedback on a given feature hypothesis from 3-5 respondents.

Description of informed consent

At the beginning of each interview session, respondents will be made aware of the interview process. Researchers will inform the participant that they have the right to not answer any question with which they aren’t comfortable, and will request to record the interview. The interviewer should take time to discuss any concerns the respondent has with the interview process before initiating the conversation.

Protection of confidentiality

Respondents’ individual names will not be connected in data that is made available; however, audio recordings and transcriptions may contain full names. While individual-level information may be used, it will not be attributed, and when possible will be obfuscated through group-level data. If it becomes necessary to use individual data, pseudonyms will be used to protect the respondent’s identity.

Assessment of Risk and Benefit

While there may be no direct benefit to respondents for participating in this study, this work will gather information and prototype a tool for contextualizing regulation information. If effective, and eventually adopted, this tool may potentially alter the nature of some work or require new work as new regulation and relevant information would need to be added.

Key questions

• How do CMCS staff in a variety of roles use regulations in practice?
  • For example, what tools do they use, and how do they access and use them?
  • What is the research process for CMCS staff when trying to answer a question, including how they find subregulatory or supplementary policy information?
• For CMCS staff who need to find answers to policy questions, what is frustrating, slow, or tedious about their current practices?
• For CMCS staff who own or are experts in policy, what tools would help them help their colleagues who need to understand policy?
• How does CMCS staff help states understand requirements and regulations that they need to comply with?

See also: key use cases that we’ve identified for eRegs. One of our core questions is how to make eRegs usable for CMCS staff with those use cases.

Our assumption is that if we answer these questions for CMCS staff, our work will largely also be of value to state users and non-government users as well, because they have similar needs.

Phases

1) Establish our understanding of the landscape

To prioritize changes and improvements to eRegs. Focused on the secondary user group and our domain SME as a representative of the primary user group.

Our primary user group has limited capacity for interviews, so it helped to establish our team understanding and initial hypotheses before going in depth with them.

2) Test hypotheses about core interfaces that would serve users. In parallel, deepen our understanding of the needs of our primary user group.

We need to make eRegs match and exceed the core usability of eCFR for many CMCS research needs, so we need to test our core readability and navigation for reg and key subreg content.

3) Test enriched interpretive features

Test hypotheses for adding a variety of helpful shortcuts for researchers to find the information they need.

4) Test policy contribution and maintenance experience

As part of accuracy, viability, and sustainability for eRegs, we need policy experts to be able to contribute to eRegs. This means we need to develop and test usable authoring and updating experiences for content.

Research Schedule

Features and prioritization identified through generative research in the first quarter will be the foundation of the user testing that will be the focus of the second quarter.

Dates for phase 2

Round	Planning	Recruitment	Testing
Round 1	Jan 25 - Feb 3	Feb 1 - Feb 19	Feb 8 - Feb 24
Round 2	Feb 22 - Feb 26	Feb 22 - Mar 12	Mar 1 - Mar 12
Round 3	Mar 8 - Mar 19	Mar 8 - Apr 2	Mar 29 - Apr 9
Round 4	Apr 19 - Apr 23	Apr 5 - April 23	Apr 26 - Apr 30

Features will be refined and prioritized continuously in response to user feedback, but some early identified features are listed alphabetically below. For more information, please see the Feature list