Usability Study SOP

What this is: Wikipedia defines SOPs as “a set of step-by-step instructions compiled by an organization to help workers carry out routine operations. SOPs aim to achieve efficiency, quality output and uniformity of performance, while reducing miscommunication and failure to comply with industry regulations.”

Who this is for: Anyone on the eRegs team who takes a part in organizing and participating in usability testing.

Why are we doing this: Support shared team understanding around the steps it takes to plan, organize, carry out, and analyze results from a usability test. This reduces the complexity of planning each study, supports clarity on who leads each task, and helps people identify when/where they can pitch in on subtasks.

---

Draft possible usability study topics

Title: Draft possible usability study topics ([KEYWORDS])

In order to recruit study participants aligned with our goals, we need to sketch our possibilities for what to test and make the best determination we can with the information that we have.

This should happen 4 weeks ahead of target study week.

Implementation sketch

• Create Mural board (using template) and populate areas for learning goals and scope of test, using possibilities from design status board
• Schedule planning meeting with product/research/design team members to discuss scope [30 min]
• During planning meeting, identify best guess at scope together + identify likely appropriate types of participants or even names (and document it on the board)

ACs

• We have documented our best guess at study scope + ideal participants

Recruit usability test participants

Title: Recruit usability test participants ([KEYWORDS])

In order to gather data, we need to line up sessions with appropriate user representatives. This should align with our upcoming likely features to test (guided by our design status board): [LINK]

This should start 3-4 weeks ahead of target study week.

Implementation sketch

Context: [If available, link to feature page on wiki as well]

• Determine timeframe for testing
• Identify who to recruit and how to recruit them (based on our research plan and informed by our personas)
• Determine who will be emailing the invitation
• Send recruitment emails
• Ensure the appointment is on the eRegs team calendar
• Schedule debrief sessions for each test
• Update respondent list
• Send recruitment email reminder the day before
• Close session registration once desired number is reached
• After each completed usability test, send a thank you email to the participant

This includes identifying a dry run participant from our coworkers, reaching out to them, and scheduling with them as well.

ACs

• We've filled sufficient interview spots with appropriate participants (ideal is 5; we do the best we can)

Plan usability study scope and goals

Title: Plan usability study scope and goals ([KEYWORDS])

In order to gather the data that we need, we need to have a rough plan for what to test and (broadly) how.

This should be 2 weeks ahead of target study week.

Implementation sketch

• Review existing Mural board from our earlier drafting session
• Schedule planning meeting with product/research/design team members, with link to Mural board
• During planning meeting, review learning goals and scope of test (refine if needed), and identify what aspects of the design(s) we can test and potential ways to test them
• Determine appropriate type of study
  • Evaluation type: formative (primarily) or summative
  • Moderation technique: primarily moderated/synchronous (usually using Concurrent Think Aloud and Concurrent Probing techniques), occasionally unmoderated asynchronous
  • Methodology: qualitative/mixed method

ACs

• The team has documented shared goals and ideas for upcoming test (sufficient for people to start drafting the plan document)

Write usability study plan

Title: Write usability study plan documents ([KEYWORDS])

In order to gather the data that we need, we need to have a specific plan for what to test and how, and what to record to support interpretation of the test.

This should be a minimum of 1 week ahead of target study week.

Implementation sketch

Start work from a template or previous doc.

• Write test goals and research questions based on what we want to learn and what we can test
  • Ensure future note-taker has a structured document for recording observations relevant to our specific goals and questions
• Write script (scenarios with open-ended, simple, and short tasks)
• Share draft for feedback
• Refine draft and/or design(s) as needed

ACs

• We have a written usability plan that has been reviewed by at least another team member, sufficient for dry run.
• We have a structured document for note-taking that supports our goals.

Conduct dry-run session

Title: Conduct dry-run session ([KEYWORDS])

In order to have well-planned usability study sessions, we want to try out our plan/script on a coworker before we use it with our key participants.

Implementation sketch

This should be a minimum of 1 week ahead of target study week.

• Make sure we've recruited and scheduled an internal A1M or Fearless participant (usually done ahead of time)
• Facilitate dry-run
• Debrief after session
• Refine draft and/or design(s) as needed - coordinate with plan drafter

ACs

• We've run a dry run session.
• We've incorporated what we learned into our plan (and other materials as needed, such as designs).

Gather and manage data

Title: Gather data for usability sessions ([KEYWORDS])

In order to gather data about [KEYWORDS], we need to record our tests in structured ways for our analysis. This should be aligned with our usability study plan: [LINK]

Implementation sketch

Context: [If available, link to feature page on wiki as well]

• Create docs for each participant, including organizational/role context (or make sure they've already been created) and note-taking structure
  • Include observations from researchers
• Upload recordings to Dovetail and initiate transcription
• Process recordings
  • Correct jargon in transcript
  • Add names of everyone who speaks
• Can add tags for specific quotes of interest (bulk of this will be during synthesis later)

ACs

• Notes docs (including debriefs) are completed (prepared for synthesis)
• Dovetail recordings are uploaded and processed (prepared for synthesis)

Conduct usability sessions

Title: Conduct usability sessions ([KEYWORDS])

In order to gather data about [KEYWORDS], we need to test them with our participants. This should be aligned with our usability study plan: [LINK]

Implementation sketch

Context: [If available, link to feature page on wiki as well]

Steps for each session:

• Determine study facilitator (by default, the owner of this story is the facilitator, but can delegate as needed)
• Check with note-taker before session to make sure they're available (by default, this is the collect-and-manage-data person)
• Share prep/notes doc in the Slack channel
• Facilitate study sessions
• Facilitate debrief after session (or delegate facilitation to another team member), including first impressions of core research questions

ACs

• Every planned session has a planned facilitator and note-taker
• The sessions for this round are completed, including debrief sessions

Synthesize data for findings and recommendations

Title: Synthesize usability study data ([KEYWORDS])

In order to produce recommendations about [KEYWORDS], we need to analyze our data. This should be aligned with our usability study plan: [LINK]

This should be the week after the study.

Implementation sketch

• Compile data from all sessions
• Conduct affinity mapping (synthesis)
  • Positive and negative themes, participant suggestions, usability issues, and/or bugs
• Tally metrics, if any
  • Typically the task success rate
• Identify 1-2 most important "insights" to develop and share with the team (within Dovetail)
• Write key findings and recommendations document on our wiki, including key screenshots
  • Export "insights" from Dovetail and include in document
• Find a time for a review with the team

ACs

• We've shared key findings and recommendations with the team in a document
• We've reviewed this analysis with the team, so that the team can make product/content/design/development decisions
• If we made decisions during the review, we recorded them in the analysis document

Distribute findings [draft]

We want to share a summary of findings with our Product Owner(s) and usability study participants as well. TBD: exact steps