Timeline findings and recommendations

Decisions based on 3/23/21 team discussion

Move forward with changing to a subpart view (from the legacy section-by-section view)

• Change "See Next" difference from going section-by-section to jump links within a subpart

Move forward with the Top bar approach

• Revisit an earlier wireframe version with two regs side-by side
  • Split view like GitHub, but users might not have the screen space for this
  • Try collapsing a sidebar for more screen space
• Revise date labeling in the Top bar and main content area
  • Emphasize the date in the top so it’s easier to notice
  • Put final rule in the top right to make it more prominent
  • Iterate on the UI copy in the Top bar
  • Change “Other dates” in the Top bar dropdown to “choose another date”
• Add hover text to diff additions and subtractions
  • Look into feasibility of adding additional text explaining how to understand which version is reflecting the overlaid changes

Compare mode should consistently show chronological older stuff vs newer stuff (not reversing the adding/deleting notations based on what you selected)

• In the legacy eRegs, if you flip dates in legacy version, it DOES change what is displayed. We don't want that.

Implement history and diff views in stages

• This may help introduce some friction in the view/compare functions

---

Summary of findings

People overwhelmingly preferred the Top bar version in terms of its initial discoverability, visible activation of timeline mode, and ease of terminating the timeline mode.

• “Mode” discoverability
  • The “View and Compare” function was both more prominent and included more descriptive text in the Top bar prototype vs the Modal.
• “Mode” visibility
  • The active state of the Top bar’s timeline mode communicated more effectively than that of the Modal
• Ease of exiting out of the timeline mode
  • While all participants were able to exit out of both prototypes easily, participants preferred exiting out of the Top bar because of its location on the screen

The Modal prototype performed slightly better than the Top bar in two instances: finding an earlier version of a reg and displaying clear dates.

• One participant successfully completed the task of viewing an earlier version of a regulation with the Modal prototype (there were none for the Top bar)
• Participants preferred the Modal prototype’s treatment with the reg date in the upper right corner of the main content area

In addition to feedback on the timeline, respondents offered a wide range of thoughts regarding the prototypes. While the variety of topics precluded an abundance of consistency, several common topics emerged.

• Participants longed for a longer than section-by-section view
• Participants appreciated clear labeling of the header (part number and name)
• Participants were drawn to the supplemental content sidebar
• The link to the final rule under the View and Compare could be a bit more prominent- easy to get overlooked- may want to make it more obvious this contains the preamble.

Overall task success

Modal	Success	Partial Success*	Failure
Find an earlier reg	1	0	3
Compare two regs	4	0	0
See additional diffs	0	4	0
Stop comparing	4	0	0

Top bar	Success	Partial Success*	Failure
Find an earlier reg	0	0	4
Compare two regs	4	0	0
See additional diffs	0	4	0
Stop comparing	4	0	0

*We note partial success if a participant completes at least part of a task, but not all of it. We prefer to give context for partial successes instead of taking an absolute approach to success and failure in order to help the team understand the problem and collaboratively determine possible solutions if needed.

Findings in detail

Viewing and comparing are easily confused

**Task 1: Finding an earlier reg **

Prototype	Success	Partial Success	Failure
Modal	1	0	3
Top bar	0	0	4

Generally, participants found the Top bar prototype’s View/Compare function more quickly than the Modal prototype due to its visual prominence (regardless of order shown). However, in both prototypes, participants tended to conflate the task of Viewing an earlier reg with Comparing two regs when trying to complete Task 1.

“Everything felt visible” - P2

In 7 out of 8 times, when asked how a participant would go about viewing an earlier version of the reg they were on, participants slid immediately into comparing two regs.

Only 1 participant successfully completed the task of viewing an earlier version with the Modal prototype. P2 noticed there were two buttons divided by a line:

“Is this an either or option, View or Compare? I interpret it as two distinct things,” and then successfully completed this task. However, when the very same participant interacted with the Top bar prototype next, they went immediately into comparison mode, as all the other participants wound up doing.

It’s possible that changing the order and testing Compare first (then View) might have produced a different result. Most participants were able to successfully View after Comparing regs with facilitator redirection, however it is still too easy to slide into comparison mode, regardless of prototype.

Recommendations

Consider making the functionality of Viewing a reg distinct from Comparing a reg. Possibilities include:

• Introducing additional friction to the view/compare user flow instead of facilitating such a smooth flow into comparing. Make it more obvious to users that there is a decision to be made (Do I want to view an earlier reg or compare two reg versions?) by breaking up the flow into additional steps or prioritising one function over the other visually.
• Completely separating the two into discrete functions. Comparing as a concept makes the most sense to participants with the Top bar prototype (we’ll get to more of that below with the next section), but participants wanted confidence in knowing which version they are looking at, no matter which date they are on (current, past, or even future). If the main content area always displays what version is being shown, there is an opportunity to use that same area to switch between versions in the form of a dropdown, and comparing can remain in the sidebar, distinct.

Selecting dates is easy, but understanding is a little harder

Task 2: Comparing two regs

Prototype	Success	Partial Success	Failure
Modal	4	0	0
Top bar	4	0	0

While most participants found this function accidentally when trying to complete the first task, generally they understood the concept of comparing two regs in both prototypes. However, there was some confusion around what they were seeing once the comparison was loaded.

Note: This is an incomplete synthesis of data around dates as the prototypes were revised after P2 completed their session. P3 and P4 interacted with a slightly different version. See appendix for more details.*

Participants mostly understood the choice of dates for comparison, and some noticed the extra date options as well. P2 noticed NPRM in the Top bar prototype dropdown... “it would be really interesting to see what's coming up.” Those that noticed Other Dates (P2) in the Top down date dropdown found it useful to enter a date manually, but not all participants spotted it.

Participants struggled a little when trying to understand what they were seeing once they compared two regs: which reg version was reflecting the changes? It was ambiguous to P4 whether changes were “to” or “from” the older and newer versions.

Participants vastly preferred the Modal prototype’s “Latest Version as of” in the upper right corner of the main content area because it increased their confidence in understanding what version they were seeing.

Those that interacted with the Top bar prototype second noticed the absence of “Latest Version as of” in the main content area, which resulted in confusion around which version participants were seeing in the Top bar prototype. Additionally, most participants did not notice the link to the Final Rule below the Top bar’s date dropdowns, and seemed more drawn to the Modal prototype’s Final Rule in the upper right corner.

Recommendations

Continue to make it more clear to users what they are seeing and how things work. This should solidify our choice of having descriptive labels as described in our design standards and conventions.

• Consider using a call to action phrase instead of “Other Dates” to make the meaning a little more clear that this is a manual input.
• Make it clear what version is being displayed to users in the main content area to reassure users that they are seeing the version they expect to be seeing.
• If going with the Top bar pattern, make final rule links more prominent.
• Make it more clear how the changes from one version to another are being represented: is the newer version what’s shown?

Seeing and Understanding Additional Diffs

Task 3: Seeing diffs

Prototype	Success	Partial Success	Failure
Modal	0	4	0
Top bar	0	4	0

For this task, users were all partially successful with no-one having full successes or failures.

“It's a Track Changes view.” -P4

All participants were able to clearly identify highlighted and stricken text as meaningful, and 3 out of 4 participants were able to guess the meaning of the associated text though they all questioned their assumptions and asked if they were correct. Additionally, P4 correctly guessed the meaning of the highlighted text, however did not assert a clear understanding of the strikethrough text.

P3 indicated that they would want the diffs to clearly show when a section has been deleted or moved: “I'm wondering what that would look like using this format because it's more than just like striking language. It's like renumbering. I mean, it's possible obviously, but I would be curious what that would look like."

The “half-moon” symbol associated with the top bar prototype was recognized and more easily associated with identifying the location of other differences than the symbol used in the modal prototype. Users generally did not correlate the symbol with a specific meaning until prompted by the facilitator.

The “refresh” symbol was not as intuitive in meaning as the half-moon symbol (for those that noticed it). P1 remarked that it was very confusing and thought maybe it was a "tech" thing, so it didn't register as a marker of change.

Of the 4 users, 3 identified the “see next” button as a helpful feature when navigating differences. One specifically saying they were “drawn to the next/previous [buttons] more than the left sidebar.” When asked if there were additional ways to locate differences participant 4 didn’t identify the buttons.

Most participants were able to find multiple ways of viewing additional diffs -- at least one said they would just scroll and they expected they’d see more changes inline, which is in line with multiple requests for seeing more of the reg at a time than just section by section.

Recommendations

• Provide multiple ways to make diffs visible. Including “See more” with the half-moon symbol both tested well. Pair these two affordances with showing more scrollable content in the main area (a repeated request across usability studies and user interviews) for maximum visibility.
  • Note: “See more” functionality may change if we switch from a section view to a subpart view.
• Consider providing additional information regarding the meaning of deleted and changed text to increase user confidence, such as hover text or a tooltip.

Terminating timeline mode

Task 4: Stop Comparing Regs

Prototype	Success	Partial Success	Failure
Modal	4	0	0
Top bar	4	0	0

Both prototypes performed well here, but participants vastly preferred the Top bar version because it was more obvious and intuitive.

Recommendations

• Move forward with the overall Top bar prototype design pattern

Additional themes

We also heard a lot of feedback and interest around the following items:

Generally, participants appreciated clear labeling of the header, but one participant was confused by it

• P3 liked the subpart description explaining what the section covers in the header.
• P1 remarked: "Would the header label change? Why is it so high? Seems like it differentiates it. Like that’s the title of the whole website."

Participants longed for a longer than section-by-section view

• P1 asked: “Is there a way to have different views, of the main text, less clicks, sometimes you just want to scroll down.”
• P4 asked: “Can I click Subpart C [at left] and just scroll through all of it? Like if I know something is in Subpart C but not sure which section. Sometimes you need to roll down it or do a word search.”

Participants were drawn to the supplemental content sidebar

• P3 was enthusiastic about the resources as a whole, especially the order in which they were organized: “We’ve outlined this in training tools, so seeing it like this on screen [is great]!”
• P4 was struck by Executive Orders. They had not considered them as affecting regulations, and were curious to learn more.

Participants continue to be intrigued about SPA resources

• P1 remarked that they were “excited to see that there will be a resource for us because I think that's one of the complaints that some of my colleagues and I have sometimes. I think we know that some people in Baltimore do get a more detailed internal document that they use, and a lot of times us in the satellite offices don't have access to it."
• P4 reaffirmed that they don’t have easy access to state plan amendments. THey hoped to see under this resource “for a reg that introduces a new obligation, it'd be helpful to see as a SO whether the state has addressed the SPA first. Could be interesting to see a drop-down of states that have addressed this.”

Participants want easier access to preambles

• P4 always goes back to preambles because they “put reg language in layman terms.” and finds that the current eCFR doesn’t have links to the preambles and that they can be googled but “

Participants are curious about what “related regs” mean

• P3 noted that they were recently “helping a state and jumping around from one piece of reg to another. Not easy to jump in the tool I'm using” so having related regs linked here would have been helpful.

Interest in noting when sections were last updated

• P4 shared that it would be helpful to them “to find out here when this section was last updated. Not so much this section, but especially other sections. Sometimes states are using an old version. Probably the same date for all of Subpart C, would be helpful to have it on the page there.”

Appendix: About this study

Study goals

• To observe and learn from users in order to determine which design approach performs best and identify possible usability improvements.

Research questions

• Do users understand how the timeline works?
  • Can they view a historical version of the reg?
  • Can they compare two regs?
  • Is one prototype preferable or more intuitive over the other?
• Do users understand the purpose and meaning of diffs?
  • Do the diffs in the main body of content communicate well?
  • is "see next" helpful or do they just go to the sidebar?
  • Is one icon more understandable than the other?
• Do users understand what the terminology means?

Participants P1, P2, P3, & P4 all had some experience using regulations as part of their job but were not necessarily policy owners/SMEs. They rely on policy owners in the Central Office.

Methodology We conducted a moderated formative evaluation using the think-aloud protocol. Since two prototypes were tested, the order will be flipped for each participant so as to not bias users towards one over the other.

After P2 completed their session, the following design changes were made:

• create a hover state for final rule published
• changing the color from green to red (one is highlighted, one is just red)
• changed from current in dropdown to latest
• top bar is clearer in being able to navigate, and is good because the modal presents more challenges in working on multiple devices