[1630] Acquire UX requirements for persisted Pocket recs & updates (formerly: loading/error states)

[mcomella]

Why/User Benefit/User Problem

What / Requirements

• Get UX requirements for loading and error states (consider the approaches below, e.g. one of them removes both of these states)
  • In progress. Requirements set in [1630] Acquire UX requirements for persisted Pocket recs & updates (formerly: loading/error states) #2123 (comment) but we're addressing remaining unknowns. This comment should be updated when we are complete
• Split this into separate implementation issues

Acceptance Criteria (how do I know when I’m done?)

---

There are multiple approaches (from an eng perspective):

• Download pocket tiles on startup but if we're unable to fetch them, show an error state (e.g. that can have a "Try again" button) - this is our current behavior
• Download Pocket tiles on startup but if we're unable to fetch them, just hide the channel
• Ship with pre-defined Pocket items (to show on first run) & replace them with periodically fetched Pocket items (which are persisted). This may be the ideal user experience and creates fewer states for engineers to handle: users will never see a loading state or an error state (though first run Pocket items can only change in new builds, may not be interesting enough to users).

[brampitoyo]

@mcomella I prefer approach 3:

Ship with pre-defined Pocket items (to show on first run) & replace them with periodically fetched Pocket items (which are persisted). This may be the ideal user experience and creates fewer states for engineers to handle: users will never see a loading state or an error state (though first run Pocket items can only change in new builds, may not be interesting enough to users).

For a few reasons:

1. On initial FFTV load, users can watch something while “the rest of Pocket” is fetching
2. On all subsequent FFTV loads, users can watch last fetched Pocket videos from the last session, while new Pocket content is refreshing
3. We don’t have to design a special, empty section just for Pocket

But since the Pocket channel will be refreshed with new videos every so often, we need a content ordering behaviour that doesn’t leave the user “stranded” with no content selected, when the carousel refreshes with new content.

Content ordering behaviour

Our content order goes like this (1 is newest, 5 is oldest):

There are two possible states, either a video in the channel is selected, or nothing is selected:

If no video is selected, replace all old content when new ones are fetched:

If a video is selected:

• Content that appears after currently selected video are replaced
• Content that appears before currently selected video are not replaced

Currently selected video is not replaced, so users will never lose their current position.

Although users can also return to older videos if desired, these are mainly there as anchor points: so that the user’s position don’t get reordered every time new content are fetched.

What do you think?

[mcomella]

This is an interesting approach that I have not considered. I'm concerned about two cases:

1. The user wants to watch a video they see but does not focus the Pocket channel. The content updates, preventing them from seeing the content they wanted .
2. The user wants to watch video 4 so they focus the Pocket channel, get to item 3, and the content updates, preventing them from seeing the content they wanted.

To be explicit, we can control when we fetch Pocket updates and when we present them to the user. Currently, we fetch every 45 minutes (starting with fresh app launch) and present updates only when the overlay is reopened (i.e. never when the user is looking at it).

@brampitoyo Does this information change your thinking at all? When do you think we should be fetching new video updates and do you still think we should present the updates as in the design?

(need to think through my opinions on best approaches, will follow-up soon)

[mcomella]

(need to think through my opinions on best approaches, will follow-up soon)

Without spending too much time on it, the problem I'm trying to solve is that we want the user to be able to get the latest Pocket recommendations. From a UX/business perspective, the ideals, imo, are:

• The user sees the latest updates when they first open the app and whenever they open the overlay
• The content does not update when the user is looking at it
• There is minimal perf impact to the user's device and network
• There is minimal impact to the Pocket servers

With these in mind, my ideal solution would be to:

• Display the bundled Pocket content on first run
• Never update the Pocket tiles when the user is looking at them
• When the app is in the foreground, fetch new content every ~45 min (iirc, the content updates once every hour so this ensures the user is getting frequent updates when they're most actively using the app)
• When the app is in the background/closed, fetch new content once a day (e.g. during sleeping hours; this helps ensure that when the app is opened the next day, new content will be presented)

Unknowns:

• We should verify how often the feed gets updated to know how often we should update
• We should ensure we can fetch things when the app is in the background, like we can on regular Android

Trade-offs (+ is pro, - is con, * is neutral):

• + No new UI states to manage
• + User is never annoyed/jarred when content they want disappears
• + Users should hopefully see new content daily
• + Simpler from a programmer perspective: no complex UI updates based on what is currently focused (in addition to other fixes)
• * We have to pick great initially bundled tiles
• - For users who rarely open the overlay (e.g. YT only users), their device & our servers are doing a lot of unnecessary work (this is mostly solvable: maybe we should only trigger new updates when the user has seen the last content update?)
• - Users don't always get latest content on subsequent opens (e.g. if user has browser open for only 30 minutes and comes back later that day, they'll have the same Pocket content visible)

@brampitoyo What do you think?

[mcomella]

@brampitoyo If you want to discuss further, please throw a meeting on my calendar (I'm a little busy today - maybe tomorrow?).

[brampitoyo]

@mcomella I really like your approach as outlined above in #2123 (comment)

Let’s resolve some of the cons:

For users who rarely open the overlay (e.g. YT only users), their device & our servers are doing a lot of unnecessary work (this is mostly solvable: maybe we should only trigger new updates when the user has seen the last content update?)

Yes. The Pocket update mechanism should only trigger if, on that session, the user has accessed the homescreen (meaning: the user would have seen Pocket tiles).

Users don't always get latest content on subsequent opens (e.g. if user has browser open for only 30 minutes and comes back later that day, they'll have the same Pocket content visible)

I think this is solvable if, when the app is started, we fetch new content based on the time when the app was last opened.

In other words:

• If the app sits in the foreground, fetch new content every 45 minutes
  • If the app is quit before 45 minutes, do a time check the next time it’s open. If >45 minutes has elapsed, fetch new content
• If the app is quit and the homescreen was opened in the last session, fetch new content every 24 hours
  • If homescreen is reopened at <24 hours, fetch new content as long as >45 minutes has elapsed since the last open
• If the app is quit, but the homescreen was not opened in the last session, don’t fetch new content

This diagram may help explain the refresh flow, but I’ll invite you to a meeting tomorrow just in case!

[mcomella]

To take a step back on this specific problem:

For users who rarely open the overlay (e.g. YT only users), their device & our servers are doing a lot of unnecessary work (this is mostly solvable: maybe we should only trigger new updates when the user has seen the last content update?)

This solution...

In other words:

• If the app sits in the foreground, fetch new content every 45 minutes
  • If the app...

is starting to get complicated. Before we go further, let's make sure this is a real problem to solve:

@athomasmoz In the current Pocket implementation, any active user, even those that may never see the homescreen (i.e. YT only), will have Pocket content updates every 45 minutes: do you think this makes a non-negligible impact on the server budget and is worth addressing?

Additionally, with my new suggested implementation, we'd be adding 1 request to the server per day (assuming no failures) per installed user (i.e. even non-active users and those who don't see the homescreen). Do you think this would make a non-neglible impact on the server budget and is worth addressing?

---

Alternatively, we could keep the update scheme same as it is today (if in foreground, update every 45min) and follow-up with improvements, which will allow us to decrease regression risk and upfront discussion time: the downside is that when users open the app the following day, they may see the same content from yesterday. However, maybe that's not actually that big of a deal (maybe we need to collect data on how often users who see the homescreen see the homescreen?).

[mcomella]

I spoke with Bram and we attempted to come up with an effective but simple solution. If you find something simpler but just as effective, iterate with Bram.

Requirements

• Never update Pocket video tiles when the user has the overlay open
  • Problem solved: users may lose content they wanted to see; it is jarring to change content when the user is looking at it
• Ship a default set of Pocket video recommendations to be shown on first run
  • Problem solved: users may otherwise see loading and error screens
• Persist latest fetched recommendations to disk and display those to users
  • Problem solved: given the default set of tiles and no loading screens, users freshly starting the app would otherwise see the shipped tiles when they start the app
• If the app is foregrounded, defer the first update for ~45 minutes. After that, update on a ~45 minute interval.
  • This is the core update loop: without it, users don't get frequent fresh content
  • Problem solved by defer: a user opens the overlay to load YT. They see content they want and intend to come back to, but finish their task by loading YT. However, when they reopen the overlay, the content they wanted to see is gone (e.g. this behavior currently exists on desktop Pocket recommendations)
  • Recommended update algorithm:
    • When the app is foregrounded (onStart ?):
      • If there is no update scheduled, schedule one 45 minutes from now
      • If there is an update scheduled, do nothing
    • When the update runs:
      • If the app is foregrounded, schedule another update 45 minutes from now
      • If the app is backgrounded, do nothing
  • To summarize ^ (via Bram): when the app is foregrounded, the user will see fresh content every 45 minutes. When the app is not in the foreground, the user will see fresh content the next time it’s opened (because we’ll get one more update).
• Update Pocket videos once a day during sleeping hours (maybe 3-5am), even if the app is backgrounded
  • Problem solved: the user last used the browser on Monday. They open again on Tuesday but they've already seen this Pocket content, discouraging future engagement with Pocket. We want large novel content updates

(Note: I removed the Clarifications section because I came up with a simpler solution)

Unsolved problems / unknowns

• Update interval: how often does the Pocket video feed update? We set 45 min interval updates based on 1 hr feed updates so if it's different, we need to change the update frequency
• User's metered networks & server budget: this implementation updates more frequently than we did before so we'll need to check with Pocket if that's okay or if we should add mitigations (see more details in [1630] Acquire UX requirements for persisted Pocket recs & updates (formerly: loading/error states) #2123 (comment))
• Default Pocket videos: we need to select a default set of video recommendations to ship to all users
• Content is replaced before users can watch it sometimes: if users keep the browser open for more than 45 minutes, they may run into the situation where they open the overlay, see content they want to watch, load a site to complete the task they were originally doing, then open the overlay to watch the content but it has changed (if it's the first foreground update of the day, their content will likely be lost but their content may still be visible, just pushed off to the side, if it's a later update of the day)

Implementation tips

• Consider WorkManager: we no longer need to update when the app is first opened so we can offload all updates (and exponential back-off logic) to WorkManager
• Add randomness to scheduling: we don't want every PST user to hit the server all at 3am for their daily update

[mcomella]

@athomasmoz Can you also check with Pocket how often they update their video feed content?

@brampitoyo @athomasmoz As for the default Pocket video recommendations we ship with the application (that all users will see on first run and ~45 minutes thereafter), how do we want to source those? @athomasmoz Is that something we can commit to source or (e.g. because Pocket doesn't want that content published) something we have to bundle at build time?

[liuche]

@mcomella will split this into separate bugs once the questions are answered, and requirements are set. This will now be an investigation bug.

The bugs split off should be the simplest (rather than implementing a totally new retry/WorkManager)

[mcomella]

I spoke with Ashley who summarized these changes well: currently, we fetch Pocket updates "just-in-time" so that the user gets the latest content when they open the homescreen but this requires a loading and error state. In the proposed system, we would ship the initial content and fetch new content in the background such that there will always be content to show: it will just be out-of-date on first run.

Trade-offs for proposed solution (+ is pro, * is neutral, - is con):

• + Better user experience: no loading or error states
• + Simpler to implement/maintain and for QA test: like other home tiles, there is only one state: content
• + Simpler to change the UX of: we only need one state, not three (for loading & error)
• - Content is out-of-date on first run
  • Note: besides first run, content should generally be just as up-to-date
• - First run content potentially needs curation to make appealing
  • Alternatively, each release, we can get the latest content from the API and ship that.
• - Likely requires slightly more engineering work to implement up-front than re-using current back-end implementation
• - If other channels require loading/error states in the future, we'll still need to implement these concepts and lose the simplicity of implementation benefits (though it's still less for QA to test)

---

Another note: the content feed is updated approximately every 6 hours with one video, i.e. 4 updates a day for 4 new videos a day.

[mcomella]

We got a thumbs up from Pocket and they'll give us five default videos by the end of the week: let's split this bug up.

[mcomella]

This is broken out into:

• [1630] Source default Pocket content #2154 Source bundled Pocket content
• [1630] Persist Pocket recommendations (instead of error/loading states) #2186 persist Pocket recs
• [1630] Reduce Pocket foreground fetch interval to conserve resources #2187 CHange foreground fetch interval
• [1630] Add overnight background fetch to Pocket channels #2188 For adding background fetch