Enable a user or Export to "Download All" files to the Workstation for a given source

[ninavizz]

Problem

In order to download all messages and files for an entire source per #550, all files will need to first be downloaded from the server that have not yet been downloaded to the Workstation. This could be only some of the files for a Source, or all of the files for a Source. Likewise—sources often send multiple files, and users have reported back to the team that going "click, click, click" on multiple files when checking their Drop, can be tedious.

Solution

It would be swell if there could be a top-level menu option w/ an accompanying keyboard shortcut, to enable a user to just download all the new files for a given Source. This will alleviate the need for Yet Another UI control, while also meeting user needs—and surfacing that capability, in the kebab menu for each Source. Likewise, when a user seeks to export all Source material, they won't be forced to exit their task to "click, click, click," then retry.

TBD: Would it be helpful to offer a confirmation alert to let users know once a batch has completed? What would most "batches" look like, size-wise?

Below screen demonstrates the concept in a future-state. It has neither been discussed nor approved.

User Stories

1. As a journalist, I would like to have a single-command option to download all 7 files that I just received from a source, so that I can view them without having to mouse-to-and-click 7 different things.
2. As a journalist, I would like to be able to download all past files not yet viewed, from the past, such that I can neatly download their entire account from our SecureDrop to
3. As a journalist, I would like a single-command option to download all files from a Source that I might not have yet downloaded, before going offline for a while or needing to export the Source account. This will spare me from having to scroll through a long list and visually catching files that have yet to be downloaded, and manually clicking each "Download" command.

[ninavizz]

Arrow icon and other details, in Figma file.

[ninavizz]

Gherkinny Goodness!

Note: Towards the bottom, the scenarios are less and less thought-through; I will be updating these tomorrow when I'm in!

Scenario: New files to download
Given I have a new Source, and I have selected it in my Source List
And they have sent me 7 files
And I have not yet downloaded any of its files at all
I can click on the kebab menu, pull-down and select “Download All”
I could also invoke this same action by keying "Ctrl+D"
And the action will begin immediately; no dialog, unless the total number of downloads were to exceed XYmb*.
Yet if my client were mid-process with any Deletion action, "Ctrl-D" would be ignored—and the kebab-menu outright disabled.

Then, the state of each file's buttons will change from "Download" to "Downloading," with the little spinners—so, 7 little spinners will appear with respective downloading texts, as if the user had 1, 2, 3... clicked upon each with their mouse.
And if I click on the kebab menu after the Download action has begun yet the menu itself has closed, the "Download All" option will show as disabled

Then, one by one, as each file’s download has completed, I will see each button's in-progress state change to the dual-buttons "DownloadED" state showing “Export · Print”
And mid process should I key or select in the kebab menu a Deletion action, upon confirming my desire to execute a Deletion action, the downloads will all cancel and the Deletion will begin.
However, presuming I do not choose to interrupt my Download All action, all of the little file buttons will continue to change to Export/Print state, 1 by 1, from each's "DownloadING" state, as each file continues.
Upon completion I will not receive a confirmation message in the barre; I will just see no more spinners, and 7 available files for me to print or view.

I will also hear a "ting!" sound upon each file's availability to view/print/export.**
Because there are now no more files to download, the "Download All" option in the menu will show as disabled (yet to do state design for this in Figma).

* TBD on wording or threshold for throwing a "Yo, that will take all afternoon" dialog to users.
** @creviera @eloquence I'd prefer to have line spacing solved for, before sound—but would love to see sound considered as an enhancement for the Client, for this, and for when a new Source or Source Artifact comes in. Maybe even a singing Oscar The Grouch for Kev, or a crinkled paper sound, upon Deletion.

Scenario: New and old files to download
Given I have a longtime Source
And that Source has sent me multiple files over a span of time
And some I have downloaded some files on some days, while others I have not yet downloaded from this specific source, or to this specific Workstation machine;
And I have this Source selected in my Source List;
And I am not mid-process with executing a Deletion action;
Then I key "Command-D"
Then, all of the files from the out-of-view files waaay scrolling-up in the Source's Conversation View, to files within view that were sent more recently, that have the "Download" button shown—to indicate they have not been downloaded—will all change in appearance to have blue spinners with "Downloading" shown, instead—with all files downloading in the background.
The menu item for "Download All" will become disabled in the kebab menu.
Then, one by one, as each file’s download has completed, I will begin to see each button's in-progress state change to the dual-buttons "DownloaeED" state showing “Export · Print”
Upon completion I will not receive a confirmation message in the barre; I will just see no more spinners, and 7 available files for me to print or view.
Because there are now no more files to download, the "Download All" option in the menu will show as disabled (yet to do state design for this in Figma).

Scenario: No files to download
Given I have selected any Source
And with that Source there are either no files to download, no files at all
I will key "Command-D" and nothing will happen
I will also see the "Download All" option in the kebab menu as disabled (yet to do state design for this in Figma), should I click on it.

Scenario: User in multi-SDW using org
Given that I work for a news organization.
And all SecureDrop Journalist users in my organization each have our own Workstation machines.
And my colleagues have downloaded some, all, or no files for a particular Source—yet I have downloaded either some or no files for that Source.
My Workstation client will recognize that those files have not yet been downloaded for me;
And upon clicking the Kebab menu, I will see the "Download All" action is enabled
And upon my invocation of the "Download All" action, my Client will figure-out which files for the selected source have not yet been downloaded
And it will successfully download them.

Scenario: Connection Fail
Given I am Nina
And I have spaztic-fantastic satelitte internet connection
And it is raining outside
And I have invoked the "Download All" action against a Source with files yet to be downloaded to my Workstation
And my Client is in the middle of downloading a number of files
And the rain interferes with my Internet connection
My client will keep the good-faith effort to keep retrying, for 5 minutes
And upon deciding to throw-in the towel, will throw an error—telling me that "x of y file downloads" were not completed for z source(s)"
With X, Y, and Z all being numbers
The files that were not downloaded will have their "Download" buttons return to their static state, showing the download icon and word
And the successfully downloaded items will show their "View · Print" buttons; just as those buttons should automagically appear, once each file has been downloaded and is availble to view, export, or print.

Scenario: decryption fail
Given [decryption fail]

Scenario: absent keypair

Scenario: hyperactive journalist, +xxmb of data invoked for downloading
(show a speedbump warning that shizz gonna take for-eva)

Scenario: I've downloaded everything already, but just got a fresh Workstation install or machine (@eloquence, has this been resolved yet separate from download all?)

[ninavizz]

^^ #1362 should ideally go into the same release, as this. Not necessarily the same PR—let's keep PRs small/manageable—but definitely in the same release. :)

[sssoleileraaa]

Scenario: New files to download
Given I have a new Source, and I have selected it in my Source List
And they have sent me 7 files
And I have not yet downloaded any of its files at all
I can click on the kebab menu, pull-down and select “Download All”
I could also invoke this same action by keying "Ctrl+D"
And the action will begin immediately; no dialog, unless the total number of downloads were to exceed XYmb*.
Yet if my client were mid-process with any Deletion action, "Ctrl-D" would be ignored—and the kebab-menu outright disabled.

Currently the kebab menu is already disabled until we hear back from the server that the conversation is scheduled for deletion. I think it makes sense to disable the menu while the operation is ongoing. There is an edge case where a user could export files immediately (within seconds) after the deletion is scheduled on the server and actually export the local files, but that is an extreme edge case and will go away after we implement v2 for the journalist api.

I think we should always have a dialog, whether or not it exceeds capacity to a) give a condensed list of files about to be downloaded next to their individual size given that it's easy to lose track of files in the scroll view, and b) give the users a consistent workflow. If not as a final implementation, at least as a first implementation of "Download All."

Then, the state of each file's buttons will change from "Download" to "Downloading," with the little spinners—so, 7 little spinners will appear with respective downloading texts, as if the user had 1, 2, 3... clicked upon each with their mouse.
And if I click on the kebab menu after the Download action has begun yet the menu itself has closed, the "Download All" option will show as disabled

This makes sense to me, but we have to consider what signal to use to re-enable the menu option. Currently we fire off jobs and wait until their success and failure signals are emitted to communicate back to the GUI. If a duplicate job is submitted to the queue (say from accidentally double click a button that fires off two of the same jobs) it automatically gets dropped. So technically it's easy to handle multiple "export all" actions, and I would argue that this should go in the first iteration.

It is useful to communicate to the user when an option does not really do anything by disabling it, but we'll have to signal back to the GUI the completion of the last download. One way is by using a MultipleObjectApiJob (yes, we thought about this ahead of time by creating a SingleObjectApiJob and leaving room for a MultipleObjectApiJob, but it still doesn't exist yet) that can re-enable the menu option when finished. Also, another edge case: a new submission from the source can come in after a bunch of jobs have been enqueued, so that would also re-enable the menu option. We can have multiple ways to re-enable the menu option, so that is not a problem. We just need to add this to our list of scenarios.

Another way to do this requires more complexity in tracking which files have been enqueued and which have not per conversation. It's messy because sometimes there is an error or jobs are cleared out during a deletion operation. It would be cleaner to not have to keep track of this state.

It might make sense to just keep the option enabled for our first implementation, since we already won't add duplicate jobs to the queue. But let's discuss more ideas of how to implement this in an elegant simple way. I do like the idea of relying on a job that singles back the the GUI "I finished successfully and I'm the last job, so re-enabled the menu option if it's disabled, please."

And mid process should I key or select in the kebab menu a Deletion action, upon confirming my desire to execute a Deletion action, the downloads will all cancel and the Deletion will begin.

Our priority queue allows us to do this. If the deletion options are enabled after enqueuing all the download jobs (sounds like this is preferred but we may want to consider a way to just "cancel" the downloads instead, or in addition), we can just clear everything not currently being processed in the download_file_queue (for a particular source) when we come across a deletion job in the enqueue method.

---

This is as far as I got when going over these scenarios with @gonzalo-bulnes today. There's a lot to discuss here, and eventually we'll need to figure out how to plan all of this work. But for now, we'll start by going through all of these scenarios. The sprint ends next week, so I'm curious @gonzalo-bulnes how you're feeling about actually getting a minimal implementation of "Download All" done within the next 50 hours? First we'll have to scope this down, and plan out different phases of the work, so it would be useful to check back in on whether or not our original goal is still obtainable (https://github.com/orgs/freedomofpress/projects/1#card-73875766). Seems like there's still a lot of scoping and planning work needed, so i think it makes sense to put "Export All" (https://github.com/orgs/freedomofpress/projects/1#card-74026456) on the back burner and to redefine the "Download All" sprint commitment. Thoughts @gonzalo-bulnes? @eloquence?

[eloquence]

Thanks all. I would propose narrowing the scope for the MVP and subsequent iterations using the MoSCoW method. We can chat more about this tomorrow/Wednesday.

[ninavizz]

I didn't want to do a dialog, because of how often I expect users will be doing this @creviera (based on speaking to users in multiple interviews and testing).

We've learned from Journalists that Sources often upload multiple files in a sitting—so Journalist users, I expect will be doing command-d whenever they click on a source that has new files. They've already commented they love keyboard shortcuts (one requested shortcuts for deletion, last Thursday)—and that will be much easier than navigating to manually click on "Download" with the pointer/nubbie for each individual file. Hence, I've thought of this all along as a fast/friction-free action.

Dialog fatigue is a very thing, and we need to be very careful to not over-do it.

I think a Download Most, later, where yes, journalists can pick/choose which they want to download, could potentially have value—but it's also been observed (thus far) that Journalist users want to download either everything or nothing. Until big orgs with multiple users are all on the Workstation (so, not for the next 2-3yrs), I don't see that changing via research to date.

I thought a lot about what we'd discussed earlier (just before I left for CrossFit, and if a news organization is acting on a Source—they typically want to download & transfer everything. Not just a few things. Especially customers that have external workflows they need to transition all Source materials to, once a Source is deemed legit.

Remember: only 1/10 sources is not spam, and those Sources orgs tend to want everything from.

[ninavizz]

Ok! Figma does keyboard shortcuts, so on my Mac I did this as Command-D. Need to go open it on my Qubes machine to see if it works with the Command key, there—but it is mighty sweeet! clicking Command-D and seeing an action invoke from a prototype! Yep, it also works from the menu, too. :)

If/When you want to "replay," move your mouse outside the gray-desktop Qubes frame, to the sides where it's black... and click to see the Figma UI. In the lower-right there will be a white button to "Restart," just click that. :)

https://www.figma.com/proto/YejZGCIGZikX55LDcfvcYY/Client-%E2%89%A5-Q4-2021?page-id=432%3A46911&node-id=440%3A53291&viewport=241%2C48%2C1&scaling=scale-down&starting-point-node-id=432%3A46912

[ninavizz]

(...just added a super-sweet bit into the Source Widget that can stop the download should a user desire, while also letting them know things are still processing if they go to other Sources).

Features Shown (in above prototype)

• Ctrl/Cmmnd-D or Menu item invoke "Download All" action
• Kebab options change when "Download All" invoked
  • All Export/Delete actions, disabled
  • Shift-Ctrl-D or menu select will cancel all downloads (oh yeah!) for the selected source
• Spinner replaces paperclip in Source Widget when invoked
  • When hovering on spinner, an "x" will appear that can also cancel all the downloading files (oh yeah!).
• All not-downloaded files show to user as "Downloading" with spinners, and begin downloading (either in parallel or in sequence on the backend—but to the user, all the spinner states indicate they're all "processing")

Prototype may be a touch buggy still, but y'all get it.
No dialogs, cuz everything is start/stop easy/peasy. To cancel a download action, that should probably be Ctrl-Alt-D; but on my Mac I could only set it to be Command-Shift-D.

This is also probably where erik is chuckling and shaking his head. Minimum Usable Product, and Minimum Valuable Product; not just Minimum Viable Product, gang!

[eloquence]

Thanks for preparing! Per chat with @creviera, Gonzalo, Nina and I will discuss a bit more to narrow down the acceptance criteria for the first iteration. Some initial thoughts:

• The SecureDrop Client is already smart about handling multiple downloads one after the other while showing them all as active, so there should be no need to modify the behavior.
• Similarly, each SecureDrop Client installation has a separate download directory, and the behavior of multiple installs is not impacted.

In terms of other features proposed and implied, my initial proposal would be:

Must:

• Causes all files to be downloaded for the selected source
• Triggered via menu entry and keyboard shortcut
• Keyboard shortcut / menu entry disabled when action unavailable

Could:

• Spinner in source list [not sure of value and other implications yet]
• Some other menu actions no longer available [not sure of value and other implications yet]

Won't:

• Cancelling downloads [not a first iteration feature]
• Warning about download size [not sure of value, not a first iteration feature]
• Sounds [definitely not first iteration, separate discussion]

But we'll discuss further and then recap here.

[gonzalo-bulnes]

Summary of the MoSCoW exercise with @eloquence and @ninavizz

We are working on "Download All" (conversation for the selected Source) because we identified it as a necessary step to implement "Export All" (messages, replies and files for the selected Source). While we could develop it further as a standalone feature, this first iteration aims at enabling the "Export All" feature.

Must

• The conversation menu ("kebab") contains an action to "Download All"
• The action implements a shortcut: see Create keyboard shortcuts for Source kebab-menu items & display them in the menu #1358
• The action results in all the conversation files being downloaded (in chronological order)
• The action is enabled by default and disabled when "relevant" (e.g. no files to download, see https://gist.github.com/gonzalo-bulnes/6b981bd2cb54c18602c8a377d6b09b2b#action for inspiration)
• The menu looks reasonably similar to the Figma document, including icons, fonts, spacing and alignment. (For reference, enabled/disabled: Grimace Blue/Dusty Liac or Pasty Mauve)

Should

• The action is disabled when the source has no files
• The action is disabled when the source has no files that haven't been downloaded already
• The action is disabled when all files are either downloaded or in progress of being downloaded for the current source

Could

• The styling of the file buttons is updated, see Re-Style "Download" button in client #1362

Won't

• Cancelling downloads seems like a valuable feature, that we want to look into separately. (Both for cancelling individual downloads and cancelling all downloads at once.)
• Displaying information about the download size (whether it is in a dialog or as part of a progress indicator).
• Sound effects require further consideration
• Download indicator in the Source List. (The Figma prototype includes a spinner in each Source with downloads in progress, that could be enabled by individual or bulk downloads.)
• Enforcing prioritization between the "Delete" and "Download" actions, further than what the current UI already does

[gonzalo-bulnes]

Summary of a quick pairing session with @ninavizz today (:pear: :tada:):

• Overall the menu / behavior of the feature looks good
• We looked at a few layout options for the menu, and decided to go with "Download" section + "All Files" action.
• The section titles look currently closer to the action above (smaller top margin) than to the action below (larger bottom margin), which is the opposite of what we'd want. I'll take a look in the context of Add download conversation #1388 to see how difficult it is to adjust that.
• Adding the icons to the menu is a good follow up
• Same for adding the keyboard shortcuts to the other actions
• Both of those should likely come after releasing "Export All / Export Conversation" (remider: that feature is the reason why we're working on "Download All / Download Conversation" in the first place.)

For reference:

Note: The blue outlines are a development artifact. In production they'd be yellow because the sd-app qube has a yellow "label".