Add download conversation

[gonzalo-bulnes]

🔮 Reviewers! The commits are built individually and are meant to do one thing at a time. I left a couple of quirks when splitting them, that you'll likely notice if you review commit by commit, but I think it's still worth [reviewing commit by commit] if the overall diff feels a bit overwhelming. 🙂

Once we're there, I'll rebase the branch one more time so that it's fully up to date before merging, just let me know.

Description

Closes #1354

Please refer to the issue for context.

Summary:

• This PR enables a journalist to download all the files of the currently visible conversation at once, via the conversation menu (a.k.a "source menu", a.k.a "kebab menu") or a keyboard shortcut (Ctrl+D).
• From that single user action, the files are downloaded one by one, the oldest file first. Progress is visible to the journalist in the same way as for individual downloads: files currently being downloaded display a spinning icon, and two "export" and "print" buttons replace that icon once the download is ready.
• If a conversation has no files or all its files have already been downloaded (whether individually or all at once), the menu action is disabled. (That's also true to the keyboard shortcut.)

This feature is a prerequisite to introducing the possibility of exporting a entire conversation at once.
🍐 @creviera

Implementation Decisions

1. The state of the files is tracked in Python, in a new structure called state.State.
2. The state refers to files, source and conversations using identifiers that treat the server-generated identifiers as opaque strings. As far as the client state is concerned, it doesn't matter that files are identified by a UUID, a number, or something else as long as the identifiers are unique.
3. GUI elements interact with that new state.State structure exclusively: both the controller and the database are abstracted away from the GUI. (Scope: only for this feature. We can propagate the pattern later if we like.)
4. The GUI subscribes to changes of the state via Qt signals.
5. A limited number of GUI elements update the state via Qt slots.
6. Part of the state is persisted to the database so that it remains available across reboots. That state was already being persisted, and this PR doesn't change that (out of scope). That state is loaded when the application starts, and it is currently the only interaction of state.State with the database. (A database.Database object was introduced for clarity of the API and ease of testing.)
7. The downloading of the files is performed using the same job queue, and the same job types as individual downloads. The only addition is making sure the state gets notice when downloads succeed.
8. The action is called "All Files" within a "Download" menu section (see Enable a user or Export to "Download All" files to the Workstation for a given source #1354)

Reference diagram

           Sync --updates ---\
                (ongoing)    |
                             v
GUI  --communicates with--> State --abstracts--> Controller ------manages--> Queue of API Jobs --downloads from--> SDK | Server
                             |                        |
                      reads from                 persists state to*
                   (intialization)                    |
                              \---> Database  <------/
(*) State will do it in the future, but that change is out of scope for this PR.

Reference screenshots

This is what things should look like. (These were taken in a development environment, the original mockups are in #1354).

Test Plan

Note: Thorough the test plan, please try to activate the "All Files" action menu by clicking on it, but also using the Ctlr+D shortcut.
Note on Qubes OS: Performing the testing in at least one Workstation is necessary, because some of the UI may behave differently on Qubes OS (e.g. the conversation menu will be framed by the label color of the sd-app qube.)

It's a fairly long test plan, here we go!

• Make sure your server is up to date (see SecureDrop 2.1.0)
• Start the server, start the client, select a source.
• Confirm that the conversation and its files are shown as usual.
• Confirm that the "conversation menu" (a.k.a "source menu", a.k.a "kebab menu") contains a sub-section "Download" and an action "All Files".
• Confirm that the "All Files" action is enabled for conversations that have at least one file that hasn't been downloaded before.
• Confirm that the files can be downloaded individually (by activating their "Download" buttons) as usual.
• Confirm that downloading all the files of a conversation one by one (by activating their "Download" buttons) causes the "All Files" action to be disabled.
• Confirm that activating the "All Files" menu action causes all the not-already-downloaded files of the conversation to start downloading.
• Confirm that when downloading all files, the oldest are downloaded first.
• Confirm that once all files are downloaded, the "All Files" action is disabled.
• For later, leave some conversation with a few files to download, one file to download, no files to download, mixing those you download individually and those you download all at once.
• Stop the client.

Take a breath!

---

• Start the client.
• Confirm that the "All Files" action is disabled for conversations that have no files to download, and enabled for all the other. (Independently of the fact that the files were or not downloaded individually or via "All Files").
• Confirm that everything in the steps above is still true.

Next comes testing the sync.

---

• Stop the server (or somehow disconnect the client, this could be done emulating a network issue)
• Wait until the client displays the banner that indicates that the connection to the server was lost.
• Upload new files for existing sources. (Ideally upload file to some sources that already have files to download and other that don't have any yet, so that you can confirm later that the action was enabled as expected.)
• Create new sources, upload files from some sources, and leave other conversations without files
• Start the server (or somehow fix the network issue)
• Confirm that the banner is cleared as usual when the client is automatically sync'd
• Confirm that the new sources are loaded
• Confirm that the "All Files" action is enabled for the conversations of the new sources with files
• Confirm that the "All Files" action is disabled for the conversations of the new sources with no files
• Confirm that the new files are visible for the previously existing sources
• Confirm that the "All Files" action is enabled for the conversations that were added files
• Confirm that the "All Files" action is disabled for the conversations that didn't have files and weren't added files

Take a breath again : )

---

• Start downloading "All Files"
• Disconnect the server while the files are downloading (ideally, some were downloaded, others not yet)
• Confirm that the client behaves as usual with respect of failed download tasks. (@creviera I need your help to detail this step!)
• Stop the client

Remain some steps specific to using the keyboard shortcut.

---

• Start the client, do not select any source.
• Confirm that pressing Ctlr+D ("the shorcut") does not trigger any download (even when all conversation have files that could be downloaded)
• Select a source
• Confirm that pressing "the shortcut" causes all files to be downloaded for the conversation that's visible.
• Confirm that the "All Files" action in the conversation menu is disabled once all files were downloaded using the shortcut.
• Select a different source
• Confirm that the shortcut only causes files to be downloaded for the currently selected conversation

Thanks for sticking through! Last but not least, please let me know of any scenarios you'd test that I didn't think of : ) ⭐
Bonus points for getting the client into an inconsistent state! 🌟

Checklist

If these changes modify code paths involving cryptography, the opening of files in VMs or network (via the RPC service) traffic, Qubes testing in the staging environment is required. For fine tuning of the graphical user interface, testing in any environment in Qubes is required. Please check as applicable:

• I have tested these changes in the appropriate Qubes environment
• I do not have an appropriate Qubes OS workstation set up (the reviewer will need to test these changes)
• These changes should not need testing in Qubes

If these changes add or remove files other than client code, the AppArmor profile may need to be updated. Please check as applicable:

• I have updated the AppArmor profile
• No update to the AppArmor profile is required for these changes
• I don't know and would appreciate guidance

If these changes modify the database schema, you should include a database migration. Please check as applicable:

• I have written a migration and upgraded a test database based on main and confirmed that the migration applies cleanly
• I have written a migration but have not upgraded a test database based on main and would like the reviewer to do so
• I need help writing a database migration
• No database schema changes are needed

[sssoleileraaa]

slight pref for a rename to: DownloadConversationAction, similar to DeleteConversationAction

[gonzalo-bulnes]

I went for DownloadConversation (no ...Action suffix) because eventually I'd see the actions defined in gui/conversation/actions.py and used with a prefix:

from securedrop_client/gui/conversation import actions

#     prefix --v                          v-- suffix would be redundant in the 'actions' module
self.AddAction(actions.DownloadConversation(self, state, self.controller))

[gonzalo-bulnes]

(rebase with no changes)

[eloquence]

For the sprint starting 1/5, we will aim to get this PR fully to "Ready for review", including front-end changes and the test plan.

[gonzalo-bulnes]

(rebase)

[gonzalo-bulnes]

Consolidated commits, added tests. The last four commits are still very draft-y.

[sssoleileraaa]

Hi @creviera ! I see there are a couple of conflicts with main, let me know if you want me to rebase the PR : )

Hello! Looks like a botched rebase, so I'll fix it up and finish the approval.

[gonzalo-bulnes]

👍 (I've got a local backup too, if needed.)

[sssoleileraaa]

@gonzalo-bulnes 💯 if you're able to work today, please feel free to fix this up and ping me when finished (sorry about the extra work)

[gonzalo-bulnes]

👌 @creviera! No worries!

[gonzalo-bulnes]

Rebased @creviera - I had a few core dumps from xvfb when running make check locally, I believe it's likely to be local, but if you can run it too I think it doesn't hurt to double check.

For info, the conflicts were:

• some tests were removed that I had touched to add a parameter to the Controller constructor
• one method signature changed that is used in one test (ConversationView.on_reply_sent)

[sssoleileraaa]

I ran through the test plan again, and this still looks good. I'm going to create a followup issue for something I'd rather be worked on outside of this PR (since the PR is already pretty large) and it's a minor issue around needing to disable the "Download All" option if all conversation items have been deleted.