kind restic and other tools benchmark report

[deajan]

Hello,

I'm currently doing benchmarks for deduplication backup tools, including restic.
I decided to write a script that would:

• Install the backup programs
• Prepare the source server
• Prepare local targets / remote targets
• Run backup and restore benchmarks
• Use public available data (linux kernel sources as git repo) and checkout various git tags to simulate user changes in the dataset

The idea of the script would be to have reproductible results, the only changing factor being the machine specs & network link between sources and targets.

So far, I've run two sets of benchmarks, each done locally and remotely.
You can find the results at https://github.com/deajan/backup-bench

I'd love you to review the recipe I used for restic, and perhaps guide me on what parameters to use to get maximum performance.
Any remarks / ideas / PRs are welcome.

I've also made a comparaison table of some features of the backup solutions I'm benchmarking.
I still miss some informations for some of the backup programs.
Would you mind having a look at the comparaison table and fill the question marks related to the features of restic ?
Also, if restic has an interesting feature I didn't list, I'll be happy to extend the comparaison.

PS: I'm trying to be as unbiased as possible when it comes to those benchmarks, please forgive me if I didn't treat your program with the parameters it deserves.

Also, I've created the same issue in every git repo of the backup tools I'm testing, so every author / team / community member can judge / improve the instructions for better benchmarking.

[JsBergbau]

Just some notes.

I'd love you to review the recipe I used for restic, and perhaps guide me on what parameters to use to get maximum performance.
Any remarks / ideas / PRs are welcome.

What do you understand as maximum performance? You can disable compression or enable maximum compression for smaller backupsizes, see https://restic.readthedocs.io/en/latest/047_tuning_backup_parameters.html?highlight=max#compression

Also, if restic has an interesting feature I didn't list, I'll be happy to extend the comparaison.

Restic has its rest-server https://github.com/restic/rest-server and this has an append-only mode. With this you can make trojan safe backups, since backups can only be added but not removed. Very cool feature.

Some of your table
Are metadatas encrypted too ? Yes, see https://restic.readthedocs.io/en/latest/100_references.html#threat-model
Can encrypted / compressed data be guessed (CRIME/BREACH style attacks)? Restic is quite immune to watermark attacks https://restic.readthedocs.io/en/latest/100_references.html#backups-and-deduplication
Can a compromised client delete backups? Depends on the backend and how it is configured. With rest-server even automated backups are secure (see above)
Can a compromised client restore encrypted data? Depends if you store the password on the client or not
Can a repo be synchronized to another repo ? Yes, simplest solution is using rsync or using integrated mechanism, between different repositories https://restic.readthedocs.io/en/latest/045_working_with_repos.html#copying-snapshots-between-repositories

[deajan]

Thanks for your answers. I've updated my comparaison table.

What do you understand as maximum performance? You can disable compression or enable maximum compression for smaller backupsizes, see https://restic.readthedocs.io/en/latest/047_tuning_backup_parameters.html?highlight=max#compression

I meant best overall settings. Indeed maxing out compression would reduce filesize. But I try to not compare apples to oranges, ie have identical compression across all backup programs.

Can a compromised client restore encrypted data? Depends if you store the password on the client or not

I am supposed to have the password on the client in order to add data to the encrypted repository, ain't it ?
Other backup software have mechanisms like a "backup only" password which does not allow to decrypt data.
How is this solved using restic ?

So far restic does best on remote restore operations, due to rest_server which has amazing speeds.
I was thinking there might be some settings to improve backup speed, like thread settings or something alike.

[JsBergbau]

I meant best overall settings. Indeed maxing out compression would reduce filesize. But I try to not compare apples to oranges, ie have identical compression across all backup programs.

Best overall settings is very subjective I would say. I prefer best compression over speed, because the backup is meant for longterm storage.

I am supposed to have the password on the client in order to add data to the encrypted repository, ain't it ?
Other backup software have mechanisms like a "backup only" password which does not allow to decrypt data.
How is this solved using restic ?

Currently yes. However I think when your client is compromised you have a lot more problems than restoring data from the backup. However there is this feature request #187 that would support asymmetric encrytion. This is intended to enable deduplication between servers without the possibility to get access to the data of the other servers. But this would of course also fit your use case: Encryption key only to prevent a compromised client to restore old backups.

I was thinking there might be some settings to improve backup speed, like thread settings or something alike.
You could disable compression to increase backup speed when backup speed limit is due to CPU usage. However I would not recommend this. About one day ago there has been added a feature to tune settings to increase backup speed, especially for fast disks like SSD, see #2750

EDIT: The comparison "Can a repo be synchronized to another repo ?" needs an update, that this is possible with restic.

[MichaelEischer]

Regarding the comparison table:

• "Redundant index copies": restic can rebuild the index from the pack files if necessary
• "Continue restore on bad blocks in repository": Partially. in most cases yes, restore currently will fail if data is missing. Workaround for that are excluding the affected files during restore. Or using Add repair command #2876 to remove the file from the snapshot.
• "Data checksumming": I don't understand the answers. HMAC and reed-solomon serve totally different purposes. restic uses SHA256 hashes to verify that the data is as expected (plus an AEAD cipher to prevent tampering with the encrypted chunks).
• "Supports backup XATTRs": yes
• "Supports backup ACLs": only if these are mapped to XATTRs
• "Supports sparse files": that PR is not yet included in a released restic version
• "Does the backup software support pre/post execution hooks?": No, we consider scheduling to not be restic's responsibility
• "Can a repo be synchronized to another repo ?": the copy command can transfer snapshots between repositories

Comments to the comparison:

• For measurement durations in the single-digit second range it would be a good idea to use timestamps with a higher resolution. Otherwise the quantization can lead to a rather high measurement error.
• The average file size in the qemu image backup looks odd: "17344875" kb. That rather looks like a file size in bytes, not kilobytes. The file count also seems to be wrong. For restic 0.14.0 I'd expect about 7000.
• "Getting restic SFTP to work with a different SSH port made me roam restic forums and try various setups. Didn't succeed in getting RESTIC_REPOSITORY variable to work with that configuration.". Just add a matching entry to you ssh_config, and then use the corresponding hostname/alias. Or use the URL syntax: sftp://host:port/$path. If $path is absolute, then you'll have two consecutive slashes: sftp://host:port//path.

About one day ago there has been added a feature to tune settings to increase backup speed, especially for fast disks like SSD, see #2750

That might be interesting for SSD to SSD backups. Or when using 10Gbit Ethernet. In almost all other cases, I wouldn't expect that increasing the number of file readers makes any significant difference.

The available tuning options are listed at https://restic.readthedocs.io/en/latest/047_tuning_backup_parameters.html .

[MichaelEischer]

Further comments:

• For the initial git benchmark it is a bit misleading to show the gzip/xz compressed size of the kernel tree including the .git directory, but afterwards excluding .git for the actual backups. The latter part also seems to be missing from the description. Actually, it would be rather interesting to see how small the repo could be.
• To make the results more readable, file size in MB would be preferable.
• How high is the latency between both systems? And what is the maximum bandwidth?

[JsBergbau]

That might be interesting for SSD to SSD backups. Or when using 10Gbit Ethernet. In almost all other cases, I wouldn't expect that increasing the number of file readers makes any significant difference.

Depends. Some benchmark:
backend rest-server on encrypted spinning disk on Raspberry PI 4. Backend is limited in this configuration to about 500 MBit/s throughput.
Backupsystem: S-ATA SSD, Ryzen 5 Pro 5650GE (6 Cores, 12 Threads)
Env variables
set RESTIC_PACK_SIZE=32
set RESTIC_COMPRESSION=max

TL;DR
With default read concurreny it takes 201 seconds, with read-concurreny of 6 it takes 104 seconds. So speed / performance has almost doubled.

Feel free to move this benchmark to another thread.

Details:

With default read-concurrency of 2 adding a folder with 512 MP3s (2,52 GB) and 44 txt files (total 2,48 KB in size):

repository 0878d514 opened (repository version 2) successfully, password is correct
created new cache in C:\Users\XXX\AppData\Local\restic
no parent snapshot found, will read all files

Files:         556 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repository: 2.526 GiB (2.504 GiB stored)

processed 556 files, 2.526 GiB in 3:21
snapshot 251eecde saved

CPU usage is about 40 to 50 %.

With read concurrency of 4

repository 802b8e7c opened (repository version 2) successfully, password is correct
created new cache in C:\Users\XXX\AppData\Local\restic
no parent snapshot found, will read all files

Files:         556 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repository: 2.526 GiB (2.504 GiB stored)

processed 556 files, 2.526 GiB in 2:01
snapshot d05e3f9e saved

CPU usage is about 80 - 85 %

With read concurreny of 5:

repository 7091443a opened (repository version 2) successfully, password is correct
created new cache in C:\Users\XXX\AppData\Local\restic
no parent snapshot found, will read all files

Files:         556 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repository: 2.526 GiB (2.504 GiB stored)

processed 556 files, 2.526 GiB in 1:50
snapshot 77411c83 saved

With read concurrency of 6

repository 26c58c17 opened (repository version 2) successfully, password is correct
created new cache in C:\Users\XXX\AppData\Local\restic
no parent snapshot found, will read all files

Files:         556 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repository: 2.526 GiB (2.504 GiB stored)

processed 556 files, 2.526 GiB in 1:44
snapshot 81474633 saved

[MichaelEischer]

Hmm, you're right. The archiver pipeline can stall after reading a file while waiting for all blobs to be queued for uploading:

restic/internal/archiver/file_saver.go

Line 199 in 7112a13

for i, res := range results {

Fixing that would require fixing #3869 and then tweaking the blob saving to complete the FutureBlob right after checking whether the blob has to be uploaded or already exists. Or alternatively rework the FileSaver part which waits for the FutureBlobs to be completely asynchronous.

The first variant is probably the easier way. Although I haven't thought much about the second alternative.
[Edit]Calculating the size of the compressed blob won't be possible until after actually compressing it...[/Edit]

[MichaelEischer]

@JsBergbau Could you give PR #3955 a try? That should improve the archiver performance far enough that my initial assessment, that the --read-concurrency is mostly useful for very fast storage, is correct again.

[MichaelEischer]

Closing as we've hopefully answered the questions.