Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

btrfs-receive slows down exponentially (on large transfers) #908

Open
AmNotOnline opened this issue Oct 13, 2024 · 0 comments
Open

btrfs-receive slows down exponentially (on large transfers) #908

AmNotOnline opened this issue Oct 13, 2024 · 0 comments
Labels

Comments

@AmNotOnline
Copy link

I'm in the process of migrating my personal backups from an rsync-based approach to use snapshots. This means that I have to bootstrap a snapshot on my storage server. My home subvolume measured about 550 GiB (as reported by du -sh), and dumping that to a file via btrfs send --compressed-data -f ... resulted in a file of 419 GiB.

Initially, I tried sending the bootstrap snapshot over via an ssh connection, which more or less saturated the write speed of the receiving 1 TB HDD drive dedicated solely to backups. Yet, after multiple hours, this slows down to less than a megabyte transferred per second.

I tried this approach three times on separate days, completely wiping the receiving drive between transfer attempts.

Then, I tried dumping the snapshot to a file on an external HDD drive, and manually receiving it into my backup server. Again, for the first few hundred gigabytes of the transfer, the transfer speed was limited by the physical limitations of the drives. Eventually, this dropped to 600-800 KB/s (as reported by btop as total drive access for the backup drive).

This leads me to the conclusion that the problem is with btrfs receive, and not my flaky LAN setup.

TL;DR
Large snapshot transfers using btrfs receive slow down exponentially when far into large data loads.

Receiving System:

  • CPU: Intel Duo E7500
  • RAM: 4 GB
  • Backup Drive: Seagate Mobile ST1000LM035-1RK172 (1 TB)

Additional info:

  • Ran tests with smartctl on receiving drive, no errors were reported.
  • CPU usage only went as high as 5 %, so that shouldn't be the bottleneck.
  • Another 40 GB transfer using btrfs receive -f worked totally fine.
@kdave kdave added the bug label Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants