-
-
Notifications
You must be signed in to change notification settings - Fork 6.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
introduce chunked backups #11900
base: main
Are you sure you want to change the base?
introduce chunked backups #11900
Conversation
@Roghetti This sounds amazing, one feature i am really looking forward to! I do have a question, could you explain how it is beeing chunked? Also do the chunks change between runs? Eg. does only the last chunk change because it is the one containing the new data? |
Thanks! :-)
It's just like a normal backup, but split into multiple files, if a single file would exceed a certain size. E.g., if your normal backup has 1.5 GB, the chunked backup would create two files, one with 1GB and one with 0.5GB as 1GB is the threshold I chose.
All chunks are affected when certain data has changed. |
Ahhh thats a shame, i had hoped that this could reduce the data transferred each night and reduce wear on my flash-storage. Thanks anyway! |
c7d81e0
to
9fd5e3f
Compare
9fd5e3f
to
df5b022
Compare
df5b022
to
3175024
Compare
Hey there, appreciate the PR. I'm not sure if we'll pull this in or not. I'll bring it up with the team. |
@cody-signal There is also an discussion in the forum regarding this. I think we need to somehow adress this, because backups can easily grow beyond 4Gb, which is problematic for different reasons mentioned in the discussion |
6a92181
to
bddab19
Compare
The following feature request is related: Backup to FAT card |
Will the backups also be incremental? |
@HyperCriSiS Sadly no, this will only 'split' the backup-file into parts. I hope that they will switch to incremental backups eventually. There is a linked discussion. However, changing the backup-format should not be done very often, so we should do it properly. I think this pr should not be merged, since it fixes the symptom (backups beeing over 4Gb) and not the issue (missing segmentation/incremental backups) of ballooning backup-files |
dac29a5
to
7a45c82
Compare
I would not agree to that. Chunking would help people with devices with small internal storage but an SD-Card slot, if the Signal container size grows over 4GB and so most probably also the backup file size. Without chunking it needs 3 times >4GB (2 x existing backups + temporarily other copy until the new backup finished). The backup is encrypted so no problem to put it there. That is also the reason this issue is described initially. Looking forward that it gets merged. |
@Smojo I dont disagree with you that the growing filesize is a problem (it is, and will get even worse with time) However, the underlying issue is not that some filesystem has limitations that chunking needs to work around, but that signal-backup files grow in size indefinitively. Chunking DOES fix that, but only the symptom, not the problem. I would like to see a solution where a single backup is split by year, and then only made 'on demand' (do we ALWAYS have to store chats again for 2016?) This would solve the described issue AND the problem (as long as someone does not add more than 4Gb per year, but i deem that unlikely) |
Yeah, this might be something, which is needed and helps as well. Right now (not sure if I'm wrong) there is only a possibility to shorten chats by message-count (which also somehow results in a smaller container and so also smaller backup-file[s]), but nobody can really tell what this means in regards of "message retention" of a single chat. Some chats will not even go back a year or so if they are very talkative. So I would agree that a chunking by time, which results in a chunked container AND backup-files (which might also have a file-size limit of 4GB in addition to the split by time), will also solve the users problem described here. (without further thinking about symptom and problem ^^) Generally I would ask:
? |
7a45c82
to
40b6bcd
Compare
I think what the problem is is in the eye of the beholder. Incremental backups have the risk that an increment will break unnoticed. If, like me, you want to keep all your messages, the solution implemented here is generally reasonable and complete in itself. If you don't want to have old messages anymore, you could delete them in the application, then they wouldn't be part of a full backup anymore. Different people, different needs. Either way, this solution should be better than none. If you implement something better that would be all the cooler of course. But no reason to wait right? I would be very happy if this PR would be merged. Then I wouldn't have to delete backups by hand every day 😃 |
@lieblingsnerd Sure, different users have different needs. But there is an extremely good reason to not merge this/wait for now. As far as i can see it, signal dev's are quite conservative in terms of new features, and this will certainly be a breaking change (for backups). If the backup-system is getting changes, they will not do so lightly, and not often. That means waiting and taking time to really think through the issue is a very good reason to wait. Also: i do not consider 'delete some old messages' to be a valid solution. Backups are there because i DO want to keep old messages, deleting them cannot be the answer. And afaik you still need to delete them, this pr only splits the backup into multiple files that are smaller than the filesystem boundaries. |
I see it not as breaking change. The current status of the backup system is really bad because it is incomplete. |
@HyperCriSiS Any major change to the way backups are created are possibly a breaking change. This pr splits up the singular backup into multiple files. That requires long term support, and if not done correctly, may render old backups unusable. Therefore it could be breaking. However, your problem does not seem related at all, have you checked existing issues and opened a proper ticket for your problem if no ticket exists? |
Agree!
Agree ... doing the same as I'm out of storage + I cannot use the SD-card because of 4GB limit
Also had this issue, but this PR will probably not solve that one ^^
Agree. Why is a file chunking a breaking change? (just writing that realizing that I have no clue about coding ... still it will not break the current backup logic / way of working ... all the UI stuff can stay imho). Still @newhinton might be correct in regards of backward compatibility etc.
+1 |
Chunked backups of some kind are an important feature. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
It would be great to have SOME solution along these lines! |
Once I lost all history because backup failed of the file being too large for the file system. This is very important and highly needed functionality. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
There is still an urgent need for this functionality. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
This issue is not stale! It's just receiving the attention it deserves. The problem it addresses is not going away and is getting worse. It has resulted in me largely giving up on using Signal except in rare circumstances. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
This still remains a crucial feature. The lack of it means that I now use signal for far fewer people and purposes. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
bump |
And I second that bump. The current situation is untenable. Sadly my vote should perhaps be discounted because this deficiency has meant I have largely given up on signal. |
@greyson-signal over a year ago you commented ( #11900 (comment) ):
What happened since then? |
what is the status for incremental backups? |
I was tired waiting for smaller backup for years, I have implemented it as an external tool. So if anybody want to use my Signal Backup Deduplicator, it is available. |
Thanks for this! So nice, I'll test it! |
First time contributor checklist
Contributor checklist
Fixes #1234
syntaxDescription
I work for bevuta IT GmbH, a company based in Germany. Our boss is a Signal user on Android. He has the problem that he can't backup his Signal app data to the SD card and despite 128 GB of internal memory, he no longer has free internal storage, mainly because of the signal backups. The most reasonable explanation was that his Android version doesn't support file systems other than FAT32 on SD cards. FAT32 has some limitations, such as a maximum file size of ~4 GB. His Signal installation uses more than 4 GB. To us, the use of a sd card for backups also seems advantageous, as it is easier to get at the data if the device breaks down.
We discussed several solutions. The best solution we came up with, is to just chunk the Signal backup into parts smaller than 4 GB, if desired. I was tasked to implement it. To use this code in a real Signal installation, it has to be in the official release. Now, here is my MR! :-)