-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ability to add files to the compressed filesystem without having to decompress/recompress the whole thing #18
Comments
I'm not saying it wouldn't make sense, but doing it in a good way would likely be hard. Ideally, when adding a new set of files, you'd like to end up with a similar result than if you had recompressed the whole file system. But that would mean having to insert the files in between other files that are already in the file system in order to be able to make good use of the available redundancy. I'm not saying this can't be done, just that it's probably too much work to get done any time soon. What I'd probably do is something like:
I don't know if that would would help in your case, but it's something that could probably be automated quite easily. |
Yeah, I didn't imagine it was a five-min fix. If we do end up using DwarFS in the near future we'd very likely do something pretty similar to what you suggest. However, we already have an automated system with zstd archives of single recent commits and lrzip archives of bundles of older commits. The appeal of DwarFS (assuming it has a compression ratio similar enough) is that we could rip out all the code we have to handle decompressing the different versions when wanting to use a specific commit and just run something at a known path in the filesystem. Thanks for the quick response, and I wish we'd known about DwarFS back in 2016 when we were first creating our system! |
Just to clarify the last comment, the reason FWIW here's our journey: Raku/whateverable#23 |
Well, in 2016 it was still sitting on my laptop, I sadly didn't have the time (and energy) to publish it back then. |
Just a quick update: I'm planning to add support for "snapshots" (or whatever the feature will ultimately be called), which would definitely address this issue; in fact, it will go a lot further. You'll be able to not only mount/extract the latest update, but also all previous updates. Each update would only store the changes relative to the previous state, which would e.g. allow you to use DwarFS for incremental backups. There's no timeline, so don't hold your breath, but it'll hopefully happen before v1.0.0. :) |
If this gets implemented, what's the intention with regards to writeability? Obviously incremental backups would require providing both a source directory and existing DwarFS image, but would mounting an modifying an existing one be a consideration, too? Would that become the default behavior if that were the case or would images remain read-only by default? |
I have no plans for making the file system writable at this point. |
I'm experimenting with using DwarFS for a very similar use case as yours (a build of each Rakudo commit). However, given there are several new commits every day, creating a new DwarFS image from scratch each time doesn't make sense. Would it be possible to add new data to the compressed filesystem without having to decompress/recompress the whole thing?
The text was updated successfully, but these errors were encountered: