all: Add copy-on-write filesystem support (fixes #4271) #6746

AudriusButkevicius · 2020-06-15T18:44:54Z

I can't merge into my own branch and have a PR here, so opening the full thing.
Only the last commit is interesting here.

lib/fs/basicfs_copy_range.go

lib/fs/basicfs_copy_range_copyfilerange.go

lib/fs/basicfs_copy_range_ioctl.go

lib/fs/basicfs_copy_range_sendfile.go

imsodin

Just a few "organisational" nits.

lib/fs/basicfs_copy_range.go

lib/fs/basicfs_copy_range_allwithfallback.go

AudriusButkevicius · 2020-06-17T18:11:05Z

We gud?

calmh · 2020-06-18T06:15:40Z

Write a docs page on roughly what the different configs do?

AudriusButkevicius · 2020-06-18T07:01:57Z

Also, when we add this in the change log, we should mention that this is an option you need to enable.

Atera · 2020-06-18T07:03:11Z

I volunteered to test in the other PR. Would love to give this a spin. If the documentation's not done yet, could you briefly state what I need to tweak to enable the changes? I add a 'copyRangeMethod' to the folder options and set it to 'copy_file_range'? I seem to see options standard, ioctl, copy_file_range, sendfile, and all, but I'm not sure I'm looking in the right place or what these behaviours are.

AudriusButkevicius · 2020-06-18T07:15:43Z

If you start a master build you'll see new option in folder config after the first save, change that option to one of the mechanisms as appropriate.

* main: (218 commits) gui, man, authors: Update docs, translations, and contributors lib/protocol: Use sha256.Sum256 in NewDeviceID (syncthing#6775) lib/scanner: Fix Validate docs (syncthing#6776) build: Set CGO_ENABLED=0 by default, except on specific platforms (syncthing#6768) gui, lib/api: Add more version info to about dialog (syncthing#6773) lib/model: Use right db snap when scanning recvonly folder (syncthing#6769) build: Change version strings to not have plus in them (ref syncthing#6758) (syncthing#6760) lib/model: Don't ignore stat failure in performFinish (syncthing#6766) lib/fs: Add support for Windows duplicate extents (syncthing#6764) lib/model: Don't stay scanning forever on fail (syncthing#6761) gui, lib/ignore: Handle editing ignores with error (fixes syncthing#5425) (syncthing#6757) lib: Print nicely rounded durations (syncthing#6756) all: Add copy-on-write filesystem support (fixes syncthing#4271) (syncthing#6746) lib/rand: Various minor fixes (syncthing#6752) lib/db: Improve error message on meta inconsistency (syncthing#6751) lib/db: Use explicit byte type for type prefixes (syncthing#6754) gui, man, authors: Update docs, translations, and contributors lib/model: Check dir before deletion when pulling (syncthing#6741) build: Update integration tests and add to build.go (syncthing#6744) cmd/stdiscosrv: Don't abuse wrong header (syncthing#6749) ...

calmh · 2020-07-14T07:24:27Z

Any docs on this available? I would like to point to a page with some guidance so that we can get some testing early in the 1.8.0 RC cycle as I expect we might need iterations on this...

AudriusButkevicius · 2020-07-14T07:30:05Z

Yes, it's in the folder section.

calmh · 2020-07-14T08:12:36Z

Right... Mind if I copy/move it to a separate advanced section, to make it easier to link & refer to?

AudriusButkevicius · 2020-07-14T08:48:06Z

Sure, whatevs.

mksully22 · 2020-08-11T21:34:26Z

This pull request (commit 4812fd3) fails compile on ppc64le (compiling on Alpine).
A git bisect was used to narrow down the commit that was responsible. The Error reported:

--- FAIL: TestCopyRange (2.73s)
    --- FAIL: TestCopyRange/tmp (2.73s)
        --- FAIL: TestCopyRange/tmp/ioctl (0.34s)
            --- FAIL: TestCopyRange/tmp/ioctl/append_to_end (0.02s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/append_to_end_offsets_at_start (0.03s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/overwrite_part_of_destination_region (0.02s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/overwrite_all_of_destination (0.03s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/overwrite_part_of_destination (0.02s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/destination_gets_expanded_as_it_is_being_written_to (0.02s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/unaligned_source_offset_unaligned_size (0.01s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/unaligned_source_offset_aligned_size (0.03s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/unaligned_destination_offset_unaligned_size (0.03s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/unaligned_destination_offset_aligned_size (0.03s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/aligned_offsets_unaligned_size (0.03s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/last_block (0.01s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/whole_file_copy_block_aligned (0.01s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
            --- FAIL: TestCopyRange/tmp/ioctl/whole_file_copy_not_block_aligned (0.01s)
                filesystem_copy_range_test.go:332: inappropriate ioctl for device
FAIL
FAIL    github.com/syncthing/syncthing/lib/fs   3.214s

AudriusButkevicius · 2020-08-11T21:37:57Z

There was a follow up PR in 1.8.0 that probably fixes this.

mksully22 · 2020-08-11T21:48:46Z

Thank you Audrius. I just re-attempted to build the latest (main branch) with git log showing the latest commit being:
'''
commit fd8bea6 (HEAD -> main, origin/main, origin/HEAD)
Author: Simon Frei [email protected]
Date: Mon Aug 10 18:48:37 2020 +0200

lib/osutil: Preserve perms in AtomicWriter (fixes #tbd) (#6885)

'''

It also fails in the same way at this level. Is the PR you are thinking of uncommitted? If so can you give me a link to it? I'd be happy to give it a try.

AudriusButkevicius · 2021-06-29T17:01:28Z

For people finding this ticket in the future.
This is now implemented as an optional configuration, but the default behaviour remains the same, copy data into application memory.

You can find the docs for the feature here: https://docs.syncthing.net/advanced/folder-copyrangemethod.html

AudriusButkevicius and others added 30 commits February 16, 2019 17:45

lib/fs: Add fs.CopyRange

e9f797f

gofmt

c162d7c

Fix unix

673af18

Fix types

7a445de

More type fixes

1ef0573

Refactor tests

57a07a3

Fix builds

e09b280

Make slices big enough

9182709

Fix types yet again

aecddfe

Fix yet again

a9c310f

Rename files

51839aa

Merge and stuff

e409a21

More fixes

1c8a6ff

Fix sendfile calling convention, allow tests to run in a custom dir

55936fc

Tweaks

1f0e37e

Tweak tests

451f937

Massage sendfile

b525541

More massage

7dac4a5

Optimise offset tweaks, use fs.CopyRange where we can

ef50a37

Dont use it yet

2903248

Rething registration

5f7dad0

Missing files

4de1614

Dead code

82eabdd

Fix builds

9f9b043

Dont use it yet

814ec3d

noop change

d166509

Update basicfs_copy_range_generic.go

fc3ca28

Fix copy_file_range exit condition, add test, normalise errors

5f72ed8

Review comments

3163fcb

Typo

c5526ef

AudriusButkevicius added 2 commits June 16, 2020 09:10

Push down config

24bb3dd

Fix tests

e9bd31a

AudriusButkevicius changed the title ~~Use copy range~~ all: Add copy-on-write filesystem support (fixes #4271) Jun 16, 2020

calmh reviewed Jun 16, 2020

View reviewed changes

AudriusButkevicius added 2 commits June 16, 2020 10:46

Tweaks

3d6837d

Add EAGAIN handling back

57db96f

imsodin reviewed Jun 16, 2020

View reviewed changes

lib/fs/basicfs_copy_range.go Outdated Show resolved Hide resolved

lib/fs/basicfs_copy_range_allwithfallback.go Outdated Show resolved Hide resolved

Everyday Im shufflin

201925d

calmh approved these changes Jun 18, 2020

View reviewed changes

calmh merged commit 4812fd3 into syncthing:main Jun 18, 2020

calmh added this to the v1.8.0 milestone Jun 18, 2020

AudriusButkevicius deleted the use_copy_range branch June 18, 2020 07:02

This was referenced Aug 11, 2020

model: folderconfiguration/CopyRangeMethod Catfriend1/syncthing-android#674

Closed

model: Add [folder]/copyRangeMethod for compatibility with Syncthing v1.8.0 (fixes #674) Catfriend1/syncthing-android#685

Merged

st-review added the frozen-due-to-age Issues closed and untouched for a long time, together with being locked for discussion label Jun 18, 2021

syncthing locked and limited conversation to collaborators Jun 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

all: Add copy-on-write filesystem support (fixes #4271) #6746

all: Add copy-on-write filesystem support (fixes #4271) #6746

AudriusButkevicius commented Jun 15, 2020 •

edited

Loading

imsodin left a comment

AudriusButkevicius commented Jun 17, 2020

calmh commented Jun 18, 2020

AudriusButkevicius commented Jun 18, 2020

Atera commented Jun 18, 2020

AudriusButkevicius commented Jun 18, 2020

calmh commented Jul 14, 2020

AudriusButkevicius commented Jul 14, 2020

calmh commented Jul 14, 2020

AudriusButkevicius commented Jul 14, 2020

mksully22 commented Aug 11, 2020

AudriusButkevicius commented Aug 11, 2020

mksully22 commented Aug 11, 2020

AudriusButkevicius commented Jun 29, 2021

all: Add copy-on-write filesystem support (fixes #4271) #6746

all: Add copy-on-write filesystem support (fixes #4271) #6746

Conversation

AudriusButkevicius commented Jun 15, 2020 • edited Loading

imsodin left a comment

Choose a reason for hiding this comment

AudriusButkevicius commented Jun 17, 2020

calmh commented Jun 18, 2020

AudriusButkevicius commented Jun 18, 2020

Atera commented Jun 18, 2020

AudriusButkevicius commented Jun 18, 2020

calmh commented Jul 14, 2020

AudriusButkevicius commented Jul 14, 2020

calmh commented Jul 14, 2020

AudriusButkevicius commented Jul 14, 2020

mksully22 commented Aug 11, 2020

AudriusButkevicius commented Aug 11, 2020

mksully22 commented Aug 11, 2020

AudriusButkevicius commented Jun 29, 2021

AudriusButkevicius commented Jun 15, 2020 •

edited

Loading