macOS: modification and creation dates reset #229

alexanderharm · 2018-04-28T19:26:12Z

I started using gocryptfs (latest version) on macOS High Sierra and noticed that each time I eject my encrypted volume and remount it, that the creation and modification dates are reset to 1970-01-01. Is this expected behaviour?

I mount my dirs using

gocryptfs -ko local /path/to/encDir /Volumes/encDir

Also mounting them without the -ko local option has the same result.

The text was updated successfully, but these errors were encountered:

rfjakob · 2018-04-28T19:36:14Z

Huh. What happens to the dates of the encrypted files?

alexanderharm · 2018-04-28T19:42:19Z

The date of the metadata file (?) is correct, the date of the data file is also reset.

rfjakob · 2018-04-28T19:46:00Z

When you write to a file, the encrypted file stays at 1970?

alexanderharm · 2018-04-28T19:46:56Z

For a very brief moment it shows the correct datetime but after max. one second it is reset.

rfjakob · 2018-04-28T19:53:45Z

Well, that's funny. I'll see if I can get my hands on a macos box next week to reproduce this.

I the meantime, can you run gocryptfs with

"-fg -nosyslog -fusedebug"

an see what happens when you modify a file?

alexanderharm · 2018-04-28T20:02:40Z

I don't see anything unusual but then I don't have a clue about this :-)
gocryptfs.log

All but the newest OSX versions (xnu-4570.1.46, released 2017) lack utimensat() and UTIME_OMIT, see: https://github.com/apple/darwin-xnu/blame/0a798f6738bc1db01281fc08ae024145e84df927/bsd/sys/stat.h#L537 The UTIME_OMIT value is intepreted literally by darwin, resulting in all files getting a 1970 timestamp ( rfjakob/gocryptfs#229 ). Emulate UTIME_OMIT by filling in the missing values using an extra GetAttr call.

rfjakob · 2018-04-29T19:24:59Z

Maybe found it. Can you try this build? gocryptfs.gz
gocryptfs v1.4.4-29-g996d2f1 without_openssl; go-fuse v20170619-32-g4ff4497; 2018-04-29 go1.10

alexanderharm · 2018-04-29T19:40:22Z

I tried and noticed the following:

Mounted decrypted new file: creation date still 1970, modification date: fine

Encrypted new file: both dates seem to look fine
BUT: I cannot modify existing files anymore (see attached screenshot)

rfjakob · 2018-04-29T20:39:46Z

Ok, thanks for testing! So the problem is where I though it is, but the fix is causing other problems

alexanderharm · 2018-04-29T20:41:46Z

Well, thank you for all your efforts and prompt replies. Really appreciated!

All but the newest OSX versions (xnu-4570.1.46, released 2017) lack utimensat() and UTIME_OMIT, see: https://github.com/apple/darwin-xnu/blame/0a798f6738bc1db01281fc08ae024145e84df927/bsd/sys/stat.h#L537 The UTIME_OMIT value is intepreted literally by darwin, resulting in all files getting a 1970 timestamp ( rfjakob/gocryptfs#229 ). Emulate UTIME_OMIT by filling in the missing values using an extra GetAttr call.

rfjakob · 2018-05-01T15:21:34Z

Ok, I have that Mac box. And I can reproduce the save-existing-files problem: Happens in TextEdit as well, but does not seem to be related the fix. Also happens with older gocryptfs versions.

On the command line, writing files works fine, so it seems to be that the MacOS GUI file save functions seem to be unhappy about something.

alexanderharm · 2018-05-01T16:27:29Z

Did you use the Homebrew version? Because with the latest version from Homebrew I don't have any problem apart from the file dates, never had. So I cannot confirm that it also happens with older gocrypt versions, at least not on my systems.

We previously returned EPERM to prevent the kernel from blacklisting our xattr support once we get an unsupported flag, but this causes lots of trouble on MacOS: Cannot save files from GUI apps, see #229 Returning ENOSYS triggers the dotfiles fallback on MacOS and fixes the issue.

rfjakob · 2018-05-01T21:46:37Z

Ok it seems that xattr support is what causes the The file is locked problems. This is not yet released and not yet in homebrew. Change fe38906 should fix this, care to test the new build (attached)?

gocryptfs.gz

alexanderharm · 2018-05-02T05:39:53Z

Tested the build and it works for me. However, creation date is still 1970-01-01 but all the rest works fine!

All but the newest OSX versions (xnu-4570.1.46, released 2017) lack utimensat() and UTIME_OMIT, see: https://github.com/apple/darwin-xnu/blame/0a798f6738bc1db01281fc08ae024145e84df927/bsd/sys/stat.h#L537 The UTIME_OMIT value is intepreted literally by darwin, resulting in all files getting a 1970 timestamp ( rfjakob/gocryptfs#229 ). Emulate UTIME_OMIT by filling in the missing values using an extra GetAttr call.

Tests loopbackFileSystem.Utimens() and loopbackfile.Utimens() at 1-second precision. The exising TestUtimesNano() test only works on Linux because it relies on syscall.UtimesNano(), which is not available on Darwin. The new tests call the Utimens() functions directly, bypassing FUSE, and work on all platforms. Because Darwin does not have syscall.UtimesNano(), getting the Utimens() implementation right is hard. The tests currently fail on Darwin, underlining the need for them ( rfjakob/gocryptfs#229 ): $ go test ./fuse/nodefs [...] --- FAIL: TestLoopbackFileUtimens (0.00s) files_test.go:51: mtime has changed: 1525378914 -> 1073 files_test.go:70: atime has changed: 1525291058 -> 1073 [...] $ go test ./fuse/pathfs --- FAIL: TestLoopbackFileSystemUtimens (0.00s) loopback_test.go:55: mtime has changed: 1525378929 -> 1073 loopback_test.go:74: atime has changed: 1525291058 -> 1073 [...]

Tests loopbackFileSystem.Utimens() and loopbackfile.Utimens() at 1-second precision. The exising TestUtimesNano() test only works on Linux because it relies on syscall.UtimesNano(), which is not available on Darwin. The new tests call the Utimens() functions directly, bypassing FUSE, and work on all platforms. Because Darwin does not have syscall.UtimesNano(), getting the Utimens() implementation right is hard. The tests currently fail on Darwin, underlining the need for them ( rfjakob/gocryptfs#229 ): $ go test ./fuse/nodefs [...] --- FAIL: TestLoopbackFileUtimens (0.00s) helpers.go:51: mtime has changed: 1525384186 -> 1073 helpers.go:70: atime has changed: 1525291058 -> 1073 [...] $ go test ./fuse/pathfs --- FAIL: TestLoopbackFileSystemUtimens (0.00s) helpers.go:51: mtime has changed: 1525384379 -> 1073 helpers.go:70: atime has changed: 1525291058 -> 1073 [...]

rfjakob · 2018-05-03T22:01:47Z

Just to let you know, this needs fixing in the go-fuse library and will take some time, step one is this: hanwen/go-fuse@bb848eb

alexanderharm · 2018-05-04T06:20:44Z

Indeed, this can take a while. Any chance of getting your "fixed" version into Homebrew. Somehow I don't feel comfortable that I published a non-working version there... And I have to come back to ctime because I don't see it addressed anywhere. Any chance of getting that fixed as well?

rfjakob · 2018-05-04T07:44:44Z

The "some time" this will take are probably two weeks. The issue is not too serious, and I believe it only affects High Sierra users. This is also why it was not noticed before. Things are mostly working, I don't think we need to rush out a fix.

ctime: yes, I will try to get this fixed as well!

alexanderharm · 2018-05-04T07:48:11Z

I think High Sierra is the worst bug-fix/maintenance release ever...

rfjakob · 2018-05-04T08:54:15Z

Well, maybe, but in that case, the bug is in gocryptfs / go-fuse ;)

All but the newest OSX versions (xnu-4570.1.46, released 2017) lack utimensat() and UTIME_OMIT, see: https://github.com/apple/darwin-xnu/blame/0a798f6738bc1db01281fc08ae024145e84df927/bsd/sys/stat.h#L537 The UTIME_OMIT value is intepreted literally by darwin, resulting in all files getting a 1970 timestamp ( rfjakob/gocryptfs#229 ). Emulate UTIME_OMIT by filling in the missing values using an extra GetAttr call.

All but the newest MacOS versions (xnu-4570.1.46 / High Sierra) lack utimensat() and UTIME_OMIT, see: https://github.com/apple/darwin-xnu/blame/0a798f6738bc1db01281fc08ae024145e84df927/bsd/sys/stat.h#L537 The UTIME_OMIT value is intepreted literally by MacOS, resulting in all files getting a 1970 timestamp ( rfjakob/gocryptfs#229 ). Emulate UTIME_OMIT by filling in the missing values using an extra GetAttr call. To not duplicate the logic in pathfs and nodefs, an internal "utimens" helper package is created.

All but the newest MacOS versions (xnu-4570.1.46 / High Sierra) lack utimensat() and UTIME_OMIT, see: https://github.com/apple/darwin-xnu/blame/0a798f6738bc1db01281fc08ae024145e84df927/bsd/sys/stat.h#L537 The UTIME_OMIT value is intepreted literally by MacOS, resulting in all files getting a 1970 timestamp ( rfjakob/gocryptfs#229 ). Emulate UTIME_OMIT by filling in the missing values using an extra GetAttr call. To not duplicate the logic in pathfs and nodefs, an internal "utimens" helper package is created. This patch fixes the failing utimens tests.

rfjakob · 2018-05-06T19:47:33Z

Does this one fix the creation time issue? gocryptfs.gz

alexanderharm · 2018-05-06T21:06:36Z

Unfortunately not.

rfjakob · 2018-05-06T21:29:57Z

Thanks. The good news is, I have reproduced it here:

$ stat b/Untitled.rtf 
855638233 4572339 -rw-r--r-- 1 user group 0 375 [...] "Jan  1 01:00:00 1970" 131072 2048 0 b/Untitled.rtf

rfjakob · 2018-05-06T21:40:02Z

The bad news is that this is unfixable, as FUSE has no support for file creation ("birth") time.
Struct fuse_attr just does not have a field for it: https://github.com/libfuse/libfuse/blob/21b55a05a158b1c225ba312529bc068cadd5431d/include/fuse_kernel.h#L158

Anyway, I'll keep this ticket open until the timestamp fixes are merged into go-fuse.

alexanderharm · 2018-05-08T06:44:47Z

According to the author of osxfuse this should not be an issue:

FUSE for macOS sends FUSE_GETXTIMES messages to your user space file system daemon. You need to reply with the creation time. I don't know how this is handled by the Go library you are using. But this is definitely no t a limitation of FUSE.

Tests loopbackFileSystem.Utimens() and loopbackfile.Utimens() at 1-second precision. The exising TestUtimesNano() test only works on Linux because it relies on syscall.UtimesNano(), which is not available on Darwin. The new tests call the Utimens() functions directly, bypassing FUSE, and work on all platforms. Because Darwin does not have syscall.UtimesNano(), getting the Utimens() implementation right is hard. The tests currently fail on Darwin, underlining the need for them ( rfjakob/gocryptfs#229 ): $ go test ./fuse/nodefs [...] --- FAIL: TestLoopbackFileUtimens (0.00s) helpers.go:51: mtime has changed: 1525384186 -> 1073 helpers.go:70: atime has changed: 1525291058 -> 1073 [...] $ go test ./fuse/pathfs --- FAIL: TestLoopbackFileSystemUtimens (0.00s) helpers.go:51: mtime has changed: 1525384379 -> 1073 helpers.go:70: atime has changed: 1525291058 -> 1073 [...]

All but the newest MacOS versions (xnu-4570.1.46 / High Sierra) lack utimensat() and UTIME_OMIT, see: https://github.com/apple/darwin-xnu/blame/0a798f6738bc1db01281fc08ae024145e84df927/bsd/sys/stat.h#L537 The UTIME_OMIT value is intepreted literally by MacOS, resulting in all files getting a 1970 timestamp ( rfjakob/gocryptfs#229 ). Emulate UTIME_OMIT by filling in the missing values using an extra GetAttr call. To not duplicate the logic in pathfs and nodefs, an internal "utimens" helper package is created. This patch fixes the failing utimens tests.

rfjakob · 2018-05-08T08:17:46Z

Thanks for finding out! Not supported at the moment by either gocryptfs or go-fuse, though. This is a mac-specific extension of the FUSE protocol.

alexanderharm · 2018-05-08T08:31:28Z

I know, but maybe we can get hanwen to implement it in go-fuse...

rfjakob · 2018-05-08T20:48:34Z

Maybe another day...

Fix was merged into go-fuse (hanwen/go-fuse@76c2303), I'll close the issue.

rfjakob · 2018-06-12T21:21:18Z

Released as v1.5

warmup72 · 2019-04-14T08:43:28Z

is this problem still under work?
I´ve seen yesterday that all my files unter macOS have the creation date 1.1.1970.
The encrypted-folder is fine/works but the files in the mounted folder show the wrong creation date :-(

rfjakob · 2019-04-14T08:51:38Z

The original problem was fixed in gocryptfs 1.5. You see the same problem again? Which gocryptfs and macos version?

alexanderharm · 2019-04-14T08:53:10Z

@rfjakob I think he talks about the birth date issue. In version 1.5 you fixed the modification dates.

@warmup72 Nope, afaik no one tackled it. The problem is not in gocryptfs but in hanwen/go-fuse, see hanwen/go-fuse#218. According to the author of osxfuse this should be an easy fix but I didn't manage since I lack the skills. It would be great if someone could tackle this because I rather like gocryptfs but I also started using cryptomator since that one also supports versioning (and you don't get the complaints about unsupported volume when using Number, Pages etc.

warmup72 · 2019-04-14T12:57:52Z

I´m using 1.7 on macOS 10.14.4
Yes, as I said the creation date is set to 1970:
Mounted Path
stat ./data/*

905969707 12897042013 -rw-r--r-- 1 Simon staff 0 264826 "Apr  9 18:05:16 2019" "Dec  4 16:03:39 2018" "Apr  9 14:54:24 2019" "**Jan  1 01:00:00 1970**" 131072 2048 0 ./data/181204_180012_OSRAM_LED_STAR_PAR16.pdf
905969707 12895479981 drwxr-xr-x 4 Simon staff 0 128 "Apr 14 10:35:07 2019" "Mar 17 16:46:35 2019" "Mar 17 16:46:35 2019" "**Jan  1 01:00:00 1970**" 131072 2048 0 ./data/Neuer Ordner
905969707 12897042014 -rw-r--r-- 1 Simon staff 0 2171559 "Apr 14 14:51:38 2019" "Nov 20 10:03:00 2017" "Apr  9 14:54:24 2019" "**Jan  1 01:00:00 1970**" 131072 6144 0 ./data/eHZ_Stromzähler_Produkthandbuch.pdf

encrypted Path
stat ./crypt/*

16777220 12897042014 -rw-r--r-- 1 Simon staff 0 2188569 "Apr 14 14:51:38 2019" "Nov 20 10:03:00 2017" "Apr  9 14:54:24 2019" "**Nov 20 10:03:00 2017**" 4096 4280 0 ./crypt/9GYIE_K6h2D4GVfJNfPfOzWwB6MDlNnHSMxzvbBCm3wpKkdiwTTRp1YLYyGLkKGV
16777220 12895479981 drwxr-xr-x 4 Simon staff 0 128 "Apr 14 10:35:07 2019" "Mar 17 16:46:35 2019" "Mar 17 16:46:35 2019" "**Mar 17 16:46:35 2019**" 4096 0 0 ./crypt/lMsNrys1ZotFOcRU5lDvKg
16777220 12897042013 -rw-r--r-- 1 Simon staff 0 266924 "Apr  9 18:05:16 2019" "Dec  4 16:03:39 2018" "Apr  9 14:54:24 2019" "**Dec  4 16:03:39 2018**" 4096 528 0 ./crypt/uLmlTA1sPWA8fr3yDA9X6fMVLZBmLItn9KCgwo17PzNyo2ewOkqRIKQmfewCgbg4

warmup72 · 2019-04-14T13:03:32Z

BTW: I also tried cryptomator but they didn´t support tags which I need for my DMS

warmup72 · 2019-04-17T08:26:57Z

Maybe this issue should be reopend?

rfjakob · 2019-04-17T20:29:34Z

Could you create a new one? I would prefer that, as the modification time issue was fixed already.

This was referenced May 1, 2018

Xattr: Darwin and Apple native applications compatibility improvement #230

Closed

Added passing of the flags parameter to the lower-level syscall pkg/xattr#26

Merged

rfjakob mentioned this issue May 3, 2018

fuse: add unit-tests for loopback Utimens() hanwen/go-fuse#215

Closed

rfjakob added the bug label May 6, 2018

alexanderharm mentioned this issue May 7, 2018

How to set creation time osxfuse/osxfuse#501

Closed

rfjakob closed this as completed May 8, 2018

alexanderharm mentioned this issue May 9, 2018

Support Btime (birth time, creation time) on macOS hanwen/go-fuse#218

Open

ysmolski mentioned this issue Feb 9, 2022

Mac: Creation Date is set to 1.1.1970 #396

Open

macOS: modification and creation dates reset #229

macOS: modification and creation dates reset #229

Comments

alexanderharm commented Apr 28, 2018

rfjakob commented Apr 28, 2018

alexanderharm commented Apr 28, 2018

rfjakob commented Apr 28, 2018

alexanderharm commented Apr 28, 2018

rfjakob commented Apr 28, 2018

alexanderharm commented Apr 28, 2018

rfjakob commented Apr 29, 2018

alexanderharm commented Apr 29, 2018

rfjakob commented Apr 29, 2018

alexanderharm commented Apr 29, 2018

rfjakob commented May 1, 2018

alexanderharm commented May 1, 2018

rfjakob commented May 1, 2018

alexanderharm commented May 2, 2018

rfjakob commented May 3, 2018

alexanderharm commented May 4, 2018

rfjakob commented May 4, 2018

alexanderharm commented May 4, 2018

rfjakob commented May 4, 2018

rfjakob commented May 6, 2018

alexanderharm commented May 6, 2018

rfjakob commented May 6, 2018 • edited Loading

rfjakob commented May 6, 2018

alexanderharm commented May 8, 2018

rfjakob commented May 8, 2018

alexanderharm commented May 8, 2018

rfjakob commented May 8, 2018

rfjakob commented Jun 12, 2018

warmup72 commented Apr 14, 2019

rfjakob commented Apr 14, 2019

alexanderharm commented Apr 14, 2019

warmup72 commented Apr 14, 2019 • edited Loading

warmup72 commented Apr 14, 2019

warmup72 commented Apr 17, 2019

rfjakob commented Apr 17, 2019

rfjakob commented May 6, 2018 •

edited

Loading

warmup72 commented Apr 14, 2019 •

edited

Loading