feat: support UnixFS 1.5 file mode and modification times #653
Conversation
Replaces PR #34 written by @kstuart Adds support for storing and retrieving file mode and last modification time. Support added to: - [X] Files - [X] LinkFiles - [X] Webfiles - [X] Directories - [X] Tar Archives When the TAR archive (headers) include a file mode or modification time, the extractor will restore that metadata when supported for the underlying filesystem. The Golang runtime currently does not support changing the times on a symlink, for Linux and some BSDs a custom solution has been implemented, for Darwin this is not the case so when copying a symlink to the filesystem the last modification time is not updated. Since for concrete files and directories stored modes and modification times are faithfully restored to the filesystem this should not be a breaking issue, a similar solution to that implemented for Linux/BSDs is likely implementable by a developer with access to a Darwin platform. Replaces PRs: - ipfs/go-ipfs-files#31 - ipfs/tar-utils#11 - #34 Relates to ipfs/kubo#6920
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Codecov ReportAttention: Patch coverage is
@@ Coverage Diff @@
## main #653 +/- ##
==========================================
+ Coverage 59.89% 60.10% +0.21%
==========================================
Files 237 241 +4
Lines 29955 30600 +645
==========================================
+ Hits 17942 18393 +451
- Misses 10398 10563 +165
- Partials 1615 1644 +29
|
Replaces #7754 written by @kstuart - ipfs/boxo#653 - ipfs/boxo#658 - [X] Can `ipfs add` with preserved mode and/or last modification time - [X] on files - [X] on directories - [X] Can `ipfs add` with custom mode and/or last modification time - [X] on files - [X] on directories - [X] Can `ipfs get` restoring mode and/or last modification time - [X] on files - [X] on directories - [X] in archives - [X] Can `ipfs files chmod` to change mode - [X] on files - [X] on directories - [X] Can `ipfs files touch` to change last modification time - [X] on files - [X] on directories - [X] Automatically update the last modification time when file data is changed or truncated (e.g. `ipfs files write`) - [X] Can add files and directories with mode and/or modification time using multipart-form data - [X] `ipfs files stat` reports mode and last modification time **Note:** - [X] Adds support to `kubo/core/rpc` (may require additional tests). - ~ipfs/interface-go-ipfs-core/pull/66~ replace by this PR - ~ipfs/go-unixfs/pull/85~ replaced by: ipfs/boxo#658 - ~ipfs/go-mfs/pull/93~ replaced by: ipfs/boxo#658 - ~ipfs/go-ipfs-files/pull/31~ replaced by: ipfs/boxo#653 - ~ipfs/tar-utils/pull/11~ replaced by: ipfs/boxo#653 - When adding files and directories without opting to store a mode or modification time the same CIDs are generated that would have been created before this feature was implemented (opt-in). - The Go runtime currently has no native support for restoring file mode and modification time on symbolic-links, support for restoring the last modification time has been added for Linux distributions and the following BSDs: freebsd, netbsd, openbsd, dragonflybsd. - Automatically updating a modification time will only occur if a modification time was previously stored. - When creating an archive, for compatibility, time resolution is to the second; Nanoseconds are not supported. The `ipfs add` options `--preserve-mode` and `--preserve-mtime` are used to store the original mode and last modified time of the file being added, the options `--mode`, `--mtime` and `--mtime-nsecs` are used to store custom values, a custom value of 0 is a no-op as is providing `--mtime-nsecs` without `--mtime`. The preserve flags and custom options are mutually exclusive, if both are provided the custom options take precedence. --- Closes #6920
…658) * feat: Support UnixFS mode and modification times in ipld dag and mfs Adds support for storing and retrieving file mode and last modification time. Support added to: - Files - LinkFiles - Webfiles - Directories Tar archives are supported by the parent branch.
There was a problem hiding this comment.
Did bunch of testing with ipfs/kubo#10478 and I think we are mostly good to go.
Need to decide if/how we address below issues.
| // mostly copied from proto 3 - with int32 nanos changed to fixed32 for js-ipfs compatibility | ||
| // https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/timestamp.proto | ||
| message IPFSTimestamp { | ||
| // Represents seconds of UTC time since Unix epoch | ||
| // 1970-01-01T00:00:00Z. Must be from 0001-01-01T00:00:00Z to | ||
| // 9999-12-31T23:59:59Z inclusive. | ||
| required int64 seconds = 1; | ||
|
|
||
| // Non-negative fractions of a second at nanosecond resolution. Negative | ||
| // second values with fractions must still have non-negative nanos values | ||
| // that count forward in time. Must be from 0 to 999,999,999 | ||
| // inclusive. | ||
| optional fixed32 nanos = 2; |
There was a problem hiding this comment.
I feel we should have some GO-JS interop tests for nanos somewhere.
Initially planned to do that in gateway-conformance after #659 is done, but realized that the Last-Modified HTTP header in gateway responses will not include nanosecond precision. It only supports second precision with an optional millisecond component.
@achingbrain any suggestions? my initial idea is to add mode and mtime tests (live creation and with static dag-pb fixture) to https://github.com/ipfs/helia/tree/main/packages/interop#readme after we have kubo 0.30.0-rc1 ad updated kubo-rpc-client that works with new RPC in JS, but lmk if there is a better place/way.
| func Chmod(rt *Root, pth string, mode os.FileMode) error { | ||
| nd, err := Lookup(rt, pth) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| return nd.SetMode(mode) | ||
| } | ||
|
|
||
| func Touch(rt *Root, pth string, ts time.Time) error { | ||
| nd, err := Lookup(rt, pth) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| return nd.SetModTime(ts) | ||
| } |
There was a problem hiding this comment.
@gammazero found a bug: ipfs files touch and ipfs files chmod fail when we execute them against raw blocks.
Repro steps with ipfs/kubo#10478:
$ echo -n "hello path gw" | ipfs add --cid-version 1 -Q --raw-leaves --to-files /raw-test-1
bafkreiehuhfvxn7w4i5hexlhq4a3y7jf5kfogsvmm5zlfxg5poj4oqozyu
$ ipfs files touch /raw-test-1
Error: expected a ProtoNode as internal node
$ ipfs files chmod 0644 /raw-test-1
Error: expected a ProtoNode as internal nodelmk if you feel this is fixable in this PR before Wednesday, or should we move ahead with RC1, fill bug for this and fix later.
(hopefully we can add code that detects when node is a raw block and upgrade it to dag-pb before setting mode/time. otherwise, might cause a bigger refactor, because MFS hardcodes rawleaves=true for mutating files that use CIDv1)
There was a problem hiding this comment.
To unblock RC testing, I've extracted it into #660.
There was a problem hiding this comment.
Thanks! Bumped and added end-to-end test in Kubo PR as well ipfs/kubo@55dad60
gammazero
force-pushed
the
feat/unixfs15
branch
from
68902be
to
47dca25
Compare
gammazero
force-pushed
the
feat/unixfs15
branch
from
47dca25
to
f4259d6
Compare
switching to version with fix from ipfs/boxo#653 (comment) and adding tests
There was a problem hiding this comment.
Thank you @kstuart for initial version, and @gammazero for refactoring and implementing the missing parts and pushing over finish line.
This should be ready for shipping in kubo 0.30.0-rc1 and getting user feedback from folks that asked for mode and mtime.
We have a bunch of tests here and in ipfs/kubo#10478 (ignore sharness failing here, it passes in kubo PR), and these extra attributes have been around in JS for years, and are always opt-in, so relatively low risk.
commit after merging ipfs/boxo#653
Replaces PR #34 written by @kstuart
Adds support for storing and retrieving file mode and last modification time.
Support added to:
When the TAR archive (headers) include a file mode or modification time, the extractor will restore that metadata when supported for the underlying filesystem.
The Golang runtime currently does not support changing the times on a symlink, for Linux and some BSDs a custom solution has been implemented, for Darwin this is not the case so when copying a symlink to the filesystem the last modification time is not updated. Since for concrete files and directories stored modes and modification times are faithfully restored to the filesystem this should not be a breaking issue, a similar solution to that implemented for Linux/BSDs is likely implementable by a developer with access to a Darwin platform.
Replaces PRs:
Relates to ipfs/kubo#6920