New --links parameter not working with mount?

[ErAzOr2k]

What is the problem you are having with rclone?

I uploaded some symlinks with "rclone copy" and parameter "--links" to my gdrive remote.
As expected, files with the extension "rclonelink" are generated.

I now expect that the rclonelink files are not visible via rclone mount. Instead they should be translated to symlinks. But actually it is not the case. Only plain rclonelink files are visible.

Additionally I'm also not able to create symlink. Instead I get I/O error (see below).

What is your rclone version (output from rclone version)

rclone v1.46-DEV (latest commit as of today)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux x64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --config /config/rclone.conf mount gdrive_decrypted:/ /mnt --allow-other --allow-non-empty --uid=1000 --gid=1000 --umask=007 --links --fast-list --rc--verbose

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

ln -s /tmp/testfile testlink
ln: testlink: I/O error

[ErAzOr2k]

Ok if I undetstand it correctly, it only works with local remote.
Is it possible to implement this feature for drive remote?

[ncw]

I think this would have to be a new flag for mount, say --use-links or something like that which inteprets the .rclonelink files and turns them back into symlinks.

Therefore you would be able to mount your drive remote, and have the symlinks appear as symlinks.

Is that what is required do you think?

[ncw]

This probably wouldn't be too tricky if someone wanted to have a go?

[ErAzOr2k]

I think this would have to be a new flag for mount, say --use-links or something like that which inteprets the .rclonelink files and turns them back into symlinks.

Therefore you would be able to mount your drive remote, and have the symlinks appear as symlinks.

Is that what is required do you think?

yes, this is exactly the expected behavior :)

[felisucoibi]

I hope this i implemented soon, i would like too mount to follow symlinks instead of show .rclonesymlink and e able to reate symlinks in the cloud (internally .rlclonelink) but for mount a symbolik one.

[diamondsw]

I was just wondering about this myself. It would certainly be useful if the mount layer could seamlessly translate rclonelink files back into symlinks (and bonus points if creating symlinks on a rw rclone mount would create rclonelink files behind the scenes).

[thekiefs]

I would also love this feature!

[kocane]

@ncw Any plans on implementing this? would be very useful. Had plans on using 'UrBackup' with Google Drive mount but as it's built completely on symlinks, it don't seem possible :-(

[mikwaluk]

It would be great if you could add this feature! :)

[dinoboy197]

I think this would have to be a new flag for mount, say --use-links or something like that which inteprets the .rclonelink files and turns them back into symlinks.

Therefore you would be able to mount your drive remote, and have the symlinks appear as symlinks.

Is that what is required do you think?

Hi @ncw - from the documentation for the --links flag in local mode:

--links, -l
...
If you supply this flag then rclone will copy symbolic links from the local storage, and store them as text files, with a '.rclonelink' suffix in the remote storage.
...
This flag applies to all commands.

Considering that the documentation written indicates that the flag applies to all commands, do you think it would reasonable instead to simply allow the pre-existing --links flag to work both for sync as well as mount operations?

[dinoboy197]

I've created a draft PR of a code addition to have the mount command support symbolic links. No extra mount parameters are needed; this works without any extra --links command line arguments.

I still need to add documentation and tests, but it should be ready for use. For anyone who is still interested in this feature, please try out my branch and let me know if that works for you! I hope to have the PR ready for review in the next week or two.

[ncw]

I've created a draft PR of a code addition to have the mount command support symbolic links.

Nice one :-)

No extra mount parameters are needed; this works without any extra --links command line arguments.

I'm not sure about that, but we can discuss more!

I still need to add documentation and tests, but it should be ready for use. For anyone who is still interested in this feature, please try out my branch and let me know if that works for you! I hope to have the PR ready for review in the next week or two.

I note you've done this in the mount layer. There are 3 mount backends - mount/cmount/mount2 and this only applies to one of them. It would be better to do this in the VFS layer which will work with all mount backends. The mount backends will still need a tweak to recognise symlinks though.

[eatnumber1]

For folks following along like myself, the PR is #4933. (thanks for working on this!)

[dinoboy197]

There are 3 mount backends - mount/cmount/mount2 and this only applies to one of them.

Wow - I wondered a bit about those, and didn't realize those were any more than prototypes. Thanks for the note!

It would be better to do this in the VFS layer which will work with all mount backends. The mount backends will still need a tweak to recognise symlinks though.

I doubt that will be any issue, though I admit that I'm not quite sure how the other two mount backends that I didn't tweak actually get enabled. Would you be able to point me to any existing documentation or tips on how to enable the two alternate mount backends so that I can be sure that any changes work with those?

[ncw]

There are 3 mount backends - mount/cmount/mount2 and this only applies to one of them.

Wow - I wondered a bit about those, and didn't realize those were any more than prototypes. Thanks for the note!

No worries!

Note also that the VFS layer is used for the rclone servers too, eg rclone serve http/sftp/webdav

It would be better to do this in the VFS layer which will work with all mount backends. The mount backends will still need a tweak to recognise symlinks though.

I doubt that will be any issue, though I admit that I'm not quite sure how the other two mount backends that I didn't tweak actually get enabled. Would you be able to point me to any existing documentation or tips on how to enable the two alternate mount backends so that I can be sure that any changes work with those?

Sure...

You use rclone mount ..., rclone mount2 ... and rclone cmount ....

You need to supply the build tag -tags cmount to build with cmount support, and you'll need the FUSE headers and a C compiler. cmount uses cgo with the FUSE library, whereas mount talks straight to the kernel.

Note that rclone mount is the default mount for linux/bsd/other whereas rclone cmount is the default mount for Windows/macOS as rclone mount doesn't work there.

At some point I plan to drop mount2 but rclone needs mount and cmount!

[dinoboy197]

After some initial refactoring, I realized that symlink support down into the VFS layer means building symlink support for several other components - not just mount and cmount. I realized that I would need to make some fairly significant design choices regarding how to represent symlinks at that layer - should they be a File, but have a new isSymlink method, or should there be a new type that derives from Node entirely; should symlinks support propagate further down chain into the FS layer, etc. While I was aware earlier that my symlink addition for the mount command was fairly trivial and somewhat superficial, I didn't have a proper appreciation for what all would go into proper support in the VFS layer. Not surprising considering I'm a first-time rclone contributor!

I don't believe I have the right experience or vision to be making these kinds of changes in the VFS layer as a first-time contributor to rclone - I'd have to ask many questions and take up a considerable amount of time trying to sync with the future vision of VFS in order to come up with a solution that I thought was satisfactory. Rather, I'd feel much more comfortable if someone with more of an ownership stake in the VFS architecture and vision implemented the symlink support they'd like to see in that layer, then I'd be comfortable retrofitting my mount solution to use whatever VFS hooks were built out.

I say this not a way to suggest that my changes should go to the rclone mainline just because I'd rather not implement things in the VFS layer. Rather, I do think that VFS support is the right way to implement this and look forward to someone with more experience (both with rclone and with Golang) to drive that a bit. I'm sorry if this comes as a disappointment for anyone waiting for a full solution.

I'll keep an eye out here, and should I become more well-versed with the codebase and VFS direction at a later date, perhaps will pick this up again!

[ivandeex]

The attempt PR #4933 was abandoned and closed.
New developers can pick it by the link above and use its patch as a basis.

[ncw]

This feature is approaching ready - please try this beta if possible

v1.69.0-beta.8381.204b90a79.fix-2975-mount-symlinks on branch fix-2975-mount-symlinks (uploaded in 15-30 mins)

Use the --links flag on rclone mount to enable.

[pjhfggij]

fantastic, I'm looking forward for be able to setup python virtual environments directly on rclone mounts (they require symlinks by default)

[hongkongkiwi]

This feature is approaching ready - please try this beta if possible

v1.69.0-beta.8381.204b90a79.fix-2975-mount-symlinks on branch fix-2975-mount-symlinks (uploaded in 15-30 mins)

Use the --links flag on rclone mount to enable.

is this in the official beta build installed from the website?

[ncw]

is this in the official beta build installed from the website?

Not yet, please try from the link above

[mikklfr]

Hello, a big thank you for working on this feature.
Unfortunately provided beta is not "mounting" the .rclonelink file on my end, folder does not contain the symlink.

rclone mount --links local:/mnt/symlinks/test /mnt/test

/mnt/symlinks/test contains two file one is a symlink of the other.
Only the "regular" file is showing up using ls.

logs:

2024/11/12 17:57:10 NOTICE: Local file system at /mnt/symlinks/test: Symlinks support enabled
2024/11/12 17:57:12 NOTICE: test/testlink.txt: Can't follow symlink without -L/--copy-links

using:

rclone v1.69.0-beta.8381.204b90a79.fix-2975-mount-symlinks
- os/version: ubuntu 24.04 (64 bit)
- os/kernel: 6.8.0-1014-oracle (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.23.2
- go/linking: static
- go/tags: none

[darthShadow]

It may be colliding with the --links flag of the local FS. Can you prefix the mount command with RCLONE_LOCAL_LINKS=true and see if that works?

[mikklfr]

It may be colliding with the --links flag of the local FS. Can you prefix the mount command with RCLONE_LOCAL_LINKS=true and see if that works?

Seems you pinpointed the correct thing 👍 Working as expected now, thanks!

[ncw]

It may be colliding with the --links flag of the local FS. Can you prefix the mount command with RCLONE_LOCAL_LINKS=true and see if that works?

Well spotted @darthShadow

I'm not sure why that is happening - I would have expected the --links flag to be picked up by both the VFS and the local backend.

Assuming I can fix it - I think that is the correct think to do - would you agree?

I did consider making a separate flag --vfs-links to separate it from the local backend --links flag. But then I was thinking that is probably not what users expect.

Maybe I should stop the --links flag being a local backend specific flag and make it a global flag which the local backend reads as does the VFS - that is the usage I was thinking of. That is going to be untidy though!

Thoughts?

[darthShadow]

I think that is the correct think to do - would you agree?

Yeah, I think using the same flag would be the right approach. I would even go a step further and suggest we remove the --links flag altogether from the local backend and use the rclonelink behavior as the default, with an override as now using the copy-links flag.

I am just not sure if anyone has gotten used to the behaviour of seeing the .rclonelink file and may break their scripts/apps if they start seeing the symlink instead.

[ncw]

I've made the --links flag global so it controls both the VFS and the local backend here. There are also individual flags --local-links and --vfs-links if you want to be more selective.

If anyone wants to have a try

v1.69.0-beta.8433.67d905e40.fix-2975-mount-symlinks on branch fix-2975-mount-symlinks (uploaded in 15-30 mins)

@darthShadow wrote

Yeah, I think using the same flag would be the right approach. I would even go a step further and suggest we remove the --links flag altogether from the local backend and use the rclonelink behavior as the default, with an override as now using the copy-links flag.

So make --links the default effectively for the local backend? That would be a backwards incompatible change.

I am just not sure if anyone has gotten used to the behaviour of seeing the .rclonelink file and may break their scripts/apps if they start seeing the symlink instead.

They will only see the symlink on rclone mount everything else will remain as before. You won't see symlinks unless you enable it explicitly with --links or --vfs-links.

[darthShadow]

That would be a backwards incompatible change.

Yeah, but it would bring it inline with the behavior of the other backends. I am fine either way, just wanted to bring it up once to see if anyone else had any comments on it.

[ncw]

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.69

Thank you @pasnox

Any more testing gratefully received :-)

BTW this was the resolution to the --links flag problem which gives the most flexibility at the cost of a few more flags (!).

fs: make --links flag global and add new --local-links and --vfs-links flag

Before this change the --links flag when using the VFS override the
--links flag for the local backend which meant the local backend
needed explicit config to use links.

This fixes the problem by making the --links flag global and adding a
new --local-links flag and --vfs-links flags to control the features
individually if required.