[Feature Request] Symlink support

[JohnyDays]

Is there any performance / complexity reason for not supporting symlinks?

With the release of react-native, and the use of watchman in its packager, it's troubling that npm's (react-native's chosen package manager) endorsed method of developing local interdependent libraries with npm link is not supported, and having to commit to a remote repo and download on every change is obviously not ideal.

[wez]

Watchman recursively watches files that are contained within a filesystem tree.
Symlinks can point to an arbitrary location on the filesystem.
Changes to symlink targets that are outside the tree are not observable.

Why not simply resolve the symlink and watch its target? Because it isn't simple. Here are a handful of reasons that make this a difficult prospect. This is not an exhaustive list, it's just a few reasons I can rattle off the top of my head:

• The target of the symlink may not exist at the time we want to establish the watch, and neither may any of the parent directories in the symlink path. The operating system will not allow us to watch a path that does not exist
• Any of the directory components in the target symlink path may themselves be a symlink and may change at an arbitrary time in their lifecycle in a way that is not detectable unless we watch every single directory component in the symlink target path
• The target of the symlink, or any of the directory components of the symlink target, may be a remote filesystem that doesn't support change notifications
• Each additional watched directory consumes more filesystem watch resources. There are finite limits on these, and not every system allows raising these limits.
• Because the hypothetical watches that we'd establish for each symlink are distinct from the watch used for the root of the real tree, we lose certain ordering guarantees when processing change notifications from the kernel. We wouldn't be able to tell whether a change to the symlink target happened logically before or after changes that we observe in the main tree. This important property is used to ensure that filesystem changes have been observed up to the logical point in time of a query. More details on this can be found at https://facebook.github.io/watchman/docs/cookies.html
• Filesystem changes are fraught with TOCTOU (time of check, time of use) issues already. Adding user-space resolution of symlinks increases the chances of more TOCTOU bugs dramatically; any component of the symlink path may change while we're processing information about them.

Correctly handling all of the above would be tremendously complex and still be error prone. As a result, it is unlikely that we'll ever add support for resolving and tracking symlink targets.

[wez]

Taking a step back from this, can you describe what you're trying to do and why symlinks are important? I'm a casual and occasional node and npm user and am not familiar with npm link

[JohnyDays]

I understand why this would be difficult, and I'll close the ticket and resign to using the above solution for developing local modules.

I'll attempt to explain the process behind npm link, with my use case as an example.

I am developing a module called react-native-waterfall, it implements a generic waterfall view.
I am at the same time developing, in a sister folder, a module called react-native-social-waterfall, and this module depends on the above module.
I am at the same time developing utility modules, which both of the above modules depend on.

npm requires modules from either:

1. a globally qualified name e.g react-native-waterfall
2. a path e.g ../react-native-waterfall
3. an url e.g https://github.com/facebook/watchman/

The second solution has 2 problems:

1. Every time you change any file in a dependent module, you have to run npm install ../react-native-waterfall to reinstall the module.
2. You are checking in code that doesn't mirror the final product, where you will be using globally qualified namespaces hosted on npm.

The third solution has 2 problems as well(both are slow)

1. Every time you change any file in a dependent module, you have to commit it and push it to origin.
2. You must then go back to your dependent module and do npm update to grab the latest commit

npm link works the following way:

1. You go into the depended upon module and write npm link, this will register react-native-waterfall as a globally qualified name
2. You go into the dependent module and write npm link react-native-waterfall, this will register a symlink to the globally qualified name, using the folder where you wrote npm link.

This allows the following benefits:

1. Your checked in code mirrors the final product.
2. You do not have to do anything to update the local dependency files, meaning you can quickly test code without committing / running commands.
3. You can even edit the depended upon module in your dependent folder, and the changes will be applied to the original module.

[wez]

Thanks. How does watchman fit into this?

[JohnyDays]

I am using watchman with the react-native packager, which automatically parses dependencies, transforms code and reloads the application when you change a file. However I have to continuously restart the packager because it doesn't detect the symlinked dependencies's changes.

[JohnyDays]

Idea:
I understand why the added complexity of implicitly following symlinks would cause too many troubles, both in implementation and in maintenance. So would it be possible to instead implement an explicit/manual symlink helper in the form of:

/dir1/
/dir2/

watch dir1
watch dir2 --link-to dir1

Any change in a file at dir2 would trigger an event for an analogous file in dir1
This could easily be automated into a npm link workflow, along with some kind of unlink function

[wez]

Sorry to disappoint; it's too complex to build something to handle this that will work according to expectations.
It sounds like this is something that the react package manager should be handling, since it is the component that has special knowledge of the situation.

[wez]

I've been giving this some thought. I think full-blown symlink handling is too fraught with problems to be something we can commit to, but it doesn't mean that we can't help out in some of the easier or more common cases.

I'll collect some thoughts into a wiki page through the weekend and link to it from here

[JohnyDays]

Sounds good! Would be a great productivity boost for any npm-related consumers developing modules simultaneously, which seems to be a great deal of them nowadays

[amasad]

This came up a few times. I think we can handle it in the RN packager by detecting symlinks and starting a new watch on these symlinks.

There is also reports that the initial file system crawl (which queries watchman) is wrong if there is any linked directories. @wez: even if watchman doesn't follow the symlink shouldn't it report them like any other file?

[wez]

@amasad I've been brainstorming with @bhamiltoncx and we have a plan for this. I'd say hold off from implementing anything in RN for the moment; we have some diffs in progress.

[erickreutz]

Any updates on this? Really makes developing modules a pain.

[cgarvis]

👍

[d-vine]

+1

[atticoos]

+1, unfortunately, as you know, RN is in a bit of a bind without symlinks, as it's a common practice to npm link local modules for development. I understand the challenges you describe in #105 (comment), but I wanted to chime in with a +1 for the pain point I'm currently experiencing.

For now I've thrown in a bash script to recursively copy all my local dependencies into my project's node_modules directory. It isn't a perfect solution, but it works. So hopefully others could find this workaround useful.

[Ehesp]

+1

[jbpin]

+1

[chetstone]

What are module developers doing in general to work around this? @ajwhite, your solution sounds reasonable. Do you run your script by hand or have you wired up watchman to run it for you?

[atticoos]

I'm running them by hand currently. I'm sure there's a better way

[Ehesp]

I'm just opening a new project from the node modules directory and working on it that way... Sucks but it's the easiest way I'm finding without messing about with scripts and stuff.

[kopax-polyconseil]

Hello, I am working on a lerna project which have a web app and a react native module which also support web.

I now want to install and work in development from within a react native project app. I have been trying all the tools online and everyconfiguration, I am not able to setup a react native development environment because of this.

What's the solution in 2021 ?

[agrcrobles]

life is what happens while developers expect to have symlink support on watchman...

[kopax-polyconseil]

Thanks for the tip :) I am now trying with wl, but despite the folder being watched, it does not copy the change. Any hint perhaps ?

I expect Watchman to support symlink because this is a widely used feature used by npm, yarn and all the node package manager available out there so it is kind of expected and needed.

I am extracting a web app module so I can turn it compatible with react native, how should I configure a proper development environment within my native app if I can't use linking between the two?

[shellscape]

Meanwhile, literally every single major and minor bundler for the Node ecosystem supports following symlinks. That this has been an open request for over 6 years, that the metro team has fallen this far behind ecosystem norms, and that the team has remained stubborn on this issue is absolutely mind blowing. Maintainers: you were wrong. Let's own the mistake and do right by the community.

[chetstone]

Not sure if this workaround has been mentioned, but you can create a metro config file that explicitly adds a symlinked project to the project roots. I did it like this. In the code there's a comment with a link to the repo I stole it from.

[evelant]

7 years later, symlinked packages have long since been the standard for all JS package managers. This issue makes developing a react-native app in a monorepo unnecessarily difficult and error prone. If this issue is never going to be fixed can we please close this issue with an explanation of the rationale and possible workarounds?

[bombillazo]

Has anyone using yarn workspaces had success with options like nohoist or install --focus? Watchman lacking this feature is killing monorepo setups... 😞

[gajus]

The lack of symlink support was an unpleasant surprise while developing https://github.com/gajus/turbowatch/

[joshuat]

The lack of symlink support was an unpleasant surprise while developing https://github.com/gajus/turbowatch/

Funny to see you here! The lack of symlink support was why we couldn't consider turbowatch.

[gajus]

Funny to see you here! The lack of symlink support was why we couldn't consider turbowatch.

Huge 🤦‍♂️ We only tested Turbowatch in the context of re-building packages and apps when changes in the workspace are detected, but failed to check if it will work when the goal is to detected when symlinked dependencies change. The lack of symlink support makes Watchman unusable for our use case (monorepo with linked dependencies).

For what it is worth, I am rewriting Turbowatch to allow choosing between using Watchman or chokidar as a backend. Expect an update in the next 12 hours.

[gajus]

https://github.com/gajus/turbowatch/releases/tag/v2.0.0 Turbowatch made a switch to chokidar. However, I kept the API such that we could revert to using Watchman, or maybe even support multiple backends.

[gajus]

@chadaustin @xavierd @fanzeyi (tagging recently active contributors) Is there a chance of this issue receiving attention?

[fanzeyi]

@gajus Sorry. Likely no.

The harsh reality is that there is not really a case for symlink support internally (I recently chatted with folks about this), and we probably won't be motivated to implement this.

[gajus]

@gajus Sorry. Likely no. The harsh reality is that there is not really a case for symlink support internally (I recently chatted with folks about this), and we probably won't be motivated to implement this.

Thank you for the response. Would you consider accepting a PR?

[fanzeyi]

Would you consider accepting a PR?

Certainly. Personally I'd want to make Watchman easier to work in OSS and make contributions.