Windows Support

[daviddias]

More and more users have been trying to use js-ipfs on Windows and seeing their experiments fail. This happens because, in fact, js-ipfs doesn't have Windows support at the moment.

Reported issues:

• node js ipfs on windows  #861
• Win10 64b - binding.fcntl is not a function #1016

We welcome a captain to develop Windows support for js-ipfs. It should be really close and mostly be an issue with pathing and native deps.

[Beanow]

*cough* what about this? https://msdn.microsoft.com/en-us/commandline/wsl/install_guide

[victorb]

Did a look into this as I have a windows machine close by.

First discover was that fs-ext does not like windows, and gives a "binding.fcntl is not a function" error (as in #1016)

The only module that uses fs-ext is lock-me (https://github.com/dignifiedquire/lock-me) which is supposed to work on Windows but does not for some reason, it crashes instead of using javascript version of fs-ext.

However, fs-ext is a optional dependency, and removing it from lock-me makes both portable and unix tests on lock-me to pass.

Also, removing js-ext from node_modules in js-ipfs, makes me able to run the daemon successfully at least. Have not yet tried to add/get files though.

Code I tested with:

const ipfs = require('./src/code/index.js')
const node = new ipfs()
node.once('ready', () => console.log('ready'))
node.on('error', (err) => console.log(err))

As I usually don't develop on windows, I didn't have a dev env setup. But I found that windows-build-tools is a very useful module for setting up everything required for it to work. So after installing nodejs, running npm install --global --production windows-build-tools installs python and any .NET Framework stuff needed to develop.

[JonasDev17]

C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" install ipfs

Standard error:
npm WARN deprecated [email protected]: deprecated in favor of js-ipld-ethereum https://github.com/ipld/js-ipld-ethereum
npm WARN deprecated [email protected]: 🙌 Thanks for using Babel: we recommend using babel-preset-env now: please read babeljs.io/env to update!
npm WARN lifecycle npm is using C:\Program Files\nodejs\node.exe but there is no node binary in the current PATH. Use the --scripts-prepend-node-path option to include the path for the node binary npm was executed with.
'node' is not recognized as an internal or external command,
operable program or batch file.
'node' is not recognized as an internal or external command,
operable program or batch file.
'node' is not recognized as an internal or external command,
operable program or batch file.
'node' is not recognized as an internal or external command,
operable program or batch file.
'node' is not recognized as an internal or external command,
operable program or batch file.
'node' is not recognized as an internal or external command,
operable program or batch file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fs-ext):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] install: node-gyp configure build
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Exit status 1
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\gc-stats):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] install: node-pre-gyp install --fallback-to-build
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Exit status 1

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: prebuild-install || node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users...\AppData\Roaming\npm-cache_logs\2017-10-11T15_58_12_138Z-debug.log

Standard output:

[email protected] install C:\Users\Jonas\WebstormProjects\untitled\node_modules\fs-ext
node-gyp configure build

C:\Users...\WebstormProjects\untitled\node_modules\fs-ext>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin\....\node_modules\node-gyp\bin\node-gyp.js" configure build ) else (node "" configure build )

[email protected] install C:\Users...\WebstormProjects\untitled\node_modules\gc-stats
node-pre-gyp install --fallback-to-build

[email protected] install C:\Users\Jonas\WebstormProjects\untitled\node_modules\keccak
prebuild-install || node-gyp rebuild || echo "Keccak bindings compilation fail. Pure JS implementation will be used."

C:\Users...\WebstormProjects\untitled\node_modules\keccak>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin\....\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node "" rebuild )
"Keccak bindings compilation fail. Pure JS implementation will be used."

[email protected] install C:\Users...\WebstormProjects\untitled\node_modules\leveldown
prebuild-install || node-gyp rebuild

C:\Users...\WebstormProjects\untitled\node_modules\leveldown>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin\....\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node "" rebuild )

Process finished with exit code 1

[victorb]

 npm WARN lifecycle npm is using C:\Program Files\nodejs\node.exe but there is no node binary in the current PATH. Use the --scripts-prepend-node-path option to include the path for the node binary npm was executed with.

and

'node' is not recognized as an internal or external command,
operable program or batch file.

makes me believe that you have not installed node correctly. You should make sure the node binary is in your $PATH variable. You can find how to do that here: https://stackoverflow.com/a/27864253/360186

[mkg20001]

I finally got a windows VM setup and did some tests what works:
The gateway works after doing "jsipfs init" which crashes with Error: Invalid key: /CIQGKXXGEEETAXRWRSFJYDYW5VSDR5YLCWWGVI6N4KJ2Y6MYMKNDG6I (but "jsipfs daemon" still works fine).
After a peer connects and initializes the /ipfs/bitswap/ protocol the daemon crashes with the same error.

I suspect this error is occuring because \ or / is used in paths or path splitting where it shouldn't be used on windows. (Most likely it tries to split some string that has / with \)

I would really like help getting jsipfs to run on windows, but I don't think that I could work on that so much.

[richardschneider]

@dryajov I'm willing to spend some time on getting this to work on Windows. What would you like me to look at.

[richardschneider]

I tried installing, npm install js-ipfs, and got errors on fs-ext. It turns out that my node version (5.7.0) is the culprit. Updating it to version 8.8.1 resolved the issues and js-ipfs now installs without an error.

You can download the latest node for windows at https://nodejs.org/en/download/current/

Unfortunately, as @mkg20001 reported, I now get an Invalid Key error.

[dryajov]

Hi @richardschneider! Thanks so much for looking into this, I believe the next steps would be figuring out the error that both @mkg20001 and you are seeing as well as any other that pop up. We also need CI setup with this, which I'll look into next.

Keep up the awesome work @mkg20001 and @richardschneider! ❤️

[richardschneider]

@dryajov I recommend using AppVeyor for a CI. I've used it for both JS and C# projects to test on windows and the people are really responsive.

[dryajov]

👍 I believe that's what we're using already for some of our projects - check github.com/ipfs/aegir for example (that's our built/release system).

[pvh]

Hi guys, I've been doing some hacking on IPFS/libp2p on Windows. The ipfs-pubsub-room-demo doesn't send direct messages due to something going wrong with the direct dial. Same code works fine on my linux box. @dryajov and I did some diagnostics and isolated it to just node on Windows (example code works in the browser and on my linux machine.)

[dryajov]

@pvh yeah, windows support is precarious at best at this moment, we're just starting this work and so far it's quite possible that most things are broken. @richardschneider has been doing great work in getting the datastores to behave in windows, but there are quite possible more things broken.

It would help if you could provide a full trace of your runs in windows, we use the debug module so enabling the DEBUG=* env variable should yield some some pretty verbose output, if you could create a gist with that output, that would go a long way helping us troubleshoot this.

Thanks for helping us getting IPFS working on windows!

[daviddias]

Fixing Datastore Paths

A number of PRs have been opened to fix the lack of OS agnostic'ity of datastore. These are:

• fix: make datastore OS agnostic (path things) interface-datastore#13
• fix: sharding and query for windows interop js-datastore-core#6
• fix: Windows support js-datastore-fs#7
• fix: Windows interop js-datastore-level#4
• Windows interop js-ipfs-repo#146

I'm going to start reviewing these. To make sure we don't break anything, I would really appreciate if someone could test it all out to js-ipfs (npm link all the things) and confirm it works on windows and that the repo is still interoperable with Go)

[richardschneider]

My current understanding is that repo/blocks is interoperable with Go; even with sharding!

However, repo/datastore with levelUp is not.

[the-townsend]

I'd be happy to help here as it would be good to get js-ipfs on windows. Happy to take on direction on where to put my efforts.

[richardschneider]

@the-townsend Thanks for the offer. I'm in the middle of getting the datastores to work on windows. Hopefully will be done today.

Then we can take a look at any outstanding issues.

[daviddias]

We will need some help on adding all the badges to all the repos with appveyor enabled. I'll be going repo by repo and enabling and listing here where there is a interop bug

[daviddias]

We now have Windows support on IPFS repo! \o/ Now we need to check if all the other modules have indeed full support on Windows. Here is the list (the updates need to happen sequentially):

• https://github.com/ipfs/js-ipfs-block
• https://github.com/ipfs/js-ipfs-block-service
• https://github.com/ipld/js-ipld-dag-cbor
• https://github.com/ipld/js-ipld-dag-pb
• https://github.com/ipld/js-ipld-ethereum
• https://github.com/ipld/js-ipld-git
• https://github.com/ipld/js-ipld-raw
• https://github.com/ipld/js-ipld-resolver
• https://github.com/ipfs/js-ipfs-unixfs
• https://github.com/ipfs/js-ipfs-unixfs-engine
• https://github.com/ipfs/js-ipfs-bitswap
• https://github.com/ipfs/js-ipfs-api
• https://github.com/ipfs/js-ipfs

[the-townsend]

I have now managed to install. Details attached/linked below (*). I was using Node 9, which may have been the issue. The README for js-ipfs has 'node@6 or above'. However it could also be the change that @diasdavid made that made it work. [Note I uninstalled/deleted ''.window-build-tools'' which changed the set up issues previously, so this isn't a dependency as far as I can see.]

Unfortunately, as you can see from the attached I am getting the same issue as above coming from 'datastore' (specifically key.js); am going to try and take a look at those now.

(*)
IPFS_Window_Install.txt

[richardschneider]

@diasdavid We need ipfs/js-datastore-level#4 and then update js-ipfs-repo before checking the other modules.

[richardschneider]

@diasdavid @dignifiedquire The other modules should be run on appveyor. I'm happy to create the appveyor.yaml and do the testing but maybe one of you should create the appveyor account.

[richardschneider]

NPM reports the following dependents of js-ipfs-repo.

• ipld-resolver
• ethereumjs-ipld-dump not part of IPFS
• ipld-cli
• js-libp2p-pstn-nodenot part of IPFS
• ipfs

and these for interface-datastore

• ~~datastore-ipfs-ro-hook~ not part of IPFS
• libp2p-kad-dht
• ipfs-repo
• datastore-fs
• datastore-core
• datastore-level
• ipld-resolver
• datastore-ipfs-http-api not part of IPFS

[the-townsend]

Not sure what the protocol here is.
https://github.com/kumavis/ethereumjs-ipld-dump/blob/master/src/index.js
has '/.jsipfs' lines 23. Do we edit that code and give a pullrequest to 'kumavis'?

[richardschneider]

If its not part of IPFS we can ignore it.

[the-townsend]

Can't see that anything needs doing on

ipld-cli

[richardschneider]

@the-townsend Thanks I'll tick it off the list

[the-townsend]

js-libp2p-pstn-node is not part of ipfs

[richardschneider]

@the-townsend js-ipfsd-ctl is not working on windows. Its blocking js-ipfs-api from being tested.

Would you mind looking at it.

[the-townsend]

I’ll try to later today. Any more details available? Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

…

________________________________ From: Richard Schneider <[email protected]> Sent: Wednesday, November 8, 2017 9:17:33 PM To: ipfs/js-ipfs Cc: Christopher Townsend; Mention Subject: Re: [ipfs/js-ipfs] Windows Support (#1017) @the-townsend<https://github.com/the-townsend> js-ipfs<https://github.com/ipfs/js-ipfsd-ctl> is not working on windows. Its blocking js-ipfs-api being tested. Would you mind looking at it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1017 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AaNA4Yg6fKuM0pqntIvRfWj6t4LveERHks5s0hpsgaJpZM4PWIhG>.

[richardschneider]

It's mainly a package to spinup an IPFS server. Could you get the tests running on a windows machine

[the-townsend]

Results attached.

27 passing (55s)
3 pending
1 failing

Can't immediately see whether this is a simple problem or not.
js-ipfsd-ctl-test-results.txt

[the-townsend]

It's the same error that @diasdavid got here

https://ci.appveyor.com/project/diasdavid/js-ipfsd-ctl-a9ywu/build/job/sl48bopxa3hfs9eo

[richardschneider]

It looks like js-ipfsd-ctl is receiving some TLC this comment

[daviddias]

The next big piece should be getting https://github.com/ipfs/js-ipfs-api to work on Windows too.

[daviddias]

Windows support is here!!! 🌟🚀🎆

With #1065

BIG THANK YOU to @richardschneider who almost single-handedly defeated all the Windows dragons 🐉 and brought Windows support to js-ipfs and its ipfs dependencies.

Let's celebrate in the next all hands! @richardschneider will you be able to join and give us a demo?

[daviddias]

Closing this issue. Let's track the last js dragon on js-ipfs-api at ipfs-inactive/js-ipfs-http-client#408 (not required for js-ipfs to work on Windows)

[the-townsend]

@diasdavid @richardschneider
FYI getting below when trying to replicate locally:
...
22 passing (2m)
1 pending
7 failing
...

[details attached]

I can't access the test results above easily to try and see what's going on.

js-ipfs-test-results.txt

[barnendu]

I had same issue, jsipfs was giving me error while installing. Its has been solved after using
npm install --global --production windows-build-tools in windows 10 run power shell has admin. Then
run npm install ipfs --global