-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ipfs add hangs and stalls with large folders #3885
Comments
Try enabling sharding |
Also see: ipfs/notes#212 |
Update: It failed at 99.6% Thanks for the super fast response @Kubuxu, I will give that a crack now and let you know the results. Fingers crossed |
Got all the way through and failed with this error:
Tried a second time and it made it 1% in and threw the same error. I've restarted the IPFS daemon and get the same result still. Stopped the daemon and it seems to be running now, will report back soon. |
It bombed out again, this time saying "Killed". Trying again. I've noted since turning on the sharding it seems to really use the RAM; to the point where it's filling swap as well. Unsure if that's related but thought it might be worth mentioning. Will post the results of the second attempt. |
Interesting observation regarding sharding. |
Yeah, no good I'm afraid
It seems that it fills the RAM, then swap (other processes on the system have crashed I noticed) and then finally bombs out itself. Let me know if you guys need any info from me, I'll do my best to give logs and such |
Hrm... I'll take a look at this after breakfast... |
@dojobel If you don't mind, could you try running this branch: #3888 ? |
Not at all, I've had a go (heh) but I must be noobing something up..
I've stashed my original executable with
I've tried a |
Yep, it was me being a noob indeed - had to move the folder from "/opt/go/src/github.com/go-ipfs" to "/opt/go/src/github.com/ipfs/go-ipfs". Have gotten it to |
It's stalled at 2.98%, 764.11MB / 24.91GB done. The IPFS daemon is using a lot of memory since the import and is also using a fair bit of CPU (probably normal).
I've had to restart it as it was on the cusp of using all the RAM and crashing other processes. It's leveled out after the restart
I'm going to try importing without the Daemon running again and will report back |
Hi guys, Just a quick update - I managed to get these problem folders to import by turning off sharding again. Unsure if the patch you did had anything to do with it, but something is definitely improved. Will be trying with some much larger folders over the coming days/weeks. |
Okay so I've managed to import things ranging between 20-40GB comfortably. It is a little slow, but I suspect that's just the nature of how IPFS works (dedupe has never been fast in my experience but that's not why you use it) I have had to actually stop the IPFS daemon in order to get an I'm adding something of about 270GB in size now and will report back if it works with that or bombs out. |
@dojobel Thanks for all these updates! For now, when adding very large sets of data, its advisable to do so without the daemon running, or by using Let me know how things go on latest master, there have been several improvements since your last post (may 6th) that should help out, as well as some very exciting upcoming changes around providing and datastore performance which should land in 0.4.12. |
While trying to build an Ubuntu repository with @ElOpio, using ipfs, the add fails with the following error:
we always get the same error, in a different moment of the process. |
@rngkll what version of ipfs are you using? And are you doing the add with a running daemon? If so, does the daemon die when you get that EOF? It could be that its running out of memory. |
ipfs version 0.4.13 The daemon is running but we are using --local, the daemon showed a message at the same time.
|
Do you mind trying out latest master? We have some fixes that should improve that process a bit, It might help. Note, there is a known bug with the gateway on latest master, a fix is in progress, but if you're using this node as a public ipfs gateway, then I recommend waiting until the fix is in. |
NP, We will build the master branch and test with that one. |
Verision Output
|
@rngkll Thats.... a new bug. You shouldnt be seeing that. Thank's a bunch for reporting. |
@whyrusleeping Thanks for your help. Now we are getting something different with not a lot of output. Version Output
Is there a way that I can get more information on the error? Thanks |
@rngkll that one looks like the file in question has changed on disk. Would you mind filing a new issue for that? This thread is getting a bit long |
Though i think its a result of the |
Many things have changed since this issue was reported, including flushing some intermediate datastructures in add. |
Version information:
go-ipfs version: 0.4.8-
Repo version: 5
System version: amd64/linux
Golang version: go1.8
Type:
Bug
Severity:
High
Description:
When adding files to the repository, it gets to a seemingly arbitrary point (around 8-10GB?) in to the file transfer, then the progress bar disappears and the job stalls. I usually pipe the output into a text file to get a file and hash list, but have tried without and still no luck I'm afraid.
The directory being added is 24.91GB in size, and contains 101143 objects.
The actual content being stored can be found with an explanation on this page: climate-mirror/datasets#333
The command I'm using is:
I was originally running golang installed from the Yum repositories, which turned out to be an old version (v1.6.3). I've since removed this one and installed straight from their site (running v1.8.1 now) which has significantly improved the performance of the
ipfs add
overall.The machine is a Virtual Machine on Citrix XenServer, running Oracle Linux 7 and the datastore is on an NFS volume directly attached to the VM, mounted via /etc/fstab on boot. The VM has 2x vCPU cores pinned to it and 8GB of memory. I can provide further details on the physical hardware if it helps.
I'm running SELinux but it's not had any effect on the functionality since the start, as far as I can tell.
I installed IPFS using
ipfs-update
and am currently on the latest version (v0.4.8).I've just tried stopping the ipfs daemon and will see if that improves, so far it's looking like I've gotten further than before. I'll provide an update when it completes with the result
The text was updated successfully, but these errors were encountered: