-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OOM in a 2G box #5412
Comments
Hey @cannium, could you provide a few heap trace dumps (ideally at different uptimes, like every 3/4 hours), you can do this with:
|
Sure, I'll do. |
In case you become impatient, I'll put some updates here. |
No problem, take your time. |
It happened again. I started a file server containing all the heap files here: http://159.203.36.22:8000/ |
What commit did you build? |
I believe it's 78a32f2 |
It would seem to be related to the number of active connections.
|
Could you give more instructions about the session - stream structure? I'd like to understand how could a session take 290M memory, cache the whole file object in memory? |
It's not a single session, that's all your connections. However, that's still an absurd number. I'm getting:
Really, those channels should be pretty small. I wonder if we're leaking connection objects somewhere. Looking at this, I'm pretty sure we are. |
Yeah, there's definitely a slow leak. I've tried killing a bunch of connections but that memory doesn't move anywhere. However, this is quite a slow leak so it'll be tricky to track down where it's happening. |
Historically, this has always been a race in stream management. That is, some service is holding onto an open stream but not actually using it. When services do that, they're supposed to detect that the client has disconnected and free the stream but I'm guessing that that isn't happening here. |
Found it! If we abort the connection while the identify protocol is running, we'll keep a reference to the connection forever. This is an ancient bug. |
fix: libp2p/go-libp2p#420 |
Awesome! I'll test with the latest version to see if fixed. |
You'll have to wait for me to bubble the update to go-ipfs. I should get to that in a few days. |
I |
That's great to hear (I didn't want to have to trouble you with getting gx working but I'm glad you were able to). The remaining growth is probably the peerstore (which we're working on). However, so we don't make assumptions, mind posting a heap profile (just one should be fine) along with the IPFS binary and commit used to build it? |
Put here as before: http://159.203.36.22:8000/ |
Thanks! Yeah, that looks like a known bug: ipfs/go-ipfs-blockstore#3. Basically, we should be using a significantly more efficient "has" cache for our blockstore. |
Version information:
go-ipfs version: 0.4.18-dev-
Repo version: 7
System version: amd64/linux
Golang version: go1.10.1
Type:
Bug
Description:
I reserved a droplet(VPS in digitalocean) with 1 core, 2G memory in digitalocean, and solely run IPFS. After about 12 hours, OOM occured. The commands I used to build and run IPFS are:
kernel OOM log:
monitor graph:
As a second try, I lowered connection numbers to
I'll wait and see if it would OOM again.
I'd like to know the recommended memory configuration for a production system. On the other side, if I need to run IPFS in a 2G box, how to tune its memory usage(by both config and code)?
The text was updated successfully, but these errors were encountered: