Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

One peer occupy multiple TCP connections #2258

Closed
garyyu opened this issue Dec 30, 2018 · 2 comments · Fixed by #2262
Closed

One peer occupy multiple TCP connections #2258

garyyu opened this issue Dec 30, 2018 · 2 comments · Fixed by #2262
Labels
enhancement

Comments

@garyyu
Copy link
Contributor

garyyu commented Dec 30, 2018

Two problems here:

  1. One peer occupy multiple TCP connections, and both connections are kept. In all 30 connected peers, I got 6 peers with 2 TCP connections for each.
    For example:
$ curl -0 -XGET  -u grin:`cat ~/.grin/.api_secret` http://127.0.0.1:13413/v1/peers/connected 2>/dev/null | sed 's/,{"/\n,{"/g' | grep 139.162.150.184

{"capabilities":{"bits":15},"user_agent":"MW/Grin 0.5.0","version":1,"addr":"139.162.150.184:13414","direction":"Inbound","total_difficulty":86027740,"height":1519}

$ netstat -tlna | grep 139.162.150.184
tcp        0      0 45.118.135.254:13414    139.162.150.184:34348   ESTABLISHED
tcp        0     27 45.118.135.254:13414    139.162.150.184:34356   ESTABLISHED
  1. Disconnected peer still keep some daemon TCP connections.For example:
$ grep "clean_peers V4(180.111.249.72:13414)" grin-server.log
20181229 02:41:52.773 DEBUG grin_p2p::peers - clean_peers V4(180.111.249.72:13414), not connected
20181229 03:27:14.259 DEBUG grin_p2p::peers - clean_peers V4(180.111.249.72:13414), not connected
20181229 16:45:04.576 DEBUG grin_p2p::peers - clean_peers V4(180.111.249.72:13414), not connected

$ netstat -tlna | grep 180.111.249.72
tcp        0      0 45.118.135.254:13414    180.111.249.72:8373     ESTABLISHED

I think if we can kill case 1, then this case 2 will not happen.

So, we need an improvement that close the existing TCP connections if receiving multiple TCP connect from same IP address.

For example, the following 2 connection request is quite near, just 7ms gap:

20181229 13:41:48.765 DEBUG grin_p2p::peers - Saving newly connected peer 139.162.150.184:13414.
20181229 13:41:48.772 DEBUG grin_p2p::peers - Saving newly connected peer 139.162.150.184:13414.
@garyyu
Copy link
Contributor Author

garyyu commented Dec 30, 2018

Also saw some bad situation, one peer occupy too many TCP connections which has blocked all new connection requesting to this node:

$ netstat -tlna | grep 13414 | grep -c 35.243.216.112
63

$ netstat -tlna | grep 13414 | grep -c 35.243.216.112
64

@garyyu garyyu mentioned this issue Dec 30, 2018
Merged
@garyyu
Copy link
Contributor Author

garyyu commented Dec 30, 2018

After the fix of #2253, this issue is the main cause which make the seed nodes un-connectable (without a grin restart). I did some check on some nodes:

tcp        0      0 45.118.135.254:57196    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57200    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57216    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57226    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57242    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57248    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57264    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57274    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57292    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57304    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57318    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57330    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57340    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57352    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57360    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57374    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57394    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57406    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57422    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57436    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57444    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57464    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57480    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57494    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57510    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57518    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57536    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57550    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57574    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57590    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57610    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57620    45.61.156.122:13414     ESTABLISHED
tcp        0      0 45.118.135.254:57634    45.61.156.122:13414     ESTABLISHED

...

tcp      102      0 45.118.135.254:13414    119.237.58.248:48846    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:48866    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:48906    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:48932    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:48946    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:48958    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:48972    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:48994    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:49012    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:49034    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:49048    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:49056    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:49072    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:49082    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:49092    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:49106    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:49116    ESTABLISHED
tcp      102      0 45.118.135.254:13414    119.237.58.248:49132    ESTABLISHED

...

tcp      102      0 45.118.135.254:13414    2.95.177.91:44546       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44570       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44582       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44614       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44640       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44668       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44694       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44720       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44740       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44772       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44786       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44832       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44854       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44864       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44898       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44922       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44938       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44974       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:44998       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45012       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45042       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45066       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45078       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45104       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45124       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45148       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45172       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45208       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45236       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45248       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45272       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45292       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45324       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45336       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45364       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45398       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45412       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45456       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45464       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45488       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45506       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45532       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45554       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45572       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45592       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45616       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45642       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45664       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45686       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45720       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45752       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45778       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45808       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45828       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45864       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45878       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45902       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45936       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45948       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:45982       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46006       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46026       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46072       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46082       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46106       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46144       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46182       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46198       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46218       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46264       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46300       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46326       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46368       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46400       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46442       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46470       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46500       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46516       ESTABLISHED
tcp      102      0 45.118.135.254:13414    2.95.177.91:46618       ESTABLISHED

will debug it and try to find the root cause.

@garyyu garyyu mentioned this issue Dec 31, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: one peer occupy multiple tcp connections
1 participant
@garyyu