Replicator repeatedly can't get data from RPC

[roman-khimov]

Expected Behavior

Reconnect of some kind solving the problem.

Current Behavior

Aug 07 05:55:51 metis1 neofs-node[2289135]: error        policer/check.go:85        could not get container        {"component": "Object Policer", "cid": "3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV", "error": "could not perform test invocation (getContainerData): failed to write JSON request (invokefunction / 4): write tcp 157.90.176.145:39838->5.161.67.240:443: write: broken pipe: before sending the request"}
Aug 07 05:55:51 metis1 neofs-node[2289135]: error        policer/check.go:85        could not get container        {"component": "Object Policer", "cid": "3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV", "error": "could not perform test invocation (getContainerData): failed to write JSON request (invokefunction / 4): write tcp 157.90.176.145:39838->5.161.67.240:443: write: broken pipe: before sending the request"}
Aug 07 05:55:51 metis1 neofs-node[2289135]: error        policer/check.go:85        could not get container        {"component": "Object Policer", "cid": "3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV", "error": "could not perform test invocation (getContainerData): failed to write JSON request (invokefunction / 4): write tcp 157.90.176.145:39838->5.161.67.240:443: write: broken pipe: before sending the request"}
Aug 07 05:55:51 metis1 neofs-node[2289135]: error        policer/check.go:85        could not get container        {"component": "Object Policer", "cid": "3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV", "error": "could not perform test invocation (getContainerData): failed to write JSON request (invokefunction / 4): write tcp 157.90.176.145:39838->5.161.67.240:443: write: broken pipe: before sending the request"}

Possible Solution

Reconnect, continue operation.

Steps to Reproduce (for bugs)

Unknown. It just happened.

Context

But it happened on mainnet.

Regression

Unknown.

Your Environment

• Version used: 0.48.1
• Server setup and configuration:

fschain:
  endpoints:
  - wss://rpc.morph.fs.neo.org/ws

[roman-khimov]

For whatever reason all four nodes were stuck this way.

[roman-khimov]

9 @ 0x47942e 0x410b25 0x410777 0xd79b1b 0xea22ca 0xea221f 0xc4fa8a 0x480f81
#       0xd79b1a        github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).HandleNewEpoch+0xfa    github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:247
#       0xea22c9        main.initLocalStorage.func1+0x89                                                                        github.com/nspcc-dev/neofs-node/cmd/neofs-node/storage.go:36
#       0xea221e        main.initLocalStorage.addNewEpochAsyncNotificationHandler.WorkerPoolHandler.func11.1+0x1e               github.com/nspcc-dev/neofs-node/pkg/morph/event/utils.go:75
#       0xc4fa89        github.com/panjf2000/ants/v2.(*goWorker).run.func1+0x89                                                 github.com/panjf2000/ants/v2@v2.9.0/worker.go:67
...
1 @ 0x47942e 0x456cd7 0xbb9156 0xbad9aa 0xbb45f0 0xbb3827 0xb99a4d 0xbc6265 0xbc763a 0xbe9435 0xbe9388 0xe77983 0xea6808 0xe77ac6 0xe5f139 0xe566b0 0xaaeac6 0xa59d96 0xa5ef88 0xa5773f 0x480f81
#       0xbb9155        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*WSClient).makeWsRequest+0x315               github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/wsclient.go:767
#       0xbad9a9        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*Client).performRequest+0x149                github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/client.go:185
#       0xbb45ef        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*Client).invokeSomething+0x24f               github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/rpc.go:894
#       0xbb3826        github.com/nspcc-dev/neo-go/pkg/rpcclient.(*Client).InvokeFunction+0x1a6                github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/rpc.go:827
#       0xb99a4c        github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker.(*Invoker).Call+0xac                  github.com/nspcc-dev/neo-go@v0.111.0/pkg/rpcclient/invoker/invoker.go:161
#       0xbc6264        github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).Call+0x44                    github.com/nspcc-dev/neofs-node/pkg/morph/client/client.go:74
#       0xbc7639        github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).TestInvoke+0x39              github.com/nspcc-dev/neofs-node/pkg/morph/client/client.go:333
#       0xbe9434        github.com/nspcc-dev/neofs-node/pkg/morph/client.StaticClient.TestInvoke+0x1f4          github.com/nspcc-dev/neofs-node/pkg/morph/client/static.go:222
#       0xbe9387        github.com/nspcc-dev/neofs-node/pkg/morph/client/container.(*Client).GetEACL+0x147      github.com/nspcc-dev/neofs-node/pkg/morph/client/container/eacl.go:25
#       0xe77982        main.newCachedEACLStorage.func1+0x82                                                    github.com/nspcc-dev/neofs-node/cmd/neofs-node/cache.go:229
#       0xea6807        main.(*ttlNetCache[...]).get+0x287                                                      github.com/nspcc-dev/neofs-node/cmd/neofs-node/cache.go:109
#       0xe77ac5        main.(*ttlEACLStorage).GetEACL+0x85                                                     github.com/nspcc-dev/neofs-node/cmd/neofs-node/cache.go:237
#       0xe5f138        github.com/nspcc-dev/neofs-node/pkg/services/object/acl.(*Checker).CheckEACL+0x358      github.com/nspcc-dev/neofs-node/pkg/services/object/acl/acl.go:144
#       0xe566af        github.com/nspcc-dev/neofs-node/pkg/services/object.(*Server).SearchV2+0x26f            github.com/nspcc-dev/neofs-node/pkg/services/object/server.go:1837
#       0xaaeac5        github.com/nspcc-dev/neofs-sdk-go/proto/object._ObjectService_SearchV2_Handler+0x1a5    github.com/nspcc-dev/neofs-sdk-go@v1.0.0-rc.13.0.20250715070617-c7038b450691/proto/object/service_grpc.pb.go:762
#       0xa59d95        google.golang.org/grpc.(*Server).processUnaryRPC+0x1035                                 google.golang.org/grpc@v1.70.0/server.go:1400
#       0xa5ef87        google.golang.org/grpc.(*Server).handleStream+0xb87                                     google.golang.org/grpc@v1.70.0/server.go:1810
#       0xa5773e        google.golang.org/grpc.(*Server).serveStreams.func2.1+0x7e                              google.golang.org/grpc@v1.70.0/server.go:1030

1 @ 0x47942e 0x456cd7 0xbcb425 0x480f81
#       0xbcb424        github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).closeWaiter+0x64     github.com/nspcc-dev/neofs-node/pkg/morph/client/multi.go:62

1 @ 0x47942e 0x456cd7 0xbd271d 0x480f81
#       0xbd271c        github.com/nspcc-dev/neofs-node/pkg/morph/client.(*Client).routeNotifications+0x1dc     github.com/nspcc-dev/neofs-node/pkg/morph/client/notifications.go:239
...
1 @ 0x47942e 0x456cd7 0xd8bc09 0x480f81
#       0xd8bc08        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listenForHeaders+0xc8       github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:198

1 @ 0x47942e 0x456cd7 0xd8c045 0x480f81
#       0xd8c044        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listenForNotary+0x184       github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:230

1 @ 0x47942e 0x456cd7 0xd8c3b9 0xd8b5a5 0xd8b388 0x4969eb 0xd8b312 0xd8b2d9 0xe8cdeb 0xe8995e 0xe8cd7f 0xea54a6 0xea56b4 0x480f81
#       0xd8c3b8        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listenForNotifications+0xd8 github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:250
#       0xd8b5a4        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).listen+0x1e4                github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:145
#       0xd8b387        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).ListenWithError.func1+0x27  github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:128
#       0x4969ea        sync.(*Once).doSlow+0xaa                                                                sync/once.go:78
#       0xd8b311        sync.(*Once).Do+0x51                                                                    sync/once.go:69
#       0xd8b2d8        github.com/nspcc-dev/neofs-node/pkg/morph/event.(*listener).ListenWithError+0x18        github.com/nspcc-dev/neofs-node/pkg/morph/event/listener.go:127
#       0xe8cdea        main.listenMorphNotifications.func1.1+0x2a                                              github.com/nspcc-dev/neofs-node/cmd/neofs-node/morph.go:71
#       0xe8995d        main.runAndLog+0x9d                                                                     github.com/nspcc-dev/neofs-node/cmd/neofs-node/main.go:158
#       0xe8cd7e        main.listenMorphNotifications.func1+0x5e                                                github.com/nspcc-dev/neofs-node/cmd/neofs-node/morph.go:70
#       0xea54a5        main.(*workerFromFunc).Run+0x25                                                         github.com/nspcc-dev/neofs-node/cmd/neofs-node/worker.go:22
#       0xea56b3        main.startWorkers.func1+0x33                                                            github.com/nspcc-dev/neofs-node/cmd/neofs-node/worker.go:30

[roman-khimov]

It reproduced on m3, but restored eventually.

info client/notifications.go:230 RPC connection lost, attempting reconnect

[roman-khimov]

But before reconnect it's

error	neofs-node/netmap.go:211	could not update node state on new epoch after retries	{"epoch": 33473, "retries": 25, "error": "failed to write JSON request (invokefunction / 4): write tcp 135.181.79.214:33480->65.109.234.149:443: write: broken pipe: before sending the request"}
info	neofs-node/netmap.go:250	new epoch event, requesting new network map to sync SN connection caches...	{"epoch": 33473}

[roman-khimov]

So it fails requests, but receives an event. Pretty strange combination.

[roman-khimov]

It's as if writer thread is done in WSClient, but reader is still active. Which shouldn't happen, broken connection is broken for both sides. We also have some ping-ponging.