PermissionDenied error causes traffic loss

[rpiceage]

Hi,

During a 24 hour stability run we found that 2 NSC pods out of 4 had traffic loss.
In the stability test, we run basically the kernel2kernel example with 4 NSC-s.

In NSC logs the following errors can be found like this:
[ERRO] �[0m(15.1) rpc error: code = Unknown desc = Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/authorize/authorizeServer.Request: rpc error: code = PermissionDenied desc = no sufficient privileges

In most of the cases these errors caused minimal traffic disturbances which were resolved in a few seconds. However after the error at 21:07:31.802 in the attached logs, traffic continued to be dropped, and did not recover in the next 7 minutes (after that our ctraffic tool exited in that pod because of too many connections lost).
I could not find any related error messages in the forwarder (the nsmgr logs unfortunately rolled out).

endpoint-nsc-d6c848c87-lhj4k.txt

[denis-tingaikin]

Hello!

Could you share deployments versions?

[rpiceage]

Sorry, I forgot.

nsmgr=d9c0b2c
registry-memory=f63cca1
forwarder-vpp=85e5a4f
nse-icmp-responder-vpp=a60ed79
nsc-vpp=fd917e3

[edwarnicke]

@denis-tingaikin Any thoughts on this?

[edwarnicke]

@rpiceage Could we retest this as soon as P2MP finishes landing?

[denis-tingaikin]

@NikitaSkrynnik Could you have a look when you will able?

[NikitaSkrynnik]

@rpiceage Hello! Could you provide more logs from other components?

[rpiceage]

Hi, unfortunately the logs have rotated out by the end of the test, and it's hard to reproduce, as the error occurred after quite a few hours. I'll do what I can.

[edwarnicke]

@rpiceage Is it possible to retain more logs given that its a long term soak test?

[NikitaSkrynnik]

@rpiceage Hello. From logs i can see that server rejects client request because client has old certificate. We discussed it and it looks like this Pull Request should fix the problem.

[rpiceage]

Thanks, will try to re-run as soon as possible, with periodic log collection.

[edwarnicke]

@rpiceage Is this still an issue?

[rpiceage]

We haven't seen the issue in the last run. I think we can close this. Thanks for the effort.

[denis-tingaikin]

@rpiceage Be free to reopen if you will face this :)