Concurrency issue causing the NSMgr to crash

[LionelJouin]

Current Behavior

The NSMgr crashes during the deployment of NSCs and NSEs, not sure what is the exact cause and how we could reproduce, but I believe from the logs it is due to this part: https://github.com/networkservicemesh/sdk/blob/v1.4.0/pkg/networkservice/common/metrics/server.go#L79.
I included logs below.

Failure Information (for bugs)

2022-06-15T12:46:13.045665873Z stderr F fatal error: concurrent map read and map write
2022-06-15T12:46:13.048873743Z stderr F 
2022-06-15T12:46:13.048940244Z stderr F goroutine 886 [running]:
2022-06-15T12:46:13.048945407Z stderr F runtime.throw({0xd2ef0d?, 0x4af46f?})
2022-06-15T12:46:13.048960268Z stderr F 	/usr/local/go/src/runtime/panic.go:992 +0x71 fp=0xc0008f5ea8 sp=0xc0008f5e78 pc=0x4359b1
2022-06-15T12:46:13.048971173Z stderr F runtime.mapaccess2_faststr(0x133eb00?, 0x1?, {0xc0003da7f8, 0x11})
2022-06-15T12:46:13.048986036Z stderr F 	/usr/local/go/src/runtime/map_faststr.go:117 +0x3d4 fp=0xc0008f5f10 sp=0xc0008f5ea8 pc=0x412d54
2022-06-15T12:46:13.04899616Z stderr F github.com/networkservicemesh/sdk/pkg/networkservice/common/metrics.(*metricServer).writeMetrics(0xc00000ddd0, {0xe44880, 0xc0004b9a40}, 0xc00075b090)
2022-06-15T12:46:13.049000884Z stderr F 	/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/metrics/server.go:79 +0x199 fp=0xc0008f60d0 sp=0xc0008f5f10 pc=0xb1a339
2022-06-15T12:46:13.049004563Z stderr F github.com/networkservicemesh/sdk/pkg/networkservice/common/metrics.(*metricServer).Request(0xe44880?, {0xe44880, 0xc0004b9a40}, 0xc0007dc410?)
2022-06-15T12:46:13.049011318Z stderr F 	/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/metrics/server.go:52 +0x85 fp=0xc0008f6118 sp=0xc0008f60d0 pc=0xb1a065
2022-06-15T12:46:13.049021859Z stderr F github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request(0xc0002efe00, {0xe44880, 0xc0004b9830}, 0xc0007dc410)
...

Context

• Kubernetes Version: 1.24
• NSM Version: v1.4.0

Failure Logs

0.log

[denis-tingaikin]

@LionelJouin How is this critical for you?

@edwarnicke Should we schedule v1.4.1?

[LionelJouin]

It's not critical, I haven't managed to reproduce it yet.