vpp-agent failes when reading statistics

[miroslavKovacPantheon]

agent version: v2.1.1-1-g80401e6" BuildDate="2019-05-27T23:57-07:00" CommitHash=80401e66a0ef370ac61c404590de51d86f78bf4c

vpp version 1904

Agent failes at some point after reading statistics with following log:

vswitch.log

[ondrej-fabry]

This is actually GoVPP's wrapper for stats API provided by vppapiclient from VPP.

The crash is happening in dump of stats and is easily reproducible in VPP 19.08. Ole suspects some corrupted/mangled data in is being dumped.

2019-05-30 13:29:12,939 DEBG 'agent' stdout output:
fatal error: unexpected signal during runtime execution

2019-05-30 13:29:12,941 DEBG 'agent' stdout output:
[signal SIGSEGV: segmentation violation code=0x2 addr=0x7fc1eb7ff000 pc=0x7fc2785557ef]

runtime stack:
runtime.throw(0x135ee9b
2019-05-30 13:29:12,941 DEBG 'agent' stdout output:
, 0x2a)
	/usr/local/go/src/runtime/panic.go:608 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:374 +0x2f2

goroutine 150 [syscall]:
runtime.cgocall(0xfb49a0, 0xc00138d490, 0x0)
	/usr/local/go/src/runtime/cgocall.go:128 +0x5e fp=0xc00138d460 sp=0xc00138d428 pc=0x40eede
github.com/ligato/vpp-agent/vendor/git.fd.io/govpp.git/adapter/vppapiclient._Cfunc_govpp_stat_segment_dump(0x7fc2374bb53c, 0x0)
	_cgo_gotypes.go:402 +0x4a fp=0xc00138d490 sp=0xc00138d460 pc=0xbf1fba
github.com/ligato/vpp-agent/vendor/git.fd.io/govpp.git/adapter/vppapiclient.(*statClient).DumpStats(0xc0002654b0, 0xc0006ec000, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/ligato/vpp-agent/vendor/git.fd.io/govpp.git/adapter/vppapiclient/stat_client.go:114 +0xd6 fp=0xc00138d678 sp=0xc00138d490 pc=0xbf2976
github.com/ligato/vpp-agent/vendor/git.fd.io/govpp.git/core.(*StatsConnection).GetInterfaceStats(0xc0007aef20, 0x5a953bcad, 0x2150700, 0xcd71386d)
	/go/src/github.com/ligato/vpp-agent/vendor/git.fd.io/govpp.git/core/stats.go:256 +0x99 fp=0xc00138d8b8 sp=0xc00138d678 pc=0xbff3f9
github.com/ligato/vpp-agent/plugins/govppmux.(*Plugin).GetInterfaceStats(0x2151ec0, 0x5a953bcad, 0x2150700, 0xcd71386d)
	/go/src/github.com/ligato/vpp-agent/plugins/govppmux/plugin_impl_govppmux.go:325 +0x69 fp=0xc00138d8e8 sp=0xc00138d8b8 pc=0xc0b889
github.com/ligato/vpp-agent/plugins/vpp/ifplugin.(*InterfaceStateUpdater).doInterfaceStatsRead(0xc0001e0000)
	/go/src/github.com/ligato/vpp-agent/plugins/vpp/ifplugin/interface_state.go:281 +0xfc fp=0xc00138df28 sp=0xc00138d8e8 pc=0xdd652c
github.com/ligato/vpp-agent/plugins/vpp/ifplugin.(*InterfaceStateUpdater).startReadingCounters(0xc0001e0000, 0x148b160, 0xc00002f780)
	/go/src/github.com/ligato/vpp-agent/plugins/vpp/ifplugin/interface_state.go:214 +0x92 fp=0xc00138dfc8 sp=0xc00138df28
2019-05-30 13:29:12,941 DEBG 'agent' stdout output:
 pc=0xdd5de2
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00138dfd0 sp=0xc00138dfc8 pc=0x468b81
created by github.com/ligato/vpp-agent/plugins/vpp/ifplugin.(*InterfaceStateUpdater).Init
	/go/src/github.com/ligato/vpp-agent/plugins/vpp/ifplugin/interface_state.go:113 +0x40e

[ondrej-fabry]

You can now disable retrieval of interface stats by setting env var: DISABLE_INTERFACE_STATS=y. It's supported in dev branch/:dev image tag.

Commit: e39b5b6

[ondrej-fabry]

We've switched to the pure Go adapter in GoVPP - statsclient, where this issue does not occur, thus closing this issue.