Seeing a SEGV in BBS error logs

[andrew-edgar]

Summary

Getting errors in pushes and seeing this in the cf push ...

Instances information unavailable: Connection refused - Connection refused - connect(2) for "bbs.service.cf.internal" port 8889 (bbs.service.cf.internal:8889)

This is happening while rolling the cells.

Steps to Reproduce

Hard to reproduce as it's rare.

Diego repo

Environment Details

diego-release version 2.42.0

Possible Causes or Fixes (optional)

Fix nil pointer dereference as seen in the error log ...

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xc94d37]

goroutine 1629 [running]:
code.cloudfoundry.org/bbs/handlers.filterInstanceEventByCellID(0xc00185f3e0, 0x24, 0x10122c0, 0xc0008c2318, 0x0, 0x0, 0xc0022e9700)
        /var/vcap/data/compile/bbs/src/code.cloudfoundry.org/bbs/handlers/events_handlers_r0.go:256 +0x147
code.cloudfoundry.org/bbs/handlers.(*LRPInstanceEventHandler).commonSubscribe.func1(0xc0022e9710, 0xc0022e96e0, 0x2, 0x0)
        /var/vcap/data/compile/bbs/src/code.cloudfoundry.org/bbs/handlers/events_handlers_r0.go:129 +0xa3
code.cloudfoundry.org/bbs/handlers.streamSource(0xc002362360, 0xc0023623c0, 0xc002362420, 0xc0013641c0)
        /var/vcap/data/compile/bbs/src/code.cloudfoundry.org/bbs/handlers/events_handlers.go:112 +0x4b
created by code.cloudfoundry.org/bbs/handlers.(*LRPInstanceEventHandler).commonSubscribe
        /var/vcap/data/compile/bbs/src/code.cloudfoundry.org/bbs/handlers/events_handlers_r0.go:146 +0x4d0

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[jvshahid]

@andrew-edgar Can you share with us the BBS logs right before the crash?

In particular, we are curious if a corrupt DB record causes the panic. For example, do you see this message failed-unmarshaling-net-info-data in the logs?

[jvshahid]

Ping

[andrew-edgar]

sorry we have not seen this for a while and I'm not in the position to be able to regenerate or cause this to occur. at this point I think it should be closed and we can reopen if it occurs again and we can possibly get the logs associated with the failure

[jvshahid]

👍