QSRV probable ref. leak

[mdavidsaver]

So far only observed on travis-ci.org with testpdb when built against Base 3.15. Loop includes at least:

# Cnt PDBProvider = 1 (1)
# Cnt PDBSingleChannel = 1 (1)
# Cnt PDBSingleMonitor = 1 (1)
# Cnt PDBSinglePV = 1 (1)

https://travis-ci.org/mdavidsaver/pva2pva/jobs/303771863

[mdavidsaver]

I think this is resolved by epics-base/pvAccessCPP@6055dbb.

[mdavidsaver]

Not fixed.

https://travis-ci.org/mdavidsaver/pva2pva/jobs/308170903

[mdavidsaver]

Leaked refs.

# Refs after
# Cnt Channel (ABC) = 1 (1)
# Cnt ChannelBaseRequester (ABC) = 1 (1)
# Cnt ChannelProvider (ABC) = 1 (1)
# Cnt Field = 47 (17)
# Cnt PDBProvider = 1 (1)
# Cnt PDBSingleChannel = 1 (1)
# Cnt PDBSingleMonitor = 1 (1)
# Cnt PDBSinglePV = 1 (1)
# Cnt PVField = 96 (93)
# Cnt pvac::Monitor::Impl = 1 (1)

[anjohnson]

Not a blocker for 7.0.1-rc1.

[mdavidsaver]

I'm coming to the conclusion that this isn't an explicit ref. loop. Rather, an extra Monitor ref. is being held by the dbEvent worker thread (the only other involved). This ref is held through the "interested" list of Monitors to allow unlocking for Monitor callbacks. To handle adding/removing Monitors while iteration I'm used a somewhat convoluted scheme with a pair of lists (interested_add and interested_remove) to accumulate changes made while iterating. The interested_remove list holds shared_ptr<>s to keep removed Monitors alive while being iterated. So this may be involved as well.

I still don't see why a pvac::Monitor::Impl (a MonitorRequester) should be alive a well. Maybe there is a deadlock during a callback?

[mdavidsaver]

Ok, I see it. Ownership rule number 1 is being violated. PDBSingleChannel has a strong ref. to PDBProvider via PDBSinglePV. This prevents automatic closure of Channels when the Provider goes out of scope.

http://mdavidsaver.github.io/pvAccessCPP/providers.html#provides_client_ownership

[mdavidsaver]

This isn't an actual ref. leak. In practice it's a race when the a Channel is closed concurrently with a dbEvent callback. The result is that the Channel, and by extension Provider are destroyed in the dbEvent callback. The practical effect of this is a possible deadlock on process exit (db_close_events() called from dbEvent callback).

[anjohnson]

Does this trigger when you start softIocPva -x test and then pvput test:exit.PROC 1? I was trying this last night and get some interesting messages:

tux% bin/linux-x86_64/softIocPVA -x test
Starting iocInit
############################################################################
## EPICS R7.0.1-pre1-DEV
## EPICS Base built Nov 20 2017
############################################################################
iocRun: All initialization complete
epics> 2017-11-29T12:09:26.343 Server context still has 1 transport(s) active and closing...
Leaking ServerContext tcp transport use_count=5
2017-11-29T12:09:26.344 Closed transport 164.54.9.24:49128 still has use_count=5
Leaking ServerContext _responseHandler use_count=2
2017-11-29T12:09:26.344 ServerContextImpl::shutdown() doesn't break all internal ref. loops. use_count=20

The process does exit properly.

[mdavidsaver]

The first is to be expected with the exit() proc. Actually this message probably doesn't need to be printed by default.

The other two are symptoms of this issue.

[mdavidsaver]

epics-base/pvAccessCPP@457d035 demotes the still active and closing message from Info to Debug level, so it is no longer shown by default.

[mdavidsaver]

I haven't seen this in some time.