perspective-python infinite loop

[robambalu]

This may very well be related to the same bug I reported that crashed.
same setup, one thread running the perspective tornado ioloop, a separate thread pumping data into a table with update calls.
This time we get into an infinite loop on the perspective tornado thread. Side note, I noticed this stack is under GIL ( I confirmed this thread is holding GIL, thats how we noticed our process stopped responding ). You may want to release the GIL for the c++ stack calls.
Again this is hard to repro since it looks due to a race, but here is the stack nonetheless.

#0 0x00007f41186c1a73 in boost::multi_index::detail::ordered_index_node_impl<boost::multi_index::detail::null_augment_policy, std::allocator >::increment(boost::multi_index::detail::ordered_index_node_impl<boost::multi_index::detail::null_augment_policy, std::allocator >&) () from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#1 0x00007f41186c8e1c in bool boost::multi_index::detail::ordered_index_impl<boost::multi_index::member<perspective::t_stnode, unsigned long, &perspective::t_stnode::m_idx>, std::less, boost::multi_index::detail::nth_layer<1, perspective::t_stnode, boost::multi_index::indexed_by<boost::multi_index::ordered_unique<boost::multi_index::tag<perspective::by_idx, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::member<perspective::t_stnode, unsigned long, &perspective::t_stnode::m_idx>, mpl_::na>, boost::multi_index::hashed_non_unique<boost::multi_index::tag<perspective::by_depth, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::member<perspective::t_stnode, unsigned char, &perspective::t_stnode::m_depth>, mpl_::na, mpl_::na>, boost::multi_index::hashed_non_unique<boost::multi_index::tag<perspective::by_nstrands, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::member<perspective::t_stnode, unsigned long, &perspective::t_stnode::m_nstrands>, mpl_::na, mpl_::na>, boost::multi_index::ordered_unique<boost::multi_index::tag<perspective::by_pidx, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::composite_key<perspective::t_stnode, boost::multi_index::member<perspective::t_stnode, unsigned long, &perspective::t_stnode::m_pidx>, boost::multi_index::member<perspective::t_stnode, perspective::t_tscalar, &perspective::t_stnode::m_sort_value>, boost::multi_index::member<perspective::t_stnode, perspective::t_tscalar, &perspective::t_stnode::m_value>, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type>, mpl_::na>, boost::multi_index::ordered_unique<boost::multi_index::tag<perspective::by_pidx_hash, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::composite_key<perspective::t_stnode, boost::multi_index::member<perspective::t_stnode, unsigned long, &perspective::t_stnode::m_pidx>, boost::multi_index::member<perspective::t_stnode, perspective::t_tscalar, &perspective::t_stnode::m_value>, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type>, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocatorperspective::t_stnode >, boost::mpl::v_item<perspective::by_idx, boost::mpl::vector0<mpl_::na>, 0>, boost::multi_index::detail::ordered_unique_tag, boost::multi_index::detail::null_augment_policy>::replace_boost::multi_index::detail::lvalue_tag(perspective::t_stnode const&, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::hashed_index_node<boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::ordered_index_node<boost::multi_index::detail::null_augment_policy, boost::multi_index::detail::index_node_base<perspective::t_stnode, std::allocatorperspective::t_stnode > > >, boost::multi_index::detail::hashed_non_unique_tag>, boost::multi_index::detail::hashed_non_unique_tag> >, boost::multi_index::detail::lvalue_tag) ()
from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#2 0x00007f41186be0ad in perspective::t_stree::update_shape_from_static(perspective::t_dtree_ctx const&) ()
from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#3 0x00007f41186e29f4 in perspective::notify_sparse_tree_common(std::shared_ptrperspective::t_data_table, std::shared_ptrperspective::t_data_table, std::shared_ptrperspective::t_stree, std::shared_ptrperspective::t_traversal, bool, std::vector<perspective::t_aggspec, std::allocatorperspective::t_aggspec > const&, std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > const&, std::vector<perspective::t_sortspec, std::allocatorperspective::t_sortspec > const&, perspective::t_gstate const&) ()
from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#4 0x00007f41186e40d1 in perspective::notify_sparse_tree(std::shared_ptrperspective::t_stree, std::shared_ptrperspective::t_traversal, bool, std::vector<perspective::t_aggspec, std::allocatorperspective::t_aggspec > const&, std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > const&, std::vector<perspective::t_sortspec, std::allocatorperspective::t_sortspec > const&, perspective::t_data_table const&, perspective::t_data_table const&, perspective::t_data_table const&, perspective::t_data_table const&, perspective::t_data_table const&, perspective::t_data_table const&, perspective::t_config const&, perspective::t_gstate const&) () from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#5 0x00007f41185f8782 in perspective::t_ctx1::notify(perspective::t_data_table const&, perspective::t_data_table const&, perspective::t_data_table const&, perspective::t_data_table const&, perspective::t_data_table const&, perspective::t_data_table const&) () from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#6 0x00007f41186843bd in void perspective::t_gnode::notify_contextperspective::t_ctx1(perspective::t_data_table const&, perspective::t_ctx_handle const&) ()
from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#7 0x00007f411867986c in tbb::interface9::internal::start_for<tbb::blocked_range, tbb::internal::parallel_for_body<perspective::t_gnode::notify_contexts(perspective::t_data_table const&)::{lambda(int)#2}, int>, tbb::auto_partitioner const>::execute() () from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#8 0x00007f411843ab05 in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::process_bypass_loop(tbb::internal::context_guard_helper&, tbb::task*, long) ()
from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libtbb.so
#9 0x00007f411843ae86 in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::local_wait_for_all(tbb::task&, tbb::task*) ()
from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libtbb.so
#10 0x00007f4118438a10 in tbb::internal::generic_scheduler::local_spawn_root_and_wait(tbb::task*, tbb::task*&) ()
from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libtbb.so
#11 0x00007f411866fd93 in perspective::t_gnode::notify_contexts(perspective::t_data_table const&) ()
from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#12 0x00007f4118677804 in perspective::t_gnode::process(unsigned long) () from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#13 0x00007f41186e72cc in perspective::t_update_task::run() () from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#14 0x00007f4118693beb in perspective::t_pool::_process_helper() () from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#15 0x00007f4118693c0e in perspective::t_pool::_process() () from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/../../perspective_python.lib/libpsp.so
#16 0x00007f4119b6271e in ZZN8pybind1112cpp_function10initializeIZNS0_C4IvN11perspective6t_poolEJEJNS_4nameENS_9is_methodENS_7siblingEEEEMT0_FT_DpT1_EDpRKT2_EUlPS4_E_vJSI_EJS5_S6_S7_EEEvOS9_PFS8_SB_ESH_ENUlRNS_6detail13function_callEE1_4_FUNESP () from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/libbinding.so
#17 0x00007f4119b34681 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) () from /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/libbinding.so
#18 0x00000000004bf19b in _PyCFunction_FastCallDict (kwargs=0x0, nargs=, args=0x7f3fd82bbc88, func_obj=<built-in method _process of PyCapsule object at remote 0x7f4119cae1e0>) at Objects/methodobject.c:231
#19 _PyCFunction_FastCallKeywords (func=<built-in method _process of PyCapsule object at remote 0x7f4119cae1e0>, stack=0x7f3fd82bbc88, nargs=, kwnames=) at Objects/methodobject.c:294
#20 0x00000000005479b7 in call_function (pp_stack=pp_stack@entry=0x7f3fd3ffd830, oparg=, kwnames=kwnames@entry=0x0) at Python/ceval.c:4848
#21 0x000000000054ae4d in _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3322
#22 0x0000000000547861 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7f3fd82bbaf8, for file /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/_state.py, line 69, in call_process (self=<_PerspectiveStateManager(queue_process=<functools.partial at remote 0x7f3ddcb26e08>) at remote 0x7f411172ae48>, table_id=0, pool=<perspective.table.libbinding.t_pool at remote 0x7f3dea2fb110>)) at Python/ceval.c:753
#23 _PyEval_EvalCodeWithName (_co=_co@entry=<code at remote 0x7f4290fe6930>,
globals=globals@entry={'name': 'perspective.table._state', 'doc': None, 'package': 'perspective.table', 'loader': <SourceFileLoader(name='perspective.table._state', path='/data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/_state.py') at remote 0x7f4119cafac8>, 'spec': <ModuleSpec(name='perspective.table._state', loader=<...>, origin='/data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/_state.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/pycache/_state.cpython-36.pyc', _initializing=False) at remote 0x7f4119c4a208>, 'file': '/data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/_state.py', 'cached': '/data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/perspective/table/pycache/_state.cpython-36.pyc'...(truncated), locals=locals@entry=0x0,
args=args@entry=0x7f3fd3ffda80, argcount=argcount@entry=1, kwnames=kwnames@entry=0x7f3dbc280be0, kwargs=0x7f3dbc280be8, kwcount=2, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name='call_process',
qualname='_PerspectiveStateManager.call_process') at Python/ceval.c:4153
#24 0x0000000000550996 in _PyFunction_FastCallDict (func=func@entry=<function at remote 0x7f4119c4e950>, args=args@entry=0x7f3fd3ffda80, nargs=nargs@entry=1, kwargs=kwargs@entry={'table_id': 0}) at Python/ceval.c:5057
#25 0x00000000004566f1 in _PyObject_FastCallDict (kwargs={'table_id': 0}, nargs=1, args=0x7f3fd3ffda80, func=<function at remote 0x7f4119c4e950>) at Objects/abstract.c:2310
#26 _PyObject_Call_Prepend (func=<function at remote 0x7f4119c4e950>, obj=, args=(), kwargs={'table_id': 0}) at Objects/abstract.c:2373
#27 0x000000000045619e in PyObject_Call (func=func@entry=<method at remote 0x7f3db7edd6c8>, args=args@entry=(), kwargs=kwargs@entry={'table_id': 0}) at Objects/abstract.c:2261
#28 0x000000000054ab3c in do_call_core (kwdict={'table_id': 0}, callargs=(), func=<method at remote 0x7f3db7edd6c8>) at Python/ceval.c:5093
#29 _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3391
#30 0x0000000000547861 in PyEval_EvalFrameEx (throwflag=0,
f=Frame 0x7f3f0805cf68, for file /data02/apps/hfalgo_ext/10084d7/python/Python-3.6.4/lib/python3.6/site-packages/tornado/stack_context.py, line 300, in null_wrapper (args=(), kwargs={'table_id': 0}, current_state=((...), None))) at Python/ceval.c:753

[timkpaine]

we have an open ticket to audit GIL, the problem is we call back into python in a few places so we need to be a bit careful about how we do it.

[robambalu]

any thoughts on the crash and infinite loop? as is perspective-python is unusable for us

[texodus]

Thanks for the report! I believe I have encountered the same bug (in that it loops in the same place) in my own project. For this issue, I've merged a fix and it will be released in 0.5.6. This was ultimately unrelated to the GIL, threading or race condition, but is tricky to repro (requires a row and column group to be removed in the same update step on the same view).

Regarding GIL usage generally in Perspective, we have recently relaxed GIL holding for some calls, to_arrow() and update() for example. However, this does not change that Perspective is not thread-safe, so in order to enable GIL-less operation, you'll need to tell the perspective module which thread ID to expect. More details on async operation can be found here.