Which part should I change so that code execution happens in a python subprocess or equivalent instead of Jupyter kernel when using taskweaver local mode as a library #494
Description
Is your feature request related to a problem? Please describe.
I'm trying to deploy a chat app powered by TaskWeaver in the backend at work. The easiest way to go is deploying it to Databricks apps, however databricks apps cannot find Jupyter kernel to execute code in.
Error message:
| Source | Deployment | Logs |
|---|---|---|
| 2025-08-26 22:07:11 | APP | ...c05fbaf |
| 2025-08-26 22:07:11 | APP | ...c05fbaf |
| 2025-08-26 22:07:11 | APP | ...c05fbaf |
| 2025-08-26 22:07:11 | APP | ...c05fbaf |
| 2025-08-26 22:07:11 | APP | ...c05fbaf |
2025-08-26 22:07:11 | APP | ...c05fbaf | proc = Popen(cmd, **kwargs) # noqa
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/.venv/lib/python3.11/site-packages/jupyter_client/launcher.py", line 155, in launch_kernel
2025-08-26 22:07:11 | APP | ...c05fbaf | raise ex
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/.venv/lib/python3.11/site-packages/jupyter_client/launcher.py", line 170, in launch_kernel
2025-08-26 22:07:11 | APP | ...c05fbaf | self.process = launch_kernel(cmd, **scrubbed_kwargs)
2025-08-26 22:07:11 | APP | ...c05fbaf | ^^^^ File "/app/python/source_code/.venv/lib/python3.11/site-packages/jupyter_client/provisioning/local_provisioner.py", line 210, in launch_kernel
2025-08-26 22:07:11 | APP | ...c05fbaf | connection_info = await self.provisioner.launch_kernel(kernel_cmd, **kw)
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/.venv/lib/python3.11/site-packages/jupyter_client/manager.py", line 354, in _async_launch_kernel
2025-08-26 22:07:11 | APP | ...c05fbaf | await self._async_launch_kernel(kernel_cmd, **kw)
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/.venv/lib/python3.11/site-packages/jupyter_client/manager.py", line 439, in _async_start_kernel
2025-08-26 22:07:11 | APP | ...c05fbaf | out = await method(self, *args, **kwargs)
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/.venv/lib/python3.11/site-packages/jupyter_client/manager.py", line 87, in wrapper
2025-08-26 22:07:11 | APP | ...c05fbaf | raise e
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/.venv/lib/python3.11/site-packages/jupyter_client/manager.py", line 96, in wrapper
2025-08-26 22:07:11 | APP | ...c05fbaf | raise self._exception
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
2025-08-26 22:07:11 | APP | ...c05fbaf | return self.__get_result()
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/usr/lib/python3.11/concurrent/futures/_base.py", line 456, in result
2025-08-26 22:07:11 | APP | ...c05fbaf | return fut.result(None)
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/.venv/lib/python3.11/site-packages/jupyter_core/utils/init.py", line 127, in run
2025-08-26 22:07:11 | APP | ...c05fbaf | return _runner_map[name].run(inner)
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/.venv/lib/python3.11/site-packages/jupyter_core/utils/init.py", line 164, in wrapped
2025-08-26 22:07:11 | APP | ...c05fbaf | starter = ensure_async(km.start_kernel(**kwargs))
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/.venv/lib/python3.11/site-packages/jupyter_client/multikernelmanager.py", line 271, in _async_start_kernel
2025-08-26 22:07:11 | APP | ...c05fbaf | File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
2025-08-26 22:07:11 | APP | ...c05fbaf | return loop.run_until_complete(inner)
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/.venv/lib/python3.11/site-packages/jupyter_core/utils/init.py", line 158, in wrapped
2025-08-26 22:07:11 | APP | ...c05fbaf | session.kernel_id = self.multi_kernel_manager.start_kernel(
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/taskweaver/ces/environment.py", line 229, in start_session
2025-08-26 22:07:11 | APP | ...c05fbaf | self.mgr.env.start_session(self.session_id, session_dir=self.session_dir, cwd=self.cwd)
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/taskweaver/ces/manager/sub_proc.py", line 26, in start
2025-08-26 22:07:11 | APP | ...c05fbaf | client.start()
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/taskweaver/ces/manager/defer.py", line 97, in task
2025-08-26 22:07:11 | APP | ...c05fbaf | result = (init(),)
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/app/python/source_code/taskweaver/ces/manager/defer.py", line 33, in task
2025-08-26 22:07:11 | APP | ...c05fbaf | self._target(*self._args, **self._kwargs)
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/usr/lib/python3.11/threading.py", line 982, in run
2025-08-26 22:07:11 | APP | ...c05fbaf | self.run()
2025-08-26 22:07:11 | APP | ...c05fbaf | File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
2025-08-26 22:07:11 | APP | ...c05fbaf | Traceback (most recent call last):
2025-08-26 22:07:11 | APP | ...c05fbaf | During handling of the above exception, another exception occurred:
2025-08-26 22:07:11 | APP | ...c05fbaf | RuntimeError: no running event loop
2025-08-26 22:07:11 | APP | ...c05fbaf | asyncio.get_running_loop()
Describe the solution you'd like
Is there a simple patch I can apply to fix this issue? Somethings that allows code execution to happen as a python subprocess or the equivalent instead of a Jupyter kernel?
Describe alternatives you've considered
Additional context
Ideally I don't have to deploy this as a docker image because that comes with it's own difficulties in a corporate environment, but if applying patch to the code base is too difficult, I'll have to go that route, please provide some advice too on how that can be handled if I still want to use a simple app front end deployed to databricks apps.