Assign names to system threads.

Author: xavierog

CHANGELOG.md

@@ -6,6 +6,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) for its CLI (Command-Line Interface), i.e. for the `moulti` command.
 Although Moulti's Python packages, modules and functions are obviously available, they do not constitute a public API yet.
 
+## Unreleased
+
+### Changed
+
+- Unless `MOULTI_NAME_THREADS=no` is set, system threads get assigned meaningful names:
+  `moulti:main`, `network-loop`,`exec-command`, `export-to-dir`, `|step_name`, `>step_name`, `thread-pool`.
+
 ## [1.33.0] - 2025-02-26
 
 ### Changed

doc/docs/environment-variables.md

@@ -147,6 +147,11 @@ Same as `MOULTI_DIFF_VERBOSE` for `moulti manpage`
 
 `light` to start Moulti in light mode, `dark` to start Moulti in dark mode; defaults to `dark`.
 
+### MOULTI_NAME_THREADS
+
+By default, Moulti assigns names to its [system threads](threads.md).
+Setting `MOULTI_NAME_THREADS=no` disables this behavior.
+
 ### MOULTI_PASS_CONCURRENCY
 
 Define how many concurrent "moulti pass" commands is acceptable; defaults to 20.

doc/docs/threads.md

@@ -0,0 +1,181 @@
+# Threads
+
+Under the hood, Moulti uses multiple [threads](https://en.wikipedia.org/wiki/Thread_(computing)).
+This document intends to describe them.
+
+## Main thread
+
+Name: `moulti:main`
+
+Moulti's main thread runs [Textual](https://textual.textualize.io/)'s
+[asyncio](https://docs.python.org/3/library/asyncio.html)
+[event loop](https://en.wikipedia.org/wiki/Event_loop).
+Otherly put, it manages most of the TUI part.
+
+## Textual threads
+
+All Textual applications spawn two unnamed threads and Moulti is no exception:
+
+- a first thread renders the user interface to stderr;
+- a second thread reads events (e.g. keystrokes and mouse events) from stdin.
+
+## Network loop
+
+Name: `network-loop`
+
+Once the TUI part is ready, Moulti spawns a third thread that runs the network
+loop, i.e. the infinite loop that handles network events: new connections,
+incoming messages, outgoing responses, etc.
+This loop relies on non-blocking Unix sockets. Unlike the main thread, it does
+not leverage asyncio.
+Incoming messages are analyzed within this thread but actual TUI changes (e.g.
+adding or removing steps) are delegated to the main thread.
+
+## Execute command
+
+Name: `exec-command`
+
+When Moulti is launched through [moulti run](shell-scripting.md#moulti-run), and
+once the network loop is ready, Moulti spawns a fourth thread in charge of
+executing and monitoring the given command.
+
+## moulti pass
+
+Each `moulti pass` command results in Moulti spawning two extra threads.
+
+### Data ingestion
+
+Name: `|step_name`
+
+This thread reads data from the file descriptor provided by `moulti pass`.
+This file descriptor is typically a pipe, hence the `|` prefix.
+
+### Data processing
+
+Name: `>step_name`
+
+This thread converts data read by the ingestion thread so they can be
+added to the TUI.
+The `>` prefix represents the idea that lines get written to the target step.
+Unlike the `>` redirection in shell languages, this symbol does not imply creating or writing to files.
+
+### Why do step names look weird?
+
+Thread names cannot exceed:
+
+- 15 characters on Linux;
+- 63 characters on macOS;
+- 19 characters on FreeBSD;
+- 23 characters on OpenBSD;
+- 15 characters on NetBSD.
+
+Here, the first character is either `|` or `>`, leaving only:
+
+- 14 characters on Linux;
+- 62 characters on macOS;
+- 18 characters on FreeBSD;
+- 22 characters on OpenBSD;
+- 14 characters on NetBSD.
+
+This is why longer step names must be abridged in thread names.
+This is particularly true on Linux and NetBSD where abridged names keep:
+
+- the first 4 characters;
+- the middle 5 characters;
+- the last 5 characters.
+
+!!! example "Examples"
+	- `moulti pass abcdefghijklmnopqrstuvwxyz` results in threads
+        `|abcdklmnovwxyz` and `>abcdklmnovwxyz`:
+
+        ```
+        abcdefghijklmnopqrstuvwxyz
+        ^^^^      ^^^^^      ^^^^^
+        first 4   middle 5   last 5
+        ```
+
+	- The `moulti_run_output` special step results in threads `|mouli_runutput`
+	    and `>mouli_runutput`.
+
+        ```
+        moulti_run_output
+        ^^^^ ^^^^^  ^^^^^
+          |    |      `---- last 5
+          |    `----------- middle 5
+          `---------------- first 4
+        ```
+
+## Save / export contents
+
+Name: `export-to-dir`
+
+Through its "Save" action, Moulti is able to [export all shown contents to text
+files](saving-and-loading.md/#saving-a-complete-moulti-instance) stored inside
+a timestamped directory, hence the name of the thread dedicated to this operation.
+
+## Inactive threads
+
+Name: `thread-pool`
+
+Threads that have finished executing are not deleted.
+Instead, they remain inactive until the [thread pool](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor) decides to reuse them.
+
+## How to inspect thread names?
+
+The previous sections mention thread names.
+But how to inpsect threads and their names in the first place?
+
+### How to inspect thread names on Linux?
+
+Combining `pgrep` and `pstree` (typically found in the `psmisc` package) does wonders:
+
+```
+pgrep moulti:main | xargs -n 1 pstree -napt
+```
+
+Alternatively, use `htop`: hit `H` to toggle threads.
+
+### How to inspect thread names on FreeBSD?
+
+```
+ps Ho pid,tid,tdname,command
+```
+
+Thread names are shown in the `TDNAME` column.
+
+### How to inspect thread names on OpenBSD?
+
+```
+ps Ho pid,tid,command
+```
+
+Thread names are shown at the end of the `COMMAND` column, between parens, after
+the `/` character.
+
+### How to inspect thread names on NetBSD?
+
+```
+top -t -c -U $(whoami)
+```
+
+Thread names are shown in the `NAME` column, which is unfortunately truncated
+to 9 characters.
+
+### How to inspect thread names on macOS?
+
+On macOS, listing threads requires root privileges:
+
+```
+sudo htop -t -u $(whoami)
+```
+
+## How to turn off thread names?
+
+If you suspect naming threads triggers issues (e.g. crash on some niche
+operating system), you can disable it by setting the
+[`MOULTI_NAME_THREADS`](environment-variables.md/#moulti_name_threads)
+environment variable to `no`:
+
+```bash
+export MOULTI_NAME_THREADS=no
+```

doc/mkdocs.yml

@@ -31,6 +31,7 @@ nav:
   - Classes: classes.md
   - Design: design.md
   - Technical requirements: technical-requirements.md
+  - Threads: threads.md
   - Why this name?: why-this-name.md
 - Reference:
   - Commands: commands.md

src/moulti/app.py

@@ -32,6 +32,7 @@
 from .security import MoultiSecurityPolicy
 from .server import Message, FDs, MoultiServer, current_instance
 from .themes import MOULTI_THEMES
+from .thread_name import thread_name, KEEP_THREAD_NAME
 from .widgets import MoultiWidgetException
 from .widgets.tui import MoultiWidgets
 from .widgets.footer import Footer
@@ -260,6 +261,7 @@ def on_mount(self) -> None:
 			self.register_theme(theme)
 		self.theme = 'moulti-dark' if self.dark else 'moulti-light'
 
+	@thread_name('moulti:main', end=KEEP_THREAD_NAME)
 	def on_ready(self) -> None:
 		self.logconsole(f'Moulti v{MOULTI_VERSION}, Textual v{TEXTUAL_VERSION}, Python v{sys.version}')
 		self.logconsole(f'instance "{self.instance_name}", PID {os.getpid()}')
@@ -374,6 +376,7 @@ def debug(line: str) -> None:
 		step.append_from_file_descriptor_to_queue(queue, {}, helpers)
 
 	@work(thread=True, group='app-exec', name='moulti-run')
+	@thread_name('exec-command')
 	def exec(self, command: list[str]) -> None:
 		"""
 		Launch the given command with the assumption it is meant to drive the current Moulti instance.
@@ -562,6 +565,7 @@ def on_quit_dialog_exit_request(self, exit_request: QuitDialog.ExitRequest) -> N
 		self.exit()
 
 	@work(thread=True, group='app-save', name='save')
+	@thread_name('export-to-dir')
 	def action_save(self) -> None:
 		"""
 		Export everything currently shown by the instance as a bunch of files in a directory.
@@ -756,6 +760,7 @@ def debug(line: str) -> None:
 				self.reply(connection, raddr, message, done=error is None, error=error)
 
 	@work(thread=True, group='app-network', name='network-loop')
+	@thread_name('network-loop', end=KEEP_THREAD_NAME)
 	def network_loop(self) -> None:
 		current_worker = get_current_worker()
 		self.server = MoultiServer(

src/moulti/thread_name.py

@@ -0,0 +1,125 @@
+"""
+Provide helpers to assign human-friendly names to system threads.
+"""
+
+import sys
+from ctypes import CDLL, c_char_p, c_void_p
+from ctypes.util import find_library
+from functools import cache, wraps
+from threading import current_thread
+from typing import Any, Callable
+from .environ import env
+from .helpers import abridge_string
+
+def thread_name_max_len() -> int:
+	"""
+	Return the maximum name length (excluding trailing null byte) for system threads.
+	"""
+	platform = sys.platform
+	# Most Unix operating systems truncate thread names to MAXCOMLEN characters + a null byte:
+	max_len = 15
+	if platform.startswith("darwin"):
+		max_len = 63
+	elif platform.startswith("freebsd"):
+		max_len = 19
+	elif platform.startswith("openbsd"):
+		max_len = 23
+	elif platform.startswith("netbsd"):
+		max_len = 15
+	return max_len
+
+
+THREAD_NAME_MAX_LEN = thread_name_max_len()
+
+
+@cache
+def pthread_set_name_function() -> Callable:
+	"""
+	Return a simple wrapper around pthread_setname_np().
+	This wrapper expects a name as single string (str) argument and assigns it to the current thread.
+	"""
+	# Access the pthread library:
+	pthread_lib = CDLL(find_library("pthread"))
+
+	# Access the pthread_self function:
+	pthread_self = pthread_lib.pthread_self
+	pthread_self.restype = c_void_p
+
+	# Access the pthread_set_name_np function:
+	try:
+		pthread_setname_np = pthread_lib.pthread_setname_np
+	except AttributeError:
+		pthread_setname_np = pthread_lib.pthread_set_name_np
+
+	# Deal with its non-portable (hence "_np") arguments:
+	if sys.platform.startswith("netbsd"):
+		pthread_setname_np.argtypes = [c_void_p, c_char_p, c_char_p]
+		def make_args(name: bytes) -> tuple:
+			return (pthread_self(), b"%s\0", name)
+	elif sys.platform.startswith("darwin"):
+		pthread_setname_np.argtypes = [c_char_p]
+		def make_args(name: bytes) -> tuple:
+			return (name,)
+	else:
+		pthread_setname_np.argtypes = [c_void_p, c_char_p]
+		def make_args(name: bytes) -> tuple:
+			return (pthread_self(), name,)
+
+	def function(name: str) -> None:
+		name_bytes = name.encode("ascii", "replace")[:THREAD_NAME_MAX_LEN] + b"\0"
+		pthread_setname_np(*make_args(name_bytes))
+	return function
+
+
+def set_thread_name(name: str) -> None:
+	"""
+	Assign the given name to the current thread.
+	If necessary, the name gets abridged so as to fit system limitations (typically 15 or 63 characters max).
+	"""
+	if env("MOULTI_NAME_THREADS") == "no":
+		return
+	abridged_name = abridge_string(name, threshold=THREAD_NAME_MAX_LEN, sep="")
+
+	# Python >= 3.14 assigns names to system threads out of the box:
+	if sys.version_info >= (3, 14):
+		current_thread().name = abridged_name
+	else: # Previous versions of Python:
+		try:
+			pthread_set_name_np = pthread_set_name_function()
+			pthread_set_name_np(abridged_name)
+		except Exception:
+			# Best-effort: we are not even interested in what went wrong.
+			pass
+
+
+
+IDLE_THREAD_NAME = "thread-pool"
+"""
+Thread name set by the thread_name() decorator after a decorated function
+returns. This value reflects the underlying thread has become idle and has
+reintegrated the thread pool.
+"""
+
+KEEP_THREAD_NAME = None
+"""
+Legible constant, to be used with the thread_name() decorator.
+"""
+
+def thread_name(start: str|None = KEEP_THREAD_NAME, end: str|None = IDLE_THREAD_NAME) -> Callable:
+	"""
+	Decorator that calls:
+	- set_thread_name(start) before the decorated function, unless start is KEEP_THREAD_NAME;
+	- set_thread_name(end) after the decorated function, unless end is KEEP_THREAD_NAME.
+	"""
+	def thread_name_decorator(func: Callable) -> Callable:
+		@wraps(func)
+		def wrapper(*args: Any, **kwargs: Any) -> Any:
+			try:
+				if start is not KEEP_THREAD_NAME:
+					set_thread_name(start)
+				return func(*args, **kwargs)
+			finally:
+				if end is not KEEP_THREAD_NAME:
+					set_thread_name(end)
+		return wrapper
+	return thread_name_decorator