fix(terminal): acquire _env_lock for all shared dict accesses - #6655
fix(terminal): acquire _env_lock for all shared dict accesses#6655aaronlab wants to merge 1 commit into
Conversation
…st_activity accesses - get_active_environments_info: snapshot dict under _env_lock before iterating, preventing RuntimeError if a concurrent cleanup pops an entry during iteration - cleanup_all_environments: take lock when reading keys so the snapshot is consistent with the cleanup thread - _cleanup_inactive_envs: snapshot _last_activity keys under lock before the process-registry scan, and write back under lock to avoid racing with terminal_tool() which holds the lock when updating activity - _atexit_cleanup: read len(_active_environments) under lock before logging and cleanup Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
The key snapshots before iteration (lines 824-825) and the locking around reads/writes (lines 943-946, 969-970, 1040-1041 in ) look correct for preventing race conditions in this file. Have you checked whether other parts of the codebase interact with these same shared resources (, ) to ensure consistent locking patterns everywhere? Also, considering the critical nature of these locks, have you thought about adding specific concurrency tests (e.g., stress tests with multiple threads) to validate that these locks hold up under heavy load? |
|
Overlaps significantly with #6684 which bundles this same _env_lock fix along with two other fixes (dead regex, _write_count race). |
|
Overlaps significantly with #6684 which bundles this same _env_lock fix along with two other fixes. |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for identifying a real lifecycle-locking gap. The premise still holds on current main: tools/terminal_tool.py:1550-1552, :1669, and :1765-1771 access the shared environment dictionaries outside _env_lock while cleanup_vm() removes entries under the lock at :1719-1721.
Problems
get_active_environments_info()no longer exists on current main; the corresponding hunk should be dropped during salvage.- Current
_atexit_cleanup()has an additionallist(_active_environments.values())snapshot attools/terminal_tool.py:1771. The PR locks only the count, leaving this new snapshot unlocked. - The PR has no concurrency regression test for these paths.
Suggested changes
- Adapt the remaining lock changes to current
tools/terminal_tool.py. - Snapshot both
countandenvs_to_waitunder_env_lockin_atexit_cleanup(), then perform cleanup and waits outside the lock. - Add a deterministic threaded test that races
cleanup_vm()with each snapshot path.
Automated hermes-sweeper review.
| """Stop cleanup thread and shut down all remaining sandboxes on exit.""" | ||
| _stop_cleanup_thread() | ||
| if _active_environments: | ||
| with _env_lock: |
There was a problem hiding this comment.
Current main's _atexit_cleanup() also snapshots list(_active_environments.values()) before cleanup. Include that snapshot in this same _env_lock block; otherwise the current HEAD path can still race with cleanup_vm().
Summary
_env_lockbefore accessing_active_environmentsand_last_activityin four functions that previously read/iterated these dicts without the lock, riskingRuntimeError: dictionary changed size during iterationwhen the background cleanup thread concurrently pops entries.Details
get_active_environments_info()(line 938)Previously iterated
_active_environmentsdirectly without_env_lock. If_cleanup_inactive_envs()calls.pop()on the dict via the background cleanup thread during iteration, Python raisesRuntimeError. Fix: snapshotcountandtask_idsunder the lock, then iterate the snapshot for disk-usage calculation (the slowrglobcall stays outside the lock).cleanup_all_environments()(line 963)Read
.keys()without the lock. Whilelist()makes the snapshot itself safe, the pattern is inconsistent with every other function in the module. Fix: acquire_env_lockfor the snapshot._cleanup_inactive_envs()pre-scan (line 822)Read and wrote
_last_activitywithout the lock, racing withterminal_tool()which updates_last_activityinside the lock (lines 1205, 1222, 1280). Fix: snapshot keys under lock, write updates under lock._atexit_cleanup()(line 1032)Accessed
_active_environmentsdirectly for truthiness check andlen(). Fix: readlen()under lock.Test plan
get_active_environments_info()returns correct data under concurrent cleanup_cleanup_inactive_envspre-scan doesn't race withterminal_toolactivity updatespytest tests/ -qto ensure no regressions🤖 Generated with Claude Code