fix(state): preserve SQLite locks during Linux permission hardening

[bricelb]

Summary

Fixes #109728. Regression follow-up to the permission hardening merged in #109509.

On Linux, _secure_state_db_files opens and closes ordinary descriptors for the database and its WAL/SHM files. Closing such a descriptor releases the process's POSIX locks on that inode, including locks held by SQLite through other descriptors. Opening a second SessionDB can therefore silently remove the first connection's locks. Another SQLite process may then unlink its sidecars, leaving live connections on deleted generations. The deleted-generation guard correctly refuses further access, but the result is a session outage.

This behavior is documented in SQLite's corruption guidance. The regression reproduces with a throwaway database; no manual WAL deletion, update script, or session cleanup is required.

Changes

• On Linux, pin existing files with O_PATH | O_NOFOLLOW | O_CLOEXEC and apply owner-only permissions through /proc/self/fd/<fd>. Closing an O_PATH descriptor does not cancel SQLite's POSIX locks.
• Create a missing main database exclusively with mode 0600 before SQLite opens it. If another opener creates it first, use the existing-file path.
• Preserve symlink rejection and reject non-regular files, while retaining SQLite's existing error path for directories.
• Keep the existing non-Linux implementation unchanged. This fix relies on Linux procfs and does not claim to solve the equivalent lock behavior on every POSIX platform.
• Add two Linux-only invariant test functions (four cases): repeated real SQLite openers must preserve the live WAL generation, and private creation/symlink rejection must remain intact.

The shared helper also covers callers in async delegation. No WAL guard is disabled, no journal-mode default changes, no sidecars are manually removed, and no permanent descriptor cache is introduced.

Validation

Tested on Linux with Python 3.11, against main b6b53c69a6ed49cb099cf1bfe76b5e6edd718e5a, using scripts/run_tests.sh in an isolated test environment.

Red on unmodified main:

scripts/run_tests.sh -j 1 --file-retries 0 tests/test_state_permission_locks.py

Result: 1 failed, 3 passed. The live-generation test finds three deleted sidecar holders where it expects none.

With this fix:

scripts/run_tests.sh -j 1 --file-retries 0 \
  tests/test_state_permission_locks.py \
  tests/test_hermes_state.py \
  tests/tools/test_async_delegation.py \
  tests/hermes_state/test_deleted_wal_generation_guard.py \
  tests/hermes_state/test_deleted_wal_checkpoint_guard.py

Result: 328 passed, 1 failed, 5 skipped. All four new regression cases pass.

The one failure is TestFTS5Search::test_search_projection_skips_context_enrichment_queries (context_query_count() is 0 rather than 1). I independently reran that test on the unmodified base and obtained the same failure. It was also noted in #109509; this PR does not change the unrelated FTS behavior.

ruff check hermes_state.py tests/test_state_permission_locks.py and git diff --check pass.

The same fix was applied temporarily to the affected Linux installation. Gateway/dashboard operation resumed without a database restore or journal-mode change, and the user confirmed that existing sessions open and a new prompt receives a response. This contribution does not change that installation's official upstream remote or update workflow.

[teknium1]

Verdict: premise confirmed, fix is correct on Linux, one review-fix pushed. Recommend merge after Teknium's call; macOS half of the class still open (non-blocking).

Context. _secure_state_db_files (from #109509's umask hardening) runs os.open/fchmod/close on the live state.db, -wal and -shm every time a writer connection opens, including _open_writer_conn after sqlite3.connect. Per SQLite's howtocorrupt §2.2 the close() cancels every POSIX record lock this process holds on those inodes, so the next short-lived opener/closer in another process thinks it is the last holder and unlinks the WAL generation. This is the producer behind #109687 (and the "5 live SessionDB handles" precursor in #100896 makes sense now: each extra in-process handle re-runs the helper on a live generation).

Live repro (Linux 7.0, Python 3.11, SQLite from the repo venv), tmp HERMES_HOME, real SessionDB:

step	origin/main @ b6b53c6	this PR @ 64ade66
one SessionDB + external sqlite3.connect/close	sidecars survive	sidecars survive
second in-process SessionDB, then external reader close	-wal/-shm gone; process holds 3 (deleted) fds; next write → DeletedWalGenerationError	sidecars survive, 0 deleted fds, write OK, all three files 0600

So the bug fires deterministically on main with two in-process handles (the gateway's normal state), and this branch closes it. tests/hermes_state (342) + the new file pass; ruff clean.

Code notes

• O_PATH + os.chmod("/proc/self/fd/N") is the right shape on Linux: no lchmod, fchmod on an O_PATH fd is EBADF, and os.chmod(path, follow_symlinks=False) is unsupported there (os.chmod in os.supports_follow_symlinks is False). Verified.
• O_EXCL create-then-O_PATH for the missing main file keeps the "private from the first byte" property; the created fd is closed before SQLite has any lock, so that close is safe.
• Symlink handling: O_PATH|O_NOFOLLOW on a symlink yields an fd to the link itself, hence the S_ISLNK → ELOOP branch. Good; the test proves the target's mode/content are untouched.
• Pushed one commit to your branch (bdcec6b6): encoding="utf-8" on the two read_text/write_text calls in the new test. CI's lint job runs scripts/check-windows-footguns.py --all and would have flagged them; no behaviour change.

Remaining gap, not blocking this PR: the non-Linux branch still does the raw O_RDONLY open/fchmod/close on live files, and POSIX lock cancellation applies on macOS too (#104451 is a macOS report with this signature; #109641 says the holder scan is Linux-only, so macOS has neither the guard nor the fix). macOS has lchmod, so os.lstat + S_ISREG check + os.chmod(path, 0o600, follow_symlinks=False) would close the class there without any descriptor. Happy to take that as a follow-up if you'd rather keep this one Linux-scoped.

Siblings seen in the sweep: #109725 (read-only sessions stats) and #109737 (NO_CKPT_ON_CLOSE / pin writer on 3.11) attack the closer side; a bare sqlite3.connect().close() reproduces the orphan on main, so neither replaces this fix. Not merging; that decision is Teknium's.

[teknium1]

Maintainer bot review, live-verified on Linux (Py 3.11.15): #109728 repro after: 3 → 0; stricter probe (parent holds BEGIN IMMEDIATE, helper runs, child tries to write) → child gets database is locked in both WAL and journal_mode=delete. On main both fail. Tests pass locally.

Two notes, neither blocking the Linux fix:

1. Non-Linux still drops locks. path_only is Linux-only; the fallback branch keeps the O_RDONLY open/fchmod/close, so macOS (see #109752/#109759 reports) has the same bug after this lands. A plain lstat + symlink refusal + os.chmod(path, 0o600) for existing files works on every POSIX platform without O_PATH/procfs, and O_CREAT|O_EXCL for first creation keeps the private-from-first-byte property. That would make one platform-neutral helper instead of two code paths.
2. Trim to the two invariant tests you have (lock retention + symlink refusal) — good as-is.

Merge decision is Teknium's; not approving/merging from here.

[teknium1]

Superseded by #109841 (merged, e16f686): platform-independent fix to the same _secure_state_db_files helper — chmod(2) on the path for existing files, O_EXCL so only a brand-new inode ever gets a descriptor. Thank you for the clear analysis; it converged on the same mechanism.