fix(ingest): finish the non-regular-file guards left out of 3.7.1 (#2221)

[mvalentsev]

What does this PR do?

The #2221 fix reached develop through #2228 at an earlier revision of the branch, so three of its guards and their tests did not come with it. #2221 is still open. This PR carries the remainder — 6 files, +179/−10 — on top of the 3.7.1 code now on develop.

All three are covered: copy this branch's tests/test_non_regular_file_guards.py onto a pristine 906b918a and its five new tests fail 5/5 while the file's other 38 pass. Reverting each fix individually in place also kills exactly its own test, one for one.

• sweep books a failed stat as a failure again. The gate that landed probes the type with f.stat() inside a try, and its except OSError printed SKIP and continued. A dangling symlink, a symlink loop and a file unlinked between rglob and the gate all raise there. Before the gate existed every one of them reached sweep(), raised, and was appended to failures; afterwards "could not read this transcript" became a silent skip and a successful exit. A probe that fails is an error, not a benign file type — it is logged, printed as WARNING, and booked in failures. A probe that succeeds and reports a non-regular file still skips silently, unchanged.
• mempalace init no longer tracebacks on a directory it cannot enter. In _parse_gradle_root_project_name the is_file() gate sat in front of the try whose except OSError the parser already had, so a manifest under a directory with r but no x raised PermissionError out of a call that used to answer "no manifest name". The gate moves inside that try. _collect_manifest_names stats through os.path.isfile, which reports rather than raises, matching the parsers it guards. (Path.is_file() propagates EACCES on 3.9 through 3.13, all checked — EACCES is not in pathlib's _IGNORED_ERRNOS — and stops raising only on 3.14, where these tests would go quiet rather than fail.)
• split no longer blocks on a FIFO at its own output name — nor writes through a broken link. The type gate in main() covers the files the glob listed, but split_file builds its output names itself, so a pre-existing named pipe at one of them wedged write_text in the kernel waiting for a reader. The gate asks os.path.lexists, not os.path.exists: exists() follows the link, so a dangling symlink at an output name reads as "nothing there" and the write goes through it, creating the target — a chunk landing wherever the link points instead of in the output directory. The two calls differ on that one case and agree on every other (regular file, symlink to a file, missing name, FIFO, symlink to FIFO, directory). Output names that are anything but a regular file are skipped with a SKIP line.

Two smaller items:

• test_gather_origin_samples_survives_an_unreadable_directory broke under root rather than passing vacuously: CAP_DAC_OVERRIDE walks into the 0o444 directory, the walled-off file stays readable, and the count assertion sees two samples instead of one. It now carries the same needs_unprivileged_posix gate as the three new permission tests.
• Comment-only correction in miner._read_text_no_follow: F_SETLEASE on a FIFO fails EINVAL, not ENXIO (measured on Linux 6.18 / glibc 2.39). The code branches on EAGAIN and is unaffected.

How to test

# the five new tests plus the pre-existing one that broke under root
uv run pytest tests/test_non_regular_file_guards.py -v

# the modules this touches
uv run pytest tests/test_non_regular_file_guards.py tests/test_sweeper.py \
              tests/test_miner.py tests/test_split_mega_files.py -q

uv run ruff check .
uv run ruff format --check .

The three permission tests skip as root (geteuid() == 0), where CAP_DAC_OVERRIDE makes directory mode bits gate nothing — the state they need cannot be built there, so without the skip they fail rather than pass. Run them unprivileged. (unshare -r uv run pytest tests/test_non_regular_file_guards.py -q reproduces the root path: 40 passed, 3 skipped.)

To see the sweep regression, point it at a directory holding a broken symlink named like a transcript:

mkdir -p /tmp/sweepdemo && ln -s /nonexistent-target /tmp/sweepdemo/broken.jsonl
mempalace sweep /tmp/sweepdemo; echo "exit=$?"

On develop @ 906b918a the unreadable entry is reported as a skip and the command succeeds:

  SKIP: broken.jsonl (stat error: No such file or directory)
  Swept 0/1 files from /tmp/sweepdemo: +0 new, 0 already present, 0 skipped (< cursor).
exit=0

Before the gate existed — 9a3afc9d, the merge base of the #2223 branch — the same directory (plus a symlink loop) reported WARNING: sweep failed on ... for each entry, 2 file(s) failed to sweep, and exit=2. With this PR that accounting is back:

sweeper: stat failed on /tmp/sweepdemo/broken.jsonl: [Errno 2] No such file or directory: '/tmp/sweepdemo/broken.jsonl'
  WARNING: stat failed on /tmp/sweepdemo/broken.jsonl: [Errno 2] No such file or directory: '/tmp/sweepdemo/broken.jsonl'
  WARNING: 1 file(s) failed to sweep - see stderr / logs for details.
  Swept 0/1 files from /tmp/sweepdemo: +0 new, 0 already present, 0 skipped (< cursor).
exit=2

Full suite locally: 4300 passed, 31 skipped, 0 failed. One caveat for anyone running it on a loaded machine: tests/test_chroma_collection_lock.py, which this PR does not touch, flakes there — its helper re-imports mempalace.backends.chroma (and so chromadb) in a spawned child, and the parent waits for the child's ready flag for only 500 × 0.01 s before asserting "holder failed to acquire lock". A pristine 906b918a flakes the same file under load; both trees pass it 5/5 when the machine is quiet.

Checklist

• Tests pass (python -m pytest tests/ -v) — 4300 passed, 31 skipped, 0 failed
• No hardcoded paths
• Linter passes (ruff check ., ruff 0.16.1 as pinned in CI; ruff format --check . clean)

Refs

• Closes mempalace mine hangs forever on a named pipe in the mined directory (sweep, init, compress and split too) #2221 (the issue is still open; fix: 3.7.1 critical patch — hang, re-mine honesty, stale Chroma cache #2228 carried part of the fix)
• Follows fix: 3.7.1 critical patch — hang, re-mine honesty, stale Chroma cache #2228, which carried the fix(ingest): never block on a non-regular file (#2221) #2223 branch's work as db29959 and 0f3f0c6
• Supersedes fix(ingest): never block on a non-regular file (#2221) #2223, now closed. Rebasing it was the alternative, so I ran that rebase rather than reason about it. It conflicts in four files — miner.py, project_scanner.py, sweeper.py and the test file — and every one of those conflicts is the branch's own earlier revision against its later one (miner.py's single hunk, for instance, is the ENXIO/EINVAL comment). None is a collision with someone else's work. Resolved hunk by hunk it lands +144/−12 against develop — the same three fixes this PR carries, as they stood before the dangling-symlink guard was added — with fix(miner): close four re-mine safety gaps in process_file #2088 intact. Resolved file-wise it does not: taking miner.py wholesale from the branch drops fix(miner): close four re-mine safety gaps in process_file #2088's work, chunk_total going from 8 occurrences to 0, and the result balloons to +188/−122. mine --mode convos: file-level already-filed tracking can silently skip individual exchanges from an already-mined transcript #2183 and fix(chroma): reset chromadb System cache in ChromaBackend._client() on inode/mtime reopen #2032 are untouched either way. So a fresh branch was the option without that footgun, not the only workable one.