fix(ingest): never block on a non-regular file (#2221)

[mvalentsev]

What does this PR do?

Fixes #2221. A named pipe in a mined directory wedged mempalace mine forever; mine --mode convos, sweep, init, compress and split blocked the same way. All ten scenarios in the issue hang on develop under a 180 s cap and complete here.

Two shapes, neither of which asks what the file is.

Open first, check the type second. scan_project accepts notes.md on its suffix (miner.py:1683) and stat never blocks on a pipe, so process_file reaches miner.py:66:

fd = os.open(filepath, flags)          # blocks forever on a FIFO
st = os.fstat(fd)
if not stat.S_ISREG(st.st_mode) or st.st_size > MAX_FILE_SIZE:
    return None                        # correct, and unreachable

convo_miner.py:191, normalize.py:128 and repair.py:68 carry the identical dead guard.

exists() is not a type check. It is true for a pipe, and mempalace.yaml, .gitignore, entities.json and a gradle settings.gradle all reach an open behind one.

The change

O_NONBLOCK on those four opens, so the S_ISREG refusal already written there can run — the verdict is the file mode, not an errno, so a pipe with a live writer is refused too. Everywhere else a type gate: the walks drop non-regular entries with a SKIP: <name> (not a regular file) line beside the existing symlink and size lines, and fixed-name reads use is_file(). scan_project and scan_convos already stat for the size limit, so one stat() now answers both questions.

Two traps:

• The flag is not inert under a write lease — it fails EAGAIN where a blocking open waits out lease-break-time and succeeds, silently dropping a file the old code read. The retry re-checks the type first, and the kernel grants leases on regular files only (F_SETLEASE on a pipe fails EINVAL), so it is authorised by the type, not the errno.
• A gate goes where the read was, and must not eat a real error. Path.is_file() raises PermissionError on a directory without x (through 3.13), so it belongs inside whatever try the open sat in — including each project_scanner parser, which swallows its own OSError. And sweep_directory skips a non-regular type as a benign SKIP but still books a failed stat: a dangling symlink is a real error, and sweep still exits 2 on it.

Audit

Every os.open, open, read_text, read_bytes and parser entry in mempalace/ was traced to where its path comes from and probed with a real mkfifo on both commits. Nineteen sites block on develop; none block here. Two are hardening rather than live hangs, and I would rather say so than overclaim: repair's helper sits behind contains_palace_database (cli.py:1866) and repair.py:1692, hook_shell behind [ -f ] in mempal_save_hook.sh:228. detect_entities is the chokepoint for all four callers of scan_for_detection. The one site no discovery gate could reach is a write: split_file synthesises its own output names, so a pipe at one of them blocked the write with the new SKIP: line for that pipe already on screen. format_miner.extract_text is left alone — immune by ordering rather than by type, which deserves its own change; the ordering is pinned by a test here. Known limit: the stat-then-open gates stay TOCTOU-racy, unlike _read_text_no_follow, which re-checks the type on the descriptor.

How to test

mkdir -p /tmp/fifo-repro/corpus
printf 'wing: fifo-repro\nrooms:\n  - name: general\n    description: All project files\n' \
  > /tmp/fifo-repro/corpus/mempalace.yaml
python3 -c "open('/tmp/fifo-repro/corpus/real.md','w').write('# Real\n\n' + 'the quick brown fox. ' * 60)"
mkfifo /tmp/fifo-repro/corpus/notes.md

timeout 180 mempalace --palace /tmp/fifo-repro/palace mine /tmp/fifo-repro/corpus; echo "exit=$?"

develop: prints Files: 2, files real.md, then nothing — exit=124. Here: SKIP: notes.md (not a regular file), Files: 1, Drawers filed: 2, exit=0, and search afterwards returns real.md.

The exit statuses are the easiest thing here to get wrong:

mkdir -p /tmp/sweep-repro && cd /tmp/sweep-repro
printf '{"type":"user","sessionId":"s","uuid":"u","timestamp":"2026-01-01T00:00:00Z","message":{"role":"user","content":"hi"}}\n' > real.jsonl
mkfifo piped.jsonl                                    # nothing to sweep
mempalace --palace ./palace sweep .; echo "exit=$?"   # 0, with a SKIP line
rm piped.jsonl; ln -s /tmp/sweep-repro/gone.jsonl dangling.jsonl   # a real error
mempalace --palace ./palace sweep .; echo "exit=$?"   # 2, as on develop

Tests

tests/test_non_regular_file_guards.py, 42 tests, each under a SIGALRM deadline so a regression turns the file red instead of hanging pytest — the deadline raises a BaseException subclass, because except OSError guards every site here and TimeoutError is an OSError. Every production hunk was reverted on its own in an isolated copy; two are not covered by a unit test and I would rather name them than round the count up — the except OSError around detect_rooms_local in cmd_init, and the legacy mempal.yaml name in load_config. 36 of the 42 are POSIX-only (Windows has no FIFO in the filesystem namespace), and three more skip as root, matching tests/test_backups.py.

The suite's only failures here are the palace-lock family under full-suite load, and they are load artefacts rather than anything this branch does: the same commit produced one failure on a quiet machine and twelve on a busy one, and every one of them passes when the file is run on its own. CI is the arbiter. One unrelated fix rides along on a line this PR had to touch: normalize._read_transcript_file was wrapping its two refusals a second time, printing the path twice.

Checklist

• Tests pass (python -m pytest tests/ -v)
• No hardcoded paths
• Linter passes (ruff check ., and ruff format --check ., at the ruff 0.16.1 pinned in CI)

[mvalentsev]

Closing this: the #2221 work reached develop through #2228, which carried it as db29959 and 0f3f0c6.

The revision that landed predates this branch's last pass, so three guards and their tests did not come along with it:

• sweep_directory — the gate probes the file type with f.stat() inside a try, and its except OSError prints SKIP and continues. A dangling symlink named *.jsonl therefore leaves sweep reporting success on a transcript it could not read. On 9a3afc9d, the merge base of this branch, the same input was booked in failures and exited 2; on 906b918a it prints SKIP: broken.jsonl (stat error: No such file or directory) and exits 0.
• _parse_gradle_root_project_name — the is_file() gate needs to sit inside the try whose except OSError the parser already provides, and _collect_manifest_names needs to stat through os.path.isfile. Without that, a manifest under a directory with r but no x raises PermissionError out of a call that used to answer "no manifest name", so init tracebacks. (Path.is_file() propagates EACCES on 3.9 through 3.13, all checked.)
• split_file — it builds its output names itself, past the type gate in main(), so a pre-existing FIFO at one of those names still wedges write_text.

I opened #2244 with exactly that remainder — 6 files, +146/−10 on top of 906b918a, CI green — rather than rebasing this branch. On the reason, correcting myself now that I have actually run the rebase instead of reasoning about it: it conflicts in four files (miner.py, project_scanner.py, sweeper.py, the test file), and every one of those conflicts is this branch's own earlier revision against its later one — miner.py's single hunk is the ENXIO/EINVAL comment. None is a collision with someone else's work. Resolved hunk by hunk the rebase lands +144/−12 against develop, near-identical to #2244's +146/−10, with #2088 intact; #2183 and #2032 are untouched either way. Resolving file-wise is the trap — taking miner.py wholesale from this branch drops #2088's work, chunk_total going from 8 occurrences to 0. So a fresh branch avoided a footgun; it was not the only workable route.

#2221 is still open; #2244 closes it.