fix(convo-miner): skip Claude Code subagent transcripts by default (#1217)

[mvalentsev]

Problem

mempalace mine --mode convos walks ~/.claude/projects/<slug>/ recursively
and picks up every .jsonl file, including those inside <session-uuid>/subagents/
where Claude Code records Explore/Plan/Grep subagent transcripts.

Reporter data on issue #1217 (49 projects, 2 months of usage):

• 57 main session files (1.1%, 288 MB) -- the high-signal content
• 4,928 subagent files (98.8%, 1.2 GB) -- short ephemeral exchanges already
  summarized in the parent session
• Mining took 2h30+ at 100% CPU on an Apple M5 Pro

Smoke-tested against a real Claude Code project directory: 21 main + 92
subagent jsonl files (5x ratio), default-skip drops 38% of mined files with
no signal loss.

Fix

scan_convos() now skips any directory named subagents by default. New
--include-subagents flag re-enables them for users who want full history.

The filter is applied inside convo_miner.scan_convos's directory
pruning (the same place SKIP_DIRS is consulted). Pruning governs a
directory walk only, so a transcript named directly -- mine <session>/subagents/agent-x.jsonl -- is still mined without
--include-subagents. That split is deliberate: the flag decides what a
walk sweeps up, while naming a path stays an explicit request.

The shared SKIP_DIRS set in palace.py is left untouched, so project
mining (miner.scan_project) still descends into user-created
subagents/ Python modules in regular code projects.

Files

• mempalace/convo_miner.py -- scan_convos(include_subagents=False)
  parameter, threaded from mine_convos()
• mempalace/cli.py -- new --include-subagents flag on mine,
  forwarded through cmd_mine to mine_convos
• tests/test_convo_miner_unit.py -- 6 new tests (default skip, opt-in,
  any-depth match, suffix-named directories left alone, case-insensitive
  match, and an explicitly named file inside subagents/)
• tests/test_cli.py -- existing test_cmd_mine_convos_mode updated
  for the new kwarg, new test pinning the flag-threading path

Verification

End-to-end smoke test against a real Claude Code project directory:

$ python3 -m mempalace --palace /tmp/a mine <claude-projects-dir> --mode convos --dry-run
  Files:   150
$ python3 -m mempalace --palace /tmp/b mine <claude-projects-dir> --mode convos --dry-run --include-subagents
  Files:   242

Difference = 92, matches find <claude-projects-dir> -path '*/subagents/*.jsonl' | wc -l.

Out of scope

• tool-results/ directories -- already covered by the existing
  .meta.json filename filter (no .jsonl files there).
• A turn-count heuristic (Option B in the issue body) -- more fragile
  and a larger surface; default-skip + opt-in flag is the cleaner UX.
• .mempalaceignore (feat: default-exclude runtime-state files + per-file drawer cap to prevent ingestion noise #587) -- orthogonal: a future ignore-file
  mechanism would let users override the default if needed, but the
  default-safe path here is additive and doesn't conflict.

[Qodo-Free-For-OSS]

Hi, cmd_mine() directly reads args.include_subagents in convos mode, which raises AttributeError for programmatic callers that construct an argparse.Namespace without that field. This is inconsistent with cmd_mine()'s existing defensive pattern (e.g., getattr(args, "redetect_origin", False)) and can break non-CLI entrypoints/tests that call cmd_mine() directly.

Severity: remediation recommended | Category: reliability

How to fix: Use getattr with default

Agent prompt to fix - you can give this to your LLM of choice:

Issue description

mempalace.cli.cmd_mine() directly accesses args.include_subagents when args.mode == "convos". If cmd_mine() is called programmatically (tests, integrations, or internal code) with an argparse.Namespace that doesn’t include include_subagents, it will raise AttributeError.

Issue Context

cmd_mine() already treats some fields as optional/feature-gated (e.g. redetect_origin) via getattr, so include_subagents should follow the same defensive pattern.

Fix Focus Areas

• mempalace/cli.py[493-515]
  • Change include_subagents=args.include_subagents to include_subagents=getattr(args, "include_subagents", False) (or ensure the attribute is always present before branching).
  • Consider aligning other new/optional CLI fields to the same pattern to keep cmd_mine() robust for direct callers.

---

Found by Qodo code review

[igorls]

Wave 1 for 3.7.0: LGTM. Focused fix with tests (or trivial deploy/manifest change). Merging into develop for the release train.