feat(skills): add read-book and book-to-notes skills

skills/note-taking/book-to-notes/SKILL.md

@@ -0,0 +1,218 @@
+---
+name: book-to-notes
+description: Read a book (PDF/EPUB) incrementally, extract key concepts chapter by chapter, and store findings into a Zettelkasten or wiki. Optimized for long texts that exceed context window limits.
+version: 1.1.0
+author: Hermes Agent
+license: MIT
+metadata:
+  hermes:
+    tags: [reading, knowledge-extraction, zettelkasten, pdf, epubs, learning]
+    category: note-taking
+    related_skills: [read-book, ocr-and-documents, obsidian, llm-wiki]
+    requires_toolsets: [terminal, files, delegate]
+---
+
+# Book-to-Notes Workflow
+
+Read a book incrementally and extract structured knowledge for your note system.
+
+## Configuration
+
+This skill writes to a configurable vault. Set your preferred path via environment variable:
+
+```bash
+# In ~/.hermes/.env
+BOOK_VAULT_PATH="~/notes"
+```
+
+If unset, defaults to `~/notes/`.
+
+Typical vault structure:
+```
+~/notes/
+├── books/              # Source PDFs/EPUBs and extracted text
+├── reading-journal/    # Chapter summaries and reading logs
+├── zettelkasten/       # Atomic notes (one idea per file)
+└── concepts/           # Wiki-style concept pages linking atomic notes
+```
+
+## Step 1: Acquire the Book
+
+**Prefer local files over remote URLs.** URLs rot. If the user provides a URL, download it immediately to a persistent location.
+
+```bash
+# Download to persistent storage (NOT /tmp/)
+curl -sL "$URL" -o ~/books/bookname.pdf
+```
+
+**Pitfall:** `/tmp/` gets cleared between sessions or on reboot. Always use `~/books/` or another persistent path.
+
+## Step 2: Extract Text
+
+For **text-based PDFs**, `pdftotext` (from poppler-utils) is often pre-installed and faster than Python libraries:
+
+```bash
+pdftotext book.pdf book.txt
+```
+
+For **scanned PDFs**, **EPUBs**, or **complex layouts**, use the `ocr-and-documents` skill (pymupdf or marker-pdf).
+
+For **EPUB** specifically:
+```bash
+# EPUBs are ZIP files with HTML chapters
+unzip -q book.epub -d book_extracted/
+# Text is usually in OEBPS/Text/ or similar
+find book_extracted/ -name "*.html" -o -name "*.xhtml" | sort
+```
+
+## Step 3: Split into Chapters
+
+**Goal:** Create manageable chunks (~2K-10K lines each) that fit in context windows.
+
+```bash
+# Split by "CHAPTER N" markers (case-insensitive)
+python3 -c "
+import re, sys
+with open('book.txt') as f:
+    text = f.read()
+# Match CHAPTER followed by number or roman numeral
+chapters = re.split(r'\n\s*(CHAPTER\s+[0-9IVX]+|Chapter\s+[0-9]+)\s*\n', text)
+# chapters[0] = front matter, chapters[1] = 'CHAPTER 1', chapters[2] = content, etc.
+for i in range(1, len(chapters), 2):
+    num = chapters[i].strip().replace(' ', '_')
+    content = chapters[i+1] if i+1 < len(chapters) else ''
+    with open(f'ch_{num}.txt', 'w') as out:
+        out.write(content)
+        print(f'Wrote {num}: {len(content)} chars')
+"
+```
+
+**Alternative:** Split by page ranges if chapter markers are unreliable:
+```bash
+# Extract pages 1-50
+pdftotext -f 1 -l 50 book.pdf ch1.txt
+```
+
+**Store splits persistently** alongside the source PDF.
+
+## Step 4: Incremental Reading
+
+Read **one chapter per tool call** (or two if they're short). For each chapter:
+
+1. **Read the chapter** with `read_file`
+2. **Extract immediately** — don't defer synthesis. Extract:
+   - Core concepts (name + definition)
+   - Models/frameworks (list components)
+   - Key arguments and evidence
+   - Notable quotes (verbatim, with page numbers if available)
+   - Connections to other chapters or ideas
+
+3. **Write a chapter summary** as you go — a 5-10 bullet summary per chapter prevents re-reading.
+
+### Quality Checklist for Subagent Extraction
+
+When delegating chapter extraction to a subagent, provide the **Reading Tool** (`templates/reading-tool.md`) as its instructions. This enforces:
+- Two-pass extraction (structural → conceptual)
+- Concept crystallization with boundaries and implications
+- Cross-chapter threading
+- Explicit failure-mode guards against parroting, compressing, and inventing
+
+Load the tool with: `skill_view(name="book-to-notes", file_path="templates/reading-tool.md")`
+
+## Step 5: Synthesize and Store
+
+After reading all chapters (or enough for the user's purpose):
+
+1. **Create atomic notes** in the Zettelkasten (one idea per note, sentence-titled)
+2. **Create a concept page** in the wiki linking to atomic notes
+3. **Tag and link** — connect new concepts to existing notes
+
+**Storage location:** Use your configured vault (default `~/notes/`):
+- Source: `~/notes/books/Book_Title.md`
+- Notes: `~/notes/zettelkasten/`
+- Wiki: `~/notes/concepts/`
+
+## Pitfalls
+
+| Pitfall | Solution |
+|---------|----------|
+| `/tmp/` cleared | Use `~/books/` or your vault's books directory |
+| URL dies | Download immediately; keep local copy |
+| Context overflow | One chapter per call; extract as you read |
+| Re-reading same content | Write chapter summaries immediately |
+| Chapter regex fails | Inspect first 100 lines of text; adjust regex |
+| PDF is scanned/image-based | Fall back to `ocr-and-documents` skill (marker-pdf) |
+| EPUB has no clear chapter files | Unzip and inspect structure; often `OEBPS/Text/ch*.html` |
+
+## Example: Full Session
+
+```bash
+# 1. Acquire
+curl -sL "$URL" -o ~/books/flow.pdf
+
+# 2. Extract
+pdftotext ~/books/flow.pdf ~/books/flow.txt
+
+# 3. Split
+python3 scripts/split_book.py ~/books/flow.txt ~/books/flow_chapters/
+
+# 4. Read & extract (repeat per chapter)
+# ... read_file calls ...
+
+# 5. Store
+# ... write_file to vault ...
+```
+
+## Agent Architecture for Autonomous Reading
+
+When building a *reusable system* for an agent to read books unattended, prefer **subagent delegation with disk-state** over cron-based autonomous loops for books under ~400 pages.
+
+### The Pattern (v0.1)
+
+```
+state/
+├── status.json      # {"book":"Title","total_chapters":N,"current":3,"phase":"extracting"}
+└── resume.md        # Laser-focus context for the next subagent run
+
+chapters/
+├── ch_001.txt
+├── ch_002.txt
+└── ...
+
+outputs/
+├── chapter_summaries/
+├── atomic_notes/
+└── wiki_drafts/
+```
+
+**Why this works:**
+- Each subagent run reads `resume.md` + 1-2 chapters, writes outputs, updates `status.json`
+- No persistent memory needed between runs — the disk is the state machine
+- The parent agent spawns a subagent per work unit, then regains control
+- Human can inspect `status.json` at any time to see progress
+
+### Architecture Decisions
+
+| Decision | Rationale |
+|----------|-----------|
+| **Reject flush tool for < 400 pages** | Context management via `resume.md` is sufficient; flush adds complexity without benefit |
+| **Defer cron jobs to v1.0** | Premature autonomy hides failures; manual subagent runs let you verify quality first |
+| **One chapter per subagent call** | Prevents context overflow; keeps each unit focused and debuggable |
+| **Resume.md as laser-focus state** | Contains only: book thesis so far, open threads, next task, nothing else |
+| **Status.json as ground truth** | Machine-readable; cheap to parse; survives across sessions |
+
+### Incremental Roadmap
+
+- **v0.1** (today): Manual subagent runs, disk state, one chapter at a time
+- **v0.2**: Add synthesis pass — subagent reads all chapter summaries and builds the concept map
+- **v0.3**: Add triage — pre-read TOC + intro to generate `questions_to_answer.md`, skip irrelevant chapters
+- **v1.0**: Cron schedule + automatic flushing only when books exceed 400 pages or context limits demand it
+
+### When to Stop Reading
+
+You don't need to read every chapter. Stop when you have:
+- The core framework/model (usually Chapters 1-4)
+- The applied techniques (usually later chapters)
+- Enough to answer the user's specific question
+
+If the user says "use what you learn to help me do X," prioritize chapters relevant to X over completeness.

skills/note-taking/book-to-notes/scripts/split_book.py

@@ -0,0 +1,104 @@
+#!/usr/bin/env python3
+"""
+Split a book text file into chapter files.
+
+Usage:
+    python split_book.py book.txt output_dir/
+    python split_book.py book.txt output_dir/ --pattern "CHAPTER\s+[0-9]+"
+    python split_book.py book.txt output_dir/ --by-pages 50  # split every N pages (if page markers exist)
+"""
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+
+def split_by_chapter(text, pattern=None):
+    """Split text by chapter markers. Returns list of (chapter_title, content) tuples."""
+    if pattern is None:
+        # Common patterns: CHAPTER 1, Chapter I, CHAPTER ONE
+        patterns = [
+            r'(?:^|\n)\s*(CHAPTER\s+[0-9IVXLC]+)\s*\n',
+            r'(?:^|\n)\s*(Chapter\s+[0-9]+)\s*\n',
+            r'(?:^|\n)\s*(PART\s+[0-9IVXLC]+)\s*\n',
+            r'(?:^|\n)\s*(BOOK\s+[0-9IVXLC]+)\s*\n',
+        ]
+        for pat in patterns:
+            matches = list(re.finditer(pat, text))
+            if len(matches) >= 2:
+                pattern = pat
+                break
+
+        if pattern is None:
+            print("No chapter markers found. Falling back to page-based split.", file=sys.stderr)
+            return None
+
+    parts = re.split(pattern, text)
+    if len(parts) < 3:
+        return None
+
+    chapters = []
+    # parts[0] = front matter
+    for i in range(1, len(parts), 2):
+        title = parts[i].strip()
+        content = parts[i + 1] if i + 1 < len(parts) else ""
+        chapters.append((title, content))
+
+    return chapters
+
+
+def split_by_pages(text, pages_per_chunk):
+    """Split by form feed characters (page breaks in pdftotext output)."""
+    pages = text.split('\f')
+    chunks = []
+    current = []
+    for i, page in enumerate(pages):
+        current.append(page)
+        if (i + 1) % pages_per_chunk == 0:
+            chunks.append((f"Pages_{i+2-pages_per_chunk}-{i+1}", "\n".join(current)))
+            current = []
+    if current:
+        start = len(pages) - len(current) + 1
+        chunks.append((f"Pages_{start}-{len(pages)}", "\n".join(current)))
+    return chunks
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Split a book into chapter files")
+    parser.add_argument("input", help="Input text file")
+    parser.add_argument("output_dir", help="Output directory for chapter files")
+    parser.add_argument("--pattern", help="Regex pattern for chapter markers")
+    parser.add_argument("--by-pages", type=int, help="Split every N pages (if page breaks exist)")
+    parser.add_argument("--prefix", default="ch", help="Filename prefix")
+    args = parser.parse_args()
+
+    in_path = Path(args.input)
+    out_dir = Path(args.output_dir)
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    text = in_path.read_text(encoding="utf-8")
+
+    if args.by_pages:
+        chapters = split_by_pages(text, args.by_pages)
+    else:
+        chapters = split_by_chapter(text, args.pattern)
+        if chapters is None:
+            chapters = split_by_pages(text, 50)
+
+    if not chapters:
+        print("Could not split book.", file=sys.stderr)
+        sys.exit(1)
+
+    for i, (title, content) in enumerate(chapters):
+        safe_title = re.sub(r'[^\w]', '_', title)[:50]
+        filename = f"{args.prefix}_{i+1:03d}_{safe_title}.txt"
+        out_path = out_dir / filename
+        out_path.write_text(content, encoding="utf-8")
+        print(f"Wrote: {filename} ({len(content)} chars)")
+
+    print(f"\nSplit into {len(chapters)} files in {out_dir}")
+
+
+if __name__ == "__main__":
+    main()

skills/note-taking/book-to-notes/templates/reading-tool.md

@@ -0,0 +1,65 @@
+# Reading Tool — Quality Checklist for Subagent Chapter Extraction
+
+Use this checklist when extracting from a book chapter. Do not return to the parent agent until every check is satisfied.
+
+## Pre-Read Scan (30 seconds)
+- [ ] Skim the chapter: headings, first paragraph, last paragraph, any diagrams or tables
+- [ ] Identify the chapter's question or thesis in one sentence
+- [ ] Note the chapter type: argument, narrative, technical how-to, or mixed
+
+## Argument Mapping
+- [ ] Extract the author's central claim (not a summary — the actual claim)
+- [ ] List the evidence or reasoning offered (bullet points)
+- [ ] Flag any unstated assumptions
+- [ ] Note where the author shifts from observation to prescription
+
+## Two-Pass Extraction
+- [ ] Pass 1 — Structural: What is the shape of the argument? (premise → evidence → conclusion)
+- [ ] Pass 2 — Conceptual: What ideas here are portable to other contexts? (abstract the principle from the example)
+- [ ] If a concept has a name the author invented, preserve that exact term
+- [ ] If a concept has no name, coin a neutral phrase and explain why you chose it
+
+## Concept Crystallization
+For each significant concept extracted:
+- [ ] Definition: What is it? (1-2 sentences, author's own words where possible)
+- [ ] Mechanism: How does it work? (the causal chain or process)
+- [ ] Boundary: When does it apply, and when does it not?
+- [ ] Example: One concrete instance from the text
+- [ ] Implication: If this is true, what follows? (your own inference, not the author's)
+
+## Cross-Chapter Threading
+- [ ] Note any references to previous chapters (what is being built upon?)
+- [ ] Flag any promises made for future chapters (what should the reader expect?)
+- [ ] If this chapter contradicts an earlier one, name the contradiction explicitly
+
+## Failure Modes to Avoid
+- **The Parrot**: Repeating the author's words without mapping the argument structure
+- **The Compressor**: Reducing a nuanced argument to a single vague sentence
+- **The Inventor**: Adding your own ideas without clearly labeling them as yours
+- **The Skipper**: Missing the chapter's actual thesis because you focused on a colorful example
+- **The Isolator**: Extracting concepts as if they exist in a vacuum, ignoring the book's larger project
+
+## Output Format
+Return your extraction as structured markdown with these sections:
+```markdown
+## Chapter [N]: [Title]
+
+### Thesis
+[One sentence central claim]
+
+### Argument Structure
+1. [Premise/evidence]
+2. [Reasoning step]
+3. [Conclusion]
+
+### Key Concepts
+- **[Concept Name]**: [definition] — [mechanism] — [boundary] — [example] — [implication]
+
+### Threads
+- Builds on: [earlier chapter/idea]
+- Promises: [future chapter/idea]
+- Contradicts: [if applicable]
+
+### Unclear / Suspicious
+- [Anything that doesn't make sense or seems weak]
+```