feat(skills): watchers skill — poll RSS / HTTP JSON / GitHub via cron no-agent

optional-skills/devops/watchers/SKILL.md

@@ -0,0 +1,112 @@
+---
+name: watchers
+description: Poll RSS, JSON APIs, and GitHub with watermark dedup.
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+platforms: [linux, macos]
+metadata:
+  hermes:
+    tags: [cron, polling, rss, github, http, automation, monitoring]
+    category: devops
+    requires_toolsets: [terminal]
+    related_skills: []
+---
+
+# Watchers
+
+Poll external sources on an interval and react only to new items. Three ready-made scripts plus a shared watermark helper; wire them into a cron job (or run them ad-hoc from the terminal).
+
+## When to Use
+
+- User wants to watch an RSS/Atom feed and be notified of new entries
+- User wants to watch a GitHub repo's issues / pulls / releases / commits
+- User wants to poll an arbitrary JSON endpoint and get notified on new items
+- User asks for "a watcher for X" or "notify me when X changes"
+
+## Mental model
+
+A watcher is just a script that:
+
+1. Fetches data from the external source
+2. Compares against a watermark file of previously-seen IDs
+3. Writes the new watermark back
+4. Prints new items to stdout (or nothing on no-change)
+
+The scripts below handle all three. The agent runs them via the terminal tool — from a cron job, a webhook, or an interactive chat — and reports what's new.
+
+## Ready-made scripts
+
+All three live in `$HERMES_HOME/skills/devops/watchers/scripts/` once the skill is installed. Each reads `WATCHER_STATE_DIR` (defaults to `$HERMES_HOME/watcher-state/`) for its state file, keyed by the `--name` argument.
+
+| Script | What it watches | Dedup key |
+|---|---|---|
+| `watch_rss.py` | RSS 2.0 or Atom feed URL | `<guid>` / `<id>` |
+| `watch_http_json.py` | Any JSON endpoint returning a list of objects | Configurable id field |
+| `watch_github.py` | GitHub issues / pulls / releases / commits for a repo | `id` / `sha` |
+
+All three:
+
+- First run records a baseline — never replays existing feed
+- Watermark is a bounded ID set (max 500) to cap memory
+- Output format: `## <title>\n<url>\n\n<optional body>` per item
+- Empty stdout on no-new — the caller treats that as silent
+- Non-zero exit on fetch errors
+
+## Usage
+
+Run a watcher directly from the terminal tool:
+
+```bash
+python $HERMES_HOME/skills/devops/watchers/scripts/watch_rss.py \
+  --name hn --url https://news.ycombinator.com/rss --max 5
+```
+
+Watch a GitHub repo (set `GITHUB_TOKEN` in `~/.hermes/.env` to avoid the 60 req/hr anonymous rate limit):
+
+```bash
+python $HERMES_HOME/skills/devops/watchers/scripts/watch_github.py \
+  --name hermes-issues --repo NousResearch/hermes-agent --scope issues
+```
+
+Poll an arbitrary JSON API:
+
+```bash
+python $HERMES_HOME/skills/devops/watchers/scripts/watch_http_json.py \
+  --name api --url https://api.example.com/events \
+  --id-field event_id --items-path data.events
+```
+
+## Wiring into cron
+
+Ask the agent to schedule a cron job with a prompt like:
+
+> Every 15 minutes, run `watch_rss.py --name hn --url https://news.ycombinator.com/rss`. If it prints anything, summarize the headlines and deliver them. If it prints nothing, stay silent.
+
+The agent invokes the script via the terminal tool inside the cron job's agent loop; no changes to cron's built-in `--script` flag are needed.
+
+## State files
+
+Every watcher writes `$HERMES_HOME/watcher-state/<name>.json`. Inspect:
+
+```bash
+cat $HERMES_HOME/watcher-state/hn.json
+```
+
+Force a replay (next run treated as first poll):
+
+```bash
+rm $HERMES_HOME/watcher-state/hn.json
+```
+
+## Writing your own
+
+All three scripts use the same template: load watermark, fetch, diff, save, emit. `scripts/_watermark.py` is the shared helper; import it to get atomic writes + bounded ID set + first-run baseline for free. See any of the three reference scripts for how little boilerplate it takes.
+
+## Common Pitfalls
+
+1. **Printing a "no new items" header every tick.** Callers rely on empty stdout = silent. If you print anything on an empty delta, you spam the channel. The shipped scripts handle this; custom scripts must too.
+2. **Expecting the first run to emit items.** It won't — first run records a baseline. If you need an initial digest, delete the state file after the first run or add a `--prime-with-latest N` flag in your own script.
+3. **Unbounded watermark growth.** The shared helper caps at 500 IDs. Raise it for high-churn feeds; lower it on constrained filesystems.
+4. **Putting the state dir where the agent's sandbox can't write.** `$HERMES_HOME/watcher-state/` is always writable. Docker/Modal backends may not see arbitrary host paths.
+

optional-skills/devops/watchers/scripts/_watermark.py

@@ -0,0 +1,148 @@
+"""Shared watermark helper used by the three watcher scripts.
+
+A watermark is just a JSON file that records the IDs we've seen on previous
+runs, so the next run only emits items we haven't seen before.
+
+Contract:
+- First run: record all IDs from the fetched batch, emit nothing.
+- Subsequent runs: emit items whose ID isn't in the stored set.
+- Bounded: keep at most `max_seen` IDs (default 500).
+- Atomic: write to a .tmp file and rename, so a crashed script can't
+  leave a half-written state file that permanently breaks dedup.
+
+Import and use from any custom watcher script:
+
+    from _watermark import Watermark
+
+    wm = Watermark.load("my-feed-name")
+    new_items = wm.filter_new(fetched_items, id_key="id")
+    wm.save()
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+from typing import Any, Dict, Iterable, List, Optional
+
+
+def _state_dir() -> Path:
+    """Where watermark files live — respects WATCHER_STATE_DIR override."""
+    override = os.environ.get("WATCHER_STATE_DIR")
+    if override:
+        return Path(override)
+    # Default: $HERMES_HOME/watcher-state/, falling back to ~/.hermes/watcher-state/.
+    hermes_home = os.environ.get("HERMES_HOME") or str(Path.home() / ".hermes")
+    return Path(hermes_home) / "watcher-state"
+
+
+class Watermark:
+    """Per-watcher state. Persisted to <state_dir>/<name>.json."""
+
+    def __init__(self, name: str, *, max_seen: int = 500) -> None:
+        if not name or not name.replace("-", "").replace("_", "").isalnum():
+            raise ValueError(
+                f"watermark name must be alphanumeric + '-'/'_' (got {name!r})"
+            )
+        self.name = name
+        self.max_seen = max_seen
+        self._path = _state_dir() / f"{name}.json"
+        self._data: Dict[str, Any] = {"seen_ids": [], "first_run": True}
+
+    @classmethod
+    def load(cls, name: str, *, max_seen: int = 500) -> "Watermark":
+        wm = cls(name, max_seen=max_seen)
+        if wm._path.exists():
+            try:
+                wm._data = json.loads(wm._path.read_text(encoding="utf-8"))
+                wm._data.setdefault("seen_ids", [])
+                wm._data["first_run"] = False
+            except (OSError, json.JSONDecodeError):
+                # Corrupt state file — treat as a first run but don't crash.
+                wm._data = {"seen_ids": [], "first_run": True}
+        return wm
+
+    @property
+    def is_first_run(self) -> bool:
+        return bool(self._data.get("first_run", True))
+
+    @property
+    def seen(self) -> List[str]:
+        return list(self._data.get("seen_ids", []))
+
+    def filter_new(
+        self, items: Iterable[Dict[str, Any]], *, id_key: str = "id"
+    ) -> List[Dict[str, Any]]:
+        """Return items whose id isn't in the stored set.
+
+        Side effect: updates the in-memory seen set with every id in the
+        batch (so save() persists the full new watermark).  On first run,
+        records every id but returns an empty list (baseline, no replay).
+        """
+        existing = set(str(x) for x in self._data.get("seen_ids", []))
+        was_first_run = self.is_first_run
+
+        new_items: List[Dict[str, Any]] = []
+        batch_ids: List[str] = []
+        for item in items:
+            ident = item.get(id_key)
+            if ident is None:
+                continue
+            ident_str = str(ident)
+            batch_ids.append(ident_str)
+            if ident_str in existing:
+                continue
+            if was_first_run:
+                continue  # record but don't emit
+            new_items.append(item)
+
+        combined = list(existing) + [i for i in batch_ids if i not in existing]
+        if len(combined) > self.max_seen:
+            combined = combined[-self.max_seen:]
+        self._data["seen_ids"] = combined
+        self._data["first_run"] = False
+        return new_items
+
+    def save(self) -> None:
+        self._path.parent.mkdir(parents=True, exist_ok=True)
+        tmp = self._path.with_suffix(".tmp")
+        tmp.write_text(
+            json.dumps(self._data, indent=2, sort_keys=True),
+            encoding="utf-8",
+        )
+        os.replace(tmp, self._path)
+
+
+def format_items_as_markdown(
+    items: List[Dict[str, Any]],
+    *,
+    title_key: str = "title",
+    url_key: str = "url",
+    body_key: Optional[str] = None,
+    max_body_chars: int = 500,
+) -> str:
+    """Render a list of items as Markdown for cron delivery.
+
+    One heading per item + its URL + optional snippet of body.  Output is
+    empty string when items is empty — cron will then treat stdout as
+    silent and skip delivery (existing behavior).
+    """
+    if not items:
+        return ""
+    lines: List[str] = []
+    for item in items:
+        title = (item.get(title_key) or "(no title)").strip()
+        url = (item.get(url_key) or "").strip()
+        lines.append(f"## {title}")
+        if url:
+            lines.append(url)
+        if body_key:
+            body = (item.get(body_key) or "").strip()
+            if body:
+                if len(body) > max_body_chars:
+                    body = body[:max_body_chars].rstrip() + "…"
+                lines.append("")
+                lines.append(body)
+        lines.append("")
+    return "\n".join(lines).rstrip() + "\n"