fix(skills): keep HAR capture usable after a bad action

optional-skills/web-development/har-derived-api-client/SKILL.md

@@ -65,8 +65,9 @@ Rule of thumb: **if Hermes *launched* the browser, use `har_capture.py`; if it
 *connected to* one over CDP, use `har_capture_cdp.py`.** `har_capture.py` uses
 Playwright's `record_har_path`, which only works on a locally-owned context.
 `har_capture_cdp.py` attaches with `connect_over_cdp()` and assembles the HAR
-from `page.on("request"/"response")` events, because `record_har_path` is
-unavailable on a connected browser.
+from context-level request/response events, because `record_har_path` is
+unavailable on a connected browser. `--goto` opens a **new tab** so it does
+not navigate a page Hermes is already using.
 
 Then, for either path:
 
@@ -101,6 +102,7 @@ har_capture.py <url> <out.har> [--wait S] [--headed] [--action SPEC ...]
 
 har_capture_cdp.py <cdp_url> <out.har> [--goto URL] [--wait S] [--action SPEC ...]
   same action SPEC; attaches to an existing CDP browser and does NOT close it
+  --goto opens a new tab (does not reuse Hermes's current page)
   use for cloud backends (Browserbase/Browser-Use/Firecrawl) & /browser connect
 
 har_to_client.py <in.har> [--host SUBSTR] [--include-static] [--max-body N]
@@ -138,15 +140,15 @@ for p in r.json()["pages"]:
 ## Pitfalls
 
 - **Default library User-Agent gets 403.** Many sites (Wikipedia, Cloudflare-fronted APIs) reject `python-requests/x.y`. Always send the browser UA from the replay hints. This is the #1 reason a derived client fails when the browser succeeded.
-- **A failed `--action` aborts before the HAR flushes** — you get no file. If capture errors on a selector, the run produced nothing; fix the selector (use `--headed` to watch) and rerun. Don't debug a missing HAR.
+- **A failed `--action` writes a partial HAR.** Malformed specs (`fill` without text, `click` without a selector) fail with a clear error, and everything captured before the failure is still flushed to the file. Fix the selector (use `--headed` to watch) and rerun if the file is missing the request you wanted.
 - **Server-rendered pages have no XHR** to derive — `har_to_client.py` prints "No API-looking entries". The data came in the HTML; scrape it or find the interaction that does fetch JSON.
 - **Debounced/typeahead XHRs need a real pause.** Add `--action "sleep:3"` after `fill`; typing alone won't have fired the request when the HAR closes.
 - **Auth/session endpoints** need the captured `Cookie`/`Authorization` header, and those expire. The derived client is only as durable as the credential; re-capture when it 401s. HARs contain live secrets — treat `out.har` as sensitive and delete it after deriving.
 - **`record_har_content="embed"` makes big HARs.** Use `--max-body` to cap what's printed; the file itself can be large for media-heavy pages.
 - **Endpoints shift.** Sites change private APIs without notice. Re-run the capture→derive loop when a client breaks rather than patching URLs by hand.
 - **Wrong capturer = empty/no HAR.** `har_capture.py` on a cloud/CDP backend records nothing (it launches its own local browser instead of the one you meant). `har_capture_cdp.py` needs the endpoint; on Hermes get it from `/browser connect` or `BROWSER_CDP_URL`. Match the capturer to the pathway (How to Run table).
 - **Headless-Chrome UA is a weak tell.** Local/agent-browser capture yields a `HeadlessChrome/...` User-Agent; some sites sniff the "Headless" token. Cloud backends (Browserbase/Browser-Use) send a real desktop-Chrome UA, so a client derived from a cloud capture replays more reliably. If a headless-derived client 403s where the browser didn't, swap the "Headless" UA for a normal Chrome UA string before assuming the endpoint changed.
-- **CDP capture doesn't close the browser.** `har_capture_cdp.py` attaches to a browser it doesn't own and leaves it running — correct for cloud/remote sessions Hermes manages. Don't add a close; let the owning backend tear it down.
+- **CDP capture doesn't close the browser.** `har_capture_cdp.py` attaches to a browser it doesn't own and leaves it running, which is correct for cloud/remote sessions Hermes manages. Don't add a close. Let the owning backend tear it down. `--goto` opens a new tab on purpose so it cannot wipe the tab Hermes is driving.
 
 ## Verification
 

optional-skills/web-development/har-derived-api-client/scripts/har_actions.py

@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""Shared --action parser for the HAR capture scripts.
+
+Specs:
+  fill:SELECTOR:TEXT | press:SELECTOR:KEY | click:SELECTOR
+  goto:URL | sleep:SECONDS
+"""
+from __future__ import annotations
+
+import time
+
+
+def parse_action(spec: str) -> tuple[str, list[str]]:
+    """Return (kind, args). Raise ValueError on a malformed spec."""
+    if not spec:
+        raise ValueError("empty action spec")
+    parts = spec.split(":", 2)
+    kind = parts[0]
+    if kind == "fill":
+        if len(parts) < 3 or parts[1] == "":
+            raise ValueError(f"fill needs fill:SELECTOR:TEXT, got {spec!r}")
+        return kind, [parts[1], parts[2]]
+    if kind == "press":
+        if len(parts) < 3 or parts[1] == "":
+            raise ValueError(f"press needs press:SELECTOR:KEY, got {spec!r}")
+        return kind, [parts[1], parts[2]]
+    if kind == "click":
+        if len(parts) < 2 or parts[1] == "":
+            raise ValueError(f"click needs click:SELECTOR, got {spec!r}")
+        return kind, [parts[1]]
+    if kind == "goto":
+        if len(parts) < 2 or parts[1] == "":
+            raise ValueError(f"goto needs goto:URL, got {spec!r}")
+        url = parts[1] + (":" + parts[2] if len(parts) > 2 else "")
+        return kind, [url]
+    if kind == "sleep":
+        if len(parts) != 2 or parts[1] == "":
+            raise ValueError(f"sleep needs sleep:SECONDS, got {spec!r}")
+        try:
+            seconds = float(parts[1])
+        except ValueError as exc:
+            raise ValueError(f"sleep needs a number of seconds, got {spec!r}") from exc
+        return kind, [seconds]
+    raise ValueError(f"unknown action: {spec!r}")
+
+
+def run_action(page, spec: str) -> None:
+    kind, args = parse_action(spec)
+    if kind == "fill":
+        page.fill(args[0], args[1])
+    elif kind == "press":
+        page.press(args[0], args[1])
+    elif kind == "click":
+        page.click(args[0])
+    elif kind == "goto":
+        page.goto(args[0])
+    elif kind == "sleep":
+        time.sleep(args[0])
+
+
+def choose_drive_page(context, *, new_page: bool):
+    """Pick a page to drive.
+
+    ``new_page=True`` (CDP ``--goto``) always opens a tab so we do not
+    navigate a tab Hermes is already using.
+    """
+    if new_page or not getattr(context, "pages", None):
+        return context.new_page()
+    return context.pages[-1]
+
+
+def flush_pending(pending: dict, entries: list, make_entry) -> None:
+    """Turn in-flight requests into incomplete HAR entries, then clear ``pending``.
+
+    Leftovers have no completed response. Playwright's ``request.response()``
+    blocks until one arrives (no timeout), so it must never be called here.
+    An entry with a null response is the correct partial-capture outcome.
+    """
+    for req in list(pending.values()):
+        entries.append(make_entry(req, None))
+    pending.clear()

optional-skills/web-development/har-derived-api-client/scripts/har_capture.py

@@ -9,31 +9,20 @@
 Actions run in order after page load. The HAR embeds request/response bodies
 (record_har_content='embed') so derived clients can see payload shapes.
 
-NOTE: a failing action raises before the HAR is flushed -- you get no file.
-Fix the selector (try --headed to watch) and rerun.
+If an action fails, everything captured up to that point is flushed to the
+HAR anyway. Fix the selector (try --headed to watch) and rerun.
 """
+from __future__ import annotations
+
 import argparse
 import sys
 import time
+from pathlib import Path
 
 from playwright.sync_api import sync_playwright
 
-
-def run_action(page, spec: str) -> None:
-    parts = spec.split(":", 2)
-    kind = parts[0]
-    if kind == "fill":
-        page.fill(parts[1], parts[2])
-    elif kind == "press":
-        page.press(parts[1], parts[2])
-    elif kind == "click":
-        page.click(parts[1])
-    elif kind == "goto":
-        page.goto(parts[1] + (":" + parts[2] if len(parts) > 2 else ""))
-    elif kind == "sleep":
-        time.sleep(float(parts[1]))
-    else:
-        raise ValueError(f"unknown action: {spec}")
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from har_actions import run_action  # noqa: E402
 
 
 def main() -> int:
@@ -49,21 +38,25 @@ def main() -> int:
 
     with sync_playwright() as p:
         browser = p.chromium.launch(channel="chromium", headless=not args.headed)
-        context = browser.new_context(
-            record_har_path=args.har_path,
-            record_har_content="embed",  # keep response bodies in the HAR
-        )
-        page = context.new_page()
-        page.goto(args.url, wait_until="domcontentloaded")
-        for spec in args.action:
-            run_action(page, spec)
-            try:
-                page.wait_for_load_state("networkidle", timeout=15000)
-            except Exception:
-                pass  # some pages never fully idle; the trailing --wait covers it
-        time.sleep(args.wait)
-        context.close()  # flushes the HAR
-        browser.close()
+        context = None
+        try:
+            context = browser.new_context(
+                record_har_path=args.har_path,
+                record_har_content="embed",  # keep response bodies in the HAR
+            )
+            page = context.new_page()
+            page.goto(args.url, wait_until="domcontentloaded")
+            for spec in args.action:
+                run_action(page, spec)
+                try:
+                    page.wait_for_load_state("networkidle", timeout=15000)
+                except Exception:
+                    pass  # some pages never fully idle; the trailing --wait covers it
+            time.sleep(args.wait)
+        finally:
+            if context is not None:
+                context.close()  # flushes the HAR even when an action failed
+            browser.close()
     print(f"HAR written: {args.har_path}")
     return 0
 

optional-skills/web-development/har-derived-api-client/scripts/har_capture_cdp.py

@@ -8,40 +8,32 @@
 
 Why this exists: Playwright's record_har_path only works on a context you
 launched locally. connect_over_cdp() attaches to an existing browser, so
-record_har is unavailable — we assemble the HAR from CDP Network.* events
-ourselves via page.on("request"/"response").
+record_har is unavailable, so we assemble the HAR from context
+request/response events instead.
 
 Usage:
   python3 har_capture_cdp.py <cdp_url> <output.har> [--wait S] \
       [--goto URL] [--action "fill:SEL:TEXT"] [--action "click:SEL"] ...
 
 <cdp_url> is the ws:// or http:// CDP endpoint. For Hermes: run
 `/browser connect` to see the active endpoint, or read BROWSER_CDP_URL.
+
+--goto opens a NEW tab so it does not navigate a page Hermes is already using.
+Listeners are attached to every existing context (not just pages[0]).
 """
+from __future__ import annotations
+
 import argparse
 import base64
 import json
 import sys
 import time
+from pathlib import Path
 
 from playwright.sync_api import sync_playwright
 
-
-def run_action(page, spec: str) -> None:
-    parts = spec.split(":", 2)
-    kind = parts[0]
-    if kind == "fill":
-        page.fill(parts[1], parts[2])
-    elif kind == "press":
-        page.press(parts[1], parts[2])
-    elif kind == "click":
-        page.click(parts[1])
-    elif kind == "goto":
-        page.goto(parts[1] + (":" + parts[2] if len(parts) > 2 else ""))
-    elif kind == "sleep":
-        time.sleep(float(parts[1]))
-    else:
-        raise ValueError(f"unknown action: {spec}")
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from har_actions import choose_drive_page, flush_pending, run_action  # noqa: E402
 
 
 def _har_entry(req, resp):
@@ -80,22 +72,40 @@ def _har_entry(req, resp):
     }
 
 
+def _attach_network(contexts, on_request, on_response) -> list:
+    attached = []
+    for ctx in contexts:
+        ctx.on("request", on_request)
+        ctx.on("response", on_response)
+        attached.append(ctx)
+    return attached
+
+
+def _detach_network(contexts, on_request, on_response) -> None:
+    for ctx in contexts:
+        try:
+            ctx.remove_listener("request", on_request)
+            ctx.remove_listener("response", on_response)
+        except Exception:
+            pass
+
+
 def main() -> int:
     ap = argparse.ArgumentParser()
     ap.add_argument("cdp_url")
     ap.add_argument("har_path")
-    ap.add_argument("--goto", default=None, help="URL to navigate to after attaching")
+    ap.add_argument("--goto", default=None, help="URL to open in a new tab after attaching")
     ap.add_argument("--wait", type=float, default=3.0)
     ap.add_argument("--action", action="append", default=[])
     args = ap.parse_args()
 
     entries = []
     pending = {}  # id(request) -> request
+    drive_error = None
 
     with sync_playwright() as p:
         browser = p.chromium.connect_over_cdp(args.cdp_url)
-        context = browser.contexts[0] if browser.contexts else browser.new_context()
-        page = context.pages[0] if context.pages else context.new_page()
+        contexts = list(browser.contexts) or [browser.new_context()]
 
         def on_request(req):
             pending[id(req)] = req
@@ -105,29 +115,39 @@ def on_response(resp):
             pending.pop(id(req), None)
             entries.append(_har_entry(req, resp))
 
-        page.on("request", on_request)
-        page.on("response", on_response)
-
-        if args.goto:
-            page.goto(args.goto, wait_until="domcontentloaded")
-        for spec in args.action:
-            run_action(page, spec)
-            try:
-                page.wait_for_load_state("networkidle", timeout=15000)
-            except Exception:
-                pass
-        time.sleep(args.wait)
-
-        page.remove_listener("request", on_request)
-        page.remove_listener("response", on_response)
+        attached = _attach_network(contexts, on_request, on_response)
+        try:
+            page = choose_drive_page(contexts[0], new_page=bool(args.goto))
+            if args.goto:
+                page.goto(args.goto, wait_until="domcontentloaded")
+            for spec in args.action:
+                run_action(page, spec)
+                try:
+                    page.wait_for_load_state("networkidle", timeout=15000)
+                except Exception:
+                    pass
+            time.sleep(args.wait)
+        except Exception as exc:
+            drive_error = exc
+        finally:
+            # Detach before flushing so a late response event can't append a
+            # duplicate of an entry the flush is about to write.
+            _detach_network(attached, on_request, on_response)
+            flush_pending(pending, entries, _har_entry)
         # Do NOT close: we connected to someone else's browser.
 
-    har = {"log": {"version": "1.2",
-                   "creator": {"name": "har_capture_cdp", "version": "0.1"},
-                   "entries": entries}}
+    har = {
+        "log": {
+            "version": "1.2",
+            "creator": {"name": "har_capture_cdp", "version": "0.1"},
+            "entries": entries,
+        }
+    }
     with open(args.har_path, "w", encoding="utf-8") as f:
         json.dump(har, f)
     print(f"HAR written: {args.har_path} ({len(entries)} entries)")
+    if drive_error is not None:
+        raise drive_error
     return 0