fix(skills): harden HAR-derived API output

optional-skills/web-development/har-derived-api-client/SKILL.md

@@ -15,11 +15,12 @@ metadata:
 
 Drive a website once with a real browser while recording its network traffic
 to a HAR file, then distill that HAR into the site's private JSON API so you
-can call it directly with plain HTTP — far cheaper and faster than
+can call it directly with plain HTTP — often cheaper and faster than
 browser-controlling the page on every request. Credit: trick by Jared Longster,
 popularized by Dax (thdxr). This captures and replays; it does NOT bypass
 auth, solve CAPTCHAs, or defeat bot-detection — if the site needs a logged-in
-session, you carry its headers/cookies forward, you don't forge them.
+session, provide equivalent credentials at runtime from an approved secret
+source; never paste secrets from a HAR into generated code or command history.
 
 The scripts are stdlib-plus-Playwright: capture needs Playwright, derivation
 is pure stdlib, replay needs only `requests`/`httpx` (or `curl`).
@@ -44,7 +45,9 @@ HAR recording works differently in each case (see How to Run).
   - `pip install playwright` then `playwright install chromium`
   - (If a system Playwright already has browsers under `~/.cache/ms-playwright`, reuse it.)
 - `requests` or `httpx` for the replay step (stdlib `urllib` also works).
-- No API keys. Any keys/tokens the client needs are the ones the HAR captured.
+- No API keys are needed by the skill itself. Treat captured keys, tokens, and
+  cookies as secrets; `har_to_client.py` reports their presence but redacts
+  credential header values from its output.
 - For the CDP path (`har_capture_cdp.py`): a reachable CDP endpoint. On Hermes,
   run `/browser connect` to print the active endpoint, or read `BROWSER_CDP_URL`
   / `browser.cdp_url` in config. Cloud backends expose it as `cdpUrl`/`connectUrl`.
@@ -116,7 +119,7 @@ har_to_client.py <in.har> [--host SUBSTR] [--include-static] [--max-body N]
 1. **Find the interaction.** Open the site with `browser_navigate` (or `--headed` capture) to see which selector to type into / click, and confirm a JSON XHR fires in devtools/network.
 2. **Capture the HAR** via the `terminal` tool. Order `--action` to reach the request: `fill` the box, then `sleep` long enough for the debounced XHR, and always leave `--wait` at the end so late responses flush. Both capturers embed response bodies, so the derived client sees real payload shapes.
 3. **Derive** with `har_to_client.py --host <domain>`. Read off: the method, the URL/path template (numeric/UUID segments collapse to `{id}`), query params, request-body JSON, and the `### Replay hints` block.
-4. **Write the client.** Recreate the request exactly — same method, path, query params, body. Send the headers the site actually needs: at minimum copy the **User-Agent** from the replay hints. If hints report cookies or an auth/token header, resend those too.
+4. **Write the client.** Recreate the request exactly — same scheme, method, path, query params, and body. Send the non-secret headers the site actually needs, including the **User-Agent** from the replay hints. If hints report cookies or an auth/token header, obtain an equivalent current value from an approved runtime secret source; never paste the HAR value into source code or a shell command.
 5. **Test browserless.** Run the client with the `terminal` tool and confirm it returns the same data the browser saw. This is the payoff: no browser in the loop.
 6. **(Optional) Wrap as a CLI** — a small `argparse` script over the derived call, e.g. `search.py "frank herbert"`.
 
@@ -141,7 +144,7 @@ for p in r.json()["pages"]:
 - **A failed `--action` aborts before the HAR flushes** — you get no file. If capture errors on a selector, the run produced nothing; fix the selector (use `--headed` to watch) and rerun. Don't debug a missing HAR.
 - **Server-rendered pages have no XHR** to derive — `har_to_client.py` prints "No API-looking entries". The data came in the HTML; scrape it or find the interaction that does fetch JSON.
 - **Debounced/typeahead XHRs need a real pause.** Add `--action "sleep:3"` after `fill`; typing alone won't have fired the request when the HAR closes.
-- **Auth/session endpoints** need the captured `Cookie`/`Authorization` header, and those expire. The derived client is only as durable as the credential; re-capture when it 401s. HARs contain live secrets — treat `out.har` as sensitive and delete it after deriving.
+- **Auth/session endpoints** need equivalent current `Cookie`/`Authorization` values, and those expire. The derivation output redacts credential headers; load approved values at runtime from a secret manager, protected environment, or other project-approved source. HARs contain live secrets — restrict access and delete `out.har` after deriving.
 - **`record_har_content="embed"` makes big HARs.** Use `--max-body` to cap what's printed; the file itself can be large for media-heavy pages.
 - **Endpoints shift.** Sites change private APIs without notice. Re-run the capture→derive loop when a client breaks rather than patching URLs by hand.
 - **Wrong capturer = empty/no HAR.** `har_capture.py` on a cloud/CDP backend records nothing (it launches its own local browser instead of the one you meant). `har_capture_cdp.py` needs the endpoint; on Hermes get it from `/browser connect` or `BROWSER_CDP_URL`. Match the capturer to the pathway (How to Run table).

optional-skills/web-development/har-derived-api-client/scripts/har_capture_cdp.py

@@ -23,6 +23,7 @@
 import json
 import sys
 import time
+from urllib.parse import parse_qsl, urlsplit
 
 from playwright.sync_api import sync_playwright
 
@@ -64,7 +65,12 @@ def _har_entry(req, resp):
             "method": req.method,
             "url": req.url,
             "headers": [{"name": k, "value": v} for k, v in req.headers.items()],
-            "queryString": [],  # har_to_client.py re-parses the URL, so leave empty
+            "queryString": [
+                {"name": name, "value": value}
+                for name, value in parse_qsl(
+                    urlsplit(req.url).query, keep_blank_values=True
+                )
+            ],
             "postData": {"mimeType": req.headers.get("content-type", ""),
                          "text": post} if post else {},
         },

optional-skills/web-development/har-derived-api-client/scripts/har_to_client.py

@@ -9,15 +9,15 @@
 request body sample, response content-type/status, and a response body sample.
 Numeric/UUID-ish path segments are collapsed to {id} so repeated calls group.
 Also prints "### Replay hints": the browser User-Agent plus whether cookies or
-auth/token headers were present -- send those in the derived client or you may
-get a 403/401.
+auth/token headers were present. Credential header values are redacted; supply
+current values from an approved runtime secret source when needed.
 """
 import argparse
 import json
 import re
 import sys
 from collections import OrderedDict
-from urllib.parse import urlsplit
+from urllib.parse import parse_qsl, urlsplit
 
 BORING_HEADERS = {
     "accept-encoding", "accept-language", "connection", "content-length",
@@ -28,6 +28,7 @@
 }
 ID_SEG = re.compile(r"^(\d+|[0-9a-f]{8}-[0-9a-f-]{27,}|[0-9a-f]{16,})$", re.I)
 STATIC_EXT = re.compile(r"\.(js|css|png|jpe?g|gif|svg|webp|ico|woff2?|ttf|mp4|map)$", re.I)
+SENSITIVE_HEADER_MARKERS = ("authorization", "apikey", "token", "secret", "credential")
 
 
 def path_template(path: str) -> str:
@@ -54,6 +55,19 @@ def trunc(text, n: int) -> str:
     return text if len(text) <= n else text[:n] + f"... [{len(text)} chars total]"
 
 
+def is_sensitive_header(name: str) -> bool:
+    compact = re.sub(r"[^a-z0-9]", "", name.lower().lstrip(":"))
+    return any(marker in compact for marker in SENSITIVE_HEADER_MARKERS)
+
+
+def query_items(req: dict, url) -> list[tuple[str, str]]:
+    """Return HAR query items, falling back to the request URL when omitted."""
+    items = req.get("queryString")
+    if items:
+        return [(q["name"], q.get("value", "")) for q in items]
+    return parse_qsl(url.query, keep_blank_values=True)
+
+
 def main() -> int:
     ap = argparse.ArgumentParser()
     ap.add_argument("har")
@@ -76,17 +90,19 @@ def main() -> int:
         if not args.include_static:
             if STATIC_EXT.search(url.path) or not is_api_entry(entry):
                 continue
-        key = (req["method"], url.netloc, path_template(url.path))
+        key = (req["method"], url.scheme, url.netloc, path_template(url.path))
         g = groups.setdefault(key, {"count": 0, "queries": set(), "headers": {},
                                     "req_body": None, "resp": None})
         g["count"] += 1
-        for q in req.get("queryString", []):
-            g["queries"].add((q["name"], trunc(q["value"], 80)))
+        for name, value in query_items(req, url):
+            g["queries"].add((name, trunc(value, 80)))
         for h in req.get("headers", []):
             name = h["name"].lower().lstrip(":")
             if name in BORING_HEADERS or name in ("method", "path", "scheme", "authority"):
                 continue
-            g["headers"][name] = trunc(h["value"], 120)
+            g["headers"][name] = (
+                "[REDACTED]" if is_sensitive_header(name) else trunc(h["value"], 120)
+            )
         post = req.get("postData", {})
         if post.get("text") and g["req_body"] is None:
             g["req_body"] = (post.get("mimeType", ""), trunc(post["text"], args.max_body))
@@ -111,18 +127,18 @@ def main() -> int:
                 ua = h["value"]
             if n == "cookie":
                 saw_cookie = True
-            if n in ("authorization", "x-api-key") or "token" in n:
+            if is_sensitive_header(n):
                 saw_auth = True
     print("### Replay hints")
     if ua:
         print(f"  User-Agent (send this): {ua}")
     if saw_cookie:
-        print("  Cookies present -> session may be auth-gated; capture & resend the Cookie header.")
+        print("  Cookies present -> value omitted; supply an equivalent current session from an approved runtime secret source.")
     if saw_auth:
-        print("  Authorization/token header present -> extract and resend it.")
+        print("  Authorization/token header present -> value redacted; supply it from an approved runtime secret source.")
 
-    for (method, host, path), g in groups.items():
-        print(f"\n=== {method} https://{host}{path}  (x{g['count']})")
+    for (method, scheme, host, path), g in groups.items():
+        print(f"\n=== {method} {scheme}://{host}{path}  (x{g['count']})")
         if g["queries"]:
             print("  query params:")
             for name, val in sorted(g["queries"]):

tests/skills/test_har_derived_api_client_skill.py

@@ -11,6 +11,8 @@
 import importlib.util
 import json
 import re
+import sys
+import types
 from pathlib import Path
 
 import pytest
@@ -123,8 +125,6 @@ def test_derives_endpoint_and_filters_static(tmp_path, capsys):
     har = tmp_path / "t.har"
     har.write_text(json.dumps(_make_har()), encoding="utf-8")
 
-    import sys
-
     argv = sys.argv
     try:
         sys.argv = ["har_to_client.py", str(har), "--host", "example.com"]
@@ -146,6 +146,60 @@ def test_derives_endpoint_and_filters_static(tmp_path, capsys):
     assert "User-Agent (send this): Mozilla/5.0 TestBrowser/1.0" in out
 
 
+def test_preserves_scheme_parses_url_query_and_redacts_credentials(tmp_path, capsys):
+    mod = _load_module(DERIVE, "har_to_client_security_undertest")
+    fixture = _make_har()
+    request = fixture["log"]["entries"][0]["request"]
+    request["url"] = "http://api.example.com/v1/items/12345/reviews?q=one&blank="
+    request["queryString"] = []
+    request["headers"].extend([
+        {"name": "Authorization", "value": "Bearer secret-auth-value"},
+        {"name": "X-Goog-API-Key", "value": "secret-api-key-value"},
+        {"name": "X-Custom-Token", "value": "secret-token-value"},
+    ])
+    har = tmp_path / "sensitive.har"
+    har.write_text(json.dumps(fixture), encoding="utf-8")
+
+    argv = sys.argv
+    try:
+        sys.argv = ["har_to_client.py", str(har), "--host", "example.com"]
+        rc = mod.main()
+    finally:
+        sys.argv = argv
+    out = capsys.readouterr().out
+
+    assert rc == 0
+    assert "GET http://api.example.com/v1/items/{id}/reviews" in out
+    assert "q = one" in out
+    assert "blank = " in out
+    assert out.count("[REDACTED]") == 3
+    for secret in ("secret-auth-value", "secret-api-key-value", "secret-token-value"):
+        assert secret not in out
+
+
+def test_cdp_capture_populates_query_string(monkeypatch):
+    sync_api = types.ModuleType("playwright.sync_api")
+    setattr(sync_api, "sync_playwright", object())
+    playwright = types.ModuleType("playwright")
+    monkeypatch.setitem(sys.modules, "playwright", playwright)
+    monkeypatch.setitem(sys.modules, "playwright.sync_api", sync_api)
+    mod = _load_module(CAPTURE_CDP, "har_capture_cdp_query_undertest")
+
+    class Request:
+        method = "GET"
+        url = "https://example.com/api?q=one&q=two&blank="
+        headers = {}
+        post_data = None
+        resource_type = "xhr"
+
+    entry = mod._har_entry(Request(), None)
+    assert entry["request"]["queryString"] == [
+        {"name": "q", "value": "one"},
+        {"name": "q", "value": "two"},
+        {"name": "blank", "value": ""},
+    ]
+
+
 def test_path_template_collapses_ids():
     mod = _load_module(DERIVE, "har_to_client_undertest2")
     assert mod.path_template("/v1/items/12345/x") == "/v1/items/{id}/x"