Skip to content

feat(bin): replace chrome-devtools-axi with a real browser-verification helper - #3244

Draft
yagakeerthikiran wants to merge 1 commit into
kunchenguid:mainfrom
yagakeerthikiran:fm/fm-browser-verify-replacement
Draft

yagakeerthikiran wants to merge 1 commit into
kunchenguid:mainfrom
yagakeerthikiran:fm/fm-browser-verify-replacement

Conversation

@yagakeerthikiran

Copy link
Copy Markdown

Summary

chrome-devtools-axi was reporting success while not doing the work: screenshot: /tmp/shot.png while writing no file, and Required at pageId for snapshot/eval right after a successful-looking open. This replaces it in firstmate's documented workflow with bin/fm-verify-page.sh, a real-browser verification helper built on playwright-core, and records two unrelated no-mistakes defects hit the same night so the next session doesn't have to rediscover them.

What bin/fm-verify-page.sh does

bin/fm-verify-page.sh <url> [--screenshot <path>] [--timeout <ms>] [--wait-until <state>] [--text-max <n>]

Returns one JSON object on success (url, final_url, status, title, text, screenshot); exits non-zero with the real error on stderr for any failure. The single hard requirement it's built around: it must never report success for something it didn't do. A screenshot path is stat'd for existence and non-zero size before it's ever mentioned in the success output. If chromium can't start, that's a loud failure, not a silent downgrade - including a no-root, apt-get-download + dpkg-deb self-repair for this sandbox's missing libnspr4/libnss3, which itself fails loudly if it can't fix things.

Failure-mode evidence (not just the happy path)

From tests/fm-verify-page.test.sh (hermetic, local HTTP server, no internet dependency) and manual runs this session:

  • Connection-refused URL → non-zero exit, fm-verify-page: page.goto: net::ERR_... on stderr, no "status": 200 anywhere in the output.
  • Browser executable missing (PLAYWRIGHT_BROWSERS_PATH pointed at an empty dir) → non-zero exit, fm-verify-page: browserType.launch: Executable doesn't exist at ... on stderr.
  • playwright-core dependency not installed at all → non-zero exit, fm-verify-page: playwright-core is not installed (install: npm install --prefix <path>) on stderr.
  • Real page (both the hermetic test server and, per the captain's own manual run tonight, live drivelog.com.au) → real 200, real rendered title, real rendered text, and a verified non-empty screenshot (386KB on the live site).

None of the failure cases above ever wrote a screenshot file or printed a success line - the exact opposite of the chrome-devtools-axi behavior this replaces.

Reference sites repointed (chrome-devtools-axi → bin/fm-verify-page.sh)

  1. AGENTS.md (session-start tool line, section 3)
  2. bin/fm-brief.sh (both the scout and ship brief scaffolds)
  3. bin/fm-bootstrap.sh (dropped from COMMON_TOOLS required-tool detection and from install_cmd)

docs/configuration.md's "universal toolchain" description was also updated for consistency with bin/fm-bootstrap.sh (one-owner rule) since it described the same list. chrome-devtools-axi itself is untouched and still usable by hand - only its place in the documented workflow is removed.

Where the two no-mistakes defects are recorded

docs/known-tool-defects.md (new), with a one-line pointer from AGENTS.md section 7 "Validate":

  1. CI-polling blindness - the pipeline's own ci step repeated warning: could not check CI: gh pr checks: exit status 1 for 14+ minutes on PR fix: recover safe fleet sync drift #111 while gh pr checks 111 / gh-axi pr checks 111 both succeeded by hand at the same moment.
  2. Custody deadlock - on branch fm/dl-whatsapp-cloud-api, a cancelled run left gate-ref and preserved-pipeline-head pointing at different commits; both no-mistakes axi sync --recover and --recover --keep-local refused identically (confirmed inert - no refs/files changed), and no-mistakes axi run then refused to start because the previous run's custody wasn't recovered - a closed loop with no documented exit at this version.

Both are external-tool defects, out of scope to repair here; the doc records exact commands/output and the workaround used for each.

Test plan

  • tests/fm-verify-page.test.sh - real page returns 200 + rendered title/text, screenshot written and non-empty, bad URL fails loudly, unstartable browser fails loudly (all pass locally)
  • bin/fm-lint.sh (shellcheck + actionlint) clean on the full branch diff
  • bin/fm-doc-audience-check.sh clean (new doc registered, owner pointer present)
  • Manual verification against a live production site (drivelog.com.au) and a non-existent host, by the captain

Draft PR, ready to merge once green - captain merges.

…on helper

chrome-devtools-axi reported success (screenshot: /tmp/shot.png) while
writing no file and returned "Required at pageId" for snapshot/eval calls
right after a successful-looking open. Add bin/fm-verify-page.sh, built on
playwright-core, which returns the final HTTP status after redirects, the
rendered title and visible text, and a screenshot whose existence and
non-zero size are checked before success is reported. It exits non-zero
with the real error on stderr for any failure, including when chromium
itself cannot start, and repairs this sandbox's missing libnspr4/libnss3
native dependencies with an unprivileged apt-get download + dpkg-deb -x
fallback rather than silently degrading.

Repoint the three tracked chrome-devtools-axi references (AGENTS.md,
bin/fm-brief.sh, bin/fm-bootstrap.sh) at the new helper and drop it from
bootstrap's required-tool detection; chrome-devtools-axi itself is
untouched and still usable by hand. Record two reproduced no-mistakes
defects (CI-polling blindness and a branch-custody recovery deadlock) in
docs/known-tool-defects.md so the next session finds them instead of
re-discovering them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant