Jetpack CLI: fast-build command (autoloader recovery, parallel, watch, seed, verify, log-path, skip list)

[enejb]

Fixes # N/A

Proposed changes

Adds a new jetpack fast-build CLI command that restores a broken local dev site by doing only the targeted rebuild work the diff actually needs — typically composer dump-autoload in a handful of plugins, completing in seconds rather than the ~2 minutes that jetpack build jetpack takes today.

The classic pain point this addresses: after git switch or rebasing on trunk, a class added by another branch isn't in the jetpack-autoloader classmap, so localhost throws a PHP fatal. The usual fix is jetpack build <plugin>, of which ~60–90s is webpack bundling that's completely irrelevant to a stale-classmap fatal.

The command combines two independent signals:

• Log scan (Signal 1): reads /var/www/html/wp-content/debug.log from the running Jetpack docker container (or a host-side path via --log-path / JETPACK_FASTBUILD_LOG), extracts Class/Interface/Trait not found fatals, resolves each FQN to its owning project (PSR-4 prefix match first, then a namespace-aware, plausibility-filtered git grep --untracked fallback — since 91 of 96 monorepo projects use classmap autoload rather than PSR-4), and runs composer dump-autoload on every plugin that transitively depends on the owner.
• Source drift (Signal 2): diffs each project against per-project stamps and buckets the diff JSON-aware (so a composer.json/package.json version-bump-only commit doesn't trigger a heavy install). PHP-only changes → dump-autoload; JS/asset source changes → project build; require/dependencies block changes → surfaced as "run jetpack build for these" instead of auto-installed.

Two-stamp model (correctness)

Each project now has two independent stamps: last-built/<slug>.build.sha (full build, JS + PHP) and last-built/<slug>.autoload.sha (autoloader regen only). dump-autoload advances the autoload stamp without claiming the JS bundle is fresh. Without this split, a dump-autoload-only action would have marked the project as fully built, hiding any stale JS bundles from future runs.

Parallelism + helpful failure output

• Parallel dump-autoload. Each plugin's vendor/composer/jetpack_autoload_classmap.php is independent, so plans consisting entirely of dump-autoload actions run through pLimit. --concurrency N (default 4). Expected wall-clock for a 13-plugin regen: ~6–10s with the default vs. ~31s serial. Mixed plans with build actions stay serial.
• Continue-on-error is the default. A single failing project doesn't abort the rest of the plan. End-of-run summary lists failures with copy-pasteable suggestions (jetpack build … to fix root cause, or --skip … to ignore on next run). --stop-on-error restores fail-fast.
• Real composer errors surface inline. A summarizeExecError() helper extracts the actual ClassMapGenerator.php line 137: Could not scan for classes … from buffered stderr, stripping composer's noisy "dump-autoload [-o|...]" usage trailer.

Flag inventory

Flag	Purpose
--seed	Stamp every project at HEAD without building — first-time bootstrap.
--dry-run	Print the plan, don't execute.
--autoload-only	Cap Signal 2 to dump-autoload, never build.
--since <ref>	Override per-project stamps; diff every project against this ref.
--log-scan / --no-log-scan	Toggle Signal 1.
--log-path <path> (or $JETPACK_FASTBUILD_LOG)	Read debug.log from a host file instead of docker; for wp-env / Playground / custom setups.
--lines N	How much of debug.log to scan (default 500).
--skip <project> (repeatable)	Pre-exclude known-bad projects.
--concurrency N	Parallel limit for dump-autoload (default 4).
--stop-on-error	Fail-fast instead of continue-on-error.
--watch / --watch-interval N	Stay running; re-scan log every N seconds and auto-fix on each new fatal.
--production / -p	Use build-production script for build actions (mirrors jetpack build).
--verify / --no-verify	After fixes, curl --url AND grep each plugin's regenerated classmap to confirm the targeted FQNs now resolve.
--url <url>	URL for the verify-step curl (default http://localhost).

Plus .jetpack-cli/skip.txt — a persistent skip list (one slug per line, # comments allowed) merged into --skip. Saves typing the same --skip plugins/mu-wpcom-plugin --skip plugins/wpcomsh flags every run.

Files of interest

• tools/cli/commands/fast-build.js — the new command (planner + parallel runner + watch loop + reporter).
• tools/cli/helpers/logScan.js — debug.log tail via docker exec or host path, missing-symbol extraction.
• tools/cli/helpers/classOwner.js — FQN → owning-project resolution. PSR-4 match → namespace-verified grep → plausibility-filtered grep (skips test fixtures / READMEs / arbitrary "class Foo" mentions).
• tools/cli/helpers/lastBuilt.js — per-project two-stamp model + JSON-aware change classification.
• tools/cli/commands/build.js — writes a build stamp on each successful (non-mirror) build.
• tools/cli/cliRouter.js — registers the new command.
• .gitignore — ignores .jetpack-cli/.
• tools/cli/tests/unit/helpers/{logScan,lastBuilt}.test.js + tools/cli/tests/unit/commands/fast-build.test.js — 182 tests passing (16 new).

No projects/* are touched, so no changelogger entry is needed. tools/cli is a private workspace package.

Commit layout

The PR is 11 commits:

1. add fast-build command — the foundation.
2. parallelize + surface composer errors inline + planner tests — done as part of the original ship.
3. split stamp into build/autoload — fixes the stale-JS-blindness bug.
4. --seed — first-time bootstrap.
5. --verify confirms targeted classes now resolve — checks plugin classmaps, not just "no new fatals".
6. hint at jetpack docker up -d — friendlier error when no container.
7. persistent skip list — .jetpack-cli/skip.txt.
8. --log-path — non-docker setups.
9. --watch — auto-fix loop.
10. --production — mirror jetpack build.
11. tighten classOwner git-grep fallback — defends against short-name collisions across packages.

Reviewable as a sequence; each commit is self-contained.

Related product discussion/links

N/A — internal developer ergonomics improvement.

Does this pull request change what data or activity we track or use?

No.

Testing instructions

Prerequisites: Jetpack docker dev environment running (jetpack docker up -d), or a writable host path you can use via --log-path.

First-time bootstrap

1. From a clean trunk: node tools/cli/bin/jetpack.js fast-build --seed. Expect: Seeded N project(s) at <sha>. … and a populated .jetpack-cli/last-built/ directory with *.build.sha and *.autoload.sha files.
2. Run node tools/cli/bin/jetpack.js fast-build --dry-run --no-log-scan immediately after. Expect: Nothing to do.

Autoloader fatal recovery + verify

1. Reproduce a fatal: add a new PHP class under projects/packages/connection/src/, reference it from the bottom of projects/packages/connection/actions.php with new \Automattic\Jetpack\Connection\My_New_Class();, then curl http://localhost. Expect HTTP 500; jp-tail shows Fatal error: Uncaught Error: Class "..." not found.
2. Run node tools/cli/bin/jetpack.js fast-build. Expect:
   • Parsed missing class + owner project printed.
   • composer dump-autoload planned for each dependent plugin.
   • Parallel execution (default concurrency 4); per-task durations.
   • Verify: N/N targeted class(es) now resolve in their plugin classmaps.
   • Verify: http://localhost → HTTP 200.
   • Wall-clock under 15s with default concurrency.
3. Revert the test edits.

Continue-on-error + composer's real diagnostic

1. On a branch where plugins/mu-wpcom-plugin's classmap references a missing vendor dir, node tools/cli/bin/jetpack.js fast-build.
2. Expect: the failing plugin shows the actual ClassMapGenerator.php line 137 error inline, the run continues through remaining plugins, and the end-of-run summary suggests jetpack build plugins/mu-wpcom-plugin or --skip plugins/mu-wpcom-plugin.

Persistent skip list

1. echo plugins/mu-wpcom-plugin > .jetpack-cli/skip.txt.
2. Run node tools/cli/bin/jetpack.js fast-build. Expect the "Honoring N entry(s) from .jetpack-cli/skip.txt: …" notice and mu-wpcom-plugin absent from the plan.

Watch mode

1. node tools/cli/bin/jetpack.js fast-build --watch. Expect "Watching debug.log every 3s. Ctrl-C to exit."
2. Trigger a fatal in another terminal (reload localhost on a stale-class branch). Expect the watcher to print the missing symbol(s) and the fix actions, then return to watching. Ctrl-C exits cleanly.

Non-docker fallback

1. node tools/cli/bin/jetpack.js fast-build --log-path /path/to/host/debug.log --dry-run. Expect "Scanned debug.log from /path/to/host/debug.log." instead of the docker variant.

Stamp split: stale-JS bug fix

1. From a --seed-stamped state, switch to a branch that adds JS source files but no PHP fatals. Run node tools/cli/bin/jetpack.js fast-build --no-log-scan. Expect Signal 2 to detect jsSources and plan [build] for the affected project (not silently skip it).
2. After a dump-autoload-only fix completes (Signal 1 fatal recovery), inspect .jetpack-cli/last-built/: only *.autoload.sha advanced for those plugins; *.build.sha is unchanged. A subsequent run will still detect JS drift.

Parallelism comparison

1. Reproduce the fatal, then time node tools/cli/bin/jetpack.js fast-build (default concurrency 4) and time node tools/cli/bin/jetpack.js fast-build --concurrency 1. Default should be ~3–4× faster.

Source-drift planning

1. node tools/cli/bin/jetpack.js fast-build --dry-run --no-log-scan --since HEAD~5 — should produce a plan with dump-autoload/build actions for projects with PHP/JS changes and a "Skipping N project(s)…" section for projects whose require/dependencies blocks changed.

Unit tests

• cd tools/cli && NODE_OPTIONS=--experimental-vm-modules pnpm test — all 182 tests should pass.

[github-actions]

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

• To test on WoA, go to the Plugins menu on a WoA dev site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin (Jetpack or WordPress.com Site Helper), and enable the add/jetpack-fast-build-command branch.
• To test on Simple, run the following command on your sandbox:

bin/jetpack-downloader test jetpack add/jetpack-fast-build-command

bin/jetpack-downloader test jetpack-mu-wpcom-plugin add/jetpack-fast-build-command

Interested in more tips and information?

• In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
• Read more about our development workflow here: PCYsg-eg0-p2
• Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ Add a "[Status]" label (In Progress, Needs Review, ...).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

Follow this PR Review Process:

1. Ensure all required checks appearing at the bottom of this PR are passing.
2. Make sure to test your changes on all platforms that it applies to. You're responsible for the quality of the code you ship.
3. You can use GitHub's Reviewers functionality to request a review.
4. When it's reviewed and merged, you will be pinged in Slack to deploy the changes to WordPress.com simple once the build is done.

If you have questions about anything, reach out in #jetpack-developers for guidance!

[enejb]

@claude please review this PR

[claude]

Claude finished @enejb's task in 3m 47s —— View job

---

PR Review

This is a well-designed and clearly motivated command. The two-signal approach (log scan + source drift), the parallelism strategy, and the error-reporting UX are all solid. A few bugs and design issues worth addressing before merging:

---

Bug: deepEqualJson only sorts top-level keys — nested objects are compared by insertion order

tools/cli/helpers/lastBuilt.js:185

const canon = v => JSON.stringify( v, Object.keys( v || {} ).sort() );

JSON.stringify with an array replacer applies that array recursively — only keys present in the array are serialized at every nesting level. So for a call like:

deepEqualJson(
  { "psr-4": { "Automattic\\Jetpack\\": "src/" } },
  { "psr-4": { "Automattic\\Jetpack\\New\\": "src/" } }
)

Both sides produce {"psr-4":{}} (the PSR-4 prefix keys are stripped because they aren't in the top-level replacer array), so the function incorrectly returns true.

In composerJsonDiff this means a change to autoload.psr-4 or autoload-dev.psr-4 mappings won't be detected as an autoload change, and dump-autoload won't be scheduled.

A reliable fix:

function deepEqualJson( a, b ) {
    const canon = v => {
        if ( typeof v !== 'object' || v === null ) return JSON.stringify( v );
        if ( Array.isArray( v ) ) return '[' + v.map( canon ).join( ',' ) + ']';
        return '{' + Object.keys( v ).sort().map( k => JSON.stringify( k ) + ':' + canon( v[ k ] ) ).join( ',' ) + '}';
    };
    return canon( a ) === canon( b );
}

Fix this →

---

Bug: packageJsonDepsChanged treats any scripts change as "needs pnpm install"

tools/cli/helpers/lastBuilt.js:258

return ! deepEqualJson( oldJson.scripts, newJson.scripts );

A scripts change (e.g., editing the dev watch command) doesn't require pnpm install. In inspectProjectChanges, a jsDeps = true result pushes the project into the heavy array (caller warns "run jetpack build"). Renaming a watch script in package.json should not block fast-build from helping.

The function name packageJsonDepsChanged also implies it only checks deps, not scripts. The scripts comparison should either be removed or only set a lighter bucket (jsSources rather than jsDeps). Fix this →

---

Design: verify() re-scans the full debug.log tail, not just new entries

tools/cli/commands/fast-build.js:372

const scan = await scanForMissingSymbols( { lines: argv.lines } );

After running the fix, the verification re-scans the last N lines of debug.log. Pre-existing fatals from before the current session will still appear as "missing symbols", causing false negatives in the verification output even when the targeted fix worked. There's no watermark ("did this specific fatal go away") — just "are there any fatals in the tail?".

Consider capturing the set of missing symbols before the fix (it's already in scan.missing) and comparing: if all of them disappeared, verification passes, even if some unrelated fatals remain.

---

Missing tests: classOwner.js has no unit tests

classOwner.js is the most complex helper — PSR-4 longest-prefix matching, a git grep fallback, and a namespace verification pass. It's also the component most likely to produce false-positive or false-negative owner lookups. The PR adds tests for logScan and lastBuilt but not for classOwner. Even mocked unit tests for the PSR-4 path and the "multiple candidates, verify namespace" path would catch regressions.

---

Minor: currentHead() called N times in parallel tasks

tools/cli/commands/fast-build.js:465

const sha = await currentHead( cwd );
await writeStamp( slug, sha, cwd );

This runs inside the pLimit-gated task, so up to concurrency tasks may call git rev-parse HEAD simultaneously. The result is identical for every task. A single const sha = await currentHead(cwd) before the tasks array construction would be cleaner and cheaper.

---

Minor: dynamic import('fs/promises') in runBuildScriptIfAny

tools/cli/commands/fast-build.js:305

const fs = await import( 'fs/promises' );

This is called once per build action. fs/promises is a core Node module and is always cached after the first import, so this isn't a performance issue, but it's inconsistent with every other file in the codebase that uses a static top-level import. Can be moved to the top of the file.

---

Minor: silent swallow of non-task errors in Listr .catch

tools/cli/commands/fast-build.js:481

await listr.run().catch( err => {
    if ( argv.stopOnError ) {
        console.error( chalk.red( err.message ) );
        process.exit( err.exitCode || 1 );
    }
} );

When stopOnError is false, non-task errors (e.g., a bug in listr itself) are silently caught and discarded. The intent is "task failures don't abort", but this makes all errors invisible. Consider re-throwing when the error didn't originate from a tracked task failure.

---

Observation: ownsRuntimeClassmap covers only plugins/

tools/cli/commands/fast-build.js:131

return slug.startsWith( 'plugins/' );

This is probably correct for the Jetpack monorepo today, but it's worth a comment explaining why extensions/ or other project types are excluded — or a TODO if this might need expanding as the monorepo grows.

---

Summary

The core logic is sound and the parallelism design is correct. The two bugs worth fixing before merge are:

1. deepEqualJson false positives (could silently skip autoloader regen when PSR-4 mappings change)
2. scripts comparison in packageJsonDepsChanged (false positive "heavy install" warnings)

The missing tests for classOwner.js and the verify() watermark issue are lower priority but would improve reliability.

[Copilot]

Pull request overview

Adds a new jetpack fast-build CLI command intended to quickly recover local dev sites from stale autoloader/classmap fatals by planning minimal rebuild work from (1) docker debug.log scans and (2) per-project “last built” stamps.

Changes:

• Introduces fast-build command with planning, optional parallel dump-autoload, verification, and failure summarization.
• Adds helpers for log scanning, FQN→project owner resolution, and per-project last-built SHA stamping + drift classification.
• Updates jetpack build to write last-built stamps and adds unit tests for the new helpers/planner.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tools/cli/commands/fast-build.js	Implements fast-build planner/runner (parallel dump-autoload, verify, reporting).
tools/cli/helpers/logScan.js	Adds docker debug.log tail + missing-symbol extraction.
tools/cli/helpers/classOwner.js	Adds PSR-4 map + git-grep fallback to resolve a missing symbol’s owning project.
tools/cli/helpers/lastBuilt.js	Adds last-built SHA stamps + JSON-aware change inspection for planning.
tools/cli/commands/build.js	Writes last-built stamps after successful non-mirror builds.
tools/cli/cliRouter.js	Registers the new fast-build command.
tools/cli/tests/unit/helpers/logScan.test.js	Unit tests for missing-symbol extraction.
tools/cli/tests/unit/helpers/lastBuilt.test.js	Unit tests for change bucketing logic.
tools/cli/tests/unit/commands/fast-build.test.js	Unit tests for planner helpers (action merge/rank, error summarization).
.gitignore	Ignores .jetpack-cli/ stamp directory.

[tbradsha]

I'm confused by the use-case here (to be honest, the PR description is quite verbose and hard to pick out the specific problem it's trying to solve). Honest question...did you read it yourself? 😄

Glancing at the testing instructions, though, it seems perhaps the purpose is less about building things and more about fixing incomplete/outdated dependency installs. Perhaps the correct fix in these scenarios is instead using jetpack install plugins/jetpack or the like?

[enejb]

Thanks for the feedback @tbradsha!

I think this approach is probably not the right one. Will close this PR for now.