Skip to content

bench(desktop): full-picture perf — prod build, cold-start, first-token #67697

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
OutThisLife merged 1 commit into from
Jul 19, 2026
Merged

bench(desktop): full-picture perf — prod build, cold-start, first-token #67697

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 18 additions & 4 deletions apps/desktop/scripts/perf/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,21 @@ npm run perf # attaches, runs the CI suite, gates on baseline
# One scenario, with a CPU profile:
npm run perf -- stream --cpuprofile --tokens 800

# Representative PRODUCTION numbers (minified React, not the ~3x-slower dev build):
npm run perf -- cold-start stream keystroke transcript --spawn --prod

# Re-capture the baseline on your reference device, then commit baseline.json:
npm run perf -- --update-baseline
npm run perf -- cold-start stream keystroke transcript --spawn --prod --update-baseline
```

## Dev vs prod

By default the harness measures the **dev** renderer (fast to spin up, good for
relative regression checks). Pass `--prod` (with `--spawn`) to build a
production renderer *with the probe included* (`VITE_PERF_PROBE=1`) and measure
minified React — the representative shipped numbers. The committed baseline is
captured with `--prod`.

## Why isolation matters

The measurement this harness exists to run was historically blocked: a running
Expand All @@ -40,13 +51,16 @@ directly via `window.__PERF_DRIVE__`, so no LLM credits are spent.
| `stream --real` | backend | same, from a real LLM stream | measure-real-stream, profile-real-stream |
| `keystroke` | ci | composer keystroke → paint latency | measure-latency, profile-typing, leak-typing |
| `transcript` | ci | large-transcript mount + paint cost | (new) |
| `cold-start` | cold | launch → CDP → driver → first paint (fresh spawn/run) | (new) |
| `first-token` | backend | Enter → first assistant token painted (TTFT) | (new) |
| `submit` | backend | Enter → cleared → user msg painted, scroll jump | measure-submit, measure-jump |
| `session-switch` | backend | route → first-paint → settle | profile-session-switch |
| `profile-switch` | backend | rail click → sidebar settled | measure-profile-switch |

`ci` scenarios need no backend/credits and are gated against `baseline.json`.
`backend` scenarios need a live backend (and `--spawn` or a real session) and
are report-only.
`ci` + `cold` scenarios need no backend/credits and are gated against
`baseline.json` (`cold-start` requires `--spawn` since it measures a fresh
launch, and must be run in its own invocation). `backend` scenarios need a live
backend (and `--spawn` or a real session/credits) and are report-only.

CPU profiling is a cross-cutting `--cpuprofile` flag on any scenario (it wraps
the run in `Profiler.start/stop` and prints a top-self-time table), replacing
Expand Down
35 changes: 23 additions & 12 deletions apps/desktop/scripts/perf/baseline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
{
"_meta": {
"note": "Median of 5 runs, darwin-arm64, `npm run perf -- --spawn` (dev renderer, gateway waited-for-connected, realistic block-settling stream chunk). Absolute values are dev-build (React dev mode is ~3x prod) so treat them as regression guards, not shipped numbers; re-baseline per device with `npm run perf -- --update-baseline`. Tolerances are loose to absorb cross-machine variance.",
"note": "Median of 5 runs, darwin-arm64, `npm run perf -- cold-start stream keystroke transcript --spawn --prod` — a PRODUCTION renderer (minified React), so these are representative shipped numbers, not dev-inflated. Re-baseline per device with the same command + `--update-baseline`. Tolerances are loose to absorb cross-machine variance; cold-start especially varies with disk/OS state.",
"platform": "darwin-arm64",
"node": "v24.11.0",
"updated": "2026-07-19T21:24:19.047Z"
"updated": "2026-07-19T21:38:19.701Z"
},
"scenarios": {
"stream": {
Expand All @@ -13,11 +13,11 @@
},
"metrics": {
"longtasks_n": 1,
"longtask_max_ms": 145,
"frame_p95_ms": 23.5,
"frame_p99_ms": 26.9,
"longtask_max_ms": 67,
"frame_p95_ms": 22,
"frame_p99_ms": 23.7,
"slow_frames_33": 1,
"intermut_p95_ms": 48.2
"intermut_p95_ms": 36.1
}
},
"keystroke": {
Expand All @@ -26,9 +26,9 @@
"tolAbs": 4
},
"metrics": {
"keystroke_p50_ms": 2,
"keystroke_p95_ms": 8.2,
"keystroke_p99_ms": 17.4,
"keystroke_p50_ms": 2.1,
"keystroke_p95_ms": 8.7,
"keystroke_p99_ms": 16.9,
"keystroke_slow_16": 2
}
},
Expand All @@ -38,9 +38,20 @@
"tolAbs": 40
},
"metrics": {
"transcript_mount_ms": 280.1,
"transcript_longtask_ms": 431,
"transcript_longtask_max_ms": 221
"transcript_mount_ms": 145,
"transcript_longtask_ms": 82,
"transcript_longtask_max_ms": 82
}
},
"cold-start": {
"tolerance": {
"tolFrac": 0.6,
"tolAbs": 150
},
"metrics": {
"spawn_to_cdp_ms": 326,
"spawn_to_driver_ms": 1647,
"fcp_ms": 500
}
}
}
Expand Down
195 changes: 125 additions & 70 deletions apps/desktop/scripts/perf/lib/launch.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,17 +105,32 @@ async function waitForConnected(cdp, timeoutMs) {
return false
}

function runNode(scriptRelPath, args = []) {
function runProcess(command, args, { env } = {}) {
return new Promise((resolveRun, reject) => {
const child = spawn(process.execPath, [join(DESKTOP_DIR, scriptRelPath), ...args], {
const child = spawn(command, args, {
cwd: DESKTOP_DIR,
stdio: 'inherit'
stdio: 'inherit',
env: env ? { ...process.env, ...env } : process.env
})
child.on('error', reject)
child.on('exit', code => (code === 0 ? resolveRun() : reject(new Error(`${scriptRelPath} exited ${code}`))))
child.on('exit', code => (code === 0 ? resolveRun() : reject(new Error(`${command} ${args[0]} exited ${code}`))))
})
}

function runNode(scriptRelPath, args = []) {
return runProcess(process.execPath, [join(DESKTOP_DIR, scriptRelPath), ...args])
}

// Build a production renderer WITH the perf probe included (VITE_PERF_PROBE=1),
// plus the prod electron-main bundle, so the harness can measure a real,
// minified React build instead of the ~3x-slower dev build. Slow (a full vite
// build); do it once, then run/attach many times.
export async function buildProdRenderer() {
const viteBin = resolveViteBin()
await runProcess(process.execPath, [viteBin, 'build'], { env: { VITE_PERF_PROBE: '1' } })
await runNode('scripts/bundle-electron-main.mjs')
}

/** Attach to a renderer already listening on `port` (launched via perf:serve or with --remote-debugging-port). */
export async function attach({ port = 9222, match } = {}) {
const cdp = await CDP.connect({ port, match })
Expand All @@ -129,9 +144,29 @@ export async function attach({ port = 9222, match } = {}) {
* and return `{ cdp, teardown, devUrl, port }`. `teardown` kills both children
* and removes any temp dirs it created.
*/
// Chromium switches that stop frame-production throttling for a window that
// isn't foregrounded (the perf window usually sits behind the IDE/terminal).
const ANTI_THROTTLE_FLAGS = [
'--disable-background-timer-throttling',
'--disable-renderer-backgrounding',
'--disable-backgrounding-occluded-windows',
'--disable-features=CalculateNativeWinOcclusion'
]

/**
* Spawn an isolated instance and connect the perf driver. Two render modes:
* · dev (default): vite dev server + dev electron-main bundle.
* · prod (`prod: true`): a production build (call buildProdRenderer first);
* electron loads dist/index.html — representative, minified React.
* `coldStart: true` skips the gateway-connect wait and settle (for launch-time
* measurement) and returns `timings` (spawn→CDP, spawn→driver) plus renderer
* boot marks (FCP, time-to-composer).
*/
export async function startIsolatedInstance({
port = 9222,
devPort = 5174,
prod = false,
coldStart = false,
hermesHome,
userDataDir,
seedConfig = true,
Expand All @@ -151,9 +186,8 @@ export async function startIsolatedInstance({

const home = hermesHome ?? mkTemp('hermes-perf-home-')
const userData = userDataDir ?? mkTemp('hermes-perf-ud-')
const devUrl = `http://127.0.0.1:${devPort}`
const devUrl = prod ? null : `http://127.0.0.1:${devPort}`

// Only seed a temp home we created — never scribble into a user-provided one.
if (seedConfig && !hermesHome) {
seedConfigFrom(join(homedir(), '.hermes'), home)
}
Expand All @@ -177,61 +211,56 @@ export async function startIsolatedInstance({
}

try {
// 1. Renderer: reuse an already-running dev server, else start one.
if (!(await reachable(devUrl))) {
const viteBin = resolveViteBin()
const vite = spawn(process.execPath, [viteBin, '--host', '127.0.0.1', '--port', String(devPort)], {
cwd: DESKTOP_DIR,
stdio: ['ignore', 'inherit', 'inherit']
})
children.push(vite)
await waitFor(() => reachable(devUrl), { timeoutMs: 60000, label: `vite dev server on :${devPort}` })
}
if (prod) {
// Renderer + main are expected pre-built (buildProdRenderer). Cheap to
// re-bundle main so an isolated run always matches current source.
await runNode('scripts/bundle-electron-main.mjs')
} else {
if (!(await reachable(devUrl))) {
const viteBin = resolveViteBin()
const vite = spawn(process.execPath, [viteBin, '--host', '127.0.0.1', '--port', String(devPort)], {
cwd: DESKTOP_DIR,
stdio: ['ignore', 'inherit', 'inherit']
})
children.push(vite)
await waitFor(() => reachable(devUrl), { timeoutMs: 60000, label: `vite dev server on :${devPort}` })
}

// 2. Electron main bundle (dev variant) — same step the dev script runs.
await runNode('scripts/bundle-electron-main.mjs', ['--dev'])
await runNode('scripts/bundle-electron-main.mjs', ['--dev'])
}

// 3. Isolated Electron. --user-data-dir gives it its own single-instance
// lock scope; HERMES_HOME gives it its own backend + sessions.
// Isolated Electron: own --user-data-dir (single-instance lock scope) + own
// HERMES_HOME (backend + sessions). No DEV_SERVER env in prod → dist load.
const electronBin = require('electron')
const env = {
...process.env,
HERMES_HOME: home,
HERMES_DESKTOP_BOOT_FAKE: '1',
HERMES_DESKTOP_BOOT_FAKE_STEP_MS: String(bootFakeStepMs),
XCURSOR_SIZE: '24'
}

if (devUrl) {
env.HERMES_DESKTOP_DEV_SERVER = devUrl
}

const spawnAt = Date.now()
const electron = spawn(
electronBin,
[
'.',
`--user-data-dir=${userData}`,
`--remote-debugging-port=${port}`,
// The perf window usually opens behind the user's other windows, and
// Chromium throttles frame production for backgrounded/occluded windows
// (~17fps), which shows up as choppy frames with ZERO longtasks and
// wrecks the stream frame-pacing metric. Disable every throttle path so
// measurements reflect real render cost regardless of window state
// (CalculateNativeWinOcclusion is the macOS/Windows occlusion detector).
'--disable-background-timer-throttling',
'--disable-renderer-backgrounding',
'--disable-backgrounding-occluded-windows',
'--disable-features=CalculateNativeWinOcclusion'
],
{
cwd: DESKTOP_DIR,
stdio: ['ignore', 'inherit', 'inherit'],
env: {
...process.env,
HERMES_HOME: home,
HERMES_DESKTOP_DEV_SERVER: devUrl,
HERMES_DESKTOP_BOOT_FAKE: '1',
HERMES_DESKTOP_BOOT_FAKE_STEP_MS: String(bootFakeStepMs),
XCURSOR_SIZE: '24'
}
}
['.', `--user-data-dir=${userData}`, `--remote-debugging-port=${port}`, ...ANTI_THROTTLE_FLAGS],
{ cwd: DESKTOP_DIR, stdio: ['ignore', 'inherit', 'inherit'], env }
)
children.push(electron)

// 4. Wait for the renderer + the perf driver to be live.
// Wait for the renderer + perf driver. In prod the target URL is file://,
// so don't match on the dev port.
let cdp = null
let cdpAt = 0
await waitFor(
async () => {
try {
cdp = await CDP.connect({ port, match: String(devPort), timeoutMs: 2000 })
cdp = await CDP.connect({ port, match: devUrl ? String(devPort) : undefined, timeoutMs: 2000 })
cdpAt = cdpAt || Date.now()

return await cdp.eval('!!(window.__PERF_DRIVE__ && window.__PERF_DRIVE__.stream)')
} catch {
Expand All @@ -245,41 +274,46 @@ export async function startIsolatedInstance({
},
{ timeoutMs: 120000, label: 'isolated renderer + __PERF_DRIVE__' }
)
const driverAt = Date.now()

// Electron throttles rAF/timers for a window that isn't foregrounded
// (per-window backgroundThrottling, which the Chromium CLI flags above don't
// override). Focus emulation makes the renderer behave as if focused so
// frame-pacing measurements are real even though the perf window sits behind
// the user's other windows — WITHOUT actually stealing OS focus.
try {
// Behave as if focused so frame-pacing isn't throttled while the perf
// window sits behind the user's IDE/terminal — WITHOUT stealing OS focus.
await cdp.send('Emulation.setFocusEmulationEnabled', { enabled: true })
} catch {
// Older CDP / not supported — fall back to the anti-throttle flags above.
// Older CDP / not supported — fall back to the anti-throttle flags.
}

// Wait for the gateway socket to actually open. A booting/absent backend
// retries on a 1–15s backoff, and that churn contaminates frame-pacing
// (the `stream` scenario). Best-effort: proceed after the timeout so the
// backend-independent scenarios (keystroke, transcript) still run.
const connected = await waitForConnected(cdp, connectTimeoutMs)

if (!connected) {
console.warn(
`[perf] gateway did not connect within ${connectTimeoutMs}ms — ` +
'stream/frame numbers may be inflated by reconnect churn.'
)
// Renderer-side boot marks (relative to its own navigation start).
const bootMarks = await readBootMarks(cdp)
const timings = {
spawn_to_cdp_ms: cdpAt ? cdpAt - spawnAt : null,
spawn_to_driver_ms: driverAt - spawnAt,
...bootMarks
}

// Let residual cold-start work (vite dep pre-bundling, initial paint) drain.
await sleep(settleMs)
let connected = true

if (!coldStart) {
// Steady-state scenarios: wait for the gateway to connect (reconnect churn
// contaminates frame pacing) and let residual cold-start work drain.
connected = await waitForConnected(cdp, connectTimeoutMs)

if (!connected) {
console.warn(
`[perf] gateway did not connect within ${connectTimeoutMs}ms — ` +
'stream/frame numbers may be inflated by reconnect churn.'
)
}

await sleep(settleMs)
}

return {
connected,
cdp,
devUrl,
port,
prod,
timings,
teardown: () => {
cdp?.close()
teardown()
Expand All @@ -291,4 +325,25 @@ export async function startIsolatedInstance({
}
}

// Read First Contentful Paint + time-to-composer from the renderer, relative to
// its navigation start (the process-spawn deltas live in `timings`).
async function readBootMarks(cdp) {
try {
return await cdp.eval(`(() => {
const paints = performance.getEntriesByType('paint')
const fcp = paints.find(p => p.name === 'first-contentful-paint')
const composer = document.querySelector('[data-slot="composer-rich-input"]')
return {
fcp_ms: fcp ? Math.round(fcp.startTime) : null,
// performance.now() at read time ≈ time since nav start; only meaningful
// right after boot (cold-start reads it immediately).
nav_to_read_ms: Math.round(performance.now()),
composer_present: !!composer
}
})()`)
} catch {
return { fcp_ms: null, nav_to_read_ms: null, composer_present: false }
}
}

export { DESKTOP_DIR }
Loading
Loading