Skip to content

chore(ci): trim darwin test profile - #12405

Merged
marius-kilocode merged 1 commit into
mainfrom
investigate-pipeline-bottlenecks
Jul 21, 2026
Merged

chore(ci): trim darwin test profile#12405
marius-kilocode merged 1 commit into
mainfrom
investigate-pipeline-bottlenecks

Conversation

@marius-kilocode

@marius-kilocode marius-kilocode commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Problem

On CLI-touching PRs the unit pipeline gates at 11-14 minutes, set by the single unit (macos) job. Linux (2 shards) and Windows (4 shards) usually finish in 5-8 minutes.

The darwin profile drifted far beyond macOS-specific coverage. It ran 100 files containing ~1300s of test time, dominated by platform-neutral application logic that the full Linux and Windows suites already cover. From a recent main junit artifact:

File macOS Linux Coverage
session/prompt.test.ts 213s 98s Session loop logic, no darwin API
kilocode/daemon.test.ts 93s 29s Daemon lifecycle over HTTP, no darwin API
cli/acp/lifecycle.test.ts 85s 26s ACP protocol logic
cli/smokes/read-only.test.ts 83s 24s Covered by the retained serve-process smoke
cli/run/run-process.test.ts 81s 22s Run command logic
snapshot/snapshot.test.ts 78s 26s Git snapshot logic
kilocode/server/config-overlay.test.ts 77s 28s Server config logic

Every file also runs roughly 3x slower on the 3-vCPU macos-15 runner, making macOS the most expensive place to duplicate platform-neutral coverage.

Change

Trim the darwin profile to 24 files whose failure would indicate a macOS-specific problem:

  • Seatbelt sandbox coverage (kilocode/sandbox/*, including tests that no-op on Linux)
  • node-pty and HTTP PTY coverage
  • FSEvents watching (core-watcher)
  • Process and terminal behavior (background-process, interactive-terminal, tool/shell, util/process)
  • Filesystem boundary behavior
  • One kilo serve boot smoke and the darwin install artifact

Every removed test continues to run in the full Linux and Windows suites. Existing Linux and Windows sharding is unchanged. Non-CLI package tests also remain on macOS because kilo-sandbox contains genuine seatbelt coverage.

The profile's recorded test workload drops from ~1300s to ~218s without committing generated test inventories or timing snapshots.

Full list of the 76 files removed from the darwin profile (all still run on Linux and Windows)
  • Session loops and snapshots: session/prompt.test.ts, snapshot/snapshot.test.ts, kilocode/snapshot-{cache,freeze-repro,revert-move,seed}.test.ts, kilocode/session-diff-restore.test.ts
  • Daemon and CLI runtime: kilocode/daemon.test.ts, cli/acp/lifecycle.test.ts, cli/run/run-process.test.ts, cli/smokes/read-only.test.ts, cli/tui/{plugin-lifecycle,plugin-loader-entrypoint,thread}.test.ts, kilocode/cli/cmd/serve.test.ts
  • HTTP API and server routing: server/{experimental-session-list,httpapi-experimental,httpapi-file,httpapi-listen,httpapi-reference,httpapi-workspace-routing,project-init-git,worktree-endpoint-repro}.test.ts, kilocode/server/{config-overlay,listener-runtime,tui-config,worktree-list}.test.ts, kilocode/mcp-oauth-callback.test.ts
  • Config and project: config/{config,tui}.test.ts, control-plane/workspace.test.ts, kilocode/config/config.test.ts, project/{instance,instance-bootstrap,migrate-global,project,project-directory,vcs,worktree,worktree-remove}.test.ts
  • Session export: kilocode/session-export/{e2e,sequence,worker,workspace-provider}.test.ts, kilocode/session-export/worker/{storage,zstd}.test.ts
  • Worktrees: kilocode/{primary-worktree,worktree-diff,worktree-diff-summary,worktree-family,worktree-family-submodule,worktree-project-skills,worktree-remove-lock}.test.ts
  • Tools and plugins: tool/{apply_patch,edit,glob,grep,read,recall,registry,skill,truncation,write}.test.ts, plugin/{install-concurrency,loader-shared}.test.ts, kilocode/tool/repo_clone.test.ts
  • Misc neutral: kilocode/{diff-full,indexing-worker,indexing-worktree,project-id,task-nesting}.test.ts, fixture/fixture.test.ts, git/git.test.ts, image/image.test.ts, util/{glob,module}.test.ts

Kept (24 files): kilocode/sandbox/* (11, seatbelt is macOS-only), pty/* + server/httpapi-pty.test.ts, kilocode/{background-process,core-watcher,external-directory-boundary,interactive-terminal,read-directory}.test.ts, kilocode/cli/install-artifact.test.ts, cli/serve/serve-process.test.ts, tool/shell.test.ts, filesystem/filesystem.test.ts, util/{filesystem,process,which}.test.ts.

@kilo-code-bot

kilo-code-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (2 files)
  • packages/opencode/script/kilocode/test-profile.ts
  • packages/opencode/test/kilocode/test-profile.test.ts

The current diff against main reduces to these two files: the darwin profile is reorganized into filesystem, pty, runtime, and sandbox groups per the PR description, and the test assertions (file-count threshold, contains/not-contains checks) are updated consistently with the new group contents. No behavioral inconsistencies found between the profile definition and its test coverage.

Previous Review Summaries (2 snapshots, latest commit dafb3c2)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit dafb3c2)

Status: No Issues Found | Recommendation: Merge

Files Reviewed (9 files)
  • packages/opencode/script/kilocode/test-durations.darwin.json (deleted)
  • packages/opencode/script/kilocode/test-durations.linux.json
  • packages/opencode/script/kilocode/test-durations.ts
  • packages/opencode/script/kilocode/test-durations.win32.json
  • packages/opencode/script/kilocode/test-profile.ts
  • packages/opencode/script/kilocode/test-shard.ts
  • packages/opencode/script/test-runner.ts
  • packages/opencode/test/kilocode/test-profile.test.ts
  • packages/opencode/test/kilocode/test-shard.test.ts

Incremental review: the latest commit only removes test-durations.darwin.json. TestShard.durations() already falls back from test-durations.<platform>.json to test-durations.linux.json (see test-shard.ts), so this deletion is safe and matches the documented fallback behavior.

Previous review (commit 58cf6ff)

Status: No Issues Found | Recommendation: Merge

Files Reviewed (9 files)
  • packages/opencode/script/kilocode/test-durations.darwin.json
  • packages/opencode/script/kilocode/test-durations.linux.json
  • packages/opencode/script/kilocode/test-durations.ts
  • packages/opencode/script/kilocode/test-durations.win32.json
  • packages/opencode/script/kilocode/test-profile.ts
  • packages/opencode/script/kilocode/test-shard.ts
  • packages/opencode/script/test-runner.ts
  • packages/opencode/test/kilocode/test-profile.test.ts
  • packages/opencode/test/kilocode/test-shard.test.ts

Reviewed by claude-sonnet-5 · Input: 30 · Output: 7.3K · Cached: 624.3K

Review guidance: REVIEW.md from base branch main

@marius-kilocode

marius-kilocode commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator Author

Final profile-only measurement from run 29815173247 (commit 4e1655da8c):

Job main #12294 (29731286534) main #12401 (29734453216) profile-only PR
unit (macos) 13m57s 7m20s 2m51s
unit (linux, 1/2) 5m16s 5m55s 5m23s
unit (linux, 2/2) 6m37s 5m39s 6m03s
unit (windows, 1/4) 7m58s 7m03s 7m21s
unit (windows, 2/4) 7m02s 6m51s 6m25s
unit (windows, 3/4) 5m49s 6m16s 6m04s
unit (windows, 4/4) 5m18s 5m42s 4m56s
HttpApi exerciser 4m16s 4m18s 6m01s
workflow wall ~14m38s ~7m58s ~8m22s

The macOS profile trim has a clear effect:

  • macOS job: 7m20s-13m57s before, 2m51s now.
  • macOS CLI test step: 11m13s in the slow baseline, 1m38s now (100 profile files reduced to 24).
  • Linux and Windows sharding is unchanged and their timings remain within their previous run-to-run range.

The full workflow is now gated by Windows 1/4 at 7m21s, not macOS. This clean run is 6m16s faster than the slow baseline but 24s slower than the fastest recent baseline because Windows and HttpApi were slower in this sample. One run does not support a claim of reliably sub-8-minute total wall time. The durable result is that the variable 7-14 minute macOS tail is removed; further reliable wall-time reduction must target Windows 1/4, HttpApi variance, or workflow startup/tail overhead.

A pre-existing kilocode/session-prompt-queue.test.ts flake occurred only on the discarded first revision and passed on rerun. The final profile-only run was green without retries.

@marius-kilocode marius-kilocode changed the title chore(ci): trim darwin test profile and balance shards by duration chore(ci): trim darwin test profile Jul 21, 2026
@marius-kilocode
marius-kilocode force-pushed the investigate-pipeline-bottlenecks branch from 9c60044 to 4e1655d Compare July 21, 2026 08:39
@marius-kilocode
marius-kilocode merged commit 80880dd into main Jul 21, 2026
28 checks passed
@marius-kilocode
marius-kilocode deleted the investigate-pipeline-bottlenecks branch July 21, 2026 11:29
t7tran pushed a commit to t7tran/kilocode that referenced this pull request Aug 14, 2026
…bottlenecks

chore(ci): trim darwin test profile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants