upstream(sync): Tier 1 perf/CPU — mounted-view, Rust hot paths, relay traffic

[devin-ai-integration]

Summary

Tier 1 performance/CPU sync from upstream Buzz onto the Crew 0.5.11 pin. Five upstream PRs, ported in scope order, each as its own commit:

Upstream PR	What lands here
block#5879	Cuts steady-state relay traffic: no-op suppression on split-slot publishes, self-echo dropped before decrypt, published-id cache capped, read-state identity extracted so read-state echo stops re-publishing unchanged state
block#5897	Observer publications now carry { agentPubkey, events } — only the newly admitted, retained events for the one agent that changed — and the active-turn bridge processes that delta instead of rescanning the whole fleet's retained journals
block#6198	Mounted-view CPU burn: shimmer animation gated, observer append fast path for in-order batches (no whole-journal Set + re-sort per envelope), low-water eviction floor so cap trims amortize, poll-tick disk reads removed
block#6330	discover_acp_providers split into a cheap cached path and an explicit forced path; hot surfaces (channel switch, composer, member bar) render from cache with a 30-minute staleTime, forced discovery is single-flighted and only runs on Settings/onboarding/post-mutation
block#6024	Five hot renderer paths moved to Rust: archive sync, relay client frame handling (native WebSocket batching), observed-unread projection + catch-up, and the persona catalog. archiveSyncManager.ts (345 lines) and the TS persona-catalog parse (242 lines) are deleted in favour of the native commands

Net: ~9.9k lines added, ~2.6k removed, mostly upstream Rust modules plus their test suites.

Prerequisites pulled in deliberately (not in #279's list)

Two upstream changes had to come along because the scoped PRs are built directly on top of them. Both are verbatim upstream code, both are in the desktop-v0.5.18 target tag, and neither is itemised in a sibling child issue:

• definition_validation.rs from upstream Harden shared agent instruction review block/buzz#4220 (a20e40301). Upstream's new persona_catalog.rs imports managed_agents::validate_agent_definition_text, which does not exist at the Crew 0.5.11 baseline. The module is copied verbatim from bbd20fae7^ and wired with only the mod + pub(crate) use lines Harden shared agent instruction review block/buzz#4220 added. None of Harden shared agent instruction review block/buzz#4220's validation boundaries (create/update/inbound paths, agent events, snapshot) are ported — that hardening is a separate PR's scope. Since this is security-sensitive code, nothing here was hand-written or re-derived.
• Observer low-water eviction floor from upstream fix(desktop): amortize observer journal eviction with a low-water mark block/buzz#5808, which perf(desktop): update active turns incrementally block/buzz#5897/fix(desktop): eliminate mounted-view CPU burn — compositor-safe shimmer, observer append fast path, poll-tick disk reads block/buzz#6198 rewrite appendAgentEvents on top of. Perf-only, same file, same function.

Crew divergences (Crew behaviour kept, upstream hunk adapted)

• upstream(sync): Guardrail — preserve channel-first IA during 0.5.18 merge #278 channel-first IA is intact — upstream's Projects/Workbench navigation hunk in AppShell.tsx is not applicable and was not taken: no onSelectProjects, no onSelectWorkbench, no restored org view. AppShell changes are additive (useArchiveAgentMetricsBridge, the archive-sync hook move, topLevelUnreadChannelIds).
• feat(desktop): debounced exact local snapshot refresh on app focus #164 / Worktree storage reclaim: aggregate view, 48h/merged-PR idle policy, suggest-and-confirm bulk sweep (completes #59 P3) #174 lifecycle hooks kept beside upstream's foreground scheduler in useAppShellLifecycleEffects.ts.
• Crew's Tauri command-table architecture kept: new commands are added additively to desktop_invoke_handler!(), new state additively to .manage(...); upstream's inline handler list is not restored.
• Relay transport: Crew keeps its transport authority, verification, reconnect/recovery state and subscription-status model; only native delivery-frame unwrapping (toRelayFrames) is adopted.
• fix(desktop): eliminate mounted-view CPU burn — compositor-safe shimmer, observer append fast path, poll-tick disk reads block/buzz#6198: Crew's managed-agent update command stays consolidated in commands/agent_models.rs instead of upstream's agent_models_update.rs; observer dedup keeps observerEventIdentity.
• perf(desktop): split discover_acp_providers into cheap and forced paths block/buzz#6330: Crew's split discovery/install modules and managed-node cache invalidation are kept; upstream's forced single-flight path and auth-status cache are added.
• perf(desktop): update active turns incrementally block/buzz#5897: the bridge hook stays in activeAgentTurnsStore because Crew owns the activeAgentTurnsHooks.ts split.
• E2E bridge: additive only — Crew fixtures and hooks untouched, four native command mocks added.
• Deep links / terminal override: Crew's existing AppShellChannelSurface and deep-link APIs stay; upstream's newer modules for those are outside this port.

desktop/scripts/file-size-baselines.json records the upstream-owned growth in observerRelayStore.ts (1031), activeAgentTurnsStore.ts (1059) and relayClientSession.ts (1070).

Related issue

Fixes #279. Part of the 0.5.18 sync stack tracked by #285; guardrail #278.

Testing

TDD per PR — tests applied first (RED), then implementation (GREEN). Full untruncated logs are attached to the session; counts:

PR	RED	GREEN
block#5879	4 failing	30 passed
block#5897	import of createActiveAgentTurnsObserverListener fails	91 passed
block#6198	failing retention/shimmer suites	passed
block#6330	21 failing	passed
block#6024	26 tests / 21 failures	33 passed

Gates (Hermit toolchain, just _ensure-sidecar-stubs before Rust):

Gate	Result
pnpm --filter buzz check	pass
pnpm --filter buzz typecheck	pass
pnpm --filter buzz test	pass — 5546 passed, 1 skipped
just check-compile	pass
cd desktop/src-tauri && cargo check --all-targets	pass
just desktop-tauri-clippy	pass
just desktop-tauri-test	pass — 2779 passed, 15 ignored
just test-unit	fails only on pre-existing buzz-cli (agents call subcommand count — Crew's call-by-name) and buzz-db (embedded_migrator 31 vs 30) assertions; neither crate is touched by this branch and both fail on main

Desktop E2E under xvfb: unread/inbox/badge suites relevant to the block#6024 rewrite — 54 passed (badge, unread-pill, sidebar-more-unread-overlap, thread-unread, inbox-live-update, inbox-edit, inbox-refactor-screenshots, mission-inbox), plus observer-archive-policy.spec.ts 6 passed. inbox-reactions and project-inbox fail on main too (project-inbox expects the removed Repositories IA, per #223/#219); dm-double-notification needs a live relay on :3000.

Quantitative delta (block#5897) — same-revision A/B of the two bridge publication paths, fleet of 20 running agents × 500 retained observer events, 200 steady-state single-event publications:

OLD full-fleet replay (pre-#5897): 1115.4 ms total (5.577 ms per publication)
NEW incremental delta (#5897):        6.0 ms total (0.030 ms per publication)
speedup: 186.6x less time in the bridge

UI evidence — full-window, before (pin 0.5.11) vs after (this branch)

The perf port is not meant to change the UI; these prove the app shell and channel-first IA render identically, including sidebar unread emphasis now sourced from the native observed-unread projection.

App shell (Inbox + channel + thread panel):

Before	After

Agents settings:

Before	After

Harness catalog (the surface block#6330's forced-discovery gate changes) — opening and ready state:

Before	After

Runtime verification — headed drive through the E2E mock bridge

Driven headed through the mock bridge with the Tauri IPC wrapped to count every native command call, focusing on the five hot paths moved to Rust (unread being the highest-risk one).

Live emits into a non-active channel bold the row and drive the badge; opening the channel clears both:

unread live	cleared after open

Adversarial isolation — a reply to an already-read root must not register as top-level unread, and the next top-level message must:

reply-only: row not bold	next top-level: row bold, badge 1

Thread unread stays separate (1 reply (1 new)), the messaging golden path works (sent reply renders, summary 1 → 2 replies), and forced ACP discovery resolves with no endless shimmer:

thread summary	sent reply	runtimes	harness catalog

No errors from start_archive_sync, observed_unread_open_scope, observed_unread_ingest, unread_catch_up or fetch_persona_catalog; only pre-existing benign mock gaps (is_shared_identity, audio-device commands, fetch_workspace_icon, discover_git_bash_prerequisite, mock-feed-mention).

Not proven / follow-ups

• Unread survival across app restart is untested. The bridge calls resetMockObservedUnread() on install and native mode no longer mirrors acknowledged state to localStorage, so the mock cannot model Rust-side persistence. Worth one manual check on a real Tauri build before merge.
• stop_archive_sync never fired in a session (start only) — the teardown side of the archive-sync port is unexercised at runtime; it appears to be unmount/sign-out only.
• The persona-catalog list rendered empty because the bridge seeds no shared personas; fetch_persona_catalog itself succeeded (covered by agents.spec.ts).
• observed_unread_ingest fires very frequently (~305 calls by boot-end). Behaviour is correct, but the per-call IPC volume is worth a glance given this is a perf PR.

Follow-up commit — Tauri test-target compile break

just check-compile runs cargo check --workspace --all-targets from the repo root, and desktop/src-tauri is not a member of that workspace, so the Tauri crate's test targets were never compiled by the documented gate and the port shipped four defects that only CI's Desktop Rust job saw. Fixed in a follow-up commit, with cargo check --all-targets + clippy + the full Tauri test suite now run in-crate:

• Restored upstream's one-line observed_unread::flush(app) in shut_down_app (WAL checkpoint on exit) that the port dropped — Crew's quit_governor line kept.
• Reverted ptt_shortcut.rs to main: the port had added a duplicate copy of Crew's inline PTT global-shortcut handler as ptt_shortcut::install, out of scope and never called.
• Narrowly #[allow(dead_code)] on validate_managed_agent_definition_text and trimmed the re-export: it is part of the verbatim upstream module, and its callers are Harden shared agent instruction review block/buzz#4220 validation-boundary changes deliberately excluded here.
• Adapted ported test fixtures to Crew's shapes rather than the reverse: ManagedAgentRecord (hermes_profile, not upstream's provider_policy_pending/effort_level), Crew's 5-argument prospective_spawn_config_snapshot, and Crew's platform-scoped managed-node path. No production types or signatures were changed to satisfy a test.

---

DONE

All five upstream changes in #279's scope are ported, on devin/issue-279-tier1-perf @ 4efdbd15350e4d92777fcfeb5abb324b13f3382c (2 commits, both -s signed):

• fix(desktop): eliminate mounted-view CPU burn — compositor-safe shimmer, observer append fast path, poll-tick disk reads block/buzz#6198 mounted-view CPU — shimmer only animates when visible, observer fast-path append with amortized low-water eviction (MAX_OBSERVER_EVENTS 3000 / 90% low water, per-agent eviction floor), no poll-tick disk reads.
• perf(desktop): move five hot renderer paths from JS into Rust block/buzz#6024 five hot renderer paths moved to Rust — native archive sync, native relay client + WebSocket frame batching, native observed-unread projection, unread catch-up, persona catalog. Wired additively into Crew's command-table macro (invoke.rs) and lib.rs managed state; the TS archive-sync manager and persona parse path were deleted in favour of the native ones.
• fix(desktop): cut steady-state relay traffic from polls and read-state echo block/buzz#5879 steady-state relay traffic — multi-frame unwrap in relayClientSession.ts, no read-state echo when unchanged (readStateIdentity.ts).
• perf(desktop): update active turns incrementally block/buzz#5897 incremental active-turn updates — typed { agentPubkey, events } observer updates, createActiveAgentTurnsObserverListener + useActiveAgentTurnsBridge hydrate once then apply deltas.
• perf(desktop): split discover_acp_providers into cheap and forced paths block/buzz#6330 cheap vs forced ACP discovery — cached cheap query (staleTime 30 min) + explicit refreshAcpRuntimes, Rust-side forced single-flight and auth-status cache.

TDD, per upstream PR (RED → GREEN, full logs): 5879-RED/GREEN.log, 5897-RED/GREEN.log, 6198-RED/GREEN.log, 6330-RED/GREEN.log, 6024-RED/GREEN.log, 6024-cargo-check.log (all under ~/evidence-279/).

Gates (Hermit toolchain; just _ensure-sidecar-stubs before Rust):

Gate	Result	Log
pnpm --filter buzz check	pass	gate-check.log
pnpm --filter buzz typecheck	pass	gate-typecheck.log
pnpm --filter buzz test	pass — 5546 passed, 1 skipped	gate-test.log
just check-compile	pass	gate-check-compile.log
cd desktop/src-tauri && cargo check --all-targets	pass	fix-desktop-rust-check.log
just desktop-tauri-clippy	pass	fix-desktop-rust-clippy.log
just desktop-tauri-test	pass — 2779 passed, 15 ignored	fix-desktop-rust-test.log
just test-unit	fails only on pre-existing buzz-cli / buzz-db assertions that also fail on main	gate-test-unit.log, base-test-unit.log

Evidence (all under ~/evidence-279/ unless noted):

• Full-window before/after app shell, Agents settings, harness catalog: shots/before-01..04-*.png, shots/after-01..04-*.png (embedded above).
• Headed runtime drive of the native paths (unread live/clear, thread-vs-top-level isolation, messaging golden path, forced discovery, native command census): screenshots ~/screenshots/ss_*.png, recording ~/screencasts/tier1-perf/tier1-perf-edited.mp4.
• E2E: e2e-unread.log, e2e-observer-archive-policy.log, e2e-after*.log, plus base-branch comparisons base-e2e-inbox-reactions.log, base-e2e-project-inbox.log.
• Quantitative perf(desktop): update active turns incrementally block/buzz#5897 delta: measure-5897.log, measure-5897.mjs — 1115.4 ms / 5.577 ms per publication (full replay) → 6.0 ms / 0.030 ms (incremental), 186.6x less bridge time at 20 agents × 500 events × 200 publications.
• Divergence register: divergences.md. CI excerpts: pr312-desktop-e2e-excerpts.log.

IN PROGRESS

Nothing is half-done. The worktree is clean at 4efdbd15, everything is pushed, and no local edit is uncommitted. One unrelated change (an update to .agents/skills/desktop-e2e-manual-drive/SKILL.md documenting how to drive unread / native-command flows) was deliberately kept out of this PR and sent as a separate skill-PR suggestion, so it is not part of this diff.

TODO / NEXT

Remaining verification, none of it blocking the diff:

1. Unread persistence across a real app restart is unproven. The E2E bridge calls resetMockObservedUnread() on install and native mode no longer mirrors acknowledged state into localStorage, so the mock cannot model the Rust store across relaunch. Worth one manual check on a real Tauri build: mark channels read, quit, relaunch, confirm they stay read.
2. stop_archive_sync was never exercised at runtime — only start_archive_sync fired in a session. Confirm the teardown path is genuinely unmount/sign-out only.
3. observed_unread_ingest fires very frequently (~305 calls by boot-end). Behaviour is correct, but the IPC call volume deserves a look given this is a perf PR.
4. Persona catalog list rendered empty under the mock because the bridge seeds no shared personas; the command itself succeeded (covered by agents.spec.ts).

Known risks and expected conflicts:

• just check-compile does not cover desktop/src-tauri (it is not a member of the root workspace), so Tauri test-target breaks are invisible to the documented gate — that is how the four defects fixed in 4efdbd15 reached CI. A tooling fix belongs in its own PR; I did not touch tooling or CI here.
• perf(desktop): move five hot renderer paths from JS into Rust block/buzz#6024 needed a bounded prerequisite: persona_catalog.rs requires validate_agent_definition_text, absent from the desktop-v0.5.11 pin, so upstream Harden shared agent instruction review block/buzz#4220's definition_validation.rs is included verbatim with only mod + one re-export of wiring. None of Harden shared agent instruction review block/buzz#4220's validation boundaries are ported, which is why validate_managed_agent_definition_text carries a narrow #[allow(dead_code)] — the allow should be removed when Harden shared agent instruction review block/buzz#4220 lands.
• Conflicts expected in the sync stack's shared files: desktop/src-tauri/src/invoke.rs and lib.rs (additive command/state registration), desktop/src/app/AppShell.tsx, desktop/src/testing/e2eBridge.ts, and desktop/scripts/file-size-baselines.json (ratcheted for activeAgentTurnsStore.ts 1059, observerRelayStore.ts 1031, relayClientSession.ts 1070). All are additive hunks; resolve by keeping both sides, never wholesale "ours".
• Crew behaviour kept over upstream throughout (channel-first IA, no Projects/Workbench, one observer notification per envelope, workspace snapshot refresh, alive heartbeats, managed-agent env injection) — see the Crew divergences section above and divergences.md.

CI status (not fixed, per instruction): the latest run shows NuncioCrew Gate failing solely because desktop-fast, desktop-rust and macos-arm were cancelled — the gate script reports desktop-fast must be success when desktop=true, got cancelled. That is the parent session's deliberate CI serialization, not a failure of this change; every non-cancelled leg that ran passed. The earlier genuine Desktop Rust compile failure was real and is fixed in 4efdbd15.

Link to Devin session: https://app.devin.ai/sessions/5e6d0da9f17741d79cc58e401375f66a
Requested by: @oscarlehuu

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

DONE

All five upstream changes in #279's scope are ported, on devin/issue-279-tier1-perf @ 4efdbd15350e4d92777fcfeb5abb324b13f3382c (2 commits, both -s signed):

• fix(desktop): eliminate mounted-view CPU burn — compositor-safe shimmer, observer append fast path, poll-tick disk reads block/buzz#6198 mounted-view CPU — shimmer only animates when visible, observer fast-path append with amortized low-water eviction (MAX_OBSERVER_EVENTS 3000 / 90% low water, per-agent eviction floor), no poll-tick disk reads.
• perf(desktop): move five hot renderer paths from JS into Rust block/buzz#6024 five hot renderer paths moved to Rust — native archive sync, native relay client + WebSocket frame batching, native observed-unread projection, unread catch-up, persona catalog. Wired additively into Crew's command-table macro (invoke.rs) and lib.rs managed state; the TS archive-sync manager and persona parse path were deleted in favour of the native ones.
• fix(desktop): cut steady-state relay traffic from polls and read-state echo block/buzz#5879 steady-state relay traffic — multi-frame unwrap in relayClientSession.ts, no read-state echo when unchanged (readStateIdentity.ts).
• perf(desktop): update active turns incrementally block/buzz#5897 incremental active-turn updates — typed { agentPubkey, events } observer updates, createActiveAgentTurnsObserverListener + useActiveAgentTurnsBridge hydrate once then apply deltas.
• perf(desktop): split discover_acp_providers into cheap and forced paths block/buzz#6330 cheap vs forced ACP discovery — cached cheap query (staleTime 30 min) + explicit refreshAcpRuntimes, Rust-side forced single-flight and auth-status cache.

TDD, per upstream PR (RED → GREEN, full logs): 5879-RED/GREEN.log, 5897-RED/GREEN.log, 6198-RED/GREEN.log, 6330-RED/GREEN.log, 6024-RED/GREEN.log, 6024-cargo-check.log (all under ~/evidence-279/).

Gates (Hermit toolchain; just _ensure-sidecar-stubs before Rust):

Gate	Result	Log
pnpm --filter buzz check	pass	gate-check.log
pnpm --filter buzz typecheck	pass	gate-typecheck.log
pnpm --filter buzz test	pass — 5546 passed, 1 skipped	gate-test.log
just check-compile	pass	gate-check-compile.log
cd desktop/src-tauri && cargo check --all-targets	pass	fix-desktop-rust-check.log
just desktop-tauri-clippy	pass	fix-desktop-rust-clippy.log
just desktop-tauri-test	pass — 2779 passed, 15 ignored	fix-desktop-rust-test.log
just test-unit	fails only on pre-existing buzz-cli / buzz-db assertions that also fail on main	gate-test-unit.log, base-test-unit.log

Evidence (all under ~/evidence-279/ unless noted):

• Full-window before/after app shell, Agents settings, harness catalog: shots/before-01..04-*.png, shots/after-01..04-*.png (embedded above).
• Headed runtime drive of the native paths (unread live/clear, thread-vs-top-level isolation, messaging golden path, forced discovery, native command census): screenshots ~/screenshots/ss_*.png, recording ~/screencasts/tier1-perf/tier1-perf-edited.mp4.
• E2E: e2e-unread.log, e2e-observer-archive-policy.log, e2e-after*.log, plus base-branch comparisons base-e2e-inbox-reactions.log, base-e2e-project-inbox.log.
• Quantitative perf(desktop): update active turns incrementally block/buzz#5897 delta: measure-5897.log, measure-5897.mjs — 1115.4 ms / 5.577 ms per publication (full replay) → 6.0 ms / 0.030 ms (incremental), 186.6x less bridge time at 20 agents × 500 events × 200 publications.
• Divergence register: divergences.md. CI excerpts: pr312-desktop-e2e-excerpts.log.

IN PROGRESS

Nothing is half-done. The worktree is clean at 4efdbd15, everything is pushed, and no local edit is uncommitted. One unrelated change (an update to .agents/skills/desktop-e2e-manual-drive/SKILL.md documenting how to drive unread / native-command flows) was deliberately kept out of this PR and sent as a separate skill-PR suggestion, so it is not part of this diff.

TODO / NEXT

Remaining verification, none of it blocking the diff:

1. Unread persistence across a real app restart is unproven. The E2E bridge calls resetMockObservedUnread() on install and native mode no longer mirrors acknowledged state into localStorage, so the mock cannot model the Rust store across relaunch. Worth one manual check on a real Tauri build: mark channels read, quit, relaunch, confirm they stay read.
2. stop_archive_sync was never exercised at runtime — only start_archive_sync fired in a session. Confirm the teardown path is genuinely unmount/sign-out only.
3. observed_unread_ingest fires very frequently (~305 calls by boot-end). Behaviour is correct, but the IPC call volume deserves a look given this is a perf PR.
4. Persona catalog list rendered empty under the mock because the bridge seeds no shared personas; the command itself succeeded (covered by agents.spec.ts).

Known risks and expected conflicts:

• just check-compile does not cover desktop/src-tauri (it is not a member of the root workspace), so Tauri test-target breaks are invisible to the documented gate — that is how the four defects fixed in 4efdbd15 reached CI. A tooling fix belongs in its own PR; I did not touch tooling or CI here.
• perf(desktop): move five hot renderer paths from JS into Rust block/buzz#6024 needed a bounded prerequisite: persona_catalog.rs requires validate_agent_definition_text, absent from the desktop-v0.5.11 pin, so upstream Harden shared agent instruction review block/buzz#4220's definition_validation.rs is included verbatim with only mod + one re-export of wiring. None of Harden shared agent instruction review block/buzz#4220's validation boundaries are ported, which is why validate_managed_agent_definition_text carries a narrow #[allow(dead_code)] — the allow should be removed when Harden shared agent instruction review block/buzz#4220 lands.
• Conflicts expected in the sync stack's shared files: desktop/src-tauri/src/invoke.rs and lib.rs (additive command/state registration), desktop/src/app/AppShell.tsx, desktop/src/testing/e2eBridge.ts, and desktop/scripts/file-size-baselines.json (ratcheted for activeAgentTurnsStore.ts 1059, observerRelayStore.ts 1031, relayClientSession.ts 1070). All are additive hunks; resolve by keeping both sides, never wholesale "ours".
• Crew behaviour kept over upstream throughout (channel-first IA, no Projects/Workbench, one observer notification per envelope, workspace snapshot refresh, alive heartbeats, managed-agent env injection) — see the Crew divergences section above and divergences.md.

CI status (not fixed, per instruction): the latest run shows NuncioCrew Gate failing solely because desktop-fast, desktop-rust and macos-arm were cancelled — the gate script reports desktop-fast must be success when desktop=true, got cancelled. That is the parent session's deliberate CI serialization, not a failure of this change; every non-cancelled leg that ran passed. The earlier genuine Desktop Rust compile failure was real and is fixed in 4efdbd15.