Skip to content

perf: reduce WindowServer compositing cost and layout measurement overhead #23515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ashleeradka merged 6 commits into from
Apr 4, 2026
Merged

perf: reduce WindowServer compositing cost and layout measurement overhead #23515

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
89df710
perf: reduce WindowServer compositing cost and layout measurement ove…
devin-ai-integration[bot] Apr 3, 2026
9775300
fix: include displayScale in attachment decode task key
devin-ai-integration[bot] Apr 3, 2026
2bad7ae
fix: preserve full-res lightbox images and use direct data→ImageIO path
devin-ai-integration[bot] Apr 3, 2026
dfdac2c
fix: decode base64 lightbox image asynchronously instead of on every …
devin-ai-integration[bot] Apr 4, 2026
3a24b91
fix: use thread-safe CGImageSource APIs for lightbox base64 decoding
devin-ai-integration[bot] Apr 4, 2026
c622748
chore: clean up docstrings — remove PR-iteration-specific language
devin-ai-integration[bot] Apr 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 3 additions & 7 deletions clients/macos/vellum-assistant/Features/Chat/ChatBubble.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -363,14 +363,10 @@ struct ChatBubble: View, Equatable {
// Uses layoutPriority instead of fixedSize to avoid forcing
// full height measurement during lazy placement.
.layoutPriority(1)
// For non-streaming, non-interleaved messages, flatten the render
// tree into a single compositing layer to reduce layout passes.
// Flatten the render tree into a single compositing layer to reduce
// the number of CALayers the WindowServer must composite per message.
// Skipped during streaming to avoid re-compositing on every token delta.
// Also skipped for interleaved messages (text + tool calls + images)
// where the complex view hierarchy makes re-compositing expensive —
// async task completions (markdown parsing, image decoding) would
// trigger full re-compositing of the entire message on every change.
.modifier(ConditionalCompositingGroup(isActive: !message.isStreaming && !cachedHasInterleavedContent))
.modifier(ConditionalCompositingGroup(isActive: !message.isStreaming))
Comment thread
ashleeradka marked this conversation as resolved.

if !isUser { Spacer(minLength: 0) }
}
Expand Down
136 changes: 115 additions & 21 deletions clients/macos/vellum-assistant/Features/Chat/ChatBubbleAttachmentContent.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import AppKit
import ImageIO
import SwiftUI
import UniformTypeIdentifiers
import VellumAssistantShared
Expand All @@ -9,6 +10,64 @@ extension Notification.Name {
static let internalImageDragStarted = Notification.Name("com.vellum.internalImageDragStarted")
}

// MARK: - Display-Resolution Image Downsampling

/// Downsample raw image data to the target display size using ImageIO's streaming
/// decoder (`CGImageSourceCreateThumbnailAtIndex`). This is the preferred path —
/// it feeds compressed bytes directly to ImageIO, avoiding a full-resolution
/// bitmap allocation entirely.
///
/// - Parameters:
/// - data: Raw image file data (JPEG, PNG, HEIC, etc.).
/// - targetSize: The maximum display-point dimensions for rendering.
/// - scale: The display scale factor (e.g., 2.0 for Retina).
/// - Returns: A downsampled `NSImage` sized for display, or `nil` if decoding fails.
///
/// Reference: [WWDC18 — Images and Graphics Best Practices](https://developer.apple.com/videos/play/wwdc2018/219/)
private func downsampleForDisplay(data: Data, targetSize: CGSize, scale: CGFloat) -> NSImage? {
let maxPixelDimension = max(targetSize.width, targetSize.height) * scale
guard maxPixelDimension > 0 else { return nil }
guard let source = CGImageSourceCreateWithData(data as CFData, nil) else { return nil }

let options: [CFString: Any] = [
kCGImageSourceCreateThumbnailFromImageAlways: true,
kCGImageSourceCreateThumbnailWithTransform: true,
kCGImageSourceThumbnailMaxPixelSize: Int(maxPixelDimension)
]
guard let cgImage = CGImageSourceCreateThumbnailAtIndex(source, 0, options as CFDictionary) else {
return nil
}
return NSImage(cgImage: cgImage, size: .zero)
}

/// Downsample an already-decoded `NSImage` to the target display size.
/// Falls back to TIFF → ImageIO when raw file bytes are unavailable (e.g. images
/// arriving from cache or tool calls as decoded `NSImage` objects).
///
/// - Parameters:
/// - image: The source image (may be arbitrarily large).
/// - targetSize: The maximum display-point dimensions for rendering.
/// - scale: The display scale factor (e.g., 2.0 for Retina).
/// - Returns: A downsampled `NSImage` sized for display, or the original if
/// downsampling fails or the image is already small enough.
private func downsampleForDisplay(_ image: NSImage, targetSize: CGSize, scale: CGFloat) -> NSImage {
let maxPixelDimension = max(targetSize.width, targetSize.height) * scale
guard maxPixelDimension > 0 else { return image }

// Check if the image is already small enough — skip downsampling to avoid
// unnecessary re-encoding and potential quality loss.
if let cgImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil) {
let sourceMaxDim = max(CGFloat(cgImage.width), CGFloat(cgImage.height))
if sourceMaxDim <= maxPixelDimension * 1.1 {
return image
}
}

// Fall back to TIFF round-trip when raw file bytes are unavailable.
guard let tiffData = image.tiffRepresentation else { return image }
return downsampleForDisplay(data: tiffData, targetSize: targetSize, scale: scale) ?? image
}
Comment thread
ashleeradka marked this conversation as resolved.

// MARK: - Inline Tool Call Image

/// Renders a single tool-call-generated image at full width in the message flow.
Expand All @@ -17,11 +76,13 @@ private struct InlineToolCallImageView: View {
let image: NSImage
@Environment(\.displayScale) private var displayScale
@State private var sharingServices: [NSSharingService] = []
@State private var displayImage: NSImage?

@available(macOS, deprecated: 13.0)
var body: some View {
imageContent
.onTapGesture {
// Open lightbox with the original full-resolution image.
AppDelegate.shared?.mainWindow?.windowState.showImageLightbox(
image: image, filename: "image.png"
)
Expand All @@ -36,6 +97,13 @@ private struct InlineToolCallImageView: View {
.task {
sharingServices = await ImageActions.loadSharingServices(for: "image.png")
}
.task(id: displayScale) {
let maxDim = VSpacing.chatBubbleMaxWidth
let targetSize = CGSize(width: maxDim, height: maxDim)
let scale = displayScale
let source = image
displayImage = downsampleForDisplay(source, targetSize: targetSize, scale: scale)
}
.onDrag {
NotificationCenter.default.post(name: .internalImageDragStarted, object: nil)
let provider = NSItemProvider()
Expand All @@ -55,7 +123,8 @@ private struct InlineToolCallImageView: View {

@ViewBuilder
private var imageContent: some View {
if let cgImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil) {
let renderImage = displayImage ?? image
if let cgImage = renderImage.cgImage(forProposedRect: nil, context: nil, hints: nil) {
let nativeWidth = CGFloat(cgImage.width) / displayScale
let nativeHeight = CGFloat(cgImage.height) / displayScale
let maxDim: CGFloat = VSpacing.chatBubbleMaxWidth
Expand All @@ -69,7 +138,7 @@ private struct InlineToolCallImageView: View {
)
.clipShape(RoundedRectangle(cornerRadius: VRadius.md))
} else {
Image(nsImage: image)
Image(nsImage: renderImage)
.resizable()
.aspectRatio(contentMode: .fit)
.frame(maxWidth: VSpacing.chatBubbleMaxWidth)
Expand Down Expand Up @@ -205,58 +274,83 @@ private struct AttachmentImageGrid<Fallback: View>: View {
// SwiftUI can cancel the work immediately when the bubble scrolls off-screen.
// Wrapping in Task(priority:){}.value creates an unstructured task that is NOT
// cancelled by the .task modifier, causing off-screen decodes to run to completion.
.task(id: attachment.id) {
.task(id: "\(attachment.id)-\(displayScale)") {
let scale = displayScale
if isSingleImage {
let targetSize = CGSize(width: VSpacing.chatBubbleMaxWidth, height: VSpacing.chatBubbleMaxWidth)
// Single images: prefer full-resolution data so the frame
// sizing (which uses native pixel dimensions) is accurate.
// Thumbnails are 800px max — using them gives ~400pt on
// Retina, adequate for most previews.
if let fullData = Data(base64Encoded: attachment.data), !fullData.isEmpty,
let img = NSImage(data: fullData) {
if let fullData = Data(base64Encoded: attachment.data), !fullData.isEmpty {
guard !Task.isCancelled else { return }
loadedImages[attachment.id] = img
return
// Prefer direct data → ImageIO path (no full-res bitmap allocation).
if let downsampled = downsampleForDisplay(data: fullData, targetSize: targetSize, scale: scale) {
loadedImages[attachment.id] = downsampled
return
}
// Fall back to NSImage decode if ImageIO can't handle the format.
if let img = NSImage(data: fullData) {
loadedImages[attachment.id] = downsampleForDisplay(img, targetSize: targetSize, scale: scale)
return
}
}

guard !Task.isCancelled else { return }

// Full data unavailable (lazy-load or cleared) — fall back to thumbnail.
if let img = attachment.thumbnailImage {
loadedImages[attachment.id] = img
loadedImages[attachment.id] = downsampleForDisplay(img, targetSize: targetSize, scale: scale)
return
}

guard !Task.isCancelled else { return }

if let thumbnailData = attachment.thumbnailData, !thumbnailData.isEmpty,
let img = NSImage(data: thumbnailData) {
if let thumbnailData = attachment.thumbnailData, !thumbnailData.isEmpty {
guard !Task.isCancelled else { return }
loadedImages[attachment.id] = img
return
if let downsampled = downsampleForDisplay(data: thumbnailData, targetSize: targetSize, scale: scale) {
loadedImages[attachment.id] = downsampled
return
}
if let img = NSImage(data: thumbnailData) {
loadedImages[attachment.id] = downsampleForDisplay(img, targetSize: targetSize, scale: scale)
return
}
}
} else {
let gridTargetSize = CGSize(width: 160, height: 120)
// Grid mode: prefer thumbnails for fast loading of many images.
if let img = attachment.thumbnailImage {
loadedImages[attachment.id] = img
loadedImages[attachment.id] = downsampleForDisplay(img, targetSize: gridTargetSize, scale: scale)
return
}

guard !Task.isCancelled else { return }

if let thumbnailData = attachment.thumbnailData, !thumbnailData.isEmpty,
let img = NSImage(data: thumbnailData) {
if let thumbnailData = attachment.thumbnailData, !thumbnailData.isEmpty {
guard !Task.isCancelled else { return }
loadedImages[attachment.id] = img
return
if let downsampled = downsampleForDisplay(data: thumbnailData, targetSize: gridTargetSize, scale: scale) {
loadedImages[attachment.id] = downsampled
return
}
if let img = NSImage(data: thumbnailData) {
loadedImages[attachment.id] = downsampleForDisplay(img, targetSize: gridTargetSize, scale: scale)
return
}
}

guard !Task.isCancelled else { return }

if let fullData = Data(base64Encoded: attachment.data), !fullData.isEmpty,
let img = NSImage(data: fullData) {
if let fullData = Data(base64Encoded: attachment.data), !fullData.isEmpty {
guard !Task.isCancelled else { return }
loadedImages[attachment.id] = img
return
if let downsampled = downsampleForDisplay(data: fullData, targetSize: gridTargetSize, scale: scale) {
loadedImages[attachment.id] = downsampled
return
}
if let img = NSImage(data: fullData) {
loadedImages[attachment.id] = downsampleForDisplay(img, targetSize: gridTargetSize, scale: scale)
return
}
}
}

Expand Down
47 changes: 26 additions & 21 deletions clients/macos/vellum-assistant/Features/Chat/MediaEmbeds/InlineAudioAttachmentView.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ struct InlineAudioAttachmentView: View {
@State private var hasRetriedOnce = false
@State private var isSaving = false
@State private var isHovering = false
/// Width of the progress bar track, measured via `.onGeometryChange()`.
@State private var trackWidth: CGFloat = 0

/// Coordinator object that acts as AVAudioPlayerDelegate to detect playback
/// completion and relay it back to the SwiftUI state.
Expand Down Expand Up @@ -176,31 +178,34 @@ struct InlineAudioAttachmentView: View {

private var progressBar: some View {
TimelineView(.periodic(from: .now, by: isPlaying ? 0.1 : 60)) { _ in
GeometryReader { geo in
let dur = currentDuration
let prog = currentProgress
ZStack(alignment: .leading) {
// Track
let dur = currentDuration
let prog = currentProgress
ZStack(alignment: .leading) {
// Track
RoundedRectangle(cornerRadius: 1.5)
.fill(VColor.borderBase.opacity(0.5))
.frame(height: 3)

// Filled portion
if dur > 0, trackWidth > 0 {
RoundedRectangle(cornerRadius: 1.5)
.fill(VColor.borderBase.opacity(0.5))
.frame(height: 3)

// Filled portion
if dur > 0 {
RoundedRectangle(cornerRadius: 1.5)
.fill(VColor.systemPositiveStrong)
.frame(width: max(0, geo.size.width * CGFloat(prog / dur)), height: 3)
}
}
.contentShape(Rectangle())
.onTapGesture { location in
guard dur > 0, let player = audioPlayer else { return }
let fraction = max(0, min(1, location.x / geo.size.width))
player.currentTime = fraction * dur
.fill(VColor.systemPositiveStrong)
.frame(width: max(0, trackWidth * CGFloat(prog / dur)), height: 3)
}
}
.frame(height: 3)
.contentShape(Rectangle())
.onGeometryChange(for: CGFloat.self) { proxy in
proxy.size.width
} action: { newWidth in
trackWidth = newWidth
}
.onTapGesture { location in
guard dur > 0, let player = audioPlayer, trackWidth > 0 else { return }
let fraction = max(0, min(1, location.x / trackWidth))
player.currentTime = fraction * dur
}
}
.frame(height: 3)
}

private var timeDisplay: some View {
Expand Down
26 changes: 23 additions & 3 deletions clients/macos/vellum-assistant/Features/MainWindow/MainWindowState.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,10 +359,12 @@ public final class MainWindowState: ObservableObject {
base64Data: base64Data,
lazyAttachmentId: lazyAttachmentId,
fullResImage: nil,
isLoadingFullRes: lazyAttachmentId != nil
isLoadingFullRes: lazyAttachmentId != nil || base64Data != nil

@devin-ai-integration devin-ai-integration Bot Apr 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Unnecessary loading spinner shown in lightbox when image is already full-resolution

The change to isLoadingFullRes: lazyAttachmentId != nil || base64Data != nil causes a spurious loading spinner to appear in the lightbox when callers pass an already-decoded full-resolution image alongside base64Data. Specifically, ComposerAttachments.openAttachmentPreview (ComposerAttachments.swift:102-121) decodes the full-res image synchronously via NSImage(data:) and passes both the decoded image and the raw base64Data to showImageLightbox. Before this PR, isLoadingFullRes was lazyAttachmentId != nil (false here), so no spinner appeared. Now isLoadingFullRes is true whenever base64Data is non-nil, triggering the spinner overlay (ImageLightboxOverlay.swift:31-35) on top of the already-visible full-res image while decodeBase64LightboxImage redundantly re-decodes the same data (~50-200ms). The user sees a confusing loading indicator over a fully loaded image.

Prompt for agents 
The root cause is that showImageLightbox unconditionally sets isLoadingFullRes=true when base64Data is provided, but some callers (ComposerAttachments.openAttachmentPreview) have already decoded the full-res image and pass it as the image parameter. The fix should distinguish between callers that pass a downsampled thumbnail (need async full-res decode) and callers that pass the already-decoded full-res image (no decode needed).

Possible approaches:
1. Add a parameter like needsFullResDecode: Bool to showImageLightbox so callers can opt out of the async decode when they already have full-res.
2. Have showImageLightbox only set isLoadingFullRes and call decodeBase64LightboxImage when the passed image appears to be downsampled (e.g. check pixel dimensions vs some threshold).
3. Update ComposerAttachments.openAttachmentPreview to not pass base64Data (pass nil) since it already provides the full-res image, though this would prevent the lightbox toolbar from using base64Data for copy/save actions.

Option 1 is cleanest. The new parameter defaults to true for backward compatibility with the chat bubble path (which now passes downsampled images), and ComposerAttachments passes false.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

)
if lazyAttachmentId != nil {
fetchFullResLightboxImage()
} else if let base64Data, !base64Data.isEmpty {
decodeBase64LightboxImage(base64Data)
}
}

Expand All @@ -378,8 +380,26 @@ public final class MainWindowState: ObservableObject {
guard !Task.isCancelled else { return }
if let data, let fullRes = NSImage(data: data) {
self?.imageLightbox?.fullResImage = fullRes
// Update base64Data so toolbar actions (copy, save) use full-res
// We don't have a direct setter, but fullResImage is preferred by displayImage
}
self?.imageLightbox?.isLoadingFullRes = false
}
}

/// Decode base64 image data off the main thread using thread-safe ImageIO
/// APIs and set `fullResImage` once complete.
private func decodeBase64LightboxImage(_ base64Data: String) {
lightboxFetchTask = Task { @MainActor [weak self] in
let decoded: NSImage? = await Task.detached(priority: .userInitiated) {
guard let data = Data(base64Encoded: base64Data) else { return nil }
guard let source = CGImageSourceCreateWithData(data as CFData, nil),
let cgImage = CGImageSourceCreateImageAtIndex(source, 0, nil) else {
return nil
}
return NSImage(cgImage: cgImage, size: .zero)
}.value
guard !Task.isCancelled else { return }
if let decoded {
self?.imageLightbox?.fullResImage = decoded
}
self?.imageLightbox?.isLoadingFullRes = false
}
Expand Down