diff --git a/checkpoint.txt b/checkpoint.txt new file mode 100644 index 000000000000..5b43f9f6a9e3 --- /dev/null +++ b/checkpoint.txt @@ -0,0 +1,809 @@ +# Project Checkpoint Summary + +## Attachment Handling Overview + +### Image Processing +- Supports pasting images directly (handlePaste) +- Supports drag & drop images (handleDrop) +- Supports file selection for images (handleFileSelect) +- Image processing flow: + 1. Get raw image data using getImageData + 2. Create preview using uncompressed data + 3. Compress image for storage + 4. Save to temporary directory + 5. Clean up temp files periodically (every hour) or on app exit + +### File Attachments +- Supports selecting files via dialog +- Supports drag & drop files +- File handling flow: + 1. Extract file metadata (name, extension, type) + 2. Create file reference with file:// protocol + 3. Store in attachments array with path and metadata + +### Experimental Attachments Structure +```typescript +experimental_attachments: Array<{ + name: string; + contentType: string; + url: string; +}> +``` +- Images: + - name: 'image' + - contentType: 'image/png' + - url: Base64 data URL +- Files: + - name: Original filename + - contentType: MIME type or 'application/octet-stream' + - url: file:// protocol path + +## UI Component Styling + +### Input Field (Input.tsx) +- Base dimensions: + - MIN_HEIGHT: 47px + - MAX_HEIGHT: 200px + - Padding: + - TOP: 14px + - BOTTOM: 14px + - LEFT: 16px + - RIGHT: 92px +- Dynamic height adjustment based on content +- Attachment preview area above input field +- Button container: + - Fixed position on right + - Gap between buttons: 8px (gap-2) + - Height matches MIN_HEIGHT + +### Preview Components + +#### Shared Preview Styling +- Close button (Remove button): + - Position: absolute + - Size: 20x20px (w-5 h-5) + - Background: rgba(143, 142, 147, 0.9) + - Border radius: 9999px (rounded-full) + - Icon: + - Size: 12x12px (w-3 h-3) + - Color: white + - Stroke width: 3 + - Positioning: + - Image preview: top-[6px] right-[6px] + - File preview: top-[8px] right-[8px] +- Common container styles: + - Ring: ring-1 ring-black/5 dark:ring-white/5 + - Background: gray-100 dark:bg-gray-700 + - Border radius: rounded-[12px] + +#### ImagePreview Specifics +- Message mode: + - Width: 300px + - Min height: 100px + - Max height: 300px + - Image object-fit: cover +- Preview mode: + - Standard: 120x120px square + - Adaptive height: 54x54px square +- Image container: + - overflow: hidden + - Full width/height: w-full h-full + - Object fit: cover +- Error handling: + - Console logging for failed loads + - Image source validation + - Base64 validation logging + +#### FilePreviewTile Specifics +- Container layout: + - Display: flex + - Align items: center + - Fixed height: 54px + - Horizontal padding: px-3 (12px) +- Icon container: + - Flex-shrink-0 to prevent icon compression + - SVG path: + - strokeLinecap: round + - strokeLinejoin: round + - strokeWidth: 2 +- Text container: + - Margin left: ml-3 + - Flex: 1 + - min-width: 0 (min-w-0) + - Two-line layout: + - Filename: font-medium + - Filetype: regular weight + - Text truncation: truncate class +- Responsive width: + - Message mode: 300px + - Preview mode: 200px + +### Style Differences Summary +1. Container Shape: + - Image: Square in preview (1:1 ratio), variable in message mode + - File: Always rectangular (fixed height) + +2. Content Handling: + - Image: Uses object-fit for content scaling + - File: Uses flex layout with text truncation + +3. Close Button Position: + - Image: 6px from top/right edges + - File: 8px from top/right edges + +4. Preview Adaptability: + - Image: Supports adaptive sizing (54x54px) + - File: Maintains consistent height + +5. Dark Mode Handling: + - Image: Focuses on container background + - File: Handles both icon and text colors + +### User Message Layout +- Width: 90% of container +- Max attachment width: 75% of message width +- Padding top: 16px (pt-4) +- Margin bottom: 16px (mb-[16px]) +- Message bubble: + - Background: user-bubble (dark: user-bubble-dark) + - Text color: goose-text-light (dark: goose-text-light-dark) + - Rounded corners: 2xl + - Padding: 16px (p-4) + +## Temporary File Management +- Location: os.tmpdir()/goose-images +- Cleanup criteria: + - Age: > 1 hour + - Total size: > 100MB +- Permissions: 0o777 for directory, 0o666 for files +- Automatic cleanup: + - Hourly interval + - On app exit + - Before saving new images + +## Message Content Handling +- Text content preserved as entered +- Image data included as base64 URLs +- File paths included with file:// protocol +- Content filtering in display: + - Removes file:// URLs + - Removes absolute paths + - Removes base64 image data +- Original content preserved for server use + +### Tile Behavior and Transitions + +#### Input Preview State (Input.tsx) +- Preview container: + - Padding left: matches textarea (16px) + - Flex row layout with gap-2 (8px) between tiles + - Full width with natural wrapping +- Interactive elements: + - Close buttons visible and active + - Hover states on tiles + - Click handling for removal +- Size constraints: + - Images maintain aspect ratio within bounds + - Files show full name with truncation + - Multiple attachments wrap naturally + +#### Message State (UserMessage.tsx) +- Container constraints: + - Max width: 75% of message width + - Justify content: end (right-aligned) + - Margin bottom: 8px (mb-2) +- Layout adjustments: + - Close buttons removed + - Fixed widths enforced (300px) + - Flex row with gap-2 maintained +- Content display: + - Images: + - Larger display size + - Maintains aspect ratio + - Max height enforced (300px) + - object-fit: cover for consistency + - Files: + - Wider tile (300px vs 200px) + - Same height maintained (54px) + - Full filename display prioritized + +#### State Transition Handling +- Attachment data preservation: + - Original paths maintained + - Base64 data preserved for images + - File metadata retained +- Style transitions: + - Container width adaptation + - Height adjustments for images + - Removal of interactive elements +- Layout shifts: + - Preview to message positioning + - Width and spacing adjustments + - Alignment changes + +#### Mixed Content Handling +- Multiple attachment types: + - Consistent spacing (gap-2) + - Alignment handling + - Height normalization option +- Adaptive image sizing: + - hasMixedTypes flag triggers height matching + - 54px height when mixed with files + - Maintains aspect ratio within bounds +- Order preservation: + - Maintains input sequence + - Consistent left-to-right ordering + - No type-based reordering + +## Image Data Handling + +### Preview Image Processing +- Source handling: + - Pasted images: Direct Blob to base64 + - File images: File to base64 via getImageData + - Dropped images: File to base64 via getImageData +- Data flow: + 1. Original image capture + - Maintain original quality for preview + - Store as data URL with proper MIME type + - Format: `data:image/[type];base64,[data]` + 2. Compression for storage + - Compress using compressImage utility + - Target size reduction while maintaining quality + - Used for temp file storage + 3. Preview generation + - Use uncompressed data for visual quality + - Maintain aspect ratio in preview + - Handle both light/dark mode backgrounds + +### Image Data Management +- Preview data: + - Format: Full data URL (data:image/[type];base64) + - Storage: Kept in attachment.src + - Usage: Direct source for preview img elements +- Storage data: + - Format: Compressed base64 + - Location: Temporary file system + - Reference: Path stored in attachment.path +- Validation: + - Base64 format checking + - MIME type validation + - Size constraints enforcement + +### Performance Considerations +- Memory management: + - Uncompressed data only for active previews + - Clear data when removing attachments + - Garbage collection hints via cleanup +- Loading states: + - Progressive loading for large images + - Error states for failed loads + - Fallback display for invalid data +- Size optimization: + - Preview size limits + - Compression thresholds + - Cached preview generation + +### Error Handling +- Data validation: + - Check for valid base64 encoding + - Verify MIME type matches + - Validate data URL format +- Loading failures: + - Console error logging + - User feedback + - Fallback display options +- Recovery strategies: + - Retry loading on error + - Fallback to file preview + - Clear invalid data + +## File Attachment Flow + +### Paperclip Button Implementation +- Button styling: + - Icon size: 20px + - Ghost variant with hover states + - Color: indigo-600 (dark: indigo-300) + - Hover: indigo-700 (dark: indigo-200) + - Background hover: indigo-100 (dark: indigo-800) +- State handling: + - Disabled state with opacity: 0.5 + - Drag state visual feedback + - Focus management after selection + +### File Selection Process +1. Dialog trigger: + - Uses electron.selectFileOrDirectory() + - Supports both files and directories + - Native system dialog + +2. File processing: + ```typescript + const handleFileSelect = async () => { + const filePath = await electron.selectFileOrDirectory(); + if (!filePath) return; + + const fileName = filePath.split('/').pop() || 'Unknown file'; + const fileExt = fileName.split('.').pop()?.toLowerCase() || ''; + const isImage = ['jpg', 'jpeg', 'png', 'gif', 'webp'].includes(fileExt); + ``` + +3. Type determination: + - Image detection via extension + - Fallback to generic file handling + - MIME type inference from extension + +### File Preview Generation +- For image files: + 1. Attempt image processing: + - Read as data URL + - Generate preview + - Create temporary file + 2. Fallback handling: + - Switch to file preview on error + - Preserve original path + - Set appropriate MIME type + +- For non-image files: + 1. Metadata extraction: + - Filename parsing + - Extension detection + - MIME type assignment + 2. Preview tile creation: + - File icon display + - Two-line info layout + - Path preservation + +### FilePreviewTile Lifecycle +- Creation: + ```typescript + setAttachments(prev => [...prev, { + type: 'file', + name: fileName, + fileType: fileExt ? `application/${fileExt}` : 'application/octet-stream', + path: filePath + }]); + ``` +- State management: + - Attachment array updates + - Preview tile rendering + - Remove functionality +- Focus handling: + - Return focus to textarea + - Maintain keyboard navigation + - Accessibility considerations + +## Goose Analysis Integration + +### Message Content Structure +- Combined content format: + ```typescript + const messageContent = [ + value.trim(), + ...attachments.map(att => { + if (att.type === 'image') { + return att.src; // Full base64 data URL + } else { + return `file://${att.path}`; // File protocol URL + } + }) + ].join('\n'); + ``` + +### File Type Analysis + +#### Image Analysis +- Data format: + - Full base64 data URL included in message content + - Format: `data:image/[type];base64,[data]` + - Allows direct image content analysis +- Processing flow: + 1. Image data extraction from base64 + 2. Content analysis on raw image data + 3. Results incorporated into response +- Benefits: + - No additional file system access needed + - Immediate access to image content + - Consistent cross-platform handling + +#### Document Analysis +- File access: + - Uses file:// protocol URLs + - Requires file system permissions + - Handles various document types +- Processing flow: + 1. File path extraction from URL + 2. Content reading based on file type + 3. Text extraction or binary processing + 4. Analysis of extracted content +- Supported operations: + - Text file reading + - Binary file handling + - Directory structure analysis + - File metadata access + +### Analysis Integration + +#### Resource Registration +```typescript +// Backend resource handling +const resource = Resource::new(uri.clone(), Some("text".to_string()), None); +active_resources.insert(uri.clone(), resource); +``` + +#### Content Reading +- Text files: + - Direct content reading + - UTF-8 encoding handling + - Line-by-line processing +- Binary files: + - Base64 encoding for transfer + - MIME type detection + - Size limit enforcement + +#### Error Handling +- File access errors: + - Permission issues + - Missing files + - Invalid paths +- Content processing errors: + - Encoding problems + - Size limitations + - Format validation +- Recovery strategies: + - Detailed error reporting + - Fallback processing methods + - User feedback + +### Security Considerations +- File access: + - Path validation + - Permission checking + - Directory traversal prevention +- Content processing: + - Size limits enforcement + - Memory usage monitoring + - Timeout handling +- Data handling: + - Secure temporary storage + - Content sanitization + - Resource cleanup + +## Electron Main Process Integration + +### Critical Main Process Code (main.ts) +```typescript +// Temp directory for image storage +const TEMP_DIR = path.join(os.tmpdir(), 'goose-images'); + +// Ensure temp directory exists with proper permissions +const ensureTempDir = () => { + try { + if (!fs.existsSync(TEMP_DIR)) { + fs.mkdirSync(TEMP_DIR, { + recursive: true, + mode: 0o777 // Full permissions for temp directory + }); + } + + // Verify directory permissions + const stats = fs.statSync(TEMP_DIR); + const requiredMode = 0o777; + + if ((stats.mode & 0o777) !== requiredMode) { + fs.chmodSync(TEMP_DIR, requiredMode); + } + + return true; + } catch (error) { + log.error('Error in ensureTempDir:', error); + return false; + } +}; + +// Save temporary image with retries +const saveTempImage = async (imageData: string, retries = 3): Promise => { + let lastError; + + for (let attempt = 1; attempt <= retries; attempt++) { + try { + // Ensure temp directory exists + if (!ensureTempDir()) { + throw new Error('Failed to create temp directory'); + } + + // Process base64 data + const base64Data = imageData.includes('base64,') + ? imageData.split('base64,')[1] + : imageData; + + // Generate unique filename + const timestamp = Date.now(); + const random = Math.random().toString(36).substring(2, 15); + const filename = `${timestamp}-${random}.png`; + const filepath = path.join(TEMP_DIR, filename); + + // Write file with proper permissions + await fs.promises.writeFile(filepath, Buffer.from(base64Data, 'base64'), { mode: 0o666 }); + + // Verify file was written + if (!fs.existsSync(filepath)) { + throw new Error('Failed to verify file was written'); + } + + return filepath; + } catch (error) { + lastError = error; + if (attempt === retries) { + throw new Error(`Failed to save file after ${retries} attempts: ${error.message}`); + } + // Wait before retry + await new Promise(resolve => setTimeout(resolve, 1000 * attempt)); + } + } + + throw lastError; +}; + +// Add IPC handlers for file operations +ipcMain.handle('select-file-or-directory', async () => { + const result = await dialog.showOpenDialog({ + properties: ['openFile', 'openDirectory'] + }); + + if (!result.canceled && result.filePaths.length > 0) { + return result.filePaths[0]; + } + return null; +}); + +ipcMain.handle('save-temp-image', async (_, imageData: string) => { + return saveTempImage(imageData); +}); + +// Cleanup on app exit +app.on('will-quit', () => { + try { + if (fs.existsSync(TEMP_DIR)) { + fs.rmSync(TEMP_DIR, { recursive: true, force: true }); + log.info('Successfully removed temp directory on exit'); + } + } catch (error) { + log.error('Failed to cleanup temp directory on exit:', error); + } +}); +``` + +### Main Process Features +1. Temporary Directory Management: + - Location in system temp directory + - Proper permissions (0o777) + - Automatic creation and cleanup + +2. Image Saving: + - Retry mechanism for reliability + - Unique filename generation + - Permission enforcement + - File verification + +3. IPC Handlers: + - File/directory selection + - Temporary image saving + - Error handling and logging + +4. Cleanup Mechanisms: + - Exit cleanup + - Error handling + - Logging for debugging +``` + +## Splash Screen Styling + +### Logo and Button Spacing +- Logo section: + - Container: `flex items-center` + - Logo text spacing: `ml-[8px]` + - Text styling: `ask-goose-type goose-text dark:goose-text-dark` + - Two-line "ask goose" text with `
` + +### Divider Line +- Spacing: `mt-[24px] mb-[24px]` +- Dimensions: `w-[198px] h-[17px]` +- Container: `py-2 flex-col justify-center items-start inline-flex` +- Line styling: `self-stretch h-px bg-black/5 dark:bg-white/5 rounded-sm` + +### "What can goose do?" Button +- Container width: `w-[312px]` +- Padding: `px-16 py-4` +- Text: `text-14 text-center` +- Colors: + - Text: `text-splash-pills-text dark:text-splash-pills-text-dark` + - Background: `bg-prev-goose-gradient dark:bg-dark-prev-goose-gradient` + - Text color: `text-prev-goose-text dark:text-prev-goose-text-dark` +- Styling: + - Border radius: `rounded-[14px]` + - Display: `inline-block` + - Hover: `hover:scale-[1.02]` + - Animation: `transition-all duration-150` + - White space: `whitespace-nowrap` + +### SplashPills Component +- Grid layout: + - `grid grid-cols-2 gap-4` + - Bottom margin: `mb-[8px]` + - Width: `max-w-full` + +#### Individual Pills +- Padding: `px-16 py-8` +- Text styling: + - Size: `text-14` + - Alignment: `text-center` + - Color: `text-black/60 dark:text-white/60` +- Background: + - Default: `bg-black/5 dark:bg-white/5` + - Hover: `hover:bg-black/10 dark:hover:bg-white/10` +- Shape: + - Border radius: `rounded-[1000px]` + - Display: `inline-block` +- Animation: + - Properties: `transition-all duration-150` +- Content: + - Text wrapping: `line-clamp-2` + - Cursor: `cursor-pointer` + +### Layout Structure +- Main container: `h-full flex flex-col items-center justify-center` +- Flexible spacing: + - Top/bottom space: `flex flex-1` + - Pills container: `flex items-center p-4` +- Vertical alignment: + - Logo centered + - Divider line below logo + - Action button below divider + - Pills at bottom + +### Color Scheme (from tailwind.config.ts) +- Splash pills: + - Light: `rgba(255, 255, 255, 0.60)` + - Dark: `rgba(31, 41, 55, 0.60)` +- Splash pills text: + - Light: `rgba(0, 0, 0, 0.60)` + - Dark: `rgba(255, 255, 255, 0.60)` +- Previous goose text: + - Light: `#4E52C5` + - Dark: `#9CA3AF` + +### Background Blur Effect +- Window properties: + - Vibrancy: `vibrancy: 'under-window'` + - Transparency: `transparent: true` + - Background: Removed solid white background to allow blur +- Blur characteristics: + - Type: Native system blur (macOS) + - Depth: Under-window vibrancy + - Visibility: Enhanced by removing opaque backgrounds +- Components: + - Main window: Transparent background with vibrancy + - Content areas: Semi-transparent backgrounds using opacity values + - Dark mode support: Adjusted opacity values for dark theme +- Key fixes: + - Removed white background color that was blocking vibrancy + - Ensured container backgrounds use alpha channels + - Maintained contrast while preserving transparency +- Implementation: + - BrowserWindow config: `{ vibrancy: 'under-window', transparent: true }` + - Base container: No background color to allow blur + - Child elements: Use rgba/alpha values for backgrounds + +## Implementation Examples + +### Window Configuration +```typescript +// BrowserWindow configuration in main.ts +const mainWindow = new BrowserWindow({ + titleBarStyle: 'hidden', + trafficLightPosition: { x: 16, y: 10 }, + vibrancy: 'under-window', + transparent: true, + // Remove backgroundColor to allow blur + webPreferences: { + preload: path.join(__dirname, 'preload.js'), + }, +}); +``` + +### Splash Screen Layout +```tsx +// Splash.tsx core structure +
+
+
+ + + ask
goose + +
+ + {/* Divider */} +
+
+
+ + {/* Action Button */} +
+ What can goose do? +
+
+
+``` + +### SplashPills Implementation +```tsx +// SplashPills.tsx component +function SplashPill({ content, append }) { + return ( +
{ + await append({ content, role: "user" }); + }} + > +
{content}
+
+ ); +} + +export default function SplashPills({ append }) { + return ( +
+ + + + +
+ ); +} +``` + +### Tailwind Color Configuration +```typescript +// tailwind.config.ts color definitions +colors: { + 'splash-pills': { + DEFAULT: 'rgba(255, 255, 255, 0.60)', + dark: 'rgba(31, 41, 55, 0.60)' + }, + 'splash-pills-text': { + DEFAULT: 'rgba(0, 0, 0, 0.60)', + dark: 'rgba(255, 255, 255, 0.60)' + }, + 'prev-goose-text': { + DEFAULT: '#4E52C5', + dark: '#9CA3AF' + } +} +``` + +### Background Blur Implementation +```typescript +// Main window configuration with blur +{ + titleBarStyle: 'hidden', + vibrancy: 'under-window', + transparent: true, + // Key settings for blur effect + backgroundColor: undefined, // Remove default background + webPreferences: { + preload: path.join(__dirname, 'preload.js'), + } +} + +// Component background example with transparency +
// Uses alpha channel + {/* Content */} +
+``` \ No newline at end of file diff --git a/ui/desktop/src/components/Input.tsx b/ui/desktop/src/components/Input.tsx index c189059c16c1..eef17b27a31e 100644 --- a/ui/desktop/src/components/Input.tsx +++ b/ui/desktop/src/components/Input.tsx @@ -81,7 +81,7 @@ export default function Input({ }; return ( -
+