From beb520ee86caa4eeaeaf22bb1e2405578b066ee6 Mon Sep 17 00:00:00 2001 From: trsommer Date: Mon, 17 Nov 2025 04:48:59 +0100 Subject: [PATCH 01/17] phase 1 complete --- gpuRendering.md | 421 +++++++++ package.json | 3 + pnpm-lock.yaml | 86 +- pnpm-workspace.yaml | 3 + .../maskeditor/MaskEditorContent.vue | 6 + src/composables/maskeditor/useBrushDrawing.ts | 819 +++++++++++++----- src/stores/maskEditorStore.ts | 4 + tsconfig.json | 3 +- vite.config.mts | 2 + 9 files changed, 1149 insertions(+), 198 deletions(-) create mode 100644 gpuRendering.md diff --git a/gpuRendering.md b/gpuRendering.md new file mode 100644 index 00000000000..35eac9cdbf4 --- /dev/null +++ b/gpuRendering.md @@ -0,0 +1,421 @@ +# GPU Brush Rendering: Photoshop-Like Vertex Splatter Implementation Plan + +## Photoshop's Brush System: How It Works (High-Performance Native GPU Rendering) + +Photoshop (and pro apps like Krita/Clip Studio) uses a **vertex-driven splatting pipeline** optimized for real-time stroke rendering. It's built on native GPU APIs (Metal on macOS, DirectX 12 on Windows, Vulkan cross-platform) with these core principles: + +### 1. Stroke Data → Vertex Buffer (Per-Point Instancing) +- **Input**: Mouse/touch points → smoothed path (Catmull-Rom or similar, ~5-50 pts/stroke segment). +- **Processing**: CPU generates **position + attributes** per point (pos `{x,y}`, size, opacity, rotation if textured). +- **GPU Upload**: Dynamic **vertex buffer** (`GPUBuffer` equivalent): Append/update points (~1kB/stroke). No full-canvas ops. +- **Geometry**: **Instanced quads** (or triangles): + | Primitive | Why | + |-----------|-----| + | 4-vertex quad/pt | Covers brush diameter; fragment shader fills alpha. | + | Fixed VBO (screen quad × instances) | GPU reuses tiny static buffer (4 verts); `draw(6, numPoints)` triangles. | + +### 2. Render Pipeline (Vertex + Fragment Shaders) +- **Vertex Shader** (per-vertex, runs ~4 verts/pt): + ```glsl + // Pseudo-Metal + vertex float4 vs_main(uint vid [[vertex_id]], uint iid [[instance_id]]) { + float2 quadOffset[4] = {{-r, -r}, {r, -r}, {r, r}, {-r, r}}; // r = brush radius + float2 center = points[iid].xy; // Fetch from instance buffer + return float4(center + quadOffset[vid] * points[iid].size, 0, 1); + } + ``` + - Outputs **screen-space quad** positioned/scaled per point. + - **Key**: Runs O(4 × points) = tiny (~800 invocations/stroke). + +- **Fragment Shader** (per-pixel **inside quads only**, ~brush²/pt): + ```glsl + fragment float4 fs_main(float2 pos [[position]]) { + float2 center = points[gl_InstanceID].xy; + float dist = length(pos - center); // Euclidean/Chebyshev + float alpha = opacity * smoothstep(radius, hardRadius, dist); // Falloff + return float4(color.rgb, alpha); // Src-over blend + } + ``` + - **Rasterizer Magic**: GPU hardware culls fragments outside quads (scissor/depth reject 99.9% canvas). + - **Executions**: Only ~π×r² fragments/pt (e.g., r=50 → 8k frags/pt × 200 pts = 1.6M total, fully parallel). + - **Perf**: <1ms/stroke on iPhone GPU. + - **Opacity Clamping**: For a single stroke (mouse down to up), overlapping brush points accumulate opacity, but the total added opacity per pixel is clamped to the brush opacity (e.g., if brush opacity = 50%, max added = 50%, even with heavy overlaps). + +### 3. Async Tiled Rendering + Persistent GPU Textures +- **Framebuffers**: Layer stack (base image, mask, paint) as **GPU textures** (no CPU canvas sync). + - Render-to-texture: Stroke → temp → clamp alpha → composite (ping-pong for undo). +- **Command Buffers**: Async submit (`MTLCommandBuffer.commit()`); GPU runs in background. +- **Tiling**: Metal PSMT/Vulkan subpasses → render only dirty tiles (e.g., stroke bbox). +- **Blending**: Native `src-over`/`dst-out` in pipeline state; `loadOp: 'load'` preserves prior content. +- **No Readback**: UI reads GPU via `blit` to display texture or partial staging. History: Copy layers on save. + +### 4. Multi-Threading + Optimizations +- **CPU**: Path smoothing, pressure curves (separate thread). +- **Preview**: Thin line (1px polyline) + deferred fat stroke. +- **Caches**: Pre-baked brushes (noise textures), LOD (small brushes CPU). +- **Scalability**: 8k canvas, 60FPS stylus, tablet pressure/tilt. + +**Total Cost**: **O(points × brush_pixels)** ≈ 2M ops/stroke → instant. + +## Current System: Why It's 400x Slower + Crashing + +Your WebGPU impl uses **compute shaders naively**, mimicking CPU pixel loops: + +### 1. Full-Canvas Compute Dispatch (Fatal Flaw) +```wgsl +@compute @workgroup_size(8,8) // Dispatch: canvas.w/8 × canvas.h/8 = 65k workgroups +fn main(gid) { + for (i = 0; i < numPoints; i++) { // Loops ALL points/PER PIXEL + dist = max(abs(gid.xy - points[i])); // 200x loop + } +} +``` +- **Executions**: **4M pixels × 200 pts** = **800M invocations** (vs Photoshop's 1.6M fragments). +- GPU overloads: Workgroups serialize on register pressure/memory bandwidth. + +### 2. Synchronous JS-WebGPU Queue +- `queue.submit(commands)` **blocks JS thread** until GPU ~finishes (WebGPU sync semantics). +- Browser tab hangs (GPU→CPU callback delay). + +### 3. Expensive CPU Overlaps +- **Preview**: `drawWithBetterSmoothing` → JS `drawShape()` loops (10k px/pt uncached). +- **Readback**: Full 16MB `copyTextureToBuffer` + JS `ImageData` loop per batch. +- **Batching**: Unlimited points → shader loops explode. + +### 4. No Culling/Persistence +- No quad rasterization → no hardware culling. +- CPU canvases → constant GPU↔CPU sync. + +**Result**: 500ms+ latency → freeze/crash. + +| Metric | Photoshop | Current | +|--------|-----------|---------| +| Ops/Stroke | 2M (parallel frags) | 800M (serial loops) | +| Latency | <1ms | 500ms+ | +| Sync | Async | Blocking | +| Pixels Processed | Brush area | Full canvas | + +## Rework Plan: Implement Photoshop-Like Vertex Splatter in WebGPU + +**Goal**: Replace compute → **render pipeline**. O(points × brush_px) perf. No CPU canvases (GPU-only layers). ~10x faster than canvas fallback. + +### High-Level Architecture Changes +1. **Ditch CPU Canvases**: `maskCanvas`, `rgbCanvas` → **GPU textures only** (`maskTexture`, `rgbTexture` persistent). +2. **Stroke Pipeline**: + - Points → dynamic `GPUBuffer` (vertex data). + - Fixed quad **index buffer** (static VBO for quad). + - Render stroke to temp texture with `src-over` blending. + - Clamp temp texture alpha to brush opacity using compute shader. + - Composite temp texture to main layer with `src-over`. +3. **No Preview During Stroke**: Thin polyline (CPU canvas) + fat GPU stroke async. +4. **Async Everything**: `submit()` non-blocking; readback **only** for history/export. +5. **History**: GPU texture snapshots (`copyTextureToTexture`) on `drawEnd`. +6. **Display**: Blit GPU layers → visible `imgCanvas` (CPU) via staging (once/frame or dirty). + +### Core Code Rework (`useBrushDrawing.ts`) + +This implementation uses **TypeGPU** (https://docs.swmansion.com/typegpu/) for type-safe WebGPU programming, with shaders defined as TypeScript functions that compile to WGSL. Full TypeGPU documentation is available via the MCP server "typegpu docs". + +```typescript +import tgpu from 'typegpu'; +import * as d from 'typegpu/data'; + +const root = await tgpu.init(); + +// Stroke vertex: per-point attributes +const StrokeVertex = d.struct({ + pos: d.location(0, d.vec2f), // Center x,y + size_opacity: d.location(1, d.vec2f), + color: d.location(2, d.vec3f), +}); + +// Vertex output +const VertexOut = d.struct({ + @builtin.position position: d.vec4f, + fragCenter: d.location(0, d.vec2f), + fragDistOffset: d.location(1, d.vec2f), // Relative to center + fragColor: d.location(2, d.vec3f), + fragOpacity: d.location(3, d.f32), +}); + +// Static quad vertices/indices (shared) +const QUAD_VERTICES = new Float32Array([-1,-1, 1,-1, 1,1, -1,1]); // NDC quad, 4 vec2f +const QUAD_INDICES = new Uint16Array([0,1,2, 0,2,3]); // Triangles +``` + +#### 1. New Data Structures +```typescript +// Stroke vertex: per-point attributes +interface StrokeVertex { + x: number; y: number; // Center + size: number; opacity: number; r: f32; g: f32; b: f32; +} + +// Static quad vertices/indices (shared) +const QUAD_VERTICES = new Float32Array([-1,-1, 1,-1, 1,1, -1,1]); // NDC quad +const QUAD_INDICES = new Uint16Array([0,1,2, 0,2,3]); // Triangles +``` + +#### 2. GPU Resources (Extend Existing) +```typescript +// Existing: maskTexture, rgbTexture (storage=renderable, created via TypeGPU root.createTexture) +// NEW: TypeGPU pipelines, layouts, buffers +let renderPipeline; // TypeGPUPipeline +let quadVertexLayout; +let strokeInstanceLayout; +let quadVertexBuffer; +let quadIndexBuffer; +let strokeVertexBuffer; // Preallocated large buffer + +let strokeTempTexture; // root.createTexture +let clampComputePipeline; +let clampBindGroupLayout; +let clampBindGroup; +let brushOpacityBuffer; // uniform f32 +``` + +#### 3. Shaders (TypeGPU TS Functions -> WGSL) +```typescript +// Vertex shader as TypeGPU function +const strokeVertexFn = tgpu['~unstable'].vertexFn({ + in: { + quadPos: d.vec2f, + strokeData: StrokeVertex, + }, + out: VertexOut, +})(({ quadPos, strokeData }) => { + const position = d.vec4f(strokeData.pos + quadPos * strokeData.size_opacity.x, 0.0, 1.0); + return { + position, + fragCenter: strokeData.pos, + fragDistOffset: quadPos, + fragColor: strokeData.color, + fragOpacity: strokeData.size_opacity.y, + }; +}); + +// Fragment shader (assumes params uniform bound separately) +const strokeFragmentFn = tgpu['~unstable'].fragmentFn({ + in: VertexOut, // Auto-matched from vertex out + out: d.vec4f, +})(({ fragCenter, fragDistOffset, fragColor, fragOpacity }) => { + const dist = d.length(fragDistOffset) * 0.5; + const hardRadius = 0.5 * params.hardness; // From uniform bind group + const fadeRange = 0.5 - hardRadius; + + let alpha = 0.0 as d.f32; + if (dist <= hardRadius) { + alpha = fragOpacity; + } else if (dist <= 0.5) { + const fade = (dist - hardRadius) / fadeRange; + alpha = fragOpacity * (1.0 - fade); + } else { + d.discard(); + } + + return d.vec4f(fragColor, alpha); +}); + +// Clamp compute shader +const clampComputeFn = tgpu['~unstable'].computeFn({ + in: { gid: d.builtin.globalInvocationId }, + workgroupSize: [8, 8], +})(({ gid }) => { + const texSize = d.textureDimensions(strokeTemp); + if (gid.x >= texSize.x || gid.y >= texSize.y) { return; } + + const color = d.textureLoad(strokeTemp, d.vec2i(gid.xy), 0); + const clampedAlpha = d.min(color.a, brushOpacity); + d.textureStore(strokeTemp, d.vec2i(gid.xy), 0, d.vec4f(color.rgb, clampedAlpha)); +}); +``` + + +#### 4. Pipeline Init (`initStrokeRenderPipeline`) +```typescript +const initStrokeRenderPipeline = () => { + // Vertex layouts + quadVertexLayout = tgpu.vertexLayout(d.arrayOf(d.vec2f), 'vertex'); + strokeInstanceLayout = tgpu.vertexLayout(d.disarrayOf(StrokeVertex, 1024), 'instance'); + + // Static buffers + quadVertexBuffer = root.createBuffer(d.arrayOf(d.vec2f, 4), QUAD_VERTICES).$usage('vertex'); + quadIndexBuffer = root.createBuffer(d.arrayOf(d.u16, 6), QUAD_INDICES).$usage('index'); + strokeVertexBuffer = root.createBuffer(d.disarrayOf(StrokeVertex, 1024)).$usage('vertex'); + + // Render pipeline (src-over blend) + renderPipeline = root['~unstable'] + .withVertex(strokeVertexFn, { + quadPos: quadVertexLayout.attrib, + strokeData: strokeInstanceLayout.attrib, + }) + .withFragment(strokeFragmentFn, { + format: 'rgba8unorm', + blend: { + color: { srcFactor: 'one', dstFactor: 'one-minus-src-alpha', operation: 'add' }, + alpha: { srcFactor: 'one', dstFactor: 'one-minus-src-alpha', operation: 'add' }, + }, + }) + .withPrimitive({ topology: 'triangle-list' }) + .createPipeline(); + + // Temp texture + strokeTempTexture = root.createTexture({ + size: [canvasWidth, canvasHeight], + format: 'rgba8unorm', + usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.STORAGE_BINDING, + }); + + // Clamp compute + clampBindGroupLayout = tgpu.bindGroupLayout({ + strokeTemp: { storageTexture: d.storageTexture2d('rgba8unorm') }, + brushOpacity: { uniform: d.f32 }, + }); + clampComputePipeline = root['~unstable'] + .withCompute(clampComputeFn) + .createPipeline(); +}; +``` + +#### 5. Stroke Render (`drawStrokeGPU` Rework) +```typescript +const drawStrokeGPU = async (points: Point[], opacities: number[]) => { + if (points.length === 0) return; + + // CPU: Pack vertices (batch) - TypeGPU write + const verticesData = new Float32Array(points.length * 7); // Matches StrokeVertex stride + for (let i = 0; i < points.length; i++) { + const off = i * 7; + verticesData[off + 0] = points[i].x; verticesData[off + 1] = points[i].y; + verticesData[off + 2] = store.brushSettings.size; verticesData[off + 3] = opacities[i]; + const {r,g,b} = parseToRgb(store.rgbColor); + verticesData[off + 4] = r/255; verticesData[off + 5] = g/255; verticesData[off + 6] = b/255; + } + strokeVertexBuffer.write(verticesData); // TypeGPU buffer update + + const brushOpacity = store.brushSettings.opacity / 100; + + // Update uniform + brushOpacityBuffer.write(new Float32Array([brushOpacity])); + + const clampBindGroup = root.createBindGroup(clampBindGroupLayout, { + strokeTemp: strokeTempTexture, + brushOpacity: brushOpacityBuffer, + }); + + // Step 1: Render stroke to temp texture (high-level TypeGPU) + root.clearTexture(strokeTempTexture.createView(), {r:0,g:0,b:0,a:0}); + renderPipeline + .withIndexBuffer(quadIndexBuffer) + .with(quadVertexLayout, quadVertexBuffer) + .with(strokeInstanceLayout, strokeVertexBuffer.slice(0, points.length)) // Dynamic slice + .withColorAttachment({ + view: strokeTempTexture.createView(), + loadOp: 'load', // Already cleared + storeOp: 'store', + }) + .drawIndexed(6, points.length); + + // Step 2: Clamp alpha + clampComputePipeline.with(clampBindGroup).dispatchWorkgroups( + Math.ceil(canvasWidth / 8), Math.ceil(canvasHeight / 8) + ); + + // Step 3: Composite (use separate fullscreen composite pipeline, details omitted) + // compositePipeline.with(...).draw(6); // Fullscreen quad blitting temp -> target +}; +``` +```typescript +const drawStrokeGPU = async (points: Point[], opacities: number[]) => { + if (points.length === 0) return; + + // CPU: Pack vertices (batch) + const vertices = new Float32Array(points.length * 7); // x,y,size,opacity,r,g,b + for (let i = 0; i < points.length; i++) { + const off = i * 7; + vertices[off + 0] = points[i].x; vertices[off + 1] = points[i].y; + vertices[off + 2] = store.brushSettings.size; vertices[off + 3] = opacities[i]; + // Pack color: parseToRgb(store.rgbColor || maskColor) + const {r,g,b} = parseToRgb(store.rgbColor); + vertices[off + 4] = r/255; vertices[off + 5] = g/255; vertices[off + 6] = b/255; + } + device!.queue.writeBuffer(strokeVertexBuffer, 0, vertices); + + const commandEncoder = device!.createCommandEncoder(); + + // Step 1: Render stroke to temp texture with src-over (accumulate overlaps) + const strokePass = commandEncoder.beginRenderPass({ + colorAttachments: [{ + view: strokeTempTexture.createView(), + loadOp: 'clear', storeOp: 'store', // Clear temp for each stroke + clearValue: { r: 0, g: 0, b: 0, a: 0 }, + }], + }); + strokePass.setPipeline(renderPipeline!); + strokePass.setVertexBuffer(0, quadVertexBuffer); + strokePass.setVertexBuffer(1, strokeVertexBuffer); + strokePass.setIndexBuffer(quadIndexBuffer, 'uint16'); + strokePass.drawIndexed(6, points.length); + strokePass.end(); + + // Step 2: Clamp alpha in temp texture to brush opacity + const brushOpacity = store.brushSettings.opacity / 100; // Assume 0-100 scale + device!.queue.writeBuffer(clampUniformBuffer, 0, new Float32Array([brushOpacity])); + const computePass = commandEncoder.beginComputePass(); + computePass.setPipeline(clampComputePipeline); + computePass.setBindGroup(0, clampBindGroup); + computePass.dispatchWorkgroups(Math.ceil(canvasWidth / 8), Math.ceil(canvasHeight / 8)); + computePass.end(); + + // Step 3: Composite temp to main layer with src-over + const targetTexture = store.activeLayer === 'rgb' ? rgbTexture! : maskTexture!; + const compositePass = commandEncoder.beginRenderPass({ + colorAttachments: [{ + view: targetTexture.createView(), + loadOp: 'load', storeOp: 'store', // Preserve prior content + }], + }); + // Use a full-screen quad render to composite temp onto target (src-over) + // Assuming a simple composite shader that samples temp and blends + // (Details omitted for brevity; similar to render pipeline but full-screen) + compositePass.end(); + + device!.queue.submit([commandEncoder.finish()]); +}; +``` + +#### 6. Integration Changes +- **`src/components/maskeditor/MaskEditorContent.vue`**: Remove CPU canvas refs for drawing (keep `imgCanvas` for display blit). Call `compositeGPULayers()` in RAF for display. +- **Stores**: `maskCtx`/`rgbCtx` → obsolete; use GPU compositing for history (`copyTextureToTexture(maskTexture, historyTexture)`). +- **PanZoom/Display**: New `compositeGPULayers()` → blit layers → `imgCanvas` (throttled RAF). +- **Erasing**: For erasing, render to temp with negative alpha or use dst-out blend, then clamp (ensure clamped alpha doesn't go negative). +- **Smoothing**: Unchanged, but mini-batch `drawStrokeGPU(32pts)` per `mousemove`. + +### Implementation Phases +1. **Phase 1: Render Pipeline Basics** (1-2 days) + - Init pipeline/buffers/shaders, including temp texture and clamp compute. + - `drawStrokeGPU` with render to temp, clamp, composite. + - Test: Single stroke → inspect GPU texture (Chrome GPU inspector). + +2. **Phase 2: Layer Compositing + Display** (1 day) + - Persistent textures, blit chain to display canvas. + - Async `drawStrokeGPU` in `handleDrawing` (mini-batches). + +3. **Phase 3: Smoothing, Batching, Preview** (1 day) + - Thin CPU polyline preview (`ctx.lineTo()` 1px). + - Unlimited stroke → auto mini-batch. + +4. **Phase 4: Polish + Features** (1-2 days) + - Brush shapes (texture atlas in vertex). + - Undo/History (texture copies). + - Erase mode, hardness/opacity uniforms. + - Error fallback → pure canvas. + - Perf validate: <5ms/stroke (Chrome profiler). + +**Expected Outcome**: 60FPS on 4k canvas, tablet-smooth. Matches Photoshop perf natively in WebGPU. + +**Files to Update**: +- `useBrushDrawing.ts`: Full GPU rewrite (~400 LOC). +- `MaskEditorContent.vue`: Display compositing. +- `maskEditorStore.ts`: Remove CPU ctx refs. +- Tests: Add GPU perf assertions. diff --git a/package.json b/package.json index cdc8aadd991..4e820c71712 100644 --- a/package.json +++ b/package.json @@ -75,6 +75,7 @@ "@vitest/coverage-v8": "catalog:", "@vitest/ui": "catalog:", "@vue/test-utils": "catalog:", + "@webgpu/types": "catalog:", "cross-env": "catalog:", "eslint": "catalog:", "eslint-config-prettier": "catalog:", @@ -112,6 +113,7 @@ "typescript": "catalog:", "typescript-eslint": "catalog:", "unplugin-icons": "catalog:", + "unplugin-typegpu": "catalog:", "unplugin-vue-components": "catalog:", "uuid": "^11.1.0", "vite": "catalog:", @@ -175,6 +177,7 @@ "semver": "^7.7.2", "three": "^0.170.0", "tiptap-markdown": "^0.8.10", + "typegpu": "catalog:", "vue": "catalog:", "vue-i18n": "catalog:", "vue-router": "catalog:", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 1fc27afcdd9..ef8a6e0cfc6 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -123,6 +123,9 @@ catalogs: '@vueuse/integrations': specifier: ^13.9.0 version: 13.9.0 + '@webgpu/types': + specifier: ^0.1.66 + version: 0.1.66 algoliasearch: specifier: ^5.21.0 version: 5.21.0 @@ -243,6 +246,9 @@ catalogs: tw-animate-css: specifier: ^1.3.8 version: 1.3.8 + typegpu: + specifier: ^0.8.2 + version: 0.8.2 typescript: specifier: ^5.9.2 version: 5.9.2 @@ -252,6 +258,9 @@ catalogs: unplugin-icons: specifier: ^0.22.0 version: 0.22.0 + unplugin-typegpu: + specifier: 0.8.0 + version: 0.8.0 unplugin-vue-components: specifier: ^0.28.0 version: 0.28.0 @@ -458,6 +467,9 @@ importers: tiptap-markdown: specifier: ^0.8.10 version: 0.8.10(@tiptap/core@2.10.4(@tiptap/pm@2.10.4)) + typegpu: + specifier: 'catalog:' + version: 0.8.2 vue: specifier: 'catalog:' version: 3.5.13(typescript@5.9.2) @@ -555,6 +567,9 @@ importers: '@vue/test-utils': specifier: 'catalog:' version: 2.4.6 + '@webgpu/types': + specifier: 'catalog:' + version: 0.1.66 cross-env: specifier: 'catalog:' version: 10.1.0 @@ -666,6 +681,9 @@ importers: unplugin-icons: specifier: 'catalog:' version: 0.22.0(@vue/compiler-sfc@3.5.13) + unplugin-typegpu: + specifier: 'catalog:' + version: 0.8.0(typegpu@0.8.2) unplugin-vue-components: specifier: 'catalog:' version: 0.28.0(@babel/parser@7.28.4)(rollup@4.22.4)(vue@3.5.13(typescript@5.9.2)) @@ -1425,6 +1443,10 @@ packages: resolution: {integrity: sha512-Q/N6JNWvIvPnLDvjlE1OUBLPQHH6l3CltCEsHIujp45zQUSSh8K+gHnaEX45yAT1nyngnINhvWtzN+Nb9D8RAQ==} engines: {node: '>=6.9.0'} + '@babel/standalone@7.28.5': + resolution: {integrity: sha512-1DViPYJpRU50irpGMfLBQ9B4kyfQuL6X7SS7pwTeWeZX0mNkjzPi0XFqxCjSdddZXUQy4AhnQnnesA/ZHnvAdw==} + engines: {node: '>=6.9.0'} + '@babel/template@7.27.2': resolution: {integrity: sha512-LPDZ85aEJyYSd18/DkjNh4/y1ntkE5KwUHWTiqgRxruuZL2F1yuHligVHLvcHY2vMHXttKFpJn6LwfI7cw7ODw==} engines: {node: '>=6.9.0'} @@ -3784,8 +3806,8 @@ packages: peerDependencies: vue: ^3.5.0 - '@webgpu/types@0.1.51': - resolution: {integrity: sha512-ktR3u64NPjwIViNCck+z9QeyN0iPkQCUOQ07ZCV1RzlkfP+olLTeEZ95O1QHS+v4w9vJeY9xj/uJuSphsHy5rQ==} + '@webgpu/types@0.1.66': + resolution: {integrity: sha512-YA2hLrwLpDsRueNDXIMqN9NTzD6bCDkuXbOSe0heS+f8YE8usA6Gbv1prj81pzVHrbaAma7zObnIC+I6/sXJgA==} '@xstate/fsm@1.6.5': resolution: {integrity: sha512-b5o1I6aLNeYlU/3CPlj/Z91ybk1gUsKT+5NAJI+2W4UjvS5KLG28K9v5UvNoFVjHV8PajVZ00RH3vnjyQO7ZAw==} @@ -6029,6 +6051,10 @@ packages: resolution: {integrity: sha512-h5bgJWpxJNswbU7qCrV0tIKQCaS3blPDrqKWx+QxzuzL1zGUzij9XCWLrSLsJPu5t+eWA/ycetzYAO5IOMcWAQ==} hasBin: true + magic-string-ast@1.0.3: + resolution: {integrity: sha512-CvkkH1i81zl7mmb94DsRiFeG9V2fR2JeuK8yDgS8oiZSFa++wWLEgZ5ufEOyLHbvSbD1gTRKv9NdX69Rnvr9JA==} + engines: {node: '>=20.19.0'} + magic-string@0.30.19: resolution: {integrity: sha512-2N21sPY9Ws53PZvsEpVtNuSW+ScYbQdp4b9qUaL+9QkHUrGFKo56Lg9Emg5s9V/qrtNBmiR01sYhUOwu3H+VOw==} @@ -7392,6 +7418,14 @@ packages: tinybench@2.9.0: resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==} + tinyest-for-wgsl@0.1.3: + resolution: {integrity: sha512-Wm5ADG1UyDxykf42S1gLYP4U9e1QP/TdtJeovQi6y68zttpiFLKqQGioHmPs9Mjysh7YMSAr/Lpuk0cD2MVdGA==} + engines: {node: '>=12.20.0'} + + tinyest@0.1.2: + resolution: {integrity: sha512-aHRmouyowIq1P5jrTF+YK6pGX+WuvFtSCLbqk91yHnU3SWQRIcNIamZLM5XF6lLqB13AWz0PGPXRff2QGDsxIg==} + engines: {node: '>=12.20.0'} + tinyexec@0.3.2: resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==} @@ -7518,6 +7552,13 @@ packages: resolution: {integrity: sha512-3KS2b+kL7fsuk/eJZ7EQdnEmQoaho/r6KUef7hxvltNA5DR8NAUM+8wJMbJyZ4G9/7i3v5zPBIMN5aybAh2/Jg==} engines: {node: '>= 0.4'} + typed-binary@4.3.2: + resolution: {integrity: sha512-HT3pIBM2njCZUmeczDaQUUErGiM6GXFCqMsHegE12HCoBtvHCkfR10JJni0TeGOTnLilTd6YFyj+YhflqQDrDQ==} + + typegpu@0.8.2: + resolution: {integrity: sha512-wkMJWhJE0pSkw2G/FesjqjbtHkREyOKu1Zmyj19xfmaX5+65YFwgfQNKSK8CxqN4kJkP7JFelLDJTSYY536TYg==} + engines: {node: '>=12.20.0'} + typescript-eslint@8.44.0: resolution: {integrity: sha512-ib7mCkYuIzYonCq9XWF5XNw+fkj2zg629PSa9KNIQ47RXFF763S5BIX4wqz1+FLPogTZoiw8KmCiRPRa8bL3qw==} engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} @@ -7622,6 +7663,11 @@ packages: vue-template-es2015-compiler: optional: true + unplugin-typegpu@0.8.0: + resolution: {integrity: sha512-VJHdXSXGOkAx0WhwFczhVUjAI6HyDkrQXk20HnwyuzIE3FdqE5l9sJTCYZzoVGo3z8i/IA5TMHCDzzP0Bc97Cw==} + peerDependencies: + typegpu: ^0.8.0 + unplugin-vue-components@0.28.0: resolution: {integrity: sha512-jiTGtJ3JsRFBjgvyilfrX7yUoGKScFgbdNw+6p6kEXU+Spf/rhxzgvdfuMcvhCcLmflB/dY3pGQshYBVGOUx7Q==} engines: {node: '>=14'} @@ -8950,6 +8996,8 @@ snapshots: '@babel/runtime@7.28.4': {} + '@babel/standalone@7.28.5': {} + '@babel/template@7.27.2': dependencies: '@babel/code-frame': 7.27.1 @@ -10997,7 +11045,7 @@ snapshots: '@tweenjs/tween.js': 23.1.3 '@types/stats.js': 0.17.3 '@types/webxr': 0.5.20 - '@webgpu/types': 0.1.51 + '@webgpu/types': 0.1.66 fflate: 0.8.2 meshoptimizer: 0.18.1 @@ -11500,7 +11548,7 @@ snapshots: dependencies: vue: 3.5.13(typescript@5.9.2) - '@webgpu/types@0.1.51': {} + '@webgpu/types@0.1.66': {} '@xstate/fsm@1.6.5': {} @@ -13979,6 +14027,10 @@ snapshots: lz-string@1.5.0: {} + magic-string-ast@1.0.3: + dependencies: + magic-string: 0.30.19 + magic-string@0.30.19: dependencies: '@jridgewell/sourcemap-codec': 1.5.5 @@ -15834,6 +15886,12 @@ snapshots: tinybench@2.9.0: {} + tinyest-for-wgsl@0.1.3: + dependencies: + tinyest: 0.1.2 + + tinyest@0.1.2: {} + tinyexec@0.3.2: {} tinyexec@1.0.1: {} @@ -15965,6 +16023,13 @@ snapshots: reflect.getprototypeof: 1.0.10 optional: true + typed-binary@4.3.2: {} + + typegpu@0.8.2: + dependencies: + tinyest: 0.1.2 + typed-binary: 4.3.2 + typescript-eslint@8.44.0(eslint@9.35.0(jiti@2.4.2))(typescript@5.9.2): dependencies: '@typescript-eslint/eslint-plugin': 8.44.0(@typescript-eslint/parser@8.44.0(eslint@9.35.0(jiti@2.4.2))(typescript@5.9.2))(eslint@9.35.0(jiti@2.4.2))(typescript@5.9.2) @@ -16060,6 +16125,19 @@ snapshots: transitivePeerDependencies: - supports-color + unplugin-typegpu@0.8.0(typegpu@0.8.2): + dependencies: + '@babel/standalone': 7.28.5 + defu: 6.1.4 + estree-walker: 3.0.3 + magic-string-ast: 1.0.3 + pathe: 2.0.3 + picomatch: 4.0.3 + tinyest: 0.1.2 + tinyest-for-wgsl: 0.1.3 + typegpu: 0.8.2 + unplugin: 2.3.5 + unplugin-vue-components@0.28.0(@babel/parser@7.28.4)(rollup@4.22.4)(vue@3.5.13(typescript@5.9.2)): dependencies: '@antfu/utils': 0.7.10 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index d2d081e8914..08acc11b095 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -42,6 +42,7 @@ catalog: '@vue/test-utils': ^2.4.6 '@vueuse/core': ^11.0.0 '@vueuse/integrations': ^13.9.0 + '@webgpu/types': ^0.1.66 algoliasearch: ^5.21.0 axios: ^1.8.2 cross-env: ^10.1.0 @@ -82,9 +83,11 @@ catalog: tailwindcss-primeui: ^0.6.1 tsx: ^4.15.6 tw-animate-css: ^1.3.8 + typegpu: ^0.8.2 typescript: ^5.9.2 typescript-eslint: ^8.44.0 unplugin-icons: ^0.22.0 + unplugin-typegpu: 0.8.0 unplugin-vue-components: ^0.28.0 vite: ^5.4.19 vite-plugin-dts: ^4.5.4 diff --git a/src/components/maskeditor/MaskEditorContent.vue b/src/components/maskeditor/MaskEditorContent.vue index 421ef8a548a..284a344cc56 100644 --- a/src/components/maskeditor/MaskEditorContent.vue +++ b/src/components/maskeditor/MaskEditorContent.vue @@ -149,6 +149,11 @@ const initUI = async () => { store.canvasHistory.saveInitialState() + // Initialize GPU resources after canvases are fully set up (Phase 1 prep) + if (toolManager?.brushDrawing) { + await toolManager.brushDrawing.initGPUResources() + } + initialized.value = true } catch (error) { console.error('[MaskEditorContent] Initialization failed:', error) @@ -172,6 +177,7 @@ onMounted(() => { }) onBeforeUnmount(() => { + toolManager?.brushDrawing.destroy() toolManager?.brushDrawing.saveBrushSettings() keyboard?.removeListeners() diff --git a/src/composables/maskeditor/useBrushDrawing.ts b/src/composables/maskeditor/useBrushDrawing.ts index c55d9b784f9..0bf6f7201a2 100644 --- a/src/composables/maskeditor/useBrushDrawing.ts +++ b/src/composables/maskeditor/useBrushDrawing.ts @@ -11,6 +11,20 @@ import { import type { Brush, Point } from '@/extensions/core/maskeditor/types' import { useMaskEditorStore } from '@/stores/maskEditorStore' import { useCoordinateTransform } from './useCoordinateTransform' +import TGPU from 'typegpu' + +// Phase 1 GPU stroke resources +let quadVertexBuffer: GPUBuffer | null = null +let quadIndexBuffer: GPUBuffer | null = null +let strokePosBuffer: GPUBuffer | null = null +let uniformBuffer: GPUBuffer | null = null +let strokeBindGroup: GPUBindGroup | null = null +let strokePipeline: GPURenderPipeline | null = null + +// GPU Resources (scope fix) +let maskTexture: GPUTexture | null = null +let rgbTexture: GPUTexture | null = null +let device: GPUDevice | null = null const saveBrushToCache = debounce(function (key: string, brush: Brush): void { try { @@ -43,81 +57,6 @@ export function useBrushDrawing(initialSettings?: { const coordinateTransform = useCoordinateTransform() - const brushTextureCache = new QuickLRU({ - maxSize: 8 - }) - - const SMOOTHING_MAX_STEPS = 30 - const SMOOTHING_MIN_STEPS = 2 - - const isDrawing = ref(false) - const isDrawingLine = ref(false) - const lineStartPoint = ref(null) - const smoothingCordsArray = ref([]) - const smoothingLastDrawTime = ref(new Date()) - const initialDraw = ref(true) - - const brushStrokeCanvas = ref(null) - const brushStrokeCtx = ref(null) - - const initialPoint = ref(null) - const useDominantAxis = ref(initialSettings?.useDominantAxis ?? false) - const brushAdjustmentSpeed = ref(initialSettings?.brushAdjustmentSpeed ?? 1.0) - - const cachedBrushSettings = loadBrushFromCache('maskeditor_brush_settings') - if (cachedBrushSettings) { - store.setBrushSize(cachedBrushSettings.size) - store.setBrushOpacity(cachedBrushSettings.opacity) - store.setBrushHardness(cachedBrushSettings.hardness) - store.brushSettings.type = cachedBrushSettings.type - store.setBrushSmoothingPrecision(cachedBrushSettings.smoothingPrecision) - } - - const createBrushStrokeCanvas = async (): Promise => { - if (brushStrokeCanvas.value !== null) { - return - } - - const maskCanvas = store.maskCanvas - if (!maskCanvas) { - throw new Error('Mask canvas not initialized') - } - - const canvas = document.createElement('canvas') - canvas.width = maskCanvas.width - canvas.height = maskCanvas.height - - brushStrokeCanvas.value = canvas - brushStrokeCtx.value = canvas.getContext('2d')! - } - - const initShape = (compositionOperation: CompositionOperation) => { - const blendMode = store.maskBlendMode - const mask_ctx = store.maskCtx - const rgb_ctx = store.rgbCtx - - if (!mask_ctx || !rgb_ctx) { - throw new Error('Canvas contexts are required') - } - - mask_ctx.beginPath() - rgb_ctx.beginPath() - - if (compositionOperation === CompositionOperation.SourceOver) { - mask_ctx.fillStyle = blendMode - mask_ctx.globalCompositeOperation = CompositionOperation.SourceOver - rgb_ctx.globalCompositeOperation = CompositionOperation.SourceOver - } else if (compositionOperation === CompositionOperation.DestinationOut) { - mask_ctx.globalCompositeOperation = CompositionOperation.DestinationOut - rgb_ctx.globalCompositeOperation = CompositionOperation.DestinationOut - } - } - - const formatRgba = (hex: string, alpha: number): string => { - const { r, g, b } = hexToRgb(hex) - return `rgba(${r}, ${g}, ${b}, ${alpha})` - } - const getCachedBrushTexture = ( radius: number, hardness: number, @@ -126,6 +65,11 @@ export function useBrushDrawing(initialSettings?: { ): HTMLCanvasElement => { const cacheKey = `${radius}_${hardness}_${color}_${opacity}` + // Create new cache for textures since old one removed + const brushTextureCache = new QuickLRU({ + maxSize: 8 + }) + if (brushTextureCache.has(cacheKey)) { return brushTextureCache.get(cacheKey)! } @@ -176,49 +120,318 @@ export function useBrushDrawing(initialSettings?: { return tempCanvas } - const createBrushGradient = ( - ctx: CanvasRenderingContext2D, - x: number, - y: number, - radius: number, - hardness: number, - color: string, - opacity: number, - isErasing: boolean - ): CanvasGradient => { - const gradient = ctx.createRadialGradient(x, y, 0, x, y, radius) + const SMOOTHING_MAX_STEPS = 30 + const SMOOTHING_MIN_STEPS = 2 - if (isErasing) { - gradient.addColorStop(0, `rgba(255, 255, 255, ${opacity})`) - gradient.addColorStop(hardness, `rgba(255, 255, 255, ${opacity * 0.5})`) - gradient.addColorStop(1, `rgba(255, 255, 255, 0)`) - } else { - const { r, g, b } = parseToRgb(color) - gradient.addColorStop(0, `rgba(${r}, ${g}, ${b}, ${opacity})`) - gradient.addColorStop( - hardness, - `rgba(${r}, ${g}, ${b}, ${opacity * 0.5})` - ) - gradient.addColorStop(1, `rgba(${r}, ${g}, ${b}, 0)`) + const isDrawing = ref(false) + const isDrawingLine = ref(false) + const lineStartPoint = ref(null) + const smoothingCordsArray = ref([]) + const smoothingLastDrawTime = ref(new Date()) + const initialDraw = ref(true) + + const initialPoint = ref(null) + const useDominantAxis = ref(initialSettings?.useDominantAxis ?? false) + const brushAdjustmentSpeed = ref(initialSettings?.brushAdjustmentSpeed ?? 1.0) + + const cachedBrushSettings = loadBrushFromCache('maskeditor_brush_settings') + if (cachedBrushSettings) { + store.setBrushSize(cachedBrushSettings.size) + store.setBrushOpacity(cachedBrushSettings.opacity) + store.setBrushHardness(cachedBrushSettings.hardness) + store.brushSettings.type = cachedBrushSettings.type + store.setBrushSmoothingPrecision(cachedBrushSettings.smoothingPrecision) + } + + // GPU Resources + const initTypeGPU = async (): Promise => { + if (store.tgpuRoot) return + + try { + const root = await TGPU.init() + store.tgpuRoot = root + device = root.device + console.warn('✅ TypeGPU initialized! Root:', root) + console.warn('Device info:', root.device.limits) + } catch (error) { + console.error('Failed to initialize TypeGPU:', error) } + } - return gradient + const initGPUResources = async (): Promise => { + // Ensure TypeGPU is initialized first + await initTypeGPU() + + if (!store.tgpuRoot || !device) { + console.warn('TypeGPU not initialized, skipping GPU resource setup') + return + } + + if ( + !store.maskCanvas || + !store.rgbCanvas || + !store.maskCtx || + !store.rgbCtx + ) { + console.warn('Canvas contexts not ready, skipping GPU resource setup') + return + } + + const canvasWidth = store.maskCanvas!.width + const canvasHeight = store.maskCanvas!.height + + try { + console.warn( + `🎨 Initializing GPU resources for ${canvasWidth}x${canvasHeight} canvas` + ) + + // Create read/write textures (RGBA8Unorm, copy from canvas) + maskTexture = device.createTexture({ + size: [canvasWidth, canvasHeight], + format: 'rgba8unorm', + usage: + GPUTextureUsage.TEXTURE_BINDING | + GPUTextureUsage.STORAGE_BINDING | + GPUTextureUsage.RENDER_ATTACHMENT | + GPUTextureUsage.COPY_DST | + GPUTextureUsage.COPY_SRC + }) + + rgbTexture = device.createTexture({ + size: [canvasWidth, canvasHeight], + format: 'rgba8unorm', + usage: + GPUTextureUsage.TEXTURE_BINDING | + GPUTextureUsage.STORAGE_BINDING | + GPUTextureUsage.RENDER_ATTACHMENT | + GPUTextureUsage.COPY_DST | + GPUTextureUsage.COPY_SRC + }) + + // Upload initial canvas data to GPU + const maskImageData = store.maskCtx.getImageData( + 0, + 0, + canvasWidth, + canvasHeight + ) + device.queue.writeTexture( + { texture: maskTexture }, + maskImageData.data, + { bytesPerRow: canvasWidth * 4 }, + { width: canvasWidth, height: canvasHeight } + ) + + const rgbImageData = store.rgbCtx.getImageData( + 0, + 0, + canvasWidth, + canvasHeight + ) + device.queue.writeTexture( + { texture: rgbTexture }, + rgbImageData.data, + { bytesPerRow: canvasWidth * 4 }, + { width: canvasWidth, height: canvasHeight } + ) + + console.warn('✅ GPU resources initialized successfully') + + // Phase 1: Initialize stroke render pipeline + console.warn('🎨 Phase 1: Initializing stroke pipeline & clamp...') + + // canvasWidth/Height from earlier + device = store.tgpuRoot!.device! + + // Quad verts (NDC) + const quadVerts = new Float32Array([-1, -1, 1, -1, 1, 1, -1, 1]) + quadVertexBuffer = device!.createBuffer({ + label: 'quad verts', + size: quadVerts.byteLength, + usage: GPUBufferUsage.VERTEX, + mappedAtCreation: true + }) + new Float32Array(quadVertexBuffer!.getMappedRange()).set(quadVerts) + quadVertexBuffer!.unmap() + + // Quad indices + const quadIdxs = new Uint16Array([0, 1, 2, 0, 2, 3]) + quadIndexBuffer = device!.createBuffer({ + label: 'quad idx', + size: quadIdxs.byteLength, + usage: GPUBufferUsage.INDEX, + mappedAtCreation: true + }) + new Uint16Array(quadIndexBuffer!.getMappedRange()).set(quadIdxs) + quadIndexBuffer!.unmap() + + // Stroke pos buffer (instance data) + strokePosBuffer = device!.createBuffer({ + label: 'stroke pos', + size: 4096 * 8, + usage: GPUBufferUsage.VERTEX | GPUBufferUsage.COPY_DST + }) + + // Stroke uniform (size, opacity, hardness, pad, r,g,b,pad) + uniformBuffer = device!.createBuffer({ + label: 'stroke uniform', + size: 32, + usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST + }) + + // Bind group layout + const strokeBgl = device!.createBindGroupLayout({ + label: 'stroke bgl', + entries: [ + { + binding: 0, + visibility: GPUShaderStage.VERTEX | GPUShaderStage.FRAGMENT, + buffer: { type: 'uniform' } + } + ] + }) + + // Stroke shader WGSL + const strokeWgsl = ` + struct Uniforms { + size: f32, + opacity: f32, + hardness: f32, + pad: f32, + r: f32, + g: f32, + b: f32, + pad2: f32, + } + @group(0) @binding(0) var u: Uniforms; + + struct VOut { + @builtin(position) pos: vec4, + @location(0) offset: vec2, + } + + @vertex + fn vs(@location(0) q: vec2, @location(1) p: vec2) -> VOut { + let off = q * u.size; + return VOut(vec4(p + off, 0,1), q); + } + + @fragment + fn fs(v: VOut) -> @location(0) vec4 { + let d = length(v.offset) * 0.5; + let hr = u.hardness * 0.5; + let fr = 0.5 - hr; + var a = 0.0; + if (d <= hr) { + a = u.opacity; + } else if (d <= 0.5) { + a = u.opacity * (1.0 - (d - hr) / fr); + } else { + discard; + } + return vec4(u.r, u.g, u.b, a); + } + ` + const strokeShader = device!.createShaderModule({ code: strokeWgsl }) + + // Stroke pipeline + strokePipeline = device!.createRenderPipeline({ + label: 'stroke splat', + layout: device!.createPipelineLayout({ bindGroupLayouts: [strokeBgl] }), + vertex: { + module: strokeShader, + entryPoint: 'vs', + buffers: [ + { + arrayStride: 8, + attributes: [ + { shaderLocation: 0, offset: 0, format: 'float32x2' } + ] + }, + { + arrayStride: 8, + stepMode: 'instance', + attributes: [ + { shaderLocation: 1, offset: 0, format: 'float32x2' } + ] + } + ] + }, + fragment: { + module: strokeShader, + entryPoint: 'fs', + targets: [ + { + format: 'rgba8unorm', + blend: { + color: { + operation: 'add', + srcFactor: 'src-alpha', + dstFactor: 'one-minus-src-alpha' + }, + alpha: { + operation: 'add', + srcFactor: 'src-alpha', + dstFactor: 'one-minus-src-alpha' + } + } + } + ] + }, + primitive: { topology: 'triangle-list' } + }) + + // Stroke bind group + strokeBindGroup = device!.createBindGroup({ + layout: strokeBgl, + entries: [{ binding: 0, resource: { buffer: uniformBuffer! } }] + }) + + console.warn('✅ Phase 1 complete: stroke pipeline ready') + } catch (error) { + console.error('Failed to initialize GPU resources:', error) + // Reset to null on failure + maskTexture = null + rgbTexture = null + } } - const drawShapeOnContext = ( - ctx: CanvasRenderingContext2D, - brushType: BrushShape, - x: number, - y: number, - radius: number - ): void => { - ctx.beginPath() - if (brushType === BrushShape.Rect) { - ctx.rect(x - radius, y - radius, radius * 2, radius * 2) - } else { - ctx.arc(x, y, radius, 0, Math.PI * 2, false) + const drawShape = (point: Point, overrideOpacity?: number) => { + const brush = store.brushSettings + const mask_ctx = store.maskCtx + const rgb_ctx = store.rgbCtx + + if (!mask_ctx || !rgb_ctx) { + throw new Error('Canvas contexts are required') } - ctx.fill() + + const brushType = brush.type + const brushRadius = brush.size + const hardness = brush.hardness + const opacity = overrideOpacity ?? brush.opacity + + const isErasing = mask_ctx.globalCompositeOperation === 'destination-out' + const currentTool = store.currentTool + const isRgbLayer = store.activeLayer === 'rgb' + + if ( + isRgbLayer && + currentTool && + (currentTool === Tools.Eraser || currentTool === Tools.PaintPen) + ) { + drawRgbShape(rgb_ctx, point, brushType, brushRadius, hardness, opacity) + return + } + + drawMaskShape( + mask_ctx, + point, + brushType, + brushRadius, + hardness, + opacity, + isErasing + ) } const drawRgbShape = ( @@ -315,92 +528,54 @@ export function useBrushDrawing(initialSettings?: { drawShapeOnContext(ctx, brushType, x, y, brushRadius) } - const drawShape = (point: Point, overrideOpacity?: number) => { - const brush = store.brushSettings - const mask_ctx = store.maskCtx - const rgb_ctx = store.rgbCtx - - if (!mask_ctx || !rgb_ctx) { - throw new Error('Canvas contexts are required') - } - - const brushType = brush.type - const brushRadius = brush.size - const hardness = brush.hardness - const opacity = overrideOpacity ?? brush.opacity - - const isErasing = mask_ctx.globalCompositeOperation === 'destination-out' - const currentTool = store.currentTool - const isRgbLayer = store.activeLayer === 'rgb' - - if ( - isRgbLayer && - currentTool && - (currentTool === Tools.Eraser || currentTool === Tools.PaintPen) - ) { - drawRgbShape(rgb_ctx, point, brushType, brushRadius, hardness, opacity) - return + const drawShapeOnContext = ( + ctx: CanvasRenderingContext2D, + brushType: BrushShape, + x: number, + y: number, + radius: number + ): void => { + ctx.beginPath() + if (brushType === BrushShape.Rect) { + ctx.rect(x - radius, y - radius, radius * 2, radius * 2) + } else { + ctx.arc(x, y, radius, 0, Math.PI * 2, false) } - - drawMaskShape( - mask_ctx, - point, - brushType, - brushRadius, - hardness, - opacity, - isErasing - ) + ctx.fill() } - const clampSmoothingPrecision = (value: number): number => { - return Math.min(Math.max(value, 1), 100) + const formatRgba = (hex: string, alpha: number): string => { + const { r, g, b } = hexToRgb(hex) + return `rgba(${r}, ${g}, ${b}, ${alpha})` } - const generateEquidistantPoints = ( - points: Point[], - distance: number - ): Point[] => { - const result: Point[] = [] - const cumulativeDistances: number[] = [0] - - for (let i = 1; i < points.length; i++) { - const dx = points[i].x - points[i - 1].x - const dy = points[i].y - points[i - 1].y - const dist = Math.hypot(dx, dy) - cumulativeDistances[i] = cumulativeDistances[i - 1] + dist - } - - const totalLength = cumulativeDistances[cumulativeDistances.length - 1] - const numPoints = Math.floor(totalLength / distance) - - for (let i = 0; i <= numPoints; i++) { - const targetDistance = i * distance - let idx = 0 - - while ( - idx < cumulativeDistances.length - 1 && - cumulativeDistances[idx + 1] < targetDistance - ) { - idx++ - } - - if (idx >= points.length - 1) { - result.push(points[points.length - 1]) - continue - } - - const d0 = cumulativeDistances[idx] - const d1 = cumulativeDistances[idx + 1] - const t = (targetDistance - d0) / (d1 - d0) - - const x = points[idx].x + t * (points[idx + 1].x - points[idx].x) - const y = points[idx].y + t * (points[idx + 1].y - points[idx].y) + const createBrushGradient = ( + ctx: CanvasRenderingContext2D, + x: number, + y: number, + radius: number, + hardness: number, + color: string, + opacity: number, + isErasing: boolean + ): CanvasGradient => { + const gradient = ctx.createRadialGradient(x, y, 0, x, y, radius) - result.push({ x, y }) + if (isErasing) { + gradient.addColorStop(0, `rgba(255, 255, 255, ${opacity})`) + gradient.addColorStop(hardness, `rgba(255, 255, 255, ${opacity * 0.5})`) + gradient.addColorStop(1, `rgba(255, 255, 255, 0)`) + } else { + const { r, g, b } = parseToRgb(color) + gradient.addColorStop(0, `rgba(${r}, ${g}, ${b}, ${opacity})`) + gradient.addColorStop( + hardness, + `rgba(${r}, ${g}, ${b}, ${opacity * 0.5})` + ) + gradient.addColorStop(1, `rgba(${r}, ${g}, ${b}, 0)`) } - return result + return gradient } const drawWithBetterSmoothing = (point: Point): void => { @@ -466,6 +641,7 @@ export function useBrushDrawing(initialSettings?: { } } + // Pure canvas rendering for (const p of interpolatedPoints) { drawShape(p, interpolatedOpacity) } @@ -477,6 +653,78 @@ export function useBrushDrawing(initialSettings?: { } } + const clampSmoothingPrecision = (value: number): number => { + return Math.min(Math.max(value, 1), 100) + } + + const generateEquidistantPoints = ( + points: Point[], + distance: number + ): Point[] => { + const result: Point[] = [] + const cumulativeDistances: number[] = [0] + + for (let i = 1; i < points.length; i++) { + const dx = points[i].x - points[i - 1].x + const dy = points[i].y - points[i - 1].y + const dist = Math.hypot(dx, dy) + cumulativeDistances[i] = cumulativeDistances[i - 1] + dist + } + + const totalLength = cumulativeDistances[cumulativeDistances.length - 1] + const numPoints = Math.floor(totalLength / distance) + + for (let i = 0; i <= numPoints; i++) { + const targetDistance = i * distance + let idx = 0 + + while ( + idx < cumulativeDistances.length - 1 && + cumulativeDistances[idx + 1] < targetDistance + ) { + idx++ + } + + if (idx >= points.length - 1) { + result.push(points[points.length - 1]) + continue + } + + const d0 = cumulativeDistances[idx] + const d1 = cumulativeDistances[idx + 1] + const t = (targetDistance - d0) / (d1 - d0) + + const x = points[idx].x + t * (points[idx + 1].x - points[idx].x) + const y = points[idx].y + t * (points[idx + 1].y - points[idx].y) + + result.push({ x, y }) + } + + return result + } + + const initShape = (compositionOperation: CompositionOperation) => { + const blendMode = store.maskBlendMode + const mask_ctx = store.maskCtx + const rgb_ctx = store.rgbCtx + + if (!mask_ctx || !rgb_ctx) { + throw new Error('Canvas contexts are required') + } + + mask_ctx.beginPath() + rgb_ctx.beginPath() + + if (compositionOperation === CompositionOperation.SourceOver) { + mask_ctx.fillStyle = blendMode + mask_ctx.globalCompositeOperation = CompositionOperation.SourceOver + rgb_ctx.globalCompositeOperation = CompositionOperation.SourceOver + } else if (compositionOperation === CompositionOperation.DestinationOut) { + mask_ctx.globalCompositeOperation = CompositionOperation.DestinationOut + rgb_ctx.globalCompositeOperation = CompositionOperation.DestinationOut + } + } + const drawLine = async ( p1: Point, p2: Point, @@ -517,8 +765,6 @@ export function useBrushDrawing(initialSettings?: { const coords = { x: event.offsetX, y: event.offsetY } const coords_canvas = coordinateTransform.screenToCanvas(coords) - await createBrushStrokeCanvas() - if (currentTool === 'eraser' || event.buttons === 2) { compositionOp = CompositionOperation.DestinationOut } else { @@ -586,6 +832,7 @@ export function useBrushDrawing(initialSettings?: { if (isDrawing.value) { isDrawing.value = false + store.canvasHistory.saveState() lineStartPoint.value = coords_canvas initialDraw.value = true @@ -660,12 +907,198 @@ export function useBrushDrawing(initialSettings?: { saveBrushToCache('maskeditor_brush_settings', store.brushSettings) } + // Add drawStrokeGPU function before the return statement + const drawStrokeGPU = (points: Point[]) => { + if (!strokePipeline || points.length === 0 || !device) { + console.warn('Phase 1 GPU not ready') + return + } + + const canvasWidth = store.maskCanvas!.width + const canvasHeight = store.maskCanvas!.height + + const numPoints = points.length + const testSize = 300 // px (larger for visibility) + const testOpacity = 1.0 + const testHardness = 0.8 + + // Upload stroke positions (normalized to NDC) + const posData = new Float32Array(numPoints * 2) + for (let i = 0; i < numPoints; i++) { + posData[i * 2 + 0] = (2 * points[i].x) / canvasWidth - 1 + posData[i * 2 + 1] = 1 - (2 * points[i].y) / canvasHeight + } + device!.queue.writeBuffer(strokePosBuffer!, 0, posData) + + // Upload stroke uniform (size normalized to NDC) + const ndcSize = (2 * testSize) / canvasWidth + //const hardness = testHardness + let r = 1.0, + g = 0.0, + b = 0.0 // Red for visibility + if (store.activeLayer === 'rgb') { + const rgb = parseToRgb(store.rgbColor) + r = rgb.r / 255 + g = rgb.g / 255 + b = rgb.b / 255 + } + const uData = new Float32Array([ + ndcSize, + testOpacity, + testHardness, + 0, + r, + g, + b, + 0 + ]) + device!.queue.writeBuffer(uniformBuffer!, 0, uData) + + console.warn('uData:', uData) + + // Command encoder + const encoder = device!.createCommandEncoder({ + label: 'phase1 stroke direct' + }) + + // Render stroke directly to target layer (DEBUG: clear green first) + const targetTexture = + store.activeLayer === 'rgb' ? rgbTexture! : maskTexture! + const strokePass = encoder.beginRenderPass({ + colorAttachments: [ + { + view: targetTexture.createView(), + loadOp: 'clear', + storeOp: 'store', + clearValue: { r: 0.2, g: 0.8, b: 0.2, a: 1.0 } // Green bg + } + ] + }) + strokePass.setPipeline(strokePipeline!) + strokePass.setBindGroup(0, strokeBindGroup!) + strokePass.setIndexBuffer(quadIndexBuffer!, 'uint16') + strokePass.setVertexBuffer(0, quadVertexBuffer!) + strokePass.setVertexBuffer(1, strokePosBuffer!) + strokePass.drawIndexed(6, numPoints) + strokePass.end() + + device!.queue.submit([encoder.finish()]) + + console.warn( + `🎨 Phase 1 stroke: ${numPoints} points rendered directly to ${store.activeLayer} layer` + ) + } + + // Function to inspect GPU texture by copying back to canvas + const inspectTexture = async () => { + if ( + !device || + !maskTexture || + !rgbTexture || + !store.maskCanvas || + !store.rgbCanvas + ) { + console.warn('GPU resources not ready for inspection') + return + } + + const targetTexture = store.activeLayer === 'rgb' ? rgbTexture : maskTexture + const canvasWidth = store.maskCanvas.width + const canvasHeight = store.maskCanvas.height + + // Create staging buffer for readback + const buffer = device.createBuffer({ + size: canvasWidth * canvasHeight * 4, + usage: GPUBufferUsage.COPY_DST | GPUBufferUsage.MAP_READ + }) + + // Copy texture to buffer + const encoder = device.createCommandEncoder() + encoder.copyTextureToBuffer( + { texture: targetTexture }, + { buffer, bytesPerRow: canvasWidth * 4 }, + { width: canvasWidth, height: canvasHeight } + ) + device.queue.submit([encoder.finish()]) + + // Map and copy to canvas + await buffer.mapAsync(GPUMapMode.READ) + const data = new Uint8Array(buffer.getMappedRange()) + const imageData = new ImageData( + new Uint8ClampedArray(data), + canvasWidth, + canvasHeight + ) + + // Log sample pixels: stroke start/center/end + const samples = [ + { x: 100, y: 100 }, + { x: 125, y: 125 }, + { x: 150, y: 150 }, + { x: 200, y: 200 } + ] + for (const s of samples) { + const sx = Math.floor(s.x), + sy = Math.floor(s.y) + const idx = (sy * canvasWidth + sx) * 4 + console.warn( + `Pixel (${sx},${sy}): R=${data[idx] / 255},G=${data[idx + 1] / 255},B=${data[idx + 2] / 255},A=${data[idx + 3] / 255}` + ) + } + + const ctx = store.activeLayer === 'rgb' ? store.rgbCtx : store.maskCtx + if (ctx) { + ctx.putImageData(imageData, 0, 0) + console.warn( + `✅ Texture inspected: copied ${store.activeLayer} layer to canvas` + ) + } + + buffer.unmap() + } + + // Expose for testing in development + if (typeof window !== 'undefined') { + ;(window as any).drawStrokeGPU = drawStrokeGPU + ;(window as any).inspectTexture = inspectTexture + } + + const destroy = (): void => { + // Clean up GPU textures + if (maskTexture) { + maskTexture.destroy() + maskTexture = null + } + if (rgbTexture) { + rgbTexture.destroy() + rgbTexture = null + } + + if (store.tgpuRoot) { + console.warn('Destroying TypeGPU root:', store.tgpuRoot) + store.tgpuRoot.destroy() + store.tgpuRoot = null + device = null + } + + // Cleanup Phase 1 resources + if (quadVertexBuffer) quadVertexBuffer.destroy() + if (quadIndexBuffer) quadIndexBuffer.destroy() + if (strokePosBuffer) strokePosBuffer.destroy() + if (uniformBuffer) uniformBuffer.destroy() + + // bindgroups/pipelines auto released + } + return { startDrawing, handleDrawing, drawEnd, startBrushAdjustment, handleBrushAdjustment, - saveBrushSettings + saveBrushSettings, + destroy, + drawStrokeGPU, + initGPUResources } } diff --git a/src/stores/maskEditorStore.ts b/src/stores/maskEditorStore.ts index e6546ee9685..1adc48b7db0 100644 --- a/src/stores/maskEditorStore.ts +++ b/src/stores/maskEditorStore.ts @@ -70,6 +70,8 @@ export const useMaskEditorStore = defineStore('maskEditor', () => { const canvasHistory = useCanvasHistory(20) + const tgpuRoot = ref(null) + watch(maskCanvas, (canvas) => { if (canvas) { maskCtx.value = canvas.getContext('2d', { willReadFrequently: true }) @@ -243,6 +245,8 @@ export const useMaskEditorStore = defineStore('maskEditor', () => { canvasHistory, + tgpuRoot, + setBrushSize, setBrushOpacity, setBrushHardness, diff --git a/tsconfig.json b/tsconfig.json index 5470e2c2df8..ae06ca3d11d 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -44,7 +44,8 @@ "./node_modules" ], "types": [ - "vitest/globals" + "vitest/globals", + "@webgpu/types" ], "outDir": "./dist", "rootDir": "./" diff --git a/vite.config.mts b/vite.config.mts index cffe4451d43..0ae5c8f7990 100644 --- a/vite.config.mts +++ b/vite.config.mts @@ -10,6 +10,7 @@ import { FileSystemIconLoader } from 'unplugin-icons/loaders' import IconsResolver from 'unplugin-icons/resolver' import Icons from 'unplugin-icons/vite' import Components from 'unplugin-vue-components/vite' +import typegpuPlugin from 'unplugin-typegpu/vite' import { defineConfig } from 'vite' import type { ProxyOptions, UserConfig } from 'vite' import { createHtmlPlugin } from 'vite-plugin-html' @@ -231,6 +232,7 @@ export default defineConfig({ ? [vueDevTools(), vue(), createHtmlPlugin({})] : [vue()]), tailwindcss(), + typegpuPlugin({}), comfyAPIPlugin(IS_DEV), // Twitter/Open Graph meta tags plugin (cloud distribution only) { From edaf3d5e63e0cc35d73a6b2a831ba5a778ec23ad Mon Sep 17 00:00:00 2001 From: trsommer Date: Tue, 18 Nov 2025 18:43:30 +0100 Subject: [PATCH 02/17] gpu painting works on mouse up --- gpuRendering.md | 2 +- .../maskeditor/gpu/GPUBrushRenderer.ts | 264 +++++++++ src/composables/maskeditor/gpu/gpuSchema.ts | 25 + src/composables/maskeditor/useBrushDrawing.ts | 499 +++++------------- 4 files changed, 429 insertions(+), 361 deletions(-) create mode 100644 src/composables/maskeditor/gpu/GPUBrushRenderer.ts create mode 100644 src/composables/maskeditor/gpu/gpuSchema.ts diff --git a/gpuRendering.md b/gpuRendering.md index 35eac9cdbf4..52cc89fa244 100644 --- a/gpuRendering.md +++ b/gpuRendering.md @@ -114,7 +114,7 @@ fn main(gid) { ### Core Code Rework (`useBrushDrawing.ts`) -This implementation uses **TypeGPU** (https://docs.swmansion.com/typegpu/) for type-safe WebGPU programming, with shaders defined as TypeScript functions that compile to WGSL. Full TypeGPU documentation is available via the MCP server "typegpu docs". +This implementation uses **TypeGPU** (https://docs.swmansion.com/typegpu/) for type-safe WebGPU programming, with shaders defined as TypeScript functions that compile to WGSL. Full TypeGPU documentation is available via the MCP server "typegpu". There is also webgpu docs on the mcp server. ```typescript import tgpu from 'typegpu'; diff --git a/src/composables/maskeditor/gpu/GPUBrushRenderer.ts b/src/composables/maskeditor/gpu/GPUBrushRenderer.ts new file mode 100644 index 00000000000..a56bad7dac4 --- /dev/null +++ b/src/composables/maskeditor/gpu/GPUBrushRenderer.ts @@ -0,0 +1,264 @@ +import * as d from 'typegpu/data' +import { BrushUniforms, StrokePoint } from './gpuSchema' + +const QUAD_VERTS = new Float32Array([-1, -1, 1, -1, 1, 1, -1, 1]) +const QUAD_INDICES = new Uint16Array([0, 1, 2, 0, 2, 3]) + +const UNIFORM_SIZE = d.sizeOf(BrushUniforms) // 32 +const STROKE_STRIDE = d.sizeOf(StrokePoint) // 16 +const MAX_STROKES = 10000 + +export class GPUBrushRenderer { + private device: GPUDevice + + // Buffers + private quadVertexBuffer: GPUBuffer + private indexBuffer: GPUBuffer + private instanceBuffer: GPUBuffer + private uniformBuffer: GPUBuffer + + // Shaders & Pipeline + private vertexModule: GPUShaderModule + private fragmentModule: GPUShaderModule + private uniformBindGroupLayout: GPUBindGroupLayout + private uniformBindGroup: GPUBindGroup + private pipelineLayout: GPUPipelineLayout + private renderPipeline: GPURenderPipeline + + constructor(device: GPUDevice) { + this.device = device + + // Create raw buffers + this.quadVertexBuffer = device.createBuffer({ + size: QUAD_VERTS.byteLength, + usage: GPUBufferUsage.VERTEX, + mappedAtCreation: true + }) + new Float32Array(this.quadVertexBuffer.getMappedRange()).set(QUAD_VERTS) + this.quadVertexBuffer.unmap() + + this.indexBuffer = device.createBuffer({ + size: QUAD_INDICES.byteLength, + usage: GPUBufferUsage.INDEX, + mappedAtCreation: true + }) + new Uint16Array(this.indexBuffer.getMappedRange()).set(QUAD_INDICES) + this.indexBuffer.unmap() + + this.instanceBuffer = device.createBuffer({ + size: MAX_STROKES * STROKE_STRIDE, + usage: GPUBufferUsage.VERTEX | GPUBufferUsage.COPY_DST + }) + + this.uniformBuffer = device.createBuffer({ + size: UNIFORM_SIZE, + usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST + }) + + // WGSL Shaders + const vertexCode = ` +struct BrushUniforms { + brushColor: vec3, + brushOpacity: f32, + hardness: f32, + pad: f32, + screenSize: vec2, +}; + +struct VertexOutput { + @builtin(position) position: vec4, + @location(0) localUV: vec2, + @location(1) color: vec3, + @location(2) opacity: f32, + @location(3) hardness: f32, +}; + +@group(0) @binding(0) var globals: BrushUniforms; + +@vertex +fn vs( + @location(0) quadPos: vec2, + @location(1) pos: vec2, + @location(2) size: f32, + @location(3) pressure: f32 +) -> VertexOutput { + let radius = (size * pressure) / 2.0; + let pixelPos = pos + (quadPos * radius); + let ndcX = (pixelPos.x / globals.screenSize.x) * 2.0 - 1.0; + let ndcY = 1.0 - ((pixelPos.y / globals.screenSize.y) * 2.0); + return VertexOutput( + vec4(ndcX, ndcY, 0.0, 1.0), + quadPos, + globals.brushColor, + pressure, + globals.hardness + ); +} +` + + const fragmentCode = ` +struct VertexOutput { + @builtin(position) position: vec4, + @location(0) localUV: vec2, + @location(1) color: vec3, + @location(2) opacity: f32, + @location(3) hardness: f32, +}; + +@fragment +fn fs(v: VertexOutput) -> @location(0) vec4 { + let dist = length(v.localUV); + if (dist > 1.0) { discard; } + let edge = 1.0 - (v.hardness * 0.5); + let alphaShape = 1.0 - smoothstep(edge - 0.1, 1.0, dist); + return vec4(v.color, alphaShape * v.opacity); +} +` + + this.vertexModule = device.createShaderModule({ code: vertexCode }) + this.fragmentModule = device.createShaderModule({ code: fragmentCode }) + + // Bind group layout + this.uniformBindGroupLayout = device.createBindGroupLayout({ + entries: [ + { + binding: 0, + visibility: GPUShaderStage.VERTEX | GPUShaderStage.FRAGMENT, + buffer: { type: 'uniform' } + } + ] + }) + + this.uniformBindGroup = device.createBindGroup({ + layout: this.uniformBindGroupLayout, + entries: [ + { + binding: 0, + resource: { buffer: this.uniformBuffer } + } + ] + }) + + this.pipelineLayout = device.createPipelineLayout({ + bindGroupLayouts: [this.uniformBindGroupLayout] + }) + + // Render pipeline + this.renderPipeline = device.createRenderPipeline({ + layout: this.pipelineLayout, + vertex: { + module: this.vertexModule, + entryPoint: 'vs', + buffers: [ + { + arrayStride: 8, + stepMode: 'vertex', + attributes: [{ shaderLocation: 0, offset: 0, format: 'float32x2' }] + }, + { + arrayStride: 16, + stepMode: 'instance', + attributes: [ + { shaderLocation: 1, offset: 0, format: 'float32x2' }, // pos @loc1 + { shaderLocation: 2, offset: 8, format: 'float32' }, // size @loc2 + { shaderLocation: 3, offset: 12, format: 'float32' } // pressure @loc3 + ] + } + ] + }, + fragment: { + module: this.fragmentModule, + entryPoint: 'fs', + targets: [ + { + format: 'rgba8unorm', + blend: { + color: { + srcFactor: 'src-alpha', + dstFactor: 'one-minus-src-alpha', + operation: 'add' + }, + alpha: { + srcFactor: 'one', + dstFactor: 'one-minus-src-alpha', + operation: 'add' + } + } + } + ] + }, + primitive: { topology: 'triangle-list' } + }) + } + + public renderStroke( + targetView: GPUTextureView, + points: { x: number; y: number; pressure: number }[], + settings: { + size: number + opacity: number + hardness: number + color: [number, number, number] + width: number + height: number + } + ) { + if (points.length === 0 || points.length > MAX_STROKES) return + + // Write uniform + const uData = new Float32Array(UNIFORM_SIZE / 4) + uData[0] = settings.color[0] + uData[1] = settings.color[1] + uData[2] = settings.color[2] + uData[3] = settings.opacity + uData[4] = settings.hardness + uData[5] = 0 // pad + uData[6] = settings.width + uData[7] = settings.height + this.device.queue.writeBuffer(this.uniformBuffer, 0, uData) + + // Write instance data + const iData = new Float32Array(points.length * 4) + for (let i = 0; i < points.length; i++) { + iData[i * 4 + 0] = points[i].x + iData[i * 4 + 1] = points[i].y + iData[i * 4 + 2] = settings.size + iData[i * 4 + 3] = points[i].pressure + } + this.device.queue.writeBuffer(this.instanceBuffer, 0, iData) + + const encoder = this.device.createCommandEncoder() + + const renderPass = encoder.beginRenderPass({ + colorAttachments: [ + { + view: targetView, + loadOp: 'load', + storeOp: 'store' + } + ] + }) + + renderPass.setPipeline(this.renderPipeline) + renderPass.setBindGroup(0, this.uniformBindGroup) + renderPass.setVertexBuffer(0, this.quadVertexBuffer) + renderPass.setVertexBuffer(1, this.instanceBuffer) + renderPass.setIndexBuffer(this.indexBuffer, 'uint16') + renderPass.drawIndexed(6, points.length) + renderPass.end() + + this.device.queue.submit([encoder.finish()]) + } + + public destroy() { + this.quadVertexBuffer.destroy() + this.indexBuffer.destroy() + this.instanceBuffer.destroy() + this.uniformBuffer.destroy() + // Modules and layouts are auto-cleaned + } + + public getTempTexture() { + throw new Error('No temp texture, use direct render') + } +} diff --git a/src/composables/maskeditor/gpu/gpuSchema.ts b/src/composables/maskeditor/gpu/gpuSchema.ts new file mode 100644 index 00000000000..1051702c031 --- /dev/null +++ b/src/composables/maskeditor/gpu/gpuSchema.ts @@ -0,0 +1,25 @@ +import * as d from 'typegpu/data' + +// 1. Global Brush Settings (Uniforms) +export const BrushUniforms = d.struct({ + brushColor: d.vec3f, + brushOpacity: d.f32, + hardness: d.f32, + screenSize: d.vec2f // Width/Height of the canvas +}) + +// 2. Per-Point Instance Data (Batched) +export const StrokePoint = d.struct({ + pos: d.location(0, d.vec2f), // Center x,y + size: d.location(1, d.f32), // Diameter + pressure: d.location(2, d.f32) // 0.0 - 1.0 +}) + +// 3. Vertex Shader Output +export const VertexOutput = d.struct({ + position: d.builtin.position, + localUV: d.location(0, d.vec2f), + color: d.location(1, d.vec3f), + opacity: d.location(2, d.f32), + hardness: d.location(3, d.f32) +}) diff --git a/src/composables/maskeditor/useBrushDrawing.ts b/src/composables/maskeditor/useBrushDrawing.ts index 0bf6f7201a2..3c3779c95d7 100644 --- a/src/composables/maskeditor/useBrushDrawing.ts +++ b/src/composables/maskeditor/useBrushDrawing.ts @@ -12,19 +12,13 @@ import type { Brush, Point } from '@/extensions/core/maskeditor/types' import { useMaskEditorStore } from '@/stores/maskEditorStore' import { useCoordinateTransform } from './useCoordinateTransform' import TGPU from 'typegpu' - -// Phase 1 GPU stroke resources -let quadVertexBuffer: GPUBuffer | null = null -let quadIndexBuffer: GPUBuffer | null = null -let strokePosBuffer: GPUBuffer | null = null -let uniformBuffer: GPUBuffer | null = null -let strokeBindGroup: GPUBindGroup | null = null -let strokePipeline: GPURenderPipeline | null = null +import { GPUBrushRenderer } from './gpu/GPUBrushRenderer' // GPU Resources (scope fix) let maskTexture: GPUTexture | null = null let rgbTexture: GPUTexture | null = null let device: GPUDevice | null = null +let renderer: GPUBrushRenderer | null = null const saveBrushToCache = debounce(function (key: string, brush: Brush): void { try { @@ -237,157 +231,8 @@ export function useBrushDrawing(initialSettings?: { console.warn('✅ GPU resources initialized successfully') - // Phase 1: Initialize stroke render pipeline - console.warn('🎨 Phase 1: Initializing stroke pipeline & clamp...') - - // canvasWidth/Height from earlier - device = store.tgpuRoot!.device! - - // Quad verts (NDC) - const quadVerts = new Float32Array([-1, -1, 1, -1, 1, 1, -1, 1]) - quadVertexBuffer = device!.createBuffer({ - label: 'quad verts', - size: quadVerts.byteLength, - usage: GPUBufferUsage.VERTEX, - mappedAtCreation: true - }) - new Float32Array(quadVertexBuffer!.getMappedRange()).set(quadVerts) - quadVertexBuffer!.unmap() - - // Quad indices - const quadIdxs = new Uint16Array([0, 1, 2, 0, 2, 3]) - quadIndexBuffer = device!.createBuffer({ - label: 'quad idx', - size: quadIdxs.byteLength, - usage: GPUBufferUsage.INDEX, - mappedAtCreation: true - }) - new Uint16Array(quadIndexBuffer!.getMappedRange()).set(quadIdxs) - quadIndexBuffer!.unmap() - - // Stroke pos buffer (instance data) - strokePosBuffer = device!.createBuffer({ - label: 'stroke pos', - size: 4096 * 8, - usage: GPUBufferUsage.VERTEX | GPUBufferUsage.COPY_DST - }) - - // Stroke uniform (size, opacity, hardness, pad, r,g,b,pad) - uniformBuffer = device!.createBuffer({ - label: 'stroke uniform', - size: 32, - usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST - }) - - // Bind group layout - const strokeBgl = device!.createBindGroupLayout({ - label: 'stroke bgl', - entries: [ - { - binding: 0, - visibility: GPUShaderStage.VERTEX | GPUShaderStage.FRAGMENT, - buffer: { type: 'uniform' } - } - ] - }) - - // Stroke shader WGSL - const strokeWgsl = ` - struct Uniforms { - size: f32, - opacity: f32, - hardness: f32, - pad: f32, - r: f32, - g: f32, - b: f32, - pad2: f32, - } - @group(0) @binding(0) var u: Uniforms; - - struct VOut { - @builtin(position) pos: vec4, - @location(0) offset: vec2, - } - - @vertex - fn vs(@location(0) q: vec2, @location(1) p: vec2) -> VOut { - let off = q * u.size; - return VOut(vec4(p + off, 0,1), q); - } - - @fragment - fn fs(v: VOut) -> @location(0) vec4 { - let d = length(v.offset) * 0.5; - let hr = u.hardness * 0.5; - let fr = 0.5 - hr; - var a = 0.0; - if (d <= hr) { - a = u.opacity; - } else if (d <= 0.5) { - a = u.opacity * (1.0 - (d - hr) / fr); - } else { - discard; - } - return vec4(u.r, u.g, u.b, a); - } - ` - const strokeShader = device!.createShaderModule({ code: strokeWgsl }) - - // Stroke pipeline - strokePipeline = device!.createRenderPipeline({ - label: 'stroke splat', - layout: device!.createPipelineLayout({ bindGroupLayouts: [strokeBgl] }), - vertex: { - module: strokeShader, - entryPoint: 'vs', - buffers: [ - { - arrayStride: 8, - attributes: [ - { shaderLocation: 0, offset: 0, format: 'float32x2' } - ] - }, - { - arrayStride: 8, - stepMode: 'instance', - attributes: [ - { shaderLocation: 1, offset: 0, format: 'float32x2' } - ] - } - ] - }, - fragment: { - module: strokeShader, - entryPoint: 'fs', - targets: [ - { - format: 'rgba8unorm', - blend: { - color: { - operation: 'add', - srcFactor: 'src-alpha', - dstFactor: 'one-minus-src-alpha' - }, - alpha: { - operation: 'add', - srcFactor: 'src-alpha', - dstFactor: 'one-minus-src-alpha' - } - } - } - ] - }, - primitive: { topology: 'triangle-list' } - }) - - // Stroke bind group - strokeBindGroup = device!.createBindGroup({ - layout: strokeBgl, - entries: [{ binding: 0, resource: { buffer: uniformBuffer! } }] - }) - - console.warn('✅ Phase 1 complete: stroke pipeline ready') + renderer = new GPUBrushRenderer(device!) + console.warn('✅ Brush renderer initialized') } catch (error) { console.error('Failed to initialize GPU resources:', error) // Reset to null on failure @@ -578,50 +423,32 @@ export function useBrushDrawing(initialSettings?: { return gradient } - const drawWithBetterSmoothing = (point: Point): void => { - if (!smoothingCordsArray.value) { - smoothingCordsArray.value = [] - } - - const opacityConstant = 1 / (1 + Math.exp(3)) - const interpolatedOpacity = - 1 / (1 + Math.exp(-6 * (store.brushSettings.opacity - 0.5))) - - opacityConstant - + const drawWithBetterSmoothing = async (point: Point): Promise => { smoothingCordsArray.value.push(point) const POINTS_NR = 5 - if (smoothingCordsArray.value.length < POINTS_NR) { - return - } + if (smoothingCordsArray.value.length < POINTS_NR) return let totalLength = 0 const points = smoothingCordsArray.value const len = points.length - 1 - - let dx, dy for (let i = 0; i < len; i++) { - dx = points[i + 1].x - points[i].x - dy = points[i + 1].y - points[i].y + const dx = points[i + 1].x - points[i].x + const dy = points[i + 1].y - points[i].y totalLength += Math.sqrt(dx * dx + dy * dy) } - const maxSteps = SMOOTHING_MAX_STEPS - const minSteps = SMOOTHING_MIN_STEPS - const smoothing = clampSmoothingPrecision( store.brushSettings.smoothingPrecision ) const normalizedSmoothing = (smoothing - 1) / 99 - const stepNr = Math.round( - Math.round(minSteps + (maxSteps - minSteps) * normalizedSmoothing) + SMOOTHING_MIN_STEPS + + (SMOOTHING_MAX_STEPS - SMOOTHING_MIN_STEPS) * normalizedSmoothing ) - const distanceBetweenPoints = totalLength / stepNr let interpolatedPoints = points - if (stepNr > 0) { interpolatedPoints = generateEquidistantPoints( smoothingCordsArray.value, @@ -635,15 +462,46 @@ export function useBrushDrawing(initialSettings?: { p.x === smoothingCordsArray.value[2].x && p.y === smoothingCordsArray.value[2].y ) - if (spliceIndex !== -1) { interpolatedPoints = interpolatedPoints.slice(spliceIndex + 1) } } - // Pure canvas rendering - for (const p of interpolatedPoints) { - drawShape(p, interpolatedOpacity) + // GPU render + if (renderer) { + const isErasing = + store.maskCtx!.globalCompositeOperation === 'destination-out' + if (isErasing) { + // Fallback CPU for erase + for (const p of interpolatedPoints) { + drawShape(p, 1) + } + } else { + const width = store.maskCanvas!.width + const height = store.maskCanvas!.height + const targetTexture = maskTexture! + const targetView = targetTexture.createView() + const colorStr = '#ffffff' + const { r, g, b } = parseToRgb(colorStr) + const strokePoints = interpolatedPoints.map((p) => ({ + x: p.x, + y: p.y, + pressure: 1 + })) + renderer!.renderStroke(targetView, strokePoints, { + size: store.brushSettings.size, + opacity: store.brushSettings.opacity, + hardness: store.brushSettings.hardness, + color: [r / 255, g / 255, b / 255], + width, + height + }) + } + } else { + // Fallback CPU + for (const p of interpolatedPoints) { + drawShape(p, 1) + } } if (!initialDraw.value) { @@ -730,29 +588,43 @@ export function useBrushDrawing(initialSettings?: { p2: Point, compositionOp: CompositionOperation ): Promise => { - try { - const brush_size = store.brushSettings.size - const distance = Math.sqrt((p2.x - p1.x) ** 2 + (p2.y - p1.y) ** 2) - const steps = Math.ceil( - distance / ((brush_size / store.brushSettings.smoothingPrecision) * 4) - ) - const interpolatedOpacity = - 1 / (1 + Math.exp(-6 * (store.brushSettings.opacity - 0.5))) - - 1 / (1 + Math.exp(3)) - + const brush_size = store.brushSettings.size + const distance = Math.hypot(p2.x - p1.x, p2.y - p1.y) + const steps = Math.ceil( + distance / ((brush_size / store.brushSettings.smoothingPrecision) * 4) + ) + const interpolatedOpacity = + 1 / (1 + Math.exp(-6 * (store.brushSettings.opacity - 0.5))) - + 1 / (1 + Math.exp(3)) + + const points: Point[] = [] + for (let i = 0; i <= steps; i++) { + const t = i / steps + points.push({ x: p1.x + (p2.x - p1.x) * t, y: p1.y + (p2.y - p1.y) * t }) + } + + if (renderer && compositionOp === CompositionOperation.SourceOver) { + const width = store.maskCanvas!.width + const height = store.maskCanvas!.height + const targetTexture = maskTexture! + const targetView = targetTexture.createView() + const colorStr = '#ffffff' + const { r, g, b } = parseToRgb(colorStr) + const strokePoints = points.map((p) => ({ x: p.x, y: p.y, pressure: 1 })) + renderer!.renderStroke(targetView, strokePoints, { + size: store.brushSettings.size, + opacity: store.brushSettings.opacity, + hardness: store.brushSettings.hardness, + color: [r / 255, g / 255, b / 255], + width, + height + }) + } else { + // CPU fallback initShape(compositionOp) - - for (let i = 0; i <= steps; i++) { - const t = i / steps - const x = p1.x + (p2.x - p1.x) * t - const y = p1.y + (p2.y - p1.y) * t - const point = { x, y } - + for (const point of points) { drawShape(point, interpolatedOpacity) } - } catch (error) { - console.error('[useBrushDrawing] Failed to draw line:', error) - throw error } } @@ -777,7 +649,7 @@ export function useBrushDrawing(initialSettings?: { } else { isDrawingLine.value = false initShape(compositionOp) - drawShape(coords_canvas) + await gpuDrawPoint(coords_canvas) } lineStartPoint.value = coords_canvas @@ -798,25 +670,24 @@ export function useBrushDrawing(initialSettings?: { const currentTool = store.currentTool if (diff > 20 && !isDrawing.value) { - requestAnimationFrame(() => { + requestAnimationFrame(async () => { try { initShape(CompositionOperation.SourceOver) - drawShape(coords_canvas) + await gpuDrawPoint(coords_canvas) smoothingCordsArray.value.push(coords_canvas) } catch (error) { console.error('[useBrushDrawing] Drawing error:', error) } }) } else { - requestAnimationFrame(() => { + requestAnimationFrame(async () => { try { if (currentTool === 'eraser' || event.buttons === 2) { initShape(CompositionOperation.DestinationOut) } else { initShape(CompositionOperation.SourceOver) } - - drawWithBetterSmoothing(coords_canvas) + await drawWithBetterSmoothing(coords_canvas) } catch (error) { console.error('[useBrushDrawing] Drawing error:', error) } @@ -836,6 +707,7 @@ export function useBrushDrawing(initialSettings?: { store.canvasHistory.saveState() lineStartPoint.value = coords_canvas initialDraw.value = true + await copyGpuToCanvas() } } @@ -907,164 +779,63 @@ export function useBrushDrawing(initialSettings?: { saveBrushToCache('maskeditor_brush_settings', store.brushSettings) } - // Add drawStrokeGPU function before the return statement - const drawStrokeGPU = (points: Point[]) => { - if (!strokePipeline || points.length === 0 || !device) { - console.warn('Phase 1 GPU not ready') - return - } - - const canvasWidth = store.maskCanvas!.width - const canvasHeight = store.maskCanvas!.height - - const numPoints = points.length - const testSize = 300 // px (larger for visibility) - const testOpacity = 1.0 - const testHardness = 0.8 - - // Upload stroke positions (normalized to NDC) - const posData = new Float32Array(numPoints * 2) - for (let i = 0; i < numPoints; i++) { - posData[i * 2 + 0] = (2 * points[i].x) / canvasWidth - 1 - posData[i * 2 + 1] = 1 - (2 * points[i].y) / canvasHeight - } - device!.queue.writeBuffer(strokePosBuffer!, 0, posData) - - // Upload stroke uniform (size normalized to NDC) - const ndcSize = (2 * testSize) / canvasWidth - //const hardness = testHardness - let r = 1.0, - g = 0.0, - b = 0.0 // Red for visibility - if (store.activeLayer === 'rgb') { - const rgb = parseToRgb(store.rgbColor) - r = rgb.r / 255 - g = rgb.g / 255 - b = rgb.b / 255 - } - const uData = new Float32Array([ - ndcSize, - testOpacity, - testHardness, - 0, - r, - g, - b, - 0 - ]) - device!.queue.writeBuffer(uniformBuffer!, 0, uData) - - console.warn('uData:', uData) - - // Command encoder - const encoder = device!.createCommandEncoder({ - label: 'phase1 stroke direct' - }) - - // Render stroke directly to target layer (DEBUG: clear green first) - const targetTexture = - store.activeLayer === 'rgb' ? rgbTexture! : maskTexture! - const strokePass = encoder.beginRenderPass({ - colorAttachments: [ - { - view: targetTexture.createView(), - loadOp: 'clear', - storeOp: 'store', - clearValue: { r: 0.2, g: 0.8, b: 0.2, a: 1.0 } // Green bg - } - ] - }) - strokePass.setPipeline(strokePipeline!) - strokePass.setBindGroup(0, strokeBindGroup!) - strokePass.setIndexBuffer(quadIndexBuffer!, 'uint16') - strokePass.setVertexBuffer(0, quadVertexBuffer!) - strokePass.setVertexBuffer(1, strokePosBuffer!) - strokePass.drawIndexed(6, numPoints) - strokePass.end() - - device!.queue.submit([encoder.finish()]) - - console.warn( - `🎨 Phase 1 stroke: ${numPoints} points rendered directly to ${store.activeLayer} layer` - ) - } - - // Function to inspect GPU texture by copying back to canvas - const inspectTexture = async () => { + const copyGpuToCanvas = async (): Promise => { if ( !device || !maskTexture || !rgbTexture || !store.maskCanvas || - !store.rgbCanvas - ) { - console.warn('GPU resources not ready for inspection') + !store.rgbCanvas || + !store.maskCtx || + !store.rgbCtx + ) return - } - const targetTexture = store.activeLayer === 'rgb' ? rgbTexture : maskTexture - const canvasWidth = store.maskCanvas.width - const canvasHeight = store.maskCanvas.height + const width = store.maskCanvas.width + const height = store.maskCanvas.height - // Create staging buffer for readback - const buffer = device.createBuffer({ - size: canvasWidth * canvasHeight * 4, + const bufferSize = width * height * 4 + const maskBuffer = device.createBuffer({ + size: bufferSize, + usage: GPUBufferUsage.COPY_DST | GPUBufferUsage.MAP_READ + }) + const rgbBuffer = device.createBuffer({ + size: bufferSize, usage: GPUBufferUsage.COPY_DST | GPUBufferUsage.MAP_READ }) - // Copy texture to buffer const encoder = device.createCommandEncoder() encoder.copyTextureToBuffer( - { texture: targetTexture }, - { buffer, bytesPerRow: canvasWidth * 4 }, - { width: canvasWidth, height: canvasHeight } + { texture: maskTexture }, + { buffer: maskBuffer, bytesPerRow: width * 4 }, + { width, height } ) - device.queue.submit([encoder.finish()]) - - // Map and copy to canvas - await buffer.mapAsync(GPUMapMode.READ) - const data = new Uint8Array(buffer.getMappedRange()) - const imageData = new ImageData( - new Uint8ClampedArray(data), - canvasWidth, - canvasHeight + encoder.copyTextureToBuffer( + { texture: rgbTexture }, + { buffer: rgbBuffer, bytesPerRow: width * 4 }, + { width, height } ) + device.queue.submit([encoder.finish()]) - // Log sample pixels: stroke start/center/end - const samples = [ - { x: 100, y: 100 }, - { x: 125, y: 125 }, - { x: 150, y: 150 }, - { x: 200, y: 200 } - ] - for (const s of samples) { - const sx = Math.floor(s.x), - sy = Math.floor(s.y) - const idx = (sy * canvasWidth + sx) * 4 - console.warn( - `Pixel (${sx},${sy}): R=${data[idx] / 255},G=${data[idx + 1] / 255},B=${data[idx + 2] / 255},A=${data[idx + 3] / 255}` - ) - } + await Promise.all([ + maskBuffer.mapAsync(GPUMapMode.READ), + rgbBuffer.mapAsync(GPUMapMode.READ) + ]) - const ctx = store.activeLayer === 'rgb' ? store.rgbCtx : store.maskCtx - if (ctx) { - ctx.putImageData(imageData, 0, 0) - console.warn( - `✅ Texture inspected: copied ${store.activeLayer} layer to canvas` - ) - } + const maskData = new Uint8ClampedArray(maskBuffer.getMappedRange()) + store.maskCtx.putImageData(new ImageData(maskData, width, height), 0, 0) - buffer.unmap() - } + const rgbData = new Uint8ClampedArray(rgbBuffer.getMappedRange()) + store.rgbCtx.putImageData(new ImageData(rgbData, width, height), 0, 0) - // Expose for testing in development - if (typeof window !== 'undefined') { - ;(window as any).drawStrokeGPU = drawStrokeGPU - ;(window as any).inspectTexture = inspectTexture + maskBuffer.unmap() + rgbBuffer.unmap() + maskBuffer.destroy() + rgbBuffer.destroy() } const destroy = (): void => { - // Clean up GPU textures + renderer?.destroy() if (maskTexture) { maskTexture.destroy() maskTexture = null @@ -1073,21 +844,30 @@ export function useBrushDrawing(initialSettings?: { rgbTexture.destroy() rgbTexture = null } - if (store.tgpuRoot) { - console.warn('Destroying TypeGPU root:', store.tgpuRoot) store.tgpuRoot.destroy() store.tgpuRoot = null - device = null } + device = null + } - // Cleanup Phase 1 resources - if (quadVertexBuffer) quadVertexBuffer.destroy() - if (quadIndexBuffer) quadIndexBuffer.destroy() - if (strokePosBuffer) strokePosBuffer.destroy() - if (uniformBuffer) uniformBuffer.destroy() - - // bindgroups/pipelines auto released + const gpuDrawPoint = async (point: Point, opacity: number = 1) => { + if (renderer) { + const width = store.maskCanvas!.width + const height = store.maskCanvas!.height + const targetView = maskTexture!.createView() + const strokePoints = [{ x: point.x, y: point.y, pressure: opacity }] + renderer!.renderStroke(targetView, strokePoints, { + size: store.brushSettings.size, + opacity: store.brushSettings.opacity, + hardness: store.brushSettings.hardness, + color: [1, 1, 1], // white for mask + width, + height + }) + } else { + drawShape(point, opacity) + } } return { @@ -1098,7 +878,6 @@ export function useBrushDrawing(initialSettings?: { handleBrushAdjustment, saveBrushSettings, destroy, - drawStrokeGPU, initGPUResources } } From fff8680719db098c3d1a4102cabf29fcc17536c4 Mon Sep 17 00:00:00 2001 From: trsommer Date: Tue, 18 Nov 2025 20:21:04 +0100 Subject: [PATCH 03/17] gpu painitng works much better now, but some things still need work --- .../maskeditor/MaskEditorContent.vue | 13 + .../maskeditor/dialog/TopBarHeader.vue | 1 + .../maskeditor/gpu/GPUBrushRenderer.ts | 209 +++++++++++---- src/composables/maskeditor/gpu/gpuSchema.ts | 2 +- src/composables/maskeditor/useBrushDrawing.ts | 242 ++++++++++++++---- src/stores/maskEditorStore.ts | 7 + 6 files changed, 361 insertions(+), 113 deletions(-) diff --git a/src/components/maskeditor/MaskEditorContent.vue b/src/components/maskeditor/MaskEditorContent.vue index 284a344cc56..066a018fc44 100644 --- a/src/components/maskeditor/MaskEditorContent.vue +++ b/src/components/maskeditor/MaskEditorContent.vue @@ -25,6 +25,11 @@ class="absolute top-0 left-0 w-full h-full" @contextmenu.prevent /> + +
@@ -87,6 +92,7 @@ const canvasContainerRef = ref() const imgCanvasRef = ref() const maskCanvasRef = ref() const rgbCanvasRef = ref() +const gpuCanvasRef = ref() const canvasBackgroundRef = ref() const toolPanelRef = ref>() @@ -152,6 +158,13 @@ const initUI = async () => { // Initialize GPU resources after canvases are fully set up (Phase 1 prep) if (toolManager?.brushDrawing) { await toolManager.brushDrawing.initGPUResources() + if (gpuCanvasRef.value && toolManager?.brushDrawing.initPreviewCanvas) { + // Fix: Ensure preview canvas matches the resolution of the mask canvas + gpuCanvasRef.value.width = maskCanvasRef.value.width + gpuCanvasRef.value.height = maskCanvasRef.value.height + + toolManager.brushDrawing.initPreviewCanvas(gpuCanvasRef.value) + } } initialized.value = true diff --git a/src/components/maskeditor/dialog/TopBarHeader.vue b/src/components/maskeditor/dialog/TopBarHeader.vue index 800cedca712..943b60170b6 100644 --- a/src/components/maskeditor/dialog/TopBarHeader.vue +++ b/src/components/maskeditor/dialog/TopBarHeader.vue @@ -102,6 +102,7 @@ const onInvert = () => { const onClear = () => { canvasTools.clearMask() + store.triggerClear() } const handleSave = async () => { diff --git a/src/composables/maskeditor/gpu/GPUBrushRenderer.ts b/src/composables/maskeditor/gpu/GPUBrushRenderer.ts index a56bad7dac4..293da647e09 100644 --- a/src/composables/maskeditor/gpu/GPUBrushRenderer.ts +++ b/src/composables/maskeditor/gpu/GPUBrushRenderer.ts @@ -17,18 +17,15 @@ export class GPUBrushRenderer { private instanceBuffer: GPUBuffer private uniformBuffer: GPUBuffer - // Shaders & Pipeline - private vertexModule: GPUShaderModule - private fragmentModule: GPUShaderModule - private uniformBindGroupLayout: GPUBindGroupLayout - private uniformBindGroup: GPUBindGroup - private pipelineLayout: GPUPipelineLayout + // Pipelines private renderPipeline: GPURenderPipeline + private blitPipeline: GPURenderPipeline + private uniformBindGroup: GPUBindGroup constructor(device: GPUDevice) { this.device = device - // Create raw buffers + // --- 1. Initialize Buffers --- this.quadVertexBuffer = device.createBuffer({ size: QUAD_VERTS.byteLength, usage: GPUBufferUsage.VERTEX, @@ -55,8 +52,8 @@ export class GPUBrushRenderer { usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST }) - // WGSL Shaders - const vertexCode = ` + // --- 2. Brush Shader (Drawing) --- + const brushVertex = ` struct BrushUniforms { brushColor: vec3, brushOpacity: f32, @@ -84,19 +81,22 @@ fn vs( ) -> VertexOutput { let radius = (size * pressure) / 2.0; let pixelPos = pos + (quadPos * radius); + + // Convert Pixel Space -> NDC let ndcX = (pixelPos.x / globals.screenSize.x) * 2.0 - 1.0; - let ndcY = 1.0 - ((pixelPos.y / globals.screenSize.y) * 2.0); + let ndcY = 1.0 - ((pixelPos.y / globals.screenSize.y) * 2.0); // Flip Y + return VertexOutput( vec4(ndcX, ndcY, 0.0, 1.0), quadPos, globals.brushColor, - pressure, + pressure * globals.brushOpacity, globals.hardness ); } ` - const fragmentCode = ` + const brushFragment = ` struct VertexOutput { @builtin(position) position: vec4, @location(0) localUV: vec2, @@ -109,17 +109,27 @@ struct VertexOutput { fn fs(v: VertexOutput) -> @location(0) vec4 { let dist = length(v.localUV); if (dist > 1.0) { discard; } - let edge = 1.0 - (v.hardness * 0.5); - let alphaShape = 1.0 - smoothstep(edge - 0.1, 1.0, dist); - return vec4(v.color, alphaShape * v.opacity); + + // Correct Hardness Math: + // Hardness 1.0 -> smoothstep(1.0, 1.0, dist) -> Sharp step at 1.0 + // Hardness 0.0 -> smoothstep(0.0, 1.0, dist) -> Linear fade from 0.0 + // 1.0 - smoothstep(...) inverts it so 1.0 is center, 0.0 is edge. + + let startFade = v.hardness * 0.99; // Prevent 1.0 singularity + let alphaShape = 1.0 - smoothstep(startFade, 1.0, dist); + + // Fix: Output Premultiplied Alpha + // This prevents "Black Glow" artifacts when blending semi-transparent colors. + // Standard compositing expects (r*a, g*a, b*a, a). + let alpha = alphaShape * v.opacity; + return vec4(v.color * alpha, alpha); } ` - this.vertexModule = device.createShaderModule({ code: vertexCode }) - this.fragmentModule = device.createShaderModule({ code: fragmentCode }) + const brushModuleV = device.createShaderModule({ code: brushVertex }) + const brushModuleF = device.createShaderModule({ code: brushFragment }) - // Bind group layout - this.uniformBindGroupLayout = device.createBindGroupLayout({ + const uniformBindGroupLayout = device.createBindGroupLayout({ entries: [ { binding: 0, @@ -130,51 +140,93 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { }) this.uniformBindGroup = device.createBindGroup({ - layout: this.uniformBindGroupLayout, - entries: [ - { - binding: 0, - resource: { buffer: this.uniformBuffer } - } - ] + layout: uniformBindGroupLayout, + entries: [{ binding: 0, resource: { buffer: this.uniformBuffer } }] }) - this.pipelineLayout = device.createPipelineLayout({ - bindGroupLayouts: [this.uniformBindGroupLayout] + const pipelineLayout = device.createPipelineLayout({ + bindGroupLayouts: [uniformBindGroupLayout] }) - // Render pipeline this.renderPipeline = device.createRenderPipeline({ - layout: this.pipelineLayout, + layout: pipelineLayout, vertex: { - module: this.vertexModule, + module: brushModuleV, entryPoint: 'vs', buffers: [ { arrayStride: 8, stepMode: 'vertex', - attributes: [{ shaderLocation: 0, offset: 0, format: 'float32x2' }] + attributes: [{ shaderLocation: 0, offset: 0, format: 'float32x2' }] // Quad }, { arrayStride: 16, stepMode: 'instance', attributes: [ - { shaderLocation: 1, offset: 0, format: 'float32x2' }, // pos @loc1 - { shaderLocation: 2, offset: 8, format: 'float32' }, // size @loc2 - { shaderLocation: 3, offset: 12, format: 'float32' } // pressure @loc3 + { shaderLocation: 1, offset: 0, format: 'float32x2' }, // pos + { shaderLocation: 2, offset: 8, format: 'float32' }, // size + { shaderLocation: 3, offset: 12, format: 'float32' } // pressure ] } ] }, fragment: { - module: this.fragmentModule, + module: brushModuleF, entryPoint: 'fs', targets: [ { format: 'rgba8unorm', + blend: { + // Fix: Premultiplied Alpha Blending + // Src * 1 + Dst * (1 - SrcAlpha) + color: { + srcFactor: 'one', + dstFactor: 'one-minus-src-alpha', + operation: 'add' + }, + alpha: { + srcFactor: 'one', + dstFactor: 'one-minus-src-alpha', + operation: 'add' + } + } + } + ] + }, + primitive: { topology: 'triangle-list' } + }) + + // --- 3. Blit Pipeline (For Preview) --- + const blitShader = ` + @vertex fn vs(@builtin(vertex_index) vIdx: u32) -> @builtin(position) vec4 { + var pos = array, 3>( + vec2(-1.0, -1.0), vec2(3.0, -1.0), vec2(-1.0, 3.0) + ); + return vec4(pos[vIdx], 0.0, 1.0); + } + + @group(0) @binding(0) var myTexture: texture_2d; + + @fragment fn fs(@builtin(position) pos: vec4) -> @location(0) vec4 { + return textureLoad(myTexture, vec2(pos.xy), 0); + } + ` + + this.blitPipeline = device.createRenderPipeline({ + layout: 'auto', + vertex: { + module: device.createShaderModule({ code: blitShader }), + entryPoint: 'vs' + }, + fragment: { + module: device.createShaderModule({ code: blitShader }), + entryPoint: 'fs', + targets: [ + { + format: navigator.gpu.getPreferredCanvasFormat(), blend: { color: { - srcFactor: 'src-alpha', + srcFactor: 'one', dstFactor: 'one-minus-src-alpha', operation: 'add' }, @@ -203,9 +255,9 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { height: number } ) { - if (points.length === 0 || points.length > MAX_STROKES) return + if (points.length === 0) return - // Write uniform + // 1. Update Uniforms const uData = new Float32Array(UNIFORM_SIZE / 4) uData[0] = settings.color[0] uData[1] = settings.color[1] @@ -217,9 +269,10 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { uData[7] = settings.height this.device.queue.writeBuffer(this.uniformBuffer, 0, uData) - // Write instance data - const iData = new Float32Array(points.length * 4) - for (let i = 0; i < points.length; i++) { + // 2. Batch Instance Data + const batchSize = Math.min(points.length, MAX_STROKES) + const iData = new Float32Array(batchSize * 4) + for (let i = 0; i < batchSize; i++) { iData[i * 4 + 0] = points[i].x iData[i * 4 + 1] = points[i].y iData[i * 4 + 2] = settings.size @@ -227,9 +280,9 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { } this.device.queue.writeBuffer(this.instanceBuffer, 0, iData) + // 3. Render Pass const encoder = this.device.createCommandEncoder() - - const renderPass = encoder.beginRenderPass({ + const pass = encoder.beginRenderPass({ colorAttachments: [ { view: targetView, @@ -239,26 +292,70 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { ] }) - renderPass.setPipeline(this.renderPipeline) - renderPass.setBindGroup(0, this.uniformBindGroup) - renderPass.setVertexBuffer(0, this.quadVertexBuffer) - renderPass.setVertexBuffer(1, this.instanceBuffer) - renderPass.setIndexBuffer(this.indexBuffer, 'uint16') - renderPass.drawIndexed(6, points.length) - renderPass.end() + pass.setPipeline(this.renderPipeline) + pass.setBindGroup(0, this.uniformBindGroup) + pass.setVertexBuffer(0, this.quadVertexBuffer) + pass.setVertexBuffer(1, this.instanceBuffer) + pass.setIndexBuffer(this.indexBuffer, 'uint16') + pass.drawIndexed(6, batchSize) + pass.end() + + this.device.queue.submit([encoder.finish()]) + } + + // New: Blit the working texture to a presentation canvas + public blitToCanvas( + sourceTexture: GPUTexture, + destinationCtx: GPUCanvasContext + ) { + const encoder = this.device.createCommandEncoder() + + // Dynamic BindGroup for the source texture + const bindGroup = this.device.createBindGroup({ + layout: this.blitPipeline.getBindGroupLayout(0), + entries: [{ binding: 0, resource: sourceTexture.createView() }] + }) + + const pass = encoder.beginRenderPass({ + colorAttachments: [ + { + view: destinationCtx.getCurrentTexture().createView(), + loadOp: 'clear', + clearValue: { r: 0, g: 0, b: 0, a: 0 }, + storeOp: 'store' + } + ] + }) + + pass.setPipeline(this.blitPipeline) + pass.setBindGroup(0, bindGroup) + pass.draw(3) + pass.end() this.device.queue.submit([encoder.finish()]) } + // Fix: Clear the preview canvas + public clearPreview(destinationCtx: GPUCanvasContext) { + const encoder = this.device.createCommandEncoder() + const pass = encoder.beginRenderPass({ + colorAttachments: [ + { + view: destinationCtx.getCurrentTexture().createView(), + loadOp: 'clear', + clearValue: { r: 0, g: 0, b: 0, a: 0 }, + storeOp: 'store' + } + ] + }) + pass.end() + this.device.queue.submit([encoder.finish()]) + } + public destroy() { this.quadVertexBuffer.destroy() this.indexBuffer.destroy() this.instanceBuffer.destroy() this.uniformBuffer.destroy() - // Modules and layouts are auto-cleaned - } - - public getTempTexture() { - throw new Error('No temp texture, use direct render') } } diff --git a/src/composables/maskeditor/gpu/gpuSchema.ts b/src/composables/maskeditor/gpu/gpuSchema.ts index 1051702c031..6091fddf0a6 100644 --- a/src/composables/maskeditor/gpu/gpuSchema.ts +++ b/src/composables/maskeditor/gpu/gpuSchema.ts @@ -5,7 +5,7 @@ export const BrushUniforms = d.struct({ brushColor: d.vec3f, brushOpacity: d.f32, hardness: d.f32, - screenSize: d.vec2f // Width/Height of the canvas + screenSize: d.vec2f }) // 2. Per-Point Instance Data (Batched) diff --git a/src/composables/maskeditor/useBrushDrawing.ts b/src/composables/maskeditor/useBrushDrawing.ts index 3c3779c95d7..a8d222f3fa6 100644 --- a/src/composables/maskeditor/useBrushDrawing.ts +++ b/src/composables/maskeditor/useBrushDrawing.ts @@ -1,4 +1,4 @@ -import { ref } from 'vue' +import { ref, watch } from 'vue' import QuickLRU from '@alloc/quick-lru' import { debounce } from 'es-toolkit/compat' import { hexToRgb, parseToRgb } from '@/utils/colorUtil' @@ -19,6 +19,7 @@ let maskTexture: GPUTexture | null = null let rgbTexture: GPUTexture | null = null let device: GPUDevice | null = null let renderer: GPUBrushRenderer | null = null +let previewContext: GPUCanvasContext | null = null const saveBrushToCache = debounce(function (key: string, brush: Brush): void { try { @@ -114,7 +115,6 @@ export function useBrushDrawing(initialSettings?: { return tempCanvas } - const SMOOTHING_MAX_STEPS = 30 const SMOOTHING_MIN_STEPS = 2 const isDrawing = ref(false) @@ -137,6 +137,14 @@ export function useBrushDrawing(initialSettings?: { store.setBrushSmoothingPrecision(cachedBrushSettings.smoothingPrecision) } + // Watch for external clear events + watch( + () => store.clearTrigger, + () => { + clearGPU() + } + ) + // GPU Resources const initTypeGPU = async (): Promise => { if (store.tgpuRoot) return @@ -443,8 +451,7 @@ export function useBrushDrawing(initialSettings?: { ) const normalizedSmoothing = (smoothing - 1) / 99 const stepNr = Math.round( - SMOOTHING_MIN_STEPS + - (SMOOTHING_MAX_STEPS - SMOOTHING_MIN_STEPS) * normalizedSmoothing + SMOOTHING_MIN_STEPS + (300 - SMOOTHING_MIN_STEPS) * normalizedSmoothing // 10x denser for GPU ) const distanceBetweenPoints = totalLength / stepNr @@ -452,7 +459,7 @@ export function useBrushDrawing(initialSettings?: { if (stepNr > 0) { interpolatedPoints = generateEquidistantPoints( smoothingCordsArray.value, - distanceBetweenPoints + distanceBetweenPoints * 0.1 // Much denser spacing ) } @@ -469,34 +476,7 @@ export function useBrushDrawing(initialSettings?: { // GPU render if (renderer) { - const isErasing = - store.maskCtx!.globalCompositeOperation === 'destination-out' - if (isErasing) { - // Fallback CPU for erase - for (const p of interpolatedPoints) { - drawShape(p, 1) - } - } else { - const width = store.maskCanvas!.width - const height = store.maskCanvas!.height - const targetTexture = maskTexture! - const targetView = targetTexture.createView() - const colorStr = '#ffffff' - const { r, g, b } = parseToRgb(colorStr) - const strokePoints = interpolatedPoints.map((p) => ({ - x: p.x, - y: p.y, - pressure: 1 - })) - renderer!.renderStroke(targetView, strokePoints, { - size: store.brushSettings.size, - opacity: store.brushSettings.opacity, - hardness: store.brushSettings.hardness, - color: [r / 255, g / 255, b / 255], - width, - height - }) - } + gpuRender(interpolatedPoints) } else { // Fallback CPU for (const p of interpolatedPoints) { @@ -604,21 +584,7 @@ export function useBrushDrawing(initialSettings?: { } if (renderer && compositionOp === CompositionOperation.SourceOver) { - const width = store.maskCanvas!.width - const height = store.maskCanvas!.height - const targetTexture = maskTexture! - const targetView = targetTexture.createView() - const colorStr = '#ffffff' - const { r, g, b } = parseToRgb(colorStr) - const strokePoints = points.map((p) => ({ x: p.x, y: p.y, pressure: 1 })) - renderer!.renderStroke(targetView, strokePoints, { - size: store.brushSettings.size, - opacity: store.brushSettings.opacity, - hardness: store.brushSettings.hardness, - color: [r / 255, g / 255, b / 255], - width, - height - }) + gpuRender(points) } else { // CPU fallback initShape(compositionOp) @@ -649,7 +615,9 @@ export function useBrushDrawing(initialSettings?: { } else { isDrawingLine.value = false initShape(compositionOp) - await gpuDrawPoint(coords_canvas) + // Fix: Don't draw immediately here if we are about to enter the loop. + // The handleDrawing loop starts immediately and was causing a double-draw artifact. + // await gpuDrawPoint(coords_canvas) } lineStartPoint.value = coords_canvas @@ -671,6 +639,7 @@ export function useBrushDrawing(initialSettings?: { if (diff > 20 && !isDrawing.value) { requestAnimationFrame(async () => { + if (!isDrawing.value) return // Fix: Prevent race condition try { initShape(CompositionOperation.SourceOver) await gpuDrawPoint(coords_canvas) @@ -681,6 +650,7 @@ export function useBrushDrawing(initialSettings?: { }) } else { requestAnimationFrame(async () => { + if (!isDrawing.value) return // Fix: Prevent race condition try { if (currentTool === 'eraser' || event.buttons === 2) { initShape(CompositionOperation.DestinationOut) @@ -708,6 +678,11 @@ export function useBrushDrawing(initialSettings?: { lineStartPoint.value = coords_canvas initialDraw.value = true await copyGpuToCanvas() + + // Fix: Clear the preview canvas when drawing ends + if (renderer && previewContext) { + renderer.clearPreview(previewContext) + } } } @@ -794,7 +769,13 @@ export function useBrushDrawing(initialSettings?: { const width = store.maskCanvas.width const height = store.maskCanvas.height - const bufferSize = width * height * 4 + // WebGPU requires bytesPerRow to be a multiple of 256 + const unpaddedBytesPerRow = width * 4 + const align = 256 + const paddedBytesPerRow = Math.ceil(unpaddedBytesPerRow / align) * align + + const bufferSize = paddedBytesPerRow * height + const maskBuffer = device.createBuffer({ size: bufferSize, usage: GPUBufferUsage.COPY_DST | GPUBufferUsage.MAP_READ @@ -807,12 +788,12 @@ export function useBrushDrawing(initialSettings?: { const encoder = device.createCommandEncoder() encoder.copyTextureToBuffer( { texture: maskTexture }, - { buffer: maskBuffer, bytesPerRow: width * 4 }, + { buffer: maskBuffer, bytesPerRow: paddedBytesPerRow }, { width, height } ) encoder.copyTextureToBuffer( { texture: rgbTexture }, - { buffer: rgbBuffer, bytesPerRow: width * 4 }, + { buffer: rgbBuffer, bytesPerRow: paddedBytesPerRow }, { width, height } ) device.queue.submit([encoder.finish()]) @@ -822,10 +803,53 @@ export function useBrushDrawing(initialSettings?: { rgbBuffer.mapAsync(GPUMapMode.READ) ]) - const maskData = new Uint8ClampedArray(maskBuffer.getMappedRange()) - store.maskCtx.putImageData(new ImageData(maskData, width, height), 0, 0) + const maskDataPadded = new Uint8Array(maskBuffer.getMappedRange()) + const rgbDataPadded = new Uint8Array(rgbBuffer.getMappedRange()) + + // Unpad data (row by row copy) + const maskData = new Uint8ClampedArray(width * height * 4) + const rgbData = new Uint8ClampedArray(width * height * 4) + + for (let y = 0; y < height; y++) { + const srcOffset = y * paddedBytesPerRow + const dstOffset = y * unpaddedBytesPerRow + + // Copy row and Un-premultiply Alpha + // GPU gives Premultiplied (r*a, g*a, b*a, a) + // Canvas putImageData expects Straight (r, g, b, a) + for (let x = 0; x < width; x++) { + const s = srcOffset + x * 4 + const d = dstOffset + x * 4 + + const a = maskDataPadded[s + 3] + if (a > 0) { + maskData[d] = (maskDataPadded[s] / a) * 255 + maskData[d + 1] = (maskDataPadded[s + 1] / a) * 255 + maskData[d + 2] = (maskDataPadded[s + 2] / a) * 255 + maskData[d + 3] = a + } else { + maskData[d] = 0 + maskData[d + 1] = 0 + maskData[d + 2] = 0 + maskData[d + 3] = 0 + } + + const ra = rgbDataPadded[s + 3] + if (ra > 0) { + rgbData[d] = (rgbDataPadded[s] / ra) * 255 + rgbData[d + 1] = (rgbDataPadded[s + 1] / ra) * 255 + rgbData[d + 2] = (rgbDataPadded[s + 2] / ra) * 255 + rgbData[d + 3] = ra + } else { + rgbData[d] = 0 + rgbData[d + 1] = 0 + rgbData[d + 2] = 0 + rgbData[d + 3] = 0 + } + } + } - const rgbData = new Uint8ClampedArray(rgbBuffer.getMappedRange()) + store.maskCtx.putImageData(new ImageData(maskData, width, height), 0, 0) store.rgbCtx.putImageData(new ImageData(rgbData, width, height), 0, 0) maskBuffer.unmap() @@ -870,6 +894,110 @@ export function useBrushDrawing(initialSettings?: { } } + const initPreviewCanvas = (canvas: HTMLCanvasElement) => { + if (!device) return + + const ctx = canvas.getContext('webgpu') + if (!ctx) return + + ctx.configure({ + device, + format: navigator.gpu.getPreferredCanvasFormat(), + alphaMode: 'premultiplied' + }) + previewContext = ctx + console.warn('✅ Preview Canvas Initialized') + } + + const gpuRender = (points: Point[]) => { + if (!renderer || !maskTexture || !rgbTexture) return + + const isRgb = store.activeLayer === 'rgb' + const targetTex = isRgb ? rgbTexture : maskTexture + const targetView = targetTex.createView() + + // 1. Get Correct Color + let color: [number, number, number] = [1, 1, 1] + if (isRgb) { + const c = parseToRgb(store.rgbColor) + color = [c.r / 255, c.g / 255, c.b / 255] + } else { + // Mask color - properly typed + const c = store.maskColor as { r: number; g: number; b: number } + color = [c.r / 255, c.g / 255, c.b / 255] + } + + // 2. SUPER DENSE stroke sampling with smooth pressure curve + const strokePoints: { x: number; y: number; pressure: number }[] = [] + + for (let i = 0; i < points.length - 1; i++) { + const p1 = points[i] + const p2 = points[i + 1] + const dist = Math.hypot(p2.x - p1.x, p2.y - p1.y) + + // Fix: Use step size proportional to brush size (e.g. 12% of diameter) + // Increased from 10% to reduce overlap density further + const stepSize = Math.max(1.0, store.brushSettings.size * 0.12) + const steps = Math.max(1, Math.ceil(dist / stepSize)) + + for (let step = 0; step <= steps; step++) { + const t = step / steps + + // Fix: Remove artificial tapering. + // The previous sine wave pressure caused "sausage links" on every segment. + const pressure = 1.0 + + const x = p1.x + (p2.x - p1.x) * t + const y = p1.y + (p2.y - p1.y) * t + + strokePoints.push({ x, y, pressure }) + } + } + + // Fix: Multiply opacity by a small factor to simulate "Flow". + // This prevents the brush from instantly saturating to 100% opacity/hardness + // when strokes overlap (which they always do). + // 0.1 means you need ~10 overlaps to reach full opacity, which feels natural for a soft brush. + const flowFactor = 0.08 + + renderer.renderStroke(targetView, strokePoints, { + size: store.brushSettings.size, + opacity: store.brushSettings.opacity * flowFactor, + hardness: store.brushSettings.hardness, + color: color, + width: store.maskCanvas!.width, + height: store.maskCanvas!.height + }) + + // 3. Blit to Preview (Visual Feedback) + if (previewContext) { + renderer.blitToCanvas(targetTex, previewContext) + } + } + + const clearGPU = () => { + if (!device || !maskTexture || !rgbTexture || !store.maskCanvas) return + + const width = store.maskCanvas.width + const height = store.maskCanvas.height + + // Clear Mask Texture + device.queue.writeTexture( + { texture: maskTexture }, + new Uint8Array(width * height * 4), // Zeros + { bytesPerRow: width * 4 }, + { width, height } + ) + + // Clear RGB Texture + device.queue.writeTexture( + { texture: rgbTexture }, + new Uint8Array(width * height * 4), // Zeros + { bytesPerRow: width * 4 }, + { width, height } + ) + } + return { startDrawing, handleDrawing, @@ -878,6 +1006,8 @@ export function useBrushDrawing(initialSettings?: { handleBrushAdjustment, saveBrushSettings, destroy, - initGPUResources + initGPUResources, + initPreviewCanvas, + clearGPU // Export this } } diff --git a/src/stores/maskEditorStore.ts b/src/stores/maskEditorStore.ts index 1adc48b7db0..41a3d8f5d60 100644 --- a/src/stores/maskEditorStore.ts +++ b/src/stores/maskEditorStore.ts @@ -50,6 +50,7 @@ export const useMaskEditorStore = defineStore('maskEditor', () => { const panOffset = ref({ x: 0, y: 0 }) const cursorPoint = ref({ x: 0, y: 0 }) const resetZoomTrigger = ref(0) + const clearTrigger = ref(0) const maskCanvas = ref(null) const maskCtx = ref(null) @@ -171,6 +172,10 @@ export const useMaskEditorStore = defineStore('maskEditor', () => { resetZoomTrigger.value++ } + function triggerClear(): void { + clearTrigger.value++ + } + function setMaskOpacity(opacity: number): void { maskOpacity.value = _.clamp(opacity, 0, 1) } @@ -261,6 +266,8 @@ export const useMaskEditorStore = defineStore('maskEditor', () => { setPanOffset, setCursorPoint, resetZoom, + triggerClear, + clearTrigger, setMaskOpacity, resetState } From e2c7510216525ce4cd9b1d098a04f5df2699c96e Mon Sep 17 00:00:00 2001 From: trsommer Date: Wed, 19 Nov 2025 01:43:56 +0100 Subject: [PATCH 04/17] changed interpolation to Centripetal Catmull-Rom splines, generating equidistant points using equidistant point sampling on splines, including arc length sampling for equidistance in curves --- .../maskeditor/StrokeProcessor.test.ts | 96 ++++++ src/composables/maskeditor/StrokeProcessor.ts | 114 +++++++ src/composables/maskeditor/splineUtils.ts | 272 +++++++++++++++ src/composables/maskeditor/useBrushDrawing.ts | 319 +++++++++++------- 4 files changed, 682 insertions(+), 119 deletions(-) create mode 100644 src/composables/maskeditor/StrokeProcessor.test.ts create mode 100644 src/composables/maskeditor/StrokeProcessor.ts create mode 100644 src/composables/maskeditor/splineUtils.ts diff --git a/src/composables/maskeditor/StrokeProcessor.test.ts b/src/composables/maskeditor/StrokeProcessor.test.ts new file mode 100644 index 00000000000..6b0a0365dd0 --- /dev/null +++ b/src/composables/maskeditor/StrokeProcessor.test.ts @@ -0,0 +1,96 @@ +import { describe, it, expect } from 'vitest' +import { StrokeProcessor } from './StrokeProcessor' +import type { Point } from '@/extensions/core/maskeditor/types' + +describe('StrokeProcessor', () => { + it('should generate equidistant points from irregular input', () => { + const spacing = 10 + const processor = new StrokeProcessor(spacing) + const outputPoints: Point[] = [] + + // Simulate a horizontal line drawn with irregular speed + // Points: (0,0) -> (5,0) -> (25,0) -> (30,0) -> (100,0) + const inputPoints: Point[] = [ + { x: 0, y: 0 }, + { x: 5, y: 0 }, // dist 5 + { x: 25, y: 0 }, // dist 20 + { x: 30, y: 0 }, // dist 5 + { x: 100, y: 0 } // dist 70 + ] + + for (const p of inputPoints) { + outputPoints.push(...processor.addPoint(p)) + } + outputPoints.push(...processor.endStroke()) + + // Verify we have points + expect(outputPoints.length).toBeGreaterThan(0) + + // Verify spacing + // Note: The first few points might be affected by the start condition, + // but the middle section should be perfectly spaced. + // Also, Catmull-Rom splines don't necessarily pass through control points in a straight line + // if the points are collinear, they should be straight. + + // Let's check distances between consecutive points + const distances: number[] = [] + for (let i = 1; i < outputPoints.length; i++) { + const dx = outputPoints[i].x - outputPoints[i - 1].x + const dy = outputPoints[i].y - outputPoints[i - 1].y + distances.push(Math.hypot(dx, dy)) + } + + // Check that distances are close to spacing + // We allow a small epsilon because of floating point and spline approximation + // Filter out the very last segment which might be shorter (remainder) + // But wait, our logic doesn't output the last point if it's not a full spacing step? + // resampleSegment outputs points at [start + spacing, start + 2*spacing, ...] + // It does NOT output the end point of the segment. + // So all distances between output points should be exactly `spacing`. + // EXCEPT possibly if the spline curvature makes the straight-line distance slightly different + // from the arc length. But for a straight line input, it should be exact. + + // However, catmull-rom with collinear points IS a straight line. + + // Let's log the distances for debugging if test fails + // console.log('Distances:', distances) + + // All distances should be approximately equal to spacing + // We might have a gap between segments if the logic isn't perfect, + // but within a segment it's guaranteed by resampleSegment. + // The critical part is the transition between segments. + + for (let i = 0; i < distances.length; i++) { + const d = distances[i] + if (Math.abs(d - spacing) > 0.5) { + console.log( + `Distance mismatch at index ${i}: ${d} (expected ${spacing})` + ) + console.log(`Point ${i}:`, outputPoints[i]) + console.log(`Point ${i + 1}:`, outputPoints[i + 1]) + } + expect(d).toBeCloseTo(spacing, 1) + } + }) + + it('should handle a simple 3-point stroke', () => { + const spacing = 5 + const processor = new StrokeProcessor(spacing) + const points: Point[] = [] + + points.push(...processor.addPoint({ x: 0, y: 0 })) + points.push(...processor.addPoint({ x: 10, y: 0 })) + points.push(...processor.addPoint({ x: 20, y: 0 })) + points.push(...processor.endStroke()) + + expect(points.length).toBeGreaterThan(0) + + // Check distances + for (let i = 1; i < points.length; i++) { + const dx = points[i].x - points[i - 1].x + const dy = points[i].y - points[i - 1].y + const d = Math.hypot(dx, dy) + expect(d).toBeCloseTo(spacing, 1) + } + }) +}) diff --git a/src/composables/maskeditor/StrokeProcessor.ts b/src/composables/maskeditor/StrokeProcessor.ts new file mode 100644 index 00000000000..a0e6111f368 --- /dev/null +++ b/src/composables/maskeditor/StrokeProcessor.ts @@ -0,0 +1,114 @@ +import type { Point } from '@/extensions/core/maskeditor/types' +import { catmullRomSpline, resampleSegment } from './splineUtils' + +export class StrokeProcessor { + private controlPoints: Point[] = [] + private remainder: number = 0 + private spacing: number + private isFirstPoint: boolean = true + + constructor(spacing: number) { + this.spacing = spacing + } + + /** + * Adds a point to the stroke and returns any new equidistant points generated. + * Maintains a sliding window of 4 control points to generate Catmull-Rom spline segments. + */ + public addPoint(point: Point): Point[] { + // If this is the very first point, we need to initialize the buffer + if (this.isFirstPoint) { + this.controlPoints.push(point) // p0 (phantom start) + this.controlPoints.push(point) // p1 (actual start) + this.isFirstPoint = false + return [] // No segment to draw yet + } + + this.controlPoints.push(point) + + // We need at least 4 points to generate a spline segment (p0, p1, p2, p3) + // p0: previous control point + // p1: start of segment + // p2: end of segment + // p3: next control point + if (this.controlPoints.length < 4) { + return [] + } + + // Generate the segment between p1 and p2 + const p0 = this.controlPoints[0] + const p1 = this.controlPoints[1] + const p2 = this.controlPoints[2] + const p3 = this.controlPoints[3] + + const newPoints = this.processSegment(p0, p1, p2, p3) + + // Slide the window: remove p0 + this.controlPoints.shift() + + return newPoints + } + + /** + * Ends the stroke, flushing any remaining segments. + * This adds a phantom end point to complete the last segment. + */ + public endStroke(): Point[] { + if (this.controlPoints.length < 2) { + // Not enough points to form a segment + return [] + } + + // If we have [p0, p1, p2], we need to process the segment p1->p2 + // We duplicate p2 as p3 (phantom end point) + // If we have [p0, p1], we duplicate p1 as p2 and p3 (single point stroke case handled elsewhere usually, but good for safety) + + const newPoints: Point[] = [] + + // To properly flush, we essentially pretend we added one last point which is the same as the last point + // But we might have multiple points in buffer. + // Actually, the sliding window ensures we always have [p(n-2), p(n-1), p(n)]. + // We need to process p(n-1)->p(n). + // So we just need to supply p(n) as the "next" control point (p3). + + while (this.controlPoints.length >= 3) { + const p0 = this.controlPoints[0] + const p1 = this.controlPoints[1] + const p2 = this.controlPoints[2] + const p3 = p2 // Duplicate last point + + const points = this.processSegment(p0, p1, p2, p3) + newPoints.push(...points) + + this.controlPoints.shift() + } + + return newPoints + } + + private processSegment(p0: Point, p1: Point, p2: Point, p3: Point): Point[] { + // Generate dense points for the segment + // We use a fixed high resolution for the dense curve + const densePoints: Point[] = [] + const samples = 20 // Or adaptive based on distance + + // We can use adaptive sampling if needed, but fixed is usually fine for small segments + // Let's use a simple loop for now, similar to generateSmoothCurve but for one segment + for (let i = 0; i < samples; i++) { + const t = i / samples + densePoints.push(catmullRomSpline(p0, p1, p2, p3, t)) + } + // Add the end point of the segment + densePoints.push(p2) + + // Resample using the carried-over remainder + const { points, remainder } = resampleSegment( + densePoints, + this.spacing, + this.remainder + ) + + this.remainder = remainder + return points + } +} diff --git a/src/composables/maskeditor/splineUtils.ts b/src/composables/maskeditor/splineUtils.ts new file mode 100644 index 00000000000..54691bc064a --- /dev/null +++ b/src/composables/maskeditor/splineUtils.ts @@ -0,0 +1,272 @@ +import type { Point } from '@/extensions/core/maskeditor/types' + +/** + * Evaluates a Catmull-Rom spline at parameter t between p1 and p2 + * @param p0 Previous control point + * @param p1 Start point of the curve segment + * @param p2 End point of the curve segment + * @param p3 Next control point + * @param t Parameter in range [0, 1] + * @returns Interpolated point on the curve + */ +export function catmullRomSpline( + p0: Point, + p1: Point, + p2: Point, + p3: Point, + t: number +): Point { + // Centripetal Catmull-Rom Spline (alpha = 0.5) + // This prevents loops and overshoots when control points are unevenly spaced. + const alpha = 0.5 + + const getT = (t: number, p0: Point, p1: Point) => { + const d = Math.hypot(p1.x - p0.x, p1.y - p0.y) + return t + Math.pow(d, alpha) + } + + const t0 = 0 + const t1 = getT(t0, p0, p1) + const t2 = getT(t1, p1, p2) + const t3 = getT(t2, p2, p3) + + // Map t (0..1) to the actual parameter range (t1..t2) + const tInterp = t1 + (t2 - t1) * t + + // Helper for safe interpolation when time intervals are zero (coincident points) + const interp = ( + pA: Point, + pB: Point, + tA: number, + tB: number, + t: number + ): Point => { + if (Math.abs(tB - tA) < 0.0001) return pA + const k = (t - tA) / (tB - tA) + return add(mul(pA, 1 - k), mul(pB, k)) + } + + // Barry-Goldman pyramidal formulation + const A1 = interp(p0, p1, t0, t1, tInterp) + const A2 = interp(p1, p2, t1, t2, tInterp) + const A3 = interp(p2, p3, t2, t3, tInterp) + + const B1 = interp(A1, A2, t0, t2, tInterp) + const B2 = interp(A2, A3, t1, t3, tInterp) + + const C = interp(B1, B2, t1, t2, tInterp) + + return C +} + +function add(p1: Point, p2: Point): Point { + return { x: p1.x + p2.x, y: p1.y + p2.y } +} + +function mul(p: Point, s: number): Point { + return { x: p.x * s, y: p.y * s } +} + +/** + * Generates a smooth curve through points using Catmull-Rom splines. + * Densely samples the curve for smoothness - the calling code will handle + * equidistant spacing for brush dabs. + * + * @param points Array of control points (raw mouse/touch positions) + * @param samplesPerSegment Number of samples per spline segment + * @returns Array of points forming a smooth curve + */ +export function generateSmoothCurve( + points: Point[], + samplesPerSegment: number = 20 +): Point[] { + if (points.length < 2) return points + + // For just 2 points, return linear interpolation + if (points.length === 2) { + const result: Point[] = [] + for (let i = 0; i <= samplesPerSegment; i++) { + const t = i / samplesPerSegment + result.push({ + x: points[0].x + (points[1].x - points[0].x) * t, + y: points[0].y + (points[1].y - points[0].y) * t + }) + } + return result + } + + const curvePoints: Point[] = [] + + // For each segment between consecutive points + for (let i = 0; i < points.length - 1; i++) { + // Get the 4 control points needed for Catmull-Rom + const p0 = i === 0 ? points[0] : points[i - 1] + const p1 = points[i] + const p2 = points[i + 1] + const p3 = i === points.length - 2 ? points[i + 1] : points[i + 2] + + // Sample the curve segment densely + // Use more samples for longer segments + const segLen = Math.hypot(p2.x - p1.x, p2.y - p1.y) + const samples = Math.max(samplesPerSegment, Math.ceil(segLen / 2)) + + for (let j = 0; j < samples; j++) { + const t = j / samples + curvePoints.push(catmullRomSpline(p0, p1, p2, p3, t)) + } + } + + // Add the final point + curvePoints.push(points[points.length - 1]) + + return curvePoints +} + +/** + * Resamples a curve at equidistant intervals for uniform brush dab spacing. + * This solves the density variation problem: fast/slow mouse movements will + * have the same brush density. + * + * @param points Points defining the curve (should be densely sampled) + * @param spacing Desired spacing between points in pixels + * @returns Array of equidistant points along the curve + */ +export function resampleEquidistant(points: Point[], spacing: number): Point[] { + if (points.length === 0) return [] + if (points.length === 1) return [points[0]] + + const result: Point[] = [] + const cumulativeDistances: number[] = [0] + + // Calculate cumulative distances along the curve + for (let i = 1; i < points.length; i++) { + const dx = points[i].x - points[i - 1].x + const dy = points[i].y - points[i - 1].y + const dist = Math.hypot(dx, dy) + cumulativeDistances[i] = cumulativeDistances[i - 1] + dist + } + + const totalLength = cumulativeDistances[cumulativeDistances.length - 1] + if (totalLength < spacing) { + // Curve is shorter than spacing, just return the endpoints + return [points[0], points[points.length - 1]] + } + + const numPoints = Math.floor(totalLength / spacing) + + // Sample at equidistant intervals + for (let i = 0; i <= numPoints; i++) { + const targetDistance = i * spacing + let idx = 0 + + // Find the segment containing the target distance + while ( + idx < cumulativeDistances.length - 1 && + cumulativeDistances[idx + 1] < targetDistance + ) { + idx++ + } + + if (idx >= points.length - 1) { + result.push(points[points.length - 1]) + continue + } + + // Interpolate within the segment + const d0 = cumulativeDistances[idx] + const d1 = cumulativeDistances[idx + 1] + const segmentLength = d1 - d0 + + if (segmentLength < 0.001) { + // Avoid division by zero + result.push(points[idx]) + continue + } + + const t = (targetDistance - d0) / segmentLength + + const x = points[idx].x + t * (points[idx + 1].x - points[idx].x) + const y = points[idx].y + t * (points[idx + 1].y - points[idx].y) + + result.push({ x, y }) + } + + return result +} + +/** + * Generates a smooth, equidistantly sampled stroke through the given points. + * Two-step process: + * 1. Generate dense smooth curve using Catmull-Rom splines + * 2. Resample at equidistant intervals for uniform brush dab spacing + * + * @param points Array of control points (raw mouse/touch positions) + * @param targetSpacing Desired spacing between brush dabs in pixels + * @returns Array of equidistant points forming a smooth curve + */ +export function generateSmoothStroke( + points: Point[], + targetSpacing: number +): Point[] { + if (points.length < 2) return points + + // Step 1: Generate dense smooth curve + const denseCurve = generateSmoothCurve(points, 20) + + // Step 2: Resample at equidistant intervals + const equidistantPoints = resampleEquidistant(denseCurve, targetSpacing) + + return equidistantPoints +} + +/** + * Resamples a curve segment with a starting offset (remainder from previous segment). + * Returns the resampled points and the new remainder distance. + * + * @param points Points defining the curve segment + * @param spacing Desired spacing between points + * @param startOffset Distance to travel before placing the first point (remainder) + * @returns Object containing points and new remainder + */ +export function resampleSegment( + points: Point[], + spacing: number, + startOffset: number +): { points: Point[]; remainder: number } { + if (points.length === 0) return { points: [], remainder: startOffset } + + const result: Point[] = [] + let currentDist = 0 + let nextSampleDist = startOffset + + // We iterate through the dense points of the segment + for (let i = 0; i < points.length - 1; i++) { + const p1 = points[i] + const p2 = points[i + 1] + + const dx = p2.x - p1.x + const dy = p2.y - p1.y + const segmentLen = Math.hypot(dx, dy) + + // While the next sample falls within this segment + while (nextSampleDist <= currentDist + segmentLen) { + const t = (nextSampleDist - currentDist) / segmentLen + + // Interpolate + const x = p1.x + t * dx + const y = p1.y + t * dy + result.push({ x, y }) + + nextSampleDist += spacing + } + + currentDist += segmentLen + } + + // The remainder is how far past the last point the next sample would be + // relative to the end of the segment. + // Actually, simpler: remainder = nextSampleDist - totalLength + const remainder = nextSampleDist - currentDist + + return { points: result, remainder } +} diff --git a/src/composables/maskeditor/useBrushDrawing.ts b/src/composables/maskeditor/useBrushDrawing.ts index a8d222f3fa6..c6f54bf2888 100644 --- a/src/composables/maskeditor/useBrushDrawing.ts +++ b/src/composables/maskeditor/useBrushDrawing.ts @@ -13,6 +13,7 @@ import { useMaskEditorStore } from '@/stores/maskEditorStore' import { useCoordinateTransform } from './useCoordinateTransform' import TGPU from 'typegpu' import { GPUBrushRenderer } from './gpu/GPUBrushRenderer' +import { StrokeProcessor } from './StrokeProcessor' // GPU Resources (scope fix) let maskTexture: GPUTexture | null = null @@ -115,15 +116,19 @@ export function useBrushDrawing(initialSettings?: { return tempCanvas } - const SMOOTHING_MIN_STEPS = 2 + // Debug: Track points for distance analysis + const debugPoints = ref>([]) + const debugRawPoints = ref>([]) const isDrawing = ref(false) const isDrawingLine = ref(false) const lineStartPoint = ref(null) - const smoothingCordsArray = ref([]) const smoothingLastDrawTime = ref(new Date()) const initialDraw = ref(true) + // Stroke Processor + let strokeProcessor: StrokeProcessor | null = null + const initialPoint = ref(null) const useDominantAxis = ref(initialSettings?.useDominantAxis ?? false) const brushAdjustmentSpeed = ref(initialSettings?.brushAdjustmentSpeed ?? 1.0) @@ -432,113 +437,33 @@ export function useBrushDrawing(initialSettings?: { } const drawWithBetterSmoothing = async (point: Point): Promise => { - smoothingCordsArray.value.push(point) + if (!strokeProcessor) return - const POINTS_NR = 5 - if (smoothingCordsArray.value.length < POINTS_NR) return + // Debug: Save raw input point with timestamp + debugRawPoints.value.push({ point, timestamp: performance.now() }) - let totalLength = 0 - const points = smoothingCordsArray.value - const len = points.length - 1 - for (let i = 0; i < len; i++) { - const dx = points[i + 1].x - points[i].x - const dy = points[i + 1].y - points[i].y - totalLength += Math.sqrt(dx * dx + dy * dy) - } + // Add point to processor and get new equidistant points + const newPoints = strokeProcessor.addPoint(point) - const smoothing = clampSmoothingPrecision( - store.brushSettings.smoothingPrecision - ) - const normalizedSmoothing = (smoothing - 1) / 99 - const stepNr = Math.round( - SMOOTHING_MIN_STEPS + (300 - SMOOTHING_MIN_STEPS) * normalizedSmoothing // 10x denser for GPU - ) - const distanceBetweenPoints = totalLength / stepNr - - let interpolatedPoints = points - if (stepNr > 0) { - interpolatedPoints = generateEquidistantPoints( - smoothingCordsArray.value, - distanceBetweenPoints * 0.1 // Much denser spacing - ) - } + if (newPoints.length === 0) return - if (!initialDraw.value) { - const spliceIndex = interpolatedPoints.findIndex( - (p) => - p.x === smoothingCordsArray.value[2].x && - p.y === smoothingCordsArray.value[2].y + // GPU render with pre-spaced points + if (renderer) { + // Debug: Save points with timestamps for distance analysis + const now = performance.now() + debugPoints.value.push( + ...newPoints.map((point) => ({ point, timestamp: now })) ) - if (spliceIndex !== -1) { - interpolatedPoints = interpolatedPoints.slice(spliceIndex + 1) - } - } - // GPU render - if (renderer) { - gpuRender(interpolatedPoints) + gpuRender(newPoints, true) // Pass flag to skip resampling } else { // Fallback CPU - for (const p of interpolatedPoints) { + for (const p of newPoints) { drawShape(p, 1) } } - if (!initialDraw.value) { - smoothingCordsArray.value = smoothingCordsArray.value.slice(2) - } else { - initialDraw.value = false - } - } - - const clampSmoothingPrecision = (value: number): number => { - return Math.min(Math.max(value, 1), 100) - } - - const generateEquidistantPoints = ( - points: Point[], - distance: number - ): Point[] => { - const result: Point[] = [] - const cumulativeDistances: number[] = [0] - - for (let i = 1; i < points.length; i++) { - const dx = points[i].x - points[i - 1].x - const dy = points[i].y - points[i - 1].y - const dist = Math.hypot(dx, dy) - cumulativeDistances[i] = cumulativeDistances[i - 1] + dist - } - - const totalLength = cumulativeDistances[cumulativeDistances.length - 1] - const numPoints = Math.floor(totalLength / distance) - - for (let i = 0; i <= numPoints; i++) { - const targetDistance = i * distance - let idx = 0 - - while ( - idx < cumulativeDistances.length - 1 && - cumulativeDistances[idx + 1] < targetDistance - ) { - idx++ - } - - if (idx >= points.length - 1) { - result.push(points[points.length - 1]) - continue - } - - const d0 = cumulativeDistances[idx] - const d1 = cumulativeDistances[idx + 1] - const t = (targetDistance - d0) / (d1 - d0) - - const x = points[idx].x + t * (points[idx + 1].x - points[idx].x) - const y = points[idx].y + t * (points[idx + 1].y - points[idx].y) - - result.push({ x, y }) - } - - return result + initialDraw.value = false } const initShape = (compositionOperation: CompositionOperation) => { @@ -621,7 +546,14 @@ export function useBrushDrawing(initialSettings?: { } lineStartPoint.value = coords_canvas - smoothingCordsArray.value = [coords_canvas] + + // Initialize StrokeProcessor + const targetSpacing = Math.max(1.0, store.brushSettings.size * 0.12) + strokeProcessor = new StrokeProcessor(targetSpacing) + + // Add the first point + await drawWithBetterSmoothing(coords_canvas) + smoothingLastDrawTime.value = new Date() } catch (error) { console.error('[useBrushDrawing] Failed to start drawing:', error) @@ -643,7 +575,7 @@ export function useBrushDrawing(initialSettings?: { try { initShape(CompositionOperation.SourceOver) await gpuDrawPoint(coords_canvas) - smoothingCordsArray.value.push(coords_canvas) + // smoothingCordsArray.value.push(coords_canvas) // Removed in favor of StrokeProcessor } catch (error) { console.error('[useBrushDrawing] Drawing error:', error) } @@ -677,12 +609,160 @@ export function useBrushDrawing(initialSettings?: { store.canvasHistory.saveState() lineStartPoint.value = coords_canvas initialDraw.value = true + + // Flush remaining points from StrokeProcessor + if (strokeProcessor) { + const finalPoints = strokeProcessor.endStroke() + if (finalPoints.length > 0) { + if (renderer) { + const now = performance.now() + debugPoints.value.push( + ...finalPoints.map((point) => ({ point, timestamp: now })) + ) + gpuRender(finalPoints, true) + } else { + for (const p of finalPoints) { + drawShape(p, 1) + } + } + } + strokeProcessor = null + } + await copyGpuToCanvas() // Fix: Clear the preview canvas when drawing ends if (renderer && previewContext) { renderer.clearPreview(previewContext) } + + // Debug: Calculate and log distances between points + if (debugPoints.value.length > 1 || debugRawPoints.value.length > 1) { + // Process final points + const distances: number[] = [] + const pointsWithDistances: Array<{ + index: number + point: Point + timestamp: number + distanceFromPrevious?: number + }> = [] + + if (debugPoints.value.length > 1) { + // First point + pointsWithDistances.push({ + index: 0, + point: debugPoints.value[0].point, + timestamp: debugPoints.value[0].timestamp, + distanceFromPrevious: undefined + }) + + for (let i = 1; i < debugPoints.value.length; i++) { + const dx = + debugPoints.value[i].point.x - debugPoints.value[i - 1].point.x + const dy = + debugPoints.value[i].point.y - debugPoints.value[i - 1].point.y + const dist = Math.hypot(dx, dy) + distances.push(dist) + + pointsWithDistances.push({ + index: i, + point: debugPoints.value[i].point, + timestamp: debugPoints.value[i].timestamp, + distanceFromPrevious: dist + }) + } + } + + // Process raw input points + const rawDistances: number[] = [] + const rawPointsWithDistances: Array<{ + index: number + point: Point + timestamp: number + distanceFromPrevious?: number + }> = [] + + if (debugRawPoints.value.length > 1) { + rawPointsWithDistances.push({ + index: 0, + point: debugRawPoints.value[0].point, + timestamp: debugRawPoints.value[0].timestamp, + distanceFromPrevious: undefined + }) + + for (let i = 1; i < debugRawPoints.value.length; i++) { + const dx = + debugRawPoints.value[i].point.x - + debugRawPoints.value[i - 1].point.x + const dy = + debugRawPoints.value[i].point.y - + debugRawPoints.value[i - 1].point.y + const dist = Math.hypot(dx, dy) + rawDistances.push(dist) + + rawPointsWithDistances.push({ + index: i, + point: debugRawPoints.value[i].point, + timestamp: debugRawPoints.value[i].timestamp, + distanceFromPrevious: dist + }) + } + } + + const avgDist = + distances.length > 0 + ? distances.reduce((a, b) => a + b, 0) / distances.length + : 0 + const minDist = distances.length > 0 ? Math.min(...distances) : 0 + const maxDist = distances.length > 0 ? Math.max(...distances) : 0 + const stdDev = + distances.length > 0 + ? Math.sqrt( + distances.reduce( + (sum, d) => sum + Math.pow(d - avgDist, 2), + 0 + ) / distances.length + ) + : 0 + + const rawAvgDist = + rawDistances.length > 0 + ? rawDistances.reduce((a, b) => a + b, 0) / rawDistances.length + : 0 + const rawMinDist = + rawDistances.length > 0 ? Math.min(...rawDistances) : 0 + const rawMaxDist = + rawDistances.length > 0 ? Math.max(...rawDistances) : 0 + + console.warn('🎨 Brush Spacing Debug:', { + rawInput: { + pointCount: debugRawPoints.value.length, + stats: { + avg: rawAvgDist.toFixed(2), + min: rawMinDist.toFixed(2), + max: rawMaxDist.toFixed(2) + }, + distances: rawDistances, + points: rawPointsWithDistances + }, + processed: { + pointCount: debugPoints.value.length, + stats: { + avg: avgDist.toFixed(2), + min: minDist.toFixed(2), + max: maxDist.toFixed(2), + stdDev: stdDev.toFixed(2), + targetSpacing: (store.brushSettings.size * 0.12).toFixed(2) + }, + distances, + points: pointsWithDistances + } + }) + } + + // Reset debug points for next stroke + debugPoints.value = [] + debugRawPoints.value = [] } } @@ -909,7 +989,7 @@ export function useBrushDrawing(initialSettings?: { console.warn('✅ Preview Canvas Initialized') } - const gpuRender = (points: Point[]) => { + const gpuRender = (points: Point[], skipResampling: boolean = false) => { if (!renderer || !maskTexture || !rgbTexture) return const isRgb = store.activeLayer === 'rgb' @@ -927,30 +1007,31 @@ export function useBrushDrawing(initialSettings?: { color = [c.r / 255, c.g / 255, c.b / 255] } - // 2. SUPER DENSE stroke sampling with smooth pressure curve - const strokePoints: { x: number; y: number; pressure: number }[] = [] + // 2. Prepare stroke points + let strokePoints: { x: number; y: number; pressure: number }[] = [] - for (let i = 0; i < points.length - 1; i++) { - const p1 = points[i] - const p2 = points[i + 1] - const dist = Math.hypot(p2.x - p1.x, p2.y - p1.y) - - // Fix: Use step size proportional to brush size (e.g. 12% of diameter) - // Increased from 10% to reduce overlap density further - const stepSize = Math.max(1.0, store.brushSettings.size * 0.12) - const steps = Math.max(1, Math.ceil(dist / stepSize)) + if (skipResampling) { + // Points are already properly spaced from Catmull-Rom spline interpolation + strokePoints = points.map((p) => ({ x: p.x, y: p.y, pressure: 1.0 })) + } else { + // Legacy resampling for shift+click and other cases + for (let i = 0; i < points.length - 1; i++) { + const p1 = points[i] + const p2 = points[i + 1] + const dist = Math.hypot(p2.x - p1.x, p2.y - p1.y) - for (let step = 0; step <= steps; step++) { - const t = step / steps + const stepSize = Math.max(1.0, store.brushSettings.size * 0.12) + const steps = Math.max(1, Math.ceil(dist / stepSize)) - // Fix: Remove artificial tapering. - // The previous sine wave pressure caused "sausage links" on every segment. - const pressure = 1.0 + for (let step = 0; step <= steps; step++) { + const t = step / steps + const pressure = 1.0 - const x = p1.x + (p2.x - p1.x) * t - const y = p1.y + (p2.y - p1.y) * t + const x = p1.x + (p2.x - p1.x) * t + const y = p1.y + (p2.y - p1.y) * t - strokePoints.push({ x, y, pressure }) + strokePoints.push({ x, y, pressure }) + } } } From 3619ced1c23b84f6f3196c639c4e19bee139e049 Mon Sep 17 00:00:00 2001 From: trsommer Date: Wed, 19 Nov 2025 01:51:01 +0100 Subject: [PATCH 05/17] removed logging from debugging stroke smoothing --- src/composables/maskeditor/useBrushDrawing.ts | 143 ------------------ 1 file changed, 143 deletions(-) diff --git a/src/composables/maskeditor/useBrushDrawing.ts b/src/composables/maskeditor/useBrushDrawing.ts index c6f54bf2888..1ee845d457d 100644 --- a/src/composables/maskeditor/useBrushDrawing.ts +++ b/src/composables/maskeditor/useBrushDrawing.ts @@ -117,8 +117,6 @@ export function useBrushDrawing(initialSettings?: { } // Debug: Track points for distance analysis - const debugPoints = ref>([]) - const debugRawPoints = ref>([]) const isDrawing = ref(false) const isDrawingLine = ref(false) @@ -439,9 +437,6 @@ export function useBrushDrawing(initialSettings?: { const drawWithBetterSmoothing = async (point: Point): Promise => { if (!strokeProcessor) return - // Debug: Save raw input point with timestamp - debugRawPoints.value.push({ point, timestamp: performance.now() }) - // Add point to processor and get new equidistant points const newPoints = strokeProcessor.addPoint(point) @@ -449,12 +444,6 @@ export function useBrushDrawing(initialSettings?: { // GPU render with pre-spaced points if (renderer) { - // Debug: Save points with timestamps for distance analysis - const now = performance.now() - debugPoints.value.push( - ...newPoints.map((point) => ({ point, timestamp: now })) - ) - gpuRender(newPoints, true) // Pass flag to skip resampling } else { // Fallback CPU @@ -615,10 +604,6 @@ export function useBrushDrawing(initialSettings?: { const finalPoints = strokeProcessor.endStroke() if (finalPoints.length > 0) { if (renderer) { - const now = performance.now() - debugPoints.value.push( - ...finalPoints.map((point) => ({ point, timestamp: now })) - ) gpuRender(finalPoints, true) } else { for (const p of finalPoints) { @@ -635,134 +620,6 @@ export function useBrushDrawing(initialSettings?: { if (renderer && previewContext) { renderer.clearPreview(previewContext) } - - // Debug: Calculate and log distances between points - if (debugPoints.value.length > 1 || debugRawPoints.value.length > 1) { - // Process final points - const distances: number[] = [] - const pointsWithDistances: Array<{ - index: number - point: Point - timestamp: number - distanceFromPrevious?: number - }> = [] - - if (debugPoints.value.length > 1) { - // First point - pointsWithDistances.push({ - index: 0, - point: debugPoints.value[0].point, - timestamp: debugPoints.value[0].timestamp, - distanceFromPrevious: undefined - }) - - for (let i = 1; i < debugPoints.value.length; i++) { - const dx = - debugPoints.value[i].point.x - debugPoints.value[i - 1].point.x - const dy = - debugPoints.value[i].point.y - debugPoints.value[i - 1].point.y - const dist = Math.hypot(dx, dy) - distances.push(dist) - - pointsWithDistances.push({ - index: i, - point: debugPoints.value[i].point, - timestamp: debugPoints.value[i].timestamp, - distanceFromPrevious: dist - }) - } - } - - // Process raw input points - const rawDistances: number[] = [] - const rawPointsWithDistances: Array<{ - index: number - point: Point - timestamp: number - distanceFromPrevious?: number - }> = [] - - if (debugRawPoints.value.length > 1) { - rawPointsWithDistances.push({ - index: 0, - point: debugRawPoints.value[0].point, - timestamp: debugRawPoints.value[0].timestamp, - distanceFromPrevious: undefined - }) - - for (let i = 1; i < debugRawPoints.value.length; i++) { - const dx = - debugRawPoints.value[i].point.x - - debugRawPoints.value[i - 1].point.x - const dy = - debugRawPoints.value[i].point.y - - debugRawPoints.value[i - 1].point.y - const dist = Math.hypot(dx, dy) - rawDistances.push(dist) - - rawPointsWithDistances.push({ - index: i, - point: debugRawPoints.value[i].point, - timestamp: debugRawPoints.value[i].timestamp, - distanceFromPrevious: dist - }) - } - } - - const avgDist = - distances.length > 0 - ? distances.reduce((a, b) => a + b, 0) / distances.length - : 0 - const minDist = distances.length > 0 ? Math.min(...distances) : 0 - const maxDist = distances.length > 0 ? Math.max(...distances) : 0 - const stdDev = - distances.length > 0 - ? Math.sqrt( - distances.reduce( - (sum, d) => sum + Math.pow(d - avgDist, 2), - 0 - ) / distances.length - ) - : 0 - - const rawAvgDist = - rawDistances.length > 0 - ? rawDistances.reduce((a, b) => a + b, 0) / rawDistances.length - : 0 - const rawMinDist = - rawDistances.length > 0 ? Math.min(...rawDistances) : 0 - const rawMaxDist = - rawDistances.length > 0 ? Math.max(...rawDistances) : 0 - - console.warn('🎨 Brush Spacing Debug:', { - rawInput: { - pointCount: debugRawPoints.value.length, - stats: { - avg: rawAvgDist.toFixed(2), - min: rawMinDist.toFixed(2), - max: rawMaxDist.toFixed(2) - }, - distances: rawDistances, - points: rawPointsWithDistances - }, - processed: { - pointCount: debugPoints.value.length, - stats: { - avg: avgDist.toFixed(2), - min: minDist.toFixed(2), - max: maxDist.toFixed(2), - stdDev: stdDev.toFixed(2), - targetSpacing: (store.brushSettings.size * 0.12).toFixed(2) - }, - distances, - points: pointsWithDistances - } - }) - } - - // Reset debug points for next stroke - debugPoints.value = [] - debugRawPoints.value = [] } } From 66ed91a689d56b29e9f38d1f4fc0373754a53f35 Mon Sep 17 00:00:00 2001 From: trsommer Date: Wed, 19 Nov 2025 02:13:13 +0100 Subject: [PATCH 06/17] fixed shift click line drawing --- src/composables/maskeditor/ShiftClick.test.ts | 84 +++++++++++++++++++ src/composables/maskeditor/useBrushDrawing.ts | 39 +++++---- 2 files changed, 107 insertions(+), 16 deletions(-) create mode 100644 src/composables/maskeditor/ShiftClick.test.ts diff --git a/src/composables/maskeditor/ShiftClick.test.ts b/src/composables/maskeditor/ShiftClick.test.ts new file mode 100644 index 00000000000..c96d6e2c327 --- /dev/null +++ b/src/composables/maskeditor/ShiftClick.test.ts @@ -0,0 +1,84 @@ +import { describe, it, expect } from 'vitest' +import { resampleSegment } from './splineUtils' +import type { Point } from '@/extensions/core/maskeditor/types' + +describe('Shift+Click Drawing Logic', () => { + it('should generate equidistant points across connected segments', () => { + const spacing = 4 + let remainder = spacing // Simulate start point already painted + const outputPoints: Point[] = [] + + // Define points: A -> B -> C + // A(0,0) -> B(10,0) -> C(20,0) + // Total length 20. Spacing 4. + // Expected points at x = 4, 8, 12, 16, 20 + const pA = { x: 0, y: 0 } + const pB = { x: 10, y: 0 } + const pC = { x: 20, y: 0 } + + // Segment 1: A -> B + const result1 = resampleSegment([pA, pB], spacing, remainder) + outputPoints.push(...result1.points) + remainder = result1.remainder + + // Verify intermediate state + // Length 10. Spacing 4. Start offset 4. + // Points at 4, 8. Next at 12. + // Remainder = 12 - 10 = 2. + expect(result1.points.length).toBe(2) + expect(result1.points[0].x).toBeCloseTo(4) + expect(result1.points[1].x).toBeCloseTo(8) + expect(remainder).toBeCloseTo(2) + + // Segment 2: B -> C + const result2 = resampleSegment([pB, pC], spacing, remainder) + outputPoints.push(...result2.points) + remainder = result2.remainder + + // Verify final state + // Start offset 2. Points at 2, 6, 10 (relative to B). + // Absolute x: 12, 16, 20. + expect(result2.points.length).toBe(3) + expect(result2.points[0].x).toBeCloseTo(12) + expect(result2.points[1].x).toBeCloseTo(16) + expect(result2.points[2].x).toBeCloseTo(20) + + // Verify all distances + // Note: The first point is at distance `spacing` from start (0,0) + // Subsequent points are `spacing` apart. + let prevX = 0 + for (const p of outputPoints) { + const dist = p.x - prevX + expect(dist).toBeCloseTo(spacing) + prevX = p.x + } + }) + + it('should handle segments shorter than spacing', () => { + const spacing = 10 + let remainder = spacing // Simulate start point already painted + + // A(0,0) -> B(5,0) -> C(15,0) + const pA = { x: 0, y: 0 } + const pB = { x: 5, y: 0 } + const pC = { x: 15, y: 0 } + + // Segment 1: A -> B (Length 5) + // Spacing 10. No points should be generated. + // Remainder should be 5 (next point needs 5 more units). + const result1 = resampleSegment([pA, pB], spacing, remainder) + expect(result1.points.length).toBe(0) + expect(result1.remainder).toBeCloseTo(5) + remainder = result1.remainder + + // Segment 2: B -> C (Length 10) + // Start offset 5. First point at 5 (relative to B). + // Absolute x = 10. + // Next point at 15 (relative to B). Segment ends at 10. + // Remainder = 15 - 10 = 5. + const result2 = resampleSegment([pB, pC], spacing, remainder) + expect(result2.points.length).toBe(1) + expect(result2.points[0].x).toBeCloseTo(10) + expect(result2.remainder).toBeCloseTo(5) + }) +}) diff --git a/src/composables/maskeditor/useBrushDrawing.ts b/src/composables/maskeditor/useBrushDrawing.ts index 1ee845d457d..73bdef971c4 100644 --- a/src/composables/maskeditor/useBrushDrawing.ts +++ b/src/composables/maskeditor/useBrushDrawing.ts @@ -11,6 +11,7 @@ import { import type { Brush, Point } from '@/extensions/core/maskeditor/types' import { useMaskEditorStore } from '@/stores/maskEditorStore' import { useCoordinateTransform } from './useCoordinateTransform' +import { resampleSegment } from './splineUtils' import TGPU from 'typegpu' import { GPUBrushRenderer } from './gpu/GPUBrushRenderer' import { StrokeProcessor } from './StrokeProcessor' @@ -121,6 +122,7 @@ export function useBrushDrawing(initialSettings?: { const isDrawing = ref(false) const isDrawingLine = ref(false) const lineStartPoint = ref(null) + const lineRemainder = ref(0) const smoothingLastDrawTime = ref(new Date()) const initialDraw = ref(true) @@ -480,22 +482,18 @@ export function useBrushDrawing(initialSettings?: { const drawLine = async ( p1: Point, p2: Point, - compositionOp: CompositionOperation + compositionOp: CompositionOperation, + spacing: number ): Promise => { - const brush_size = store.brushSettings.size - const distance = Math.hypot(p2.x - p1.x, p2.y - p1.y) - const steps = Math.ceil( - distance / ((brush_size / store.brushSettings.smoothingPrecision) * 4) + // Use resampleSegment to get equidistant points + // This handles the "remainder" from previous segments to ensure perfect spacing across vertices + const { points, remainder } = resampleSegment( + [p1, p2], + spacing, + lineRemainder.value ) - const interpolatedOpacity = - 1 / (1 + Math.exp(-6 * (store.brushSettings.opacity - 0.5))) - - 1 / (1 + Math.exp(3)) - const points: Point[] = [] - for (let i = 0; i <= steps; i++) { - const t = i / steps - points.push({ x: p1.x + (p2.x - p1.x) * t, y: p1.y + (p2.y - p1.y) * t }) - } + lineRemainder.value = remainder if (renderer && compositionOp === CompositionOperation.SourceOver) { gpuRender(points) @@ -503,7 +501,7 @@ export function useBrushDrawing(initialSettings?: { // CPU fallback initShape(compositionOp) for (const point of points) { - drawShape(point, interpolatedOpacity) + drawShape(point, 1) // Opacity is handled by the brush texture/layer now } } } @@ -523,12 +521,22 @@ export function useBrushDrawing(initialSettings?: { compositionOp = CompositionOperation.SourceOver } + // Calculate target spacing (same as StrokeProcessor) + const targetSpacing = Math.max(1.0, store.brushSettings.size * 0.12) + if (event.shiftKey && lineStartPoint.value) { isDrawingLine.value = true - await drawLine(lineStartPoint.value, coords_canvas, compositionOp) + await drawLine( + lineStartPoint.value, + coords_canvas, + compositionOp, + targetSpacing + ) } else { isDrawingLine.value = false initShape(compositionOp) + // Reset remainder for new unconnected stroke + lineRemainder.value = targetSpacing // Fix: Don't draw immediately here if we are about to enter the loop. // The handleDrawing loop starts immediately and was causing a double-draw artifact. // await gpuDrawPoint(coords_canvas) @@ -537,7 +545,6 @@ export function useBrushDrawing(initialSettings?: { lineStartPoint.value = coords_canvas // Initialize StrokeProcessor - const targetSpacing = Math.max(1.0, store.brushSettings.size * 0.12) strokeProcessor = new StrokeProcessor(targetSpacing) // Add the first point From 96034468f1719ec42ba5980c07030a565fbfc780 Mon Sep 17 00:00:00 2001 From: trsommer Date: Wed, 19 Nov 2025 03:05:52 +0100 Subject: [PATCH 07/17] fixed inter brush stroke accumulation, fixed bug that created inter brush cross artefacts, fixed brush size --- .../maskeditor/gpu/GPUBrushRenderer.ts | 318 +++++++++++++++++- src/composables/maskeditor/useBrushDrawing.ts | 57 +++- 2 files changed, 339 insertions(+), 36 deletions(-) diff --git a/src/composables/maskeditor/gpu/GPUBrushRenderer.ts b/src/composables/maskeditor/gpu/GPUBrushRenderer.ts index 293da647e09..c47ecbdede9 100644 --- a/src/composables/maskeditor/gpu/GPUBrushRenderer.ts +++ b/src/composables/maskeditor/gpu/GPUBrushRenderer.ts @@ -18,10 +18,16 @@ export class GPUBrushRenderer { private uniformBuffer: GPUBuffer // Pipelines - private renderPipeline: GPURenderPipeline + private renderPipeline: GPURenderPipeline // Standard alpha blend (for composite) + private accumulatePipeline: GPURenderPipeline // SourceOver blend (for accumulation) private blitPipeline: GPURenderPipeline + private compositePipeline: GPURenderPipeline // Multiplies by opacity private uniformBindGroup: GPUBindGroup + // Textures + private currentStrokeTexture: GPUTexture | null = null + private currentStrokeView: GPUTextureView | null = null + constructor(device: GPUDevice) { this.device = device @@ -79,7 +85,8 @@ fn vs( @location(2) size: f32, @location(3) pressure: f32 ) -> VertexOutput { - let radius = (size * pressure) / 2.0; + // Treat 'size' as radius to match the cursor implementation (diameter = 2 * size) + let radius = (size * pressure); let pixelPos = pos + (quadPos * radius); // Convert Pixel Space -> NDC @@ -118,9 +125,7 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { let startFade = v.hardness * 0.99; // Prevent 1.0 singularity let alphaShape = 1.0 - smoothstep(startFade, 1.0, dist); - // Fix: Output Premultiplied Alpha - // This prevents "Black Glow" artifacts when blending semi-transparent colors. - // Standard compositing expects (r*a, g*a, b*a, a). + // Output Premultiplied Alpha let alpha = alphaShape * v.opacity; return vec4(v.color * alpha, alpha); } @@ -148,6 +153,7 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { bindGroupLayouts: [uniformBindGroupLayout] }) + // Standard Render Pipeline (Alpha Blend) this.renderPipeline = device.createRenderPipeline({ layout: pipelineLayout, vertex: { @@ -177,8 +183,56 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { { format: 'rgba8unorm', blend: { - // Fix: Premultiplied Alpha Blending - // Src * 1 + Dst * (1 - SrcAlpha) + color: { + srcFactor: 'one', + dstFactor: 'one-minus-src-alpha', + operation: 'add' + }, + alpha: { + srcFactor: 'one', + dstFactor: 'one-minus-src-alpha', + operation: 'add' + } + } + } + ] + }, + primitive: { topology: 'triangle-list' } + }) + + // Accumulate Pipeline - Uses SourceOver to smooth intersections + // We rely on the composite pass to limit the opacity. + this.accumulatePipeline = device.createRenderPipeline({ + layout: pipelineLayout, + vertex: { + module: brushModuleV, + entryPoint: 'vs', + buffers: [ + { + arrayStride: 8, + stepMode: 'vertex', + attributes: [{ shaderLocation: 0, offset: 0, format: 'float32x2' }] + }, + { + arrayStride: 16, + stepMode: 'instance', + attributes: [ + { shaderLocation: 1, offset: 0, format: 'float32x2' }, + { shaderLocation: 2, offset: 8, format: 'float32' }, + { shaderLocation: 3, offset: 12, format: 'float32' } + ] + } + ] + }, + fragment: { + module: brushModuleF, + entryPoint: 'fs', + targets: [ + { + format: 'rgba8unorm', + blend: { + // SourceOver (Standard Premultiplied Alpha Blend) + // This ensures smooth intersections (no MAX creases) color: { srcFactor: 'one', dstFactor: 'one-minus-src-alpha', @@ -241,8 +295,180 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { }, primitive: { topology: 'triangle-list' } }) + + // --- 4. Composite Pipeline (Merge Accumulator to Main) --- + // Multiplies the accumulated coverage by the brush opacity + const compositeShader = ` + struct BrushUniforms { + brushColor: vec3, + brushOpacity: f32, + hardness: f32, + pad: f32, + screenSize: vec2, + }; + + @vertex fn vs(@builtin(vertex_index) vIdx: u32) -> @builtin(position) vec4 { + var pos = array, 3>( + vec2(-1.0, -1.0), vec2(3.0, -1.0), vec2(-1.0, 3.0) + ); + return vec4(pos[vIdx], 0.0, 1.0); + } + + @group(0) @binding(0) var myTexture: texture_2d; + @group(1) @binding(0) var globals: BrushUniforms; + + @fragment fn fs(@builtin(position) pos: vec4) -> @location(0) vec4 { + let sampled = textureLoad(myTexture, vec2(pos.xy), 0); + // Scale the accumulated coverage by the global brush opacity + return sampled * globals.brushOpacity; + } + ` + + this.compositePipeline = device.createRenderPipeline({ + layout: 'auto', + vertex: { + module: device.createShaderModule({ code: compositeShader }), + entryPoint: 'vs' + }, + fragment: { + module: device.createShaderModule({ code: compositeShader }), + entryPoint: 'fs', + targets: [ + { + format: 'rgba8unorm', + blend: { + color: { + srcFactor: 'one', + dstFactor: 'one-minus-src-alpha', + operation: 'add' + }, + alpha: { + srcFactor: 'one', + dstFactor: 'one-minus-src-alpha', + operation: 'add' + } + } + } + ] + }, + primitive: { topology: 'triangle-list' } + }) + } + + public prepareStroke(width: number, height: number) { + // Create or resize accumulation texture if needed + if ( + !this.currentStrokeTexture || + this.currentStrokeTexture.width !== width || + this.currentStrokeTexture.height !== height + ) { + if (this.currentStrokeTexture) this.currentStrokeTexture.destroy() + this.currentStrokeTexture = this.device.createTexture({ + size: [width, height], + format: 'rgba8unorm', + usage: + GPUTextureUsage.RENDER_ATTACHMENT | + GPUTextureUsage.TEXTURE_BINDING | + GPUTextureUsage.COPY_SRC + }) + this.currentStrokeView = this.currentStrokeTexture.createView() + } + + // Clear the accumulation texture + const encoder = this.device.createCommandEncoder() + const pass = encoder.beginRenderPass({ + colorAttachments: [ + { + view: this.currentStrokeView!, + loadOp: 'clear', + clearValue: { r: 0, g: 0, b: 0, a: 0 }, + storeOp: 'store' + } + ] + }) + pass.end() + this.device.queue.submit([encoder.finish()]) + } + + public renderStrokeToAccumulator( + points: { x: number; y: number; pressure: number }[], + settings: { + size: number + opacity: number + hardness: number + color: [number, number, number] + width: number + height: number + } + ) { + if (!this.currentStrokeView) return + // Use accumulatePipeline (SourceOver) + this.renderStrokeInternal( + this.currentStrokeView, + this.accumulatePipeline, + points, + settings + ) } + public compositeStroke( + targetView: GPUTextureView, + settings: { + opacity: number + color: [number, number, number] + hardness: number // Needed for uniforms, though unused in composite shader + screenSize: [number, number] + } + ) { + if (!this.currentStrokeTexture) return + + // Update Uniforms for Composite Pass (specifically Opacity) + const uData = new Float32Array(UNIFORM_SIZE / 4) + uData[0] = settings.color[0] + uData[1] = settings.color[1] + uData[2] = settings.color[2] + uData[3] = settings.opacity + uData[4] = settings.hardness + uData[5] = 0 // pad + uData[6] = settings.screenSize[0] + uData[7] = settings.screenSize[1] + this.device.queue.writeBuffer(this.uniformBuffer, 0, uData) + + const encoder = this.device.createCommandEncoder() + + const bindGroup0 = this.device.createBindGroup({ + layout: this.compositePipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: this.currentStrokeTexture.createView() } + ] + }) + + // Bind Group 1: Uniforms (for brushOpacity) + const bindGroup1 = this.device.createBindGroup({ + layout: this.compositePipeline.getBindGroupLayout(1), + entries: [{ binding: 0, resource: { buffer: this.uniformBuffer } }] + }) + + const pass = encoder.beginRenderPass({ + colorAttachments: [ + { + view: targetView, + loadOp: 'load', + storeOp: 'store' + } + ] + }) + + pass.setPipeline(this.compositePipeline) + pass.setBindGroup(0, bindGroup0) + pass.setBindGroup(1, bindGroup1) + pass.draw(3) + pass.end() + + this.device.queue.submit([encoder.finish()]) + } + + // Legacy direct render (still useful for single dots or non-accumulating tools) public renderStroke( targetView: GPUTextureView, points: { x: number; y: number; pressure: number }[], @@ -254,6 +480,22 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { width: number height: number } + ) { + this.renderStrokeInternal(targetView, this.renderPipeline, points, settings) + } + + private renderStrokeInternal( + targetView: GPUTextureView, + pipeline: GPURenderPipeline, + points: { x: number; y: number; pressure: number }[], + settings: { + size: number + opacity: number + hardness: number + color: [number, number, number] + width: number + height: number + } ) { if (points.length === 0) return @@ -292,7 +534,7 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { ] }) - pass.setPipeline(this.renderPipeline) + pass.setPipeline(pipeline) pass.setBindGroup(0, this.uniformBindGroup) pass.setVertexBuffer(0, this.quadVertexBuffer) pass.setVertexBuffer(1, this.instanceBuffer) @@ -309,28 +551,67 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { destinationCtx: GPUCanvasContext ) { const encoder = this.device.createCommandEncoder() + const destView = destinationCtx.getCurrentTexture().createView() + + // 1. Clear the destination first + const clearPass = encoder.beginRenderPass({ + colorAttachments: [ + { + view: destView, + loadOp: 'clear', + clearValue: { r: 0, g: 0, b: 0, a: 0 }, + storeOp: 'store' + } + ] + }) + clearPass.end() - // Dynamic BindGroup for the source texture - const bindGroup = this.device.createBindGroup({ + // 2. Draw Main Texture + const bindGroupMain = this.device.createBindGroup({ layout: this.blitPipeline.getBindGroupLayout(0), entries: [{ binding: 0, resource: sourceTexture.createView() }] }) - const pass = encoder.beginRenderPass({ + const passMain = encoder.beginRenderPass({ colorAttachments: [ { - view: destinationCtx.getCurrentTexture().createView(), - loadOp: 'clear', - clearValue: { r: 0, g: 0, b: 0, a: 0 }, + view: destView, + loadOp: 'load', storeOp: 'store' } ] }) + passMain.setPipeline(this.blitPipeline) + passMain.setBindGroup(0, bindGroupMain) + passMain.draw(3) + passMain.end() + + // 3. Draw Current Stroke Accumulator (if exists) + if (this.currentStrokeTexture) { + // Note: We should probably use the composite pipeline here to show the "limited" opacity? + // But blitPipeline is simpler. If we show the raw accumulator, it might look too bright (1.0). + // But that's fine for preview. + const bindGroupStroke = this.device.createBindGroup({ + layout: this.blitPipeline.getBindGroupLayout(0), + entries: [ + { binding: 0, resource: this.currentStrokeTexture.createView() } + ] + }) - pass.setPipeline(this.blitPipeline) - pass.setBindGroup(0, bindGroup) - pass.draw(3) - pass.end() + const passStroke = encoder.beginRenderPass({ + colorAttachments: [ + { + view: destView, + loadOp: 'load', + storeOp: 'store' + } + ] + }) + passStroke.setPipeline(this.blitPipeline) + passStroke.setBindGroup(0, bindGroupStroke) + passStroke.draw(3) + passStroke.end() + } this.device.queue.submit([encoder.finish()]) } @@ -357,5 +638,6 @@ fn fs(v: VertexOutput) -> @location(0) vec4 { this.indexBuffer.destroy() this.instanceBuffer.destroy() this.uniformBuffer.destroy() + if (this.currentStrokeTexture) this.currentStrokeTexture.destroy() } } diff --git a/src/composables/maskeditor/useBrushDrawing.ts b/src/composables/maskeditor/useBrushDrawing.ts index 73bdef971c4..3baffe2f711 100644 --- a/src/composables/maskeditor/useBrushDrawing.ts +++ b/src/composables/maskeditor/useBrushDrawing.ts @@ -510,6 +510,11 @@ export function useBrushDrawing(initialSettings?: { isDrawing.value = true try { + // Initialize Stroke Accumulator + if (renderer && store.maskCanvas) { + renderer.prepareStroke(store.maskCanvas.width, store.maskCanvas.height) + } + let compositionOp: CompositionOperation const currentTool = store.currentTool const coords = { x: event.offsetX, y: event.offsetY } @@ -522,7 +527,8 @@ export function useBrushDrawing(initialSettings?: { } // Calculate target spacing (same as StrokeProcessor) - const targetSpacing = Math.max(1.0, store.brushSettings.size * 0.12) + // Reduced to 0.05 (5%) to ensure smooth edges with MAX blending + const targetSpacing = Math.max(1.0, store.brushSettings.size * 0.05) if (event.shiftKey && lineStartPoint.value) { isDrawingLine.value = true @@ -537,9 +543,6 @@ export function useBrushDrawing(initialSettings?: { initShape(compositionOp) // Reset remainder for new unconnected stroke lineRemainder.value = targetSpacing - // Fix: Don't draw immediately here if we are about to enter the loop. - // The handleDrawing loop starts immediately and was causing a double-draw artifact. - // await gpuDrawPoint(coords_canvas) } lineStartPoint.value = coords_canvas @@ -621,6 +624,20 @@ export function useBrushDrawing(initialSettings?: { strokeProcessor = null } + // Composite the stroke accumulator into the main texture + if (renderer && maskTexture && rgbTexture) { + const isRgb = store.activeLayer === 'rgb' + const targetTex = isRgb ? rgbTexture : maskTexture + + // Use the actual brush opacity for the composite pass + renderer.compositeStroke(targetTex.createView(), { + opacity: store.brushSettings.opacity, + color: [0, 0, 0], // Color is handled by accumulator, this is just for uniforms if needed + hardness: store.brushSettings.hardness, + screenSize: [store.maskCanvas!.width, store.maskCanvas!.height] + }) + } + await copyGpuToCanvas() // Fix: Clear the preview canvas when drawing ends @@ -823,16 +840,22 @@ export function useBrushDrawing(initialSettings?: { if (renderer) { const width = store.maskCanvas!.width const height = store.maskCanvas!.height - const targetView = maskTexture!.createView() const strokePoints = [{ x: point.x, y: point.y, pressure: opacity }] - renderer!.renderStroke(targetView, strokePoints, { + + // Use accumulator with fixed high opacity to build shape + renderer.renderStrokeToAccumulator(strokePoints, { size: store.brushSettings.size, - opacity: store.brushSettings.opacity, + opacity: 0.5, // Fixed flow for smooth accumulation hardness: store.brushSettings.hardness, - color: [1, 1, 1], // white for mask + color: [1, 1, 1], width, height }) + + // Update preview + if (maskTexture && previewContext) { + renderer.blitToCanvas(maskTexture, previewContext) + } } else { drawShape(point, opacity) } @@ -858,7 +881,6 @@ export function useBrushDrawing(initialSettings?: { const isRgb = store.activeLayer === 'rgb' const targetTex = isRgb ? rgbTexture : maskTexture - const targetView = targetTex.createView() // 1. Get Correct Color let color: [number, number, number] = [1, 1, 1] @@ -884,7 +906,8 @@ export function useBrushDrawing(initialSettings?: { const p2 = points[i + 1] const dist = Math.hypot(p2.x - p1.x, p2.y - p1.y) - const stepSize = Math.max(1.0, store.brushSettings.size * 0.12) + // Reduced spacing to 0.05 for smooth blending + const stepSize = Math.max(1.0, store.brushSettings.size * 0.05) const steps = Math.max(1, Math.ceil(dist / stepSize)) for (let step = 0; step <= steps; step++) { @@ -899,15 +922,12 @@ export function useBrushDrawing(initialSettings?: { } } - // Fix: Multiply opacity by a small factor to simulate "Flow". - // This prevents the brush from instantly saturating to 100% opacity/hardness - // when strokes overlap (which they always do). - // 0.1 means you need ~10 overlaps to reach full opacity, which feels natural for a soft brush. - const flowFactor = 0.08 - - renderer.renderStroke(targetView, strokePoints, { + // Render to Accumulator (SourceOver blending) + // Use fixed opacity (0.5) to build up the shape smoothly without creases. + // The final opacity is applied in the composite pass. + renderer.renderStrokeToAccumulator(strokePoints, { size: store.brushSettings.size, - opacity: store.brushSettings.opacity * flowFactor, + opacity: 0.5, hardness: store.brushSettings.hardness, color: color, width: store.maskCanvas!.width, @@ -915,6 +935,7 @@ export function useBrushDrawing(initialSettings?: { }) // 3. Blit to Preview (Visual Feedback) + // Blits Main Texture + Accumulator if (previewContext) { renderer.blitToCanvas(targetTex, previewContext) } From fc970d6559cfa9a6e9b4691f19ebd0c5374a0481 Mon Sep 17 00:00:00 2001 From: trsommer Date: Wed, 19 Nov 2025 04:28:07 +0100 Subject: [PATCH 08/17] fixed tons of bugs --- src/components/maskeditor/BrushCursor.vue | 24 ++- .../maskeditor/BrushSettingsPanel.vue | 10 +- src/composables/maskeditor/brushUtils.test.ts | 47 +++++ src/composables/maskeditor/brushUtils.ts | 36 ++++ .../maskeditor/gpu/GPUBrushRenderer.ts | 48 ++++- src/composables/maskeditor/useBrushDrawing.ts | 183 +++++++++++++----- .../maskeditor/useCanvasHistory.ts | 1 + src/extensions/core/maskeditor/types.ts | 2 +- src/stores/maskEditorStore.ts | 12 +- 9 files changed, 293 insertions(+), 70 deletions(-) create mode 100644 src/composables/maskeditor/brushUtils.test.ts create mode 100644 src/composables/maskeditor/brushUtils.ts diff --git a/src/components/maskeditor/BrushCursor.vue b/src/components/maskeditor/BrushCursor.vue index 0f9c3463c87..f43f69b6d9e 100644 --- a/src/components/maskeditor/BrushCursor.vue +++ b/src/components/maskeditor/BrushCursor.vue @@ -26,6 +26,10 @@