fix: stabilize flaky screenshot tests that rely on search box timings#9426
Merged
christian-byrne merged 3 commits intomainfrom Mar 5, 2026
Merged
fix: stabilize flaky screenshot tests that rely on search box timings#9426christian-byrne merged 3 commits intomainfrom
christian-byrne merged 3 commits intomainfrom
Conversation
…eview Two flaky screenshot tests were identified during the PR #9400 snapshot update, where baselines regenerated without any code changes affecting them: 1. added-node-no-connection (nodeSearchBox.spec.ts): Root cause: fillAndSelectFirstNode('KSampler') selected nth(0) from the dropdown, but search result ordering is non-deterministic when the search box opens via link release (with filter chips). Sometimes 'Preview Image' appeared first instead of 'KSampler'. Fix: Add exact:true option that uses aria-label selector to click the specific matching result instead of blindly selecting the first item. 2. no-workflow-webp (loadWorkflowInMedia.spec.ts): Root cause: Dropping no_workflow.webp (no embedded workflow) creates a LoadImage node and uploads the file. The image preview then loads asynchronously via /view API. The screenshot was taken before the upload and preview load completed, causing the preview to sometimes show the image and sometimes show a black/empty area. Fix: Wait for the upload response and subsequent /view response before taking the screenshot. Both baselines are deleted for CI regeneration. Fixes #4658 Amp-Thread-ID: https://ampcode.com/threads/T-019cbc87-ce1f-7338-93c7-d0677a632805
christian-byrne
added
the
New Browser Test Expectations
dosubot
bot
added
the
size:M
Contributor
📝 WalkthroughWalkthroughThis change addresses flaky Playwright tests by introducing an exact-match selection option to the ComfyNodeSearchBox fixture method and adding deterministic rendering waits for specific file upload scenarios. The modifications enhance test stability through alternative node selection logic and proper async handling. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Comment |
🎭 Playwright: ✅ 547 passed, 0 failed · 6 flaky📊 Browser Reports
|
🎨 Storybook: ✅ Built — View Storybook |
📦 Bundle: 4.49 MB gzip ⚪ 0 BDetailsSummary
Category Glance App Entry Points — 18 kB (baseline 18 kB) • ⚪ 0 BMain entry bundles and manifests
Graph Workspace — 919 kB (baseline 919 kB) • ⚪ 0 BGraph editor runtime, canvas, workflow orchestration
Views & Navigation — 72.4 kB (baseline 72.4 kB) • ⚪ 0 BTop-level views, pages, and routed surfaces
Panels & Settings — 436 kB (baseline 436 kB) • ⚪ 0 BConfiguration panels, inspectors, and settings screens
User & Accounts — 16.1 kB (baseline 16.1 kB) • ⚪ 0 BAuthentication, profile, and account management bundles
Editors & Dialogs — 779 B (baseline 779 B) • ⚪ 0 BModals, dialogs, drawers, and in-app editors
UI Components — 47 kB (baseline 47 kB) • ⚪ 0 BReusable component library chunks
Data & Services — 2.73 MB (baseline 2.73 MB) • ⚪ 0 BStores, services, APIs, and repositories
Utilities & Hooks — 56.6 kB (baseline 56.6 kB) • ⚪ 0 BHelpers, composables, and utility bundles
Vendor & Third-Party — 8.86 MB (baseline 8.86 MB) • ⚪ 0 BExternal libraries and shared vendor chunks
Other — 7.92 MB (baseline 7.92 MB) • ⚪ 0 BBundles that do not match a named category
|
⚡ Performance Report
Raw data{
"timestamp": "2026-03-05T07:46:52.601Z",
"gitSha": "3c5f98cd9d6d876f3f2acaa2f1b76f4649546de7",
"branch": "ci/fix-flaky-screenshots-2",
"measurements": [
{
"name": "canvas-idle",
"durationMs": 2023.727000000008,
"styleRecalcs": 124,
"styleRecalcDurationMs": 19.24,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 401.971,
"heapDeltaBytes": -2647348
},
{
"name": "canvas-idle",
"durationMs": 2038.7960000000476,
"styleRecalcs": 124,
"styleRecalcDurationMs": 21.112000000000002,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 423.348,
"heapDeltaBytes": -3867552
},
{
"name": "canvas-idle",
"durationMs": 2024.3189999999913,
"styleRecalcs": 123,
"styleRecalcDurationMs": 22.990000000000002,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 418.92400000000004,
"heapDeltaBytes": -3083772
},
{
"name": "canvas-mouse-sweep",
"durationMs": 2094.102000000021,
"styleRecalcs": 188,
"styleRecalcDurationMs": 57.813,
"layouts": 12,
"layoutDurationMs": 3.6500000000000004,
"taskDurationMs": 1093.573,
"heapDeltaBytes": -2542164
},
{
"name": "canvas-mouse-sweep",
"durationMs": 1890.5749999999557,
"styleRecalcs": 173,
"styleRecalcDurationMs": 57.024,
"layouts": 12,
"layoutDurationMs": 3.6769999999999996,
"taskDurationMs": 865.388,
"heapDeltaBytes": -1592068
},
{
"name": "canvas-mouse-sweep",
"durationMs": 1913.3469999999875,
"styleRecalcs": 176,
"styleRecalcDurationMs": 60.68899999999999,
"layouts": 12,
"layoutDurationMs": 4.2540000000000004,
"taskDurationMs": 878.211,
"heapDeltaBytes": -1186064
},
{
"name": "dom-widget-clipping",
"durationMs": 563.33699999999,
"styleRecalcs": 42,
"styleRecalcDurationMs": 12.466,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 348.75999999999993,
"heapDeltaBytes": 7745916
},
{
"name": "dom-widget-clipping",
"durationMs": 601.2910000000034,
"styleRecalcs": 44,
"styleRecalcDurationMs": 13.113000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 382.682,
"heapDeltaBytes": 7648148
},
{
"name": "dom-widget-clipping",
"durationMs": 595.7500000000095,
"styleRecalcs": 44,
"styleRecalcDurationMs": 13.402000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 382.6120000000001,
"heapDeltaBytes": 7656632
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 596.5860000000021,
"styleRecalcs": 73,
"styleRecalcDurationMs": 15.412,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 424.682,
"heapDeltaBytes": -7753932
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 597.5050000000124,
"styleRecalcs": 74,
"styleRecalcDurationMs": 15.605999999999998,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 429.16700000000003,
"heapDeltaBytes": -7811884
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 632.974000000047,
"styleRecalcs": 75,
"styleRecalcDurationMs": 19.565999999999995,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 436.81000000000006,
"heapDeltaBytes": 14453068
},
{
"name": "subgraph-idle",
"durationMs": 2022.5560000000087,
"styleRecalcs": 124,
"styleRecalcDurationMs": 23.055,
"layouts": 1,
"layoutDurationMs": 0.26700000000000007,
"taskDurationMs": 402.52700000000004,
"heapDeltaBytes": -2006176
},
{
"name": "subgraph-idle",
"durationMs": 1991.5639999999826,
"styleRecalcs": 120,
"styleRecalcDurationMs": 22.011000000000003,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 401.046,
"heapDeltaBytes": -2097780
},
{
"name": "subgraph-idle",
"durationMs": 1999.5890000000145,
"styleRecalcs": 121,
"styleRecalcDurationMs": 22.826999999999998,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 426.424,
"heapDeltaBytes": -2463588
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1716.4429999999697,
"styleRecalcs": 156,
"styleRecalcDurationMs": 48.251000000000005,
"layouts": 16,
"layoutDurationMs": 4.529,
"taskDurationMs": 743.614,
"heapDeltaBytes": -4005460
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1741.6410000000155,
"styleRecalcs": 157,
"styleRecalcDurationMs": 48.986000000000004,
"layouts": 16,
"layoutDurationMs": 4.657,
"taskDurationMs": 747.549,
"heapDeltaBytes": -5316912
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1683.9650000000006,
"styleRecalcs": 154,
"styleRecalcDurationMs": 48.675,
"layouts": 16,
"layoutDurationMs": 4.388000000000001,
"taskDurationMs": 750.016,
"heapDeltaBytes": -3958568
}
]
} |
Contributor
Author
|
/update-playwright |
dosubot
bot
added
size:S
|
Updating Playwright Expectations |
benceruleanlu
approved these changes
Mar 5, 2026
Member
benceruleanlu
left a comment
benceruleanlu
left a comment
There was a problem hiding this comment.
Note: loadWorkflowInMedia.spec.ts wasn't modified here but was mentioned in the PR description. There are no new snapshots here. Stabilization LGTM.
Contributor
Author
Good point, thank you. Update the PR descript/title. @coderabbitai this was the part of the PR description I removed, we didn't end up doing it in this PR:
Can you make a followup/tracking Issue to do this later and assign it to me? |
christian-byrne
changed the title
2 tasks
Contributor
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: Length of output: 57
It's assigned to you and includes:
🧠 Learnings used |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes flaky screenshot test that was identified during the PR #9400 snapshot update, where baselines regenerated without any code changes affecting them.
Root cause:
fillAndSelectFirstNode('KSampler')selectednth(0)from the autocomplete dropdown, but search result ordering is non-deterministic when the search box opens via link release with filter chips. Sometimes 'Preview Image' appeared as the first result instead of 'KSampler', causing a completely different node to be added.Fix: Added an
exact: trueoption tofillAndSelectFirstNodethat uses anaria-labelselector to click the specific matching result instead of blindly selecting the first item.┆Issue is synchronized with this Notion page by Unito