[harness eval #34617] fix: handle existing query params in preview iframe URL#37
[harness eval #34617] fix: handle existing query params in preview iframe URL#37valentinpalkovic wants to merge 2 commits into
Conversation
|
Verify HarnessVerdict: Evidence (vision-check, Vision reasoningRecipe produced no screenshots — cannot verify visible evidence. Replay: Screenshots
|
valentinpalkovic
force-pushed
the
next
branch
3 times, most recently
from
fe2f521 to
e537022
Compare
valentinpalkovic
added
ci:verify
Verify HarnessVerdict: Evidence (vision-check, Vision reasoningThe diff primarily modifies internal URL construction logic in manager-api (query/hash handling), CI/GitHub workflow files, and script utilities. The Playwright recipe validates this logic via Replay: Screenshots
|
valentinpalkovic
added
ci:verify
Verify HarnessVerdict: Evidence (vision-check, Vision reasoningThe diff primarily consists of non-user-visible changes: removal of complex path-resolution logic in CI workflows and scripts, refactoring of URL construction in manager-api, and removal of a CLI flag. The one user-visible change is in code/core/src/manager-api/modules/url.ts which now properly handles query strings and hash fragments in previewBase URLs. The Playwright test validates this logic programmatically via page.evaluate() and inspects iframe src attributes, which are visible in the screenshots, but the screenshots show a normal Storybook UI without obvious indicators of the URL handl Replay: Screenshots
|
valentinpalkovic
added
ci:verify
valentinpalkovic
force-pushed
the
next
branch
2 times, most recently
from
80ccd7d to
745162d
Compare
Verify HarnessVerdict: Evidence (vision-check, Vision reasoningThe diff contains two categories of changes: (1) infrastructure/CI changes to GitHub Actions workflows and scripts (removing outdated path/directory logic), and (2) a functional fix in Replay: Screenshots
|
valentinpalkovic
force-pushed
the
next
branch
3 times, most recently
from
a11176d to
9de9d5b
Compare
valentinpalkovic
added
ci:verify
Verify HarnessNo verdict produced — the workflow failed before the harness ran (likely recipe-author dispatch, deny-regex, or lint). See run log for details. |
valentinpalkovic
added
ci:verify
Verify HarnessVerdict: Evidence (vision-check, Vision reasoningThe diff is primarily composed of non-user-visible changes: CI/workflow configuration, package version downgrades, test file deletions, and removal of minimum-release-age handling logic. The only potentially user-visible changes mentioned in the recipe comments (URL params handling, status icon rendering, icon swap) are internal logic fixes that may not produce observable visual differences in the Storybook UI itself. The screenshots show the sidebar and story rendering working correctly, but the diff's changes are largely invisible to end users. PR-added unit tests: ✅ passed — 6712 passed, 0 failed across 2118 suite(s) Files: How Playwright validated thistest('preview iframe URL handles existing query params; sidebar tree renders', async ({ page }, testInfo) => {
const pageErrors: string[] = [];
const consoleErrors: string[] = [];
page.on('pageerror', (err) => {
pageErrors.push(err.stack ?? err.message ?? String(err));
});
page.on('console', (msg) => {
if (msg.type() === 'error') consoleErrors.push(msg.text());
});
const baseURL =
process.env.STORYBOOK_URL ?? testInfo.project.use.baseURL ?? 'http://localhost:6006';
try {
await page.goto(`${baseURL}/?path=/story/example-button--primary`);
const recipe = new RecipePage(page, expect);
await recipe.waitUntilLoaded();
const iframe = page.locator('#storybook-preview-iframe');
await expect(iframe).toBeVisible();
const src = await iframe.getAttribute('src');
expect(src).toBeTruthy();
expect(src).toContain('id=example-button--primary');
expect(src).toContain('viewMode=story');
expect(src!).not.toMatch(/\?\?/);
expect(src!).not.toMatch(/\?&/);
const sidebar = page.locator('.sidebar-container');
await expect(sidebar).toBeVisible();
const selectedLeaf = page.locator('[data-item-id="example-button--primary"]');
await expect(selectedLeaf).toBeVisible();
await expect(selectedLeaf).toHaveAttribute('data-selected', 'true');
await expect(page.locator('#sb-errordisplay')).toBeHidden();
const previewIframe = recipe.previewIframe();
const previewRoot = previewIframe.locator('#storybook-root, #root');
await expect(previewRoot).toBeVisible();
await sidebar.screenshot({
path: testInfo.outputPath('sidebar-tree.png'),
});
await previewIframe.locator('body').screenshot({
path: testInfo.outputPath('preview.png'),
});
} finally {
await testInfo.attach('pageErrors', {
body: JSON.stringify(pageErrors),
contentType: 'application/json',
});
await testInfo.attach('consoleErrors', {
body: JSON.stringify(consoleErrors),
coReplay: Screenshots
|
valentinpalkovic
force-pushed
the
next
branch
2 times, most recently
from
ad75ba9 to
099b6f7
Compare
valentinpalkovic
added
ci:verify
Verify HarnessVerdict: Evidence (vision-check, Vision reasoningThe diff is primarily a version bump and removal of changelog entries, along with refactoring of internal CI/security configurations and test file changes. These are not user-visible UI changes. The Playwright recipe tests manager-api URL handling and sidebar status icons, but the diff shows no modifications to the actual UI components that would produce observable changes in the screenshots. PR-added unit tests: ✅ passed — 6712 passed, 0 failed across 2118 suite(s) Files: How Playwright validated thistest('sidebar review-changes Clear button shows SweepIcon', async ({ page }, testInfo) => {
const pageErrors: string[] = [];
const consoleErrors: string[] = [];
page.on('pageerror', (err) => {
pageErrors.push(err.stack ?? err.message ?? String(err));
});
page.on('console', (msg) => {
if (msg.type() === 'error') consoleErrors.push(msg.text());
});
const baseURL =
process.env.STORYBOOK_URL ?? testInfo.project.use.baseURL ?? 'http://localhost:6006';
try {
await page.goto(`${baseURL}/?path=/story/example-button--primary`);
const recipe = new RecipePage(page, expect);
await recipe.waitUntilLoaded();
await expect(page.locator('#sb-errordisplay')).toBeHidden();
const controlsTab = page.getByRole('tab', { name: /controls/i });
await controlsTab.click();
const labelInput = page.locator('input[name="label"], textarea[name="label"]').first();
await expect(labelInput).toBeVisible({ timeout: 10000 });
await labelInput.fill('Verify harness MOD');
const saveButton = page.getByRole('button', {
name: /save changes to story|update story/i,
});
await expect(saveButton).toBeVisible({ timeout: 10000 });
await saveButton.click();
const reviewToggle = page.getByRole('switch', { name: /review.+stories/i });
await expect(reviewToggle).toBeVisible({ timeout: 15000 });
await reviewToggle.click();
const clearButton = page.getByRole('button', { name: /^clear$/i });
await expect(clearButton).toBeVisible({ timeout: 10000 });
const iconSvgHtml = await clearButton.locator('svg').first().innerHTML();
expect(iconSvgHtml.length).toBeGreaterThan(0);
await page.locator('.sidebar-container').screenshot({
path: testInfo.outputPath('sidebar-clear-button-sweepicon.png'),
});
await clearButton.screenshot({
path: testInfo.outputPath('clear-button-closeup.png'),
});
} finally {
await testInfo.attach('pageErrors', {
body: JSON.stringify(pageErrors),Replay: Screenshots
|
valentinpalkovic
force-pushed
the
try-pr-34617
branch
from
3d70322 to
ee4d6a0
Compare
Verify HarnessVerdict: How Playwright validated thistest('preview iframe URL composer preserves existing query params and hash', async ({
page,
}, testInfo) => {
const pageErrors: string[] = [];
const consoleErrors: string[] = [];
page.on('pageerror', (err) => {
pageErrors.push(err.stack ?? err.message ?? String(err));
});
page.on('console', (msg) => {
if (msg.type() === 'error') consoleErrors.push(msg.text());
});
const baseURL =
process.env.STORYBOOK_URL ?? testInfo.project.use.baseURL ?? 'http://localhost:6006';
try {
await page.goto(`${baseURL}/?path=/story/example-button--primary`);
const sb = new RecipePage(page, expect);
await sb.waitUntilLoaded();
const errorDisplay = page.locator('#sb-errordisplay');
await expect(errorDisplay).toBeHidden();
const previewIframe = page.frameLocator('#storybook-preview-iframe');
const previewRoot = previewIframe.locator('#storybook-root, #root');
await expect(previewRoot).toBeVisible();
await page.goto(`${baseURL}/?path=/story/example-button--secondary`);
await sb.waitUntilLoaded();
await expect(errorDisplay).toBeHidden();
await expect(previewRoot).toBeVisible();
const cases = await page.evaluate(() => {
const buildPreviewHref = (
previewBase: string,
storyId: string,
viewMode: string
): string => {
const hashIndex = previewBase.indexOf('#');
const previewBaseWithoutHash =
hashIndex === -1 ? previewBase : previewBase.slice(0, hashIndex);
const previewHash = hashIndex === -1 ? '' : previewBase.slice(hashIndex);
const separator = previewBaseWithoutHash.includes('?') ? '&' : '?';
return `${previewBaseWithoutHash}${separator}id=${storyId}&viewMode=${viewMode}${previewHash}`;
};
return {
plain: buildPreviewHref('/iframe.html', 'example-button--primary', 'story'),
withQuery: buildPreviewHref(
'/iframe.html?foo=bar',
'example-button--primary',
'story'
Replay: Screenshots
|
valentinpalkovic
added
ci:verify
Verify HarnessVerdict: How Playwright validated thistest('preview iframe URL preserves existing query params and hash', async ({ page }, testInfo) => {
const pageErrors: string[] = [];
const consoleErrors: string[] = [];
page.on('pageerror', (err) => {
pageErrors.push(err.stack ?? err.message ?? String(err));
});
page.on('console', (msg) => {
if (msg.type() === 'error') consoleErrors.push(msg.text());
});
const baseURL =
process.env.STORYBOOK_URL ?? testInfo.project.use.baseURL ?? 'http://localhost:6006';
try {
await page.goto(`${baseURL}/?path=/story/example-button--primary`);
const sb = new RecipePage(page, expect);
await sb.waitUntilLoaded();
const iframeSrc = await page.locator('#storybook-preview-iframe').getAttribute('src');
expect(iframeSrc).toBeTruthy();
expect(iframeSrc).toContain('id=example-button--primary');
expect(iframeSrc).toContain('viewMode=story');
expect(iframeSrc).not.toMatch(/\?\?/);
const idMatches = iframeSrc!.match(/[?&]id=/g) ?? [];
expect(idMatches.length).toBe(1);
const results = await page.evaluate(() => {
const build = (previewBase: string, storyId: string, viewMode: string) => {
const hashIndex = previewBase.indexOf('#');
const previewBaseWithoutHash =
hashIndex === -1 ? previewBase : previewBase.slice(0, hashIndex);
const previewHash = hashIndex === -1 ? '' : previewBase.slice(hashIndex);
const separator = previewBaseWithoutHash.includes('?') ? '&' : '?';
return `${previewBaseWithoutHash}${separator}id=${storyId}&viewMode=${viewMode}${previewHash}`;
};
return {
noParams: build('/iframe.html', 'a--b', 'story'),
withParams: build('/iframe.html?foo=bar', 'a--b', 'story'),
withHash: build('/iframe.html#frag', 'a--b', 'story'),
withBoth: build('/iframe.html?foo=bar#frag', 'a--b', 'story'),
};
});
expect(results.noParams).toBe('/iframe.html?id=a--b&viewMode=story');
expect(results.withParams).toBe('/Replay: Screenshots
|
valentinpalkovic
force-pushed
the
try-pr-34617
branch
from
ee4d6a0 to
f89fe6c
Compare
Verify HarnessVerdict: Reason: How Playwright validated thistest('manager builds preview iframe URL with id/viewMode params (handles existing query/hash)', async ({ page }, testInfo) => {
const pageErrors: string[] = [];
const consoleErrors: string[] = [];
page.on('pageerror', (err) => {
pageErrors.push(err.stack ?? err.message ?? String(err));
});
page.on('console', (msg) => {
if (msg.type() === 'error') consoleErrors.push(msg.text());
});
const baseURL =
process.env.STORYBOOK_URL ?? testInfo.project.use.baseURL ?? 'http://localhost:6006';
try {
await page.goto(`${baseURL}/?path=/story/example-button--primary`);
const sb = new RecipePage(page, expect);
await sb.waitUntilLoaded();
await expect(page.locator('#sb-errordisplay')).toBeHidden();
const previewIframe = page.frameLocator('#storybook-preview-iframe');
const previewRoot = previewIframe.locator('#storybook-root, #root');
await expect(previewRoot).toBeVisible();
const iframeSrc = await page.locator('#storybook-preview-iframe').getAttribute('src');
expect(iframeSrc, 'preview iframe must have a src').toBeTruthy();
const src = iframeSrc as string;
expect(src).toContain('id=example-button--primary');
expect(src).toContain('viewMode=story');
const parsed = new URL(src, baseURL);
expect(parsed.searchParams.get('id')).toBe('example-button--primary');
expect(parsed.searchParams.get('viewMode')).toBe('story');
const questionMarks = (src.match(/\?/g) ?? []).length;
expect(questionMarks).toBe(1);
await page.goto(`${baseURL}/?path=/docs/example-button--docs`);
await sb.waitUntilLoaded();
const docsSrc = await page.locator('#storybook-preview-iframe').getAttribute('src');
expect(docsSrc, 'preview iframe must have a src on docs route').toBeTruthy();
expect(docsSrc as string).toContain('viewMode=docs');
const docsParsed = new URL(docsSrc as string, baseURL);
expect(docsParsed.searchParams.get('viewMode')).toBe('docs');
const docsQuestionMarks = ((docReplay: Screenshots
|
2 participants
Synthetic fork PR for agentic harness eval against storybookjs#34617.