test: automate manual test cases with wp-env E2E tests by dcalhoun · Pull Request #329 · wordpress-mobile/GutenbergKit

dcalhoun · 2026-02-20T16:35:25Z

What?

Automates all remaining manual test cases from docs/test-cases.md as Playwright E2E tests running against wp-env, and migrates the existing E2E suite to use wp-env for a more realistic testing environment.

Also switches wp-env from the Docker runtime to the WordPress Playground runtime (#344), which runs WordPress entirely in WebAssembly—eliminating the Docker dependency for both local development and CI.

Why?

Ref CMM-1253.

The manual test cases (media uploads, text alignment, embeds, color/gradient, gallery, patterns) all require a real WordPress backend. With wp-env now available, these can be automated to run in CI and reduce the manual testing burden.

Playground eliminates the Docker dependency, reducing infrastructure complexity. Our E2E tests are HTTP-based (Playwright + REST API) and don't require container-specific features. Gutenberg core is also exploring embracing Playground in WordPress/gutenberg#75506.

How?

New E2E tests

wp-env fixtures (e2e/wp-env-fixtures.js): Reads credentials from .wp-env.credentials.json and fetches/caches editor settings from the WP REST API.
EditorPage migration (e2e/editor-page.js): setup() now injects wp-env config (siteApiRoot, authHeader, editorSettings) by default. Added helper methods: getBlockAttribute(), setTextAlignment(), openBlockSettings(), waitForMediaUpload().
9 new test files: Image upload, video upload, gallery block, audio upload, file upload, text alignment, embedded content, color/gradient, pattern insertion.
Test assets: Minimal binary files (PNG, MP4, MP3, PDF) for upload tests.
Docs cleanup: Removed S.4 (save/publish — delegated to native host) and automated test case descriptions from docs/test-cases.md. Only S.1 (Undo/Redo) remains as a manual test.

wp-env Playground migration (#344)

Makefile: Add --runtime=playground to wp-env start. Remove wp-env-logs and wp-env-cli targets (unsupported by Playground). Add wp-env-android / wp-env-android-reset targets for Android emulator URL remapping.
.wp-env.json: Explicitly set WP_ENVIRONMENT_TYPE to local so WordPress enables application passwords on non-HTTPS sites.
@wordpress/env patch: Remove the --login CLI flag and Blueprint login step from the Playground runtime. Playground's auto-login causes 302 redirects on every first request and injects WordPress session cookies that interfere with application password authentication (wordpress-playground#1611). Also use the rewrite-wp-config method for defineWpConfigConsts so constants persist to wp-config.php for subsequent HTTP requests. WordPress admin login remains available via wp-login.php with admin/password.
bin/wp-env-setup.sh: Rewrite to use REST API calls (cookie-based login → nonce → application password creation) instead of wp-env run cli, which Playground does not support.
gutenbergkit-jetpack-blocks.php mu-plugin: Auto-activate the Jetpack blocks module on plugins_loaded, replacing the WP-CLI call. This is runtime-agnostic and works with both Docker and Playground.
Android emulator URL override: Add wp-env/android-url-override.php, a mu-plugin that remaps localhost/127.0.0.1 to 10.0.2.2 in WordPress URL output.
E2E test updates: Update upload URL assertions to accept both localhost and 127.0.0.1. Fix cover block test to use the active theme's palette color names (Contrast instead of Black). Remove the verse block alignment test (exercises Gutenberg core behavior outside GutenbergKit's domain).
Documentation: Update docs/code/local-wordpress.md to reflect the Playground runtime, remove Docker references, and document the Android emulator URL remapping workflow.

Testing Instructions

Run make wp-env-clean to remove any existing environment.
Run make wp-env-start — should start without Docker.
Verify credentials file exists: cat .wp-env.credentials.json
Run all E2E tests: make test-e2e
Verify all 45 tests pass (29 existing + 16 new).
Verify WordPress admin login at http://localhost:8888/wp-login.php with admin/password.

Accessibility Testing Instructions

N/A — no UI changes.

Screenshots or screencast

N/A — test infrastructure only.

GutenbergKit delegates save/publish to the native host app, so this test case does not belong in the editor's E2E suite.

Introduce `e2e/wp-env-fixtures.js` which reads wp-env credentials from `.wp-env.credentials.json` and fetches editor settings from the WP REST API. Update `EditorPage.setup()` to inject wp-env config (siteApiRoot, authHeader, editorSettings) by default so all E2E tests run against a real WordPress backend. Add helper methods to EditorPage: `getBlockAttribute()`, `setTextAlignment()`, `openBlockSettings()`, and `waitForMediaUpload()`. Update `editor-error.spec.js` to use an unreachable API root so the plugin load failure test still works with wp-env credentials present.

Add minimal test files used by the media upload E2E tests: - test-image.png (1x1 RGBA PNG) - test-image-2.png (1x1 RGBA PNG, second image for gallery) - test-video.mp4 (minimal MP4 container) - test-audio.mp3 (minimal MP3 frame) - test-file.pdf (minimal PDF document)

Add Playwright E2E tests that verify media upload flows against wp-env: - S.2 image-upload: Upload image via Image block, verify block attributes - S.3 video-upload: Upload video via Video block, verify src attribute - F.4 gallery-block: Upload multiple images, verify inner blocks and caption editing - F.6 audio-upload: Upload audio via Audio block, verify src attribute - F.7 file-upload: Upload file via File block, verify href and fileName All tests use the filechooser event to simulate file selection and `waitForMediaUpload()` to wait for the upload to complete.

Add Playwright E2E tests for non-upload editor features against wp-env: - F.1 text-alignment: Verify center/right alignment on paragraph and verse blocks via the toolbar dropdown - F.2 embed-content: Insert YouTube embed URL and verify oEmbed resolution; test fallback for non-embeddable URLs - F.3 color-gradient: Apply background color via Block Settings UI, verify theme gradients are available, apply gradient via data store - F.5 pattern-insertion: Fetch patterns from WP REST API, insert parsed pattern content via wp.blocks.parse() + insertBlocks()

Automated tests live in the E2E suite and don't need to be duplicated in the manual test cases document.

* ci: Switch wp-env from Docker to WordPress Playground runtime Replace the Docker-based wp-env runtime with WordPress Playground, which runs WordPress entirely in WebAssembly — eliminating the Docker dependency for local development and CI. - Add `--runtime=playground` flag to `wp-env start` in Makefile - Patch `@wordpress/env` to remove `--login` flag and Blueprint login step that cause 302 redirects interfering with application password auth (wordpress-playground#1611) - Rewrite `bin/wp-env-setup.sh` to use REST API (cookie-based login + nonce) instead of `wp-env run cli` (unsupported by Playground) - Add `gutenbergkit-jetpack-blocks.php` mu-plugin to auto-activate the Jetpack blocks module, replacing the WP-CLI call - Remove `wp-env-logs` and `wp-env-cli` Makefile targets (unsupported by Playground runtime) - Update documentation to reflect Playground runtime and remove Docker references * fix: Use Node.js instead of Python for JSON parsing in wp-env setup Replace python3 with node for extracting the application password from the REST API response. Node.js is guaranteed to be available since wp-env requires it, whereas python3 may not be installed on CI agents. Also prevent set -e from masking the error by allowing the node command to fail gracefully, so the subsequent error message with the raw response is displayed. * fix: Set WP_ENVIRONMENT_TYPE to local in wp-env config WordPress disables application passwords on non-HTTPS sites unless WP_ENVIRONMENT_TYPE is set to "local". While wp-env adds this constant automatically via the Blueprint's defineWpConfigConsts step, it may not persist reliably on CI with the Playground runtime. Explicitly setting it in .wp-env.json ensures it is always included. * fix: Handle Blueprint processing delay in Playground runtime The Playground runtime starts accepting HTTP requests before Blueprint steps (plugin installation, wp-config rewriting) have fully completed. This causes the credential provisioning script to fail because application passwords are not yet available. Two changes fix this: 1. Patch wp-env to use `rewrite-wp-config` method for the Blueprint `defineWpConfigConsts` step. The default `define-before-run` method only sets PHP constants during Blueprint execution — they don't persist to wp-config.php for subsequent HTTP requests. 2. Wrap the credential creation flow in a retry loop (up to 30s) so the setup script waits for the Blueprint to finish processing before giving up. * fix: Accept 127.0.0.1 in upload URL assertions The Playground runtime resolves localhost to 127.0.0.1 in WordPress's WP_SITEURL, so media upload URLs use 127.0.0.1:8888 instead of localhost:8888. Update assertions to check for :8888/wp-content/uploads/ which works regardless of hostname resolution. * fix: Select Contrast color in cover block test The active theme uses named palette colors (Contrast, Base, etc.) rather than generic names (Black, White, etc.), so the test was timing out looking for a "Black" button that doesn't exist. * fix: Ensure verse block is selected before setting alignment On slower CI machines, clicking the verse block's contenteditable element may not fully synchronize Gutenberg's block selection state before the toolbar interaction runs. Explicitly selecting the block via the data store ensures the alignment toolbar targets the correct block. * test: Add verse block debug test for CI diagnostics Temporary diagnostic test to capture block type supports, button count, and attribute state on CI to debug the verse alignment failure. * fix: Check top-level textAlign attribute for verse block The verse block uses a custom AlignmentToolbar that stores alignment in the top-level `attributes.textAlign`, unlike paragraph blocks which use the block supports system (`attributes.style.typography.textAlign`). Remove the temporary verse debug test. * test: Remove verse block alignment test The verse block's text alignment behavior is a Gutenberg core concern, not a GutenbergKit responsibility. GutenbergKit focuses on constructing a minimal editor UI and native bridge communication. * feat: Add Android emulator URL remapping for wp-env The Playground runtime resolves localhost to 127.0.0.1 in WordPress's site URL, which is unreachable from the Android emulator. Add a mu-plugin-based URL override that remaps localhost/127.0.0.1 to 10.0.2.2 (the emulator's alias for the host machine). - Add `make wp-env-android` / `make wp-env-android-reset` targets - Add `wp-env/android-url-override.php` mu-plugin that filters site_url, home_url, content_url, plugins_url, and upload_dir - Allow cleartext traffic to 127.0.0.1 in Android network security config (needed for Playground runtime) - Document the workflow in local-wordpress.md * revert: Remove 127.0.0.1 from Android network security config The 127.0.0.1 cleartext traffic entry is unnecessary — the mu-plugin remaps all URLs to 10.0.2.2 before they reach the Android app.

Replace hardcoded `:8888/wp-content/uploads/` strings in upload test assertions with a shared `uploadsPathPattern` constant derived from the wp-env credentials file. Keeps the pattern hostname-agnostic (Playground may resolve localhost to 127.0.0.1) while eliminating the hardcoded port.

- Delete pattern-insertion.spec.js (pure REST API / data store, no UI) - Rework gallery caption test to click image, use "Add caption" toolbar button, and type into the figcaption - Rework color-gradient: remove settings check and data store gradient tests, add gradient test that uses the block settings UI

Remove the gradient test that relied on preset gradient swatches (`.components-circular-option-picker__option`) which are not rendered by the Twenty Twenty-Five theme's gradient picker UI. Fix the gallery caption test by clicking the block wrapper (`[data-type="core/image"]`) instead of the `<img>` element directly, which was failing because the `<figure>` wrapper intercepts pointer events.

dcalhoun · 2026-03-03T19:45:21Z

As noted in the description and code comments, these changes largely relate to circumventing WordPress Playground's lack of support for the wp-env run capability. We must instead rely upon the REST API.

dcalhoun · 2026-03-03T19:48:40Z

+	namespaceExcludedPaths: [],
 	post: {
-		id: -1,
+		id: 1,


Relies upon the default "Hello world!" blog post included in WordPress core, which has post ID 1.

dcalhoun · 2026-03-03T21:31:01Z

These patches are required to circumvent wp-env enabling cookie-based authentication by default. Ideally, we contribute these upstream.

dcalhoun · 2026-03-03T21:31:42Z

Replace wp-env run usage (unsupported by Playground) with custom mu-plugins.

dcalhoun · 2026-03-03T21:31:53Z

Replace wp-env run usage (unsupported by Playground) with custom mu-plugins.

dcalhoun · 2026-03-03T21:32:10Z

 		"WP_DEBUG": true,
-		"WP_DEBUG_LOG": true
+		"WP_DEBUG_LOG": true,
+		"WP_ENVIRONMENT_TYPE": "local"


Ensure application passwords are enabled for the non-TLS environment.

dcalhoun · 2026-03-03T22:18:27Z

All test cases were automated or removed as unrelated to GutenbergKit's domain—e.g., post publishing belongs in host, pattern insertion/gradient selection is largely Gutenberg code.

kean

Fantastic work on automating the manual E2E testing work. I envy the extent of E2E testing in GBK 👏

I've reviewed the changes, briefly since it doesn't affect the production code, and nothing particularly stood out.

From the automated review (claude /review), one note did stand out:

Embed test depends on external YouTube oEmbed — embed-content.spec.js hits YouTube's oEmbed endpoint via the WP proxy. If YouTube rate-limits or is unreachable, this test will flake in CI. Consider whether this risk is acceptable or if the oEmbed proxy response could be mocked.

dcalhoun · 2026-03-04T18:36:56Z

Thanks for the review and praise, @kean!

Embed test depends on external YouTube oEmbed — embed-content.spec.js hits YouTube's oEmbed endpoint via the WP proxy. If YouTube rate-limits or is unreachable, this test will flake in CI. Consider whether this risk is acceptable or if the oEmbed proxy response could be mocked.

I'll follow up on this in a separate PR.

dcalhoun added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Feb 20, 2026

Base automatically changed from feat/local-wordpress-wp-env to trunk February 25, 2026 16:30

dcalhoun force-pushed the test/automate-manual-test-cases branch 2 times, most recently from 15264f3 to 5d110c5 Compare February 25, 2026 19:19

dcalhoun changed the title ~~test: Automate manual test cases with wp-env E2E tests~~ test: automate manual test cases with wp-env E2E tests Feb 27, 2026

dcalhoun force-pushed the test/automate-manual-test-cases branch from 5d110c5 to 2e1ca21 Compare February 27, 2026 15:28

dcalhoun mentioned this pull request Feb 27, 2026

docs: remove manual undo/redo test case #341

Merged

dcalhoun added 8 commits March 3, 2026 14:36

docs: Remove S.4 save/publish from manual test cases

2a4a716

GutenbergKit delegates save/publish to the native host app, so this test case does not belong in the editor's E2E suite.

docs: Remove automated test cases from test-cases.md

545f19e

Automated tests live in the E2E suite and don't need to be duplicated in the manual test cases document.

test: Remove legacy test case ID suffixes from describe blocks

8d7c8c7

dcalhoun force-pushed the test/automate-manual-test-cases branch from afa405c to 7c268fc Compare March 3, 2026 19:39

dcalhoun added 3 commits March 3, 2026 16:13

dcalhoun marked this pull request as ready for review March 4, 2026 02:42

dcalhoun commented Mar 4, 2026

View reviewed changes

dcalhoun requested a review from kean March 4, 2026 02:42

kean approved these changes Mar 4, 2026

View reviewed changes

dcalhoun merged commit aa8eaf1 into trunk Mar 4, 2026
14 checks passed

dcalhoun deleted the test/automate-manual-test-cases branch March 4, 2026 18:37

dcalhoun mentioned this pull request Mar 6, 2026

fix: mock oEmbed proxy in embed E2E test to prevent CI flakes #354

Merged

Navigation Menu

Repository navigation

test: automate manual test cases with wp-env E2E tests - #329

test: automate manual test cases with wp-env E2E tests#329
dcalhoun merged 11 commits into
trunkfrom
test/automate-manual-test-cases

dcalhoun commented Feb 20, 2026 •

edited

Loading

Uh oh!

dcalhoun Mar 3, 2026

Uh oh!

dcalhoun Mar 3, 2026

Uh oh!

dcalhoun Mar 3, 2026

Uh oh!

dcalhoun Mar 3, 2026

Uh oh!

dcalhoun Mar 3, 2026

Uh oh!

dcalhoun Mar 3, 2026

Uh oh!

dcalhoun Mar 3, 2026

Uh oh!

kean left a comment •

edited

Loading

Uh oh!

dcalhoun commented Mar 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dcalhoun commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

How?

New E2E tests

wp-env Playground migration (#344)

Testing Instructions

Accessibility Testing Instructions

Screenshots or screencast

Uh oh!

dcalhoun Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

dcalhoun Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

dcalhoun Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

dcalhoun Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

dcalhoun Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

dcalhoun Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

dcalhoun Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

kean left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dcalhoun commented Mar 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dcalhoun commented Feb 20, 2026 •

edited

Loading

kean left a comment •

edited

Loading