Performance Tests: Improve collection and reporting

[swissspidy]

What?

• Collect Server-Timing information in performance tests
• Print test results as GitHub Actions summaries for easier overview

See #33645
Fixes #61411

To-do:

• Ideally we'd prepare & reset the environment like core does (disable cron, disable external HTTP requests, flush object cache). But I don't know where to start 🤔 The setup is quite different from core's.

Why?

How?

Testing Instructions

Testing Instructions for Keyboard

Screenshots or screencast

[Mamaduka]

What's the reason for grouping these two in a single PR?

[swissspidy]

@Mamaduka I'm trying to port over some of the recent improvements from the WP core tests, having everything in one PR is more manageable for testing.

[joemcgill]

Nice! The summary on this PR already looks pretty good. A couple non-blocking observations:

• It would be nice to map the spec names for each table to a more human readable format that were more descriptive (example).
• I would flip columns for trunk and the current commit so they are consistent with the before/after format used in other places where we are reporting comparisons.
• Using the full commit hash for the header gets pretty long, possibly truncate to 7 characters, which is the default length for most short style SHA-1 hashes or go with the generic "before" and "after" like we do elsewhere.

[swissspidy]

• It would be nice to map the spec names for each table to a more human readable format that were more descriptive (example).

I think this would require more significant changes . The way the performance reporter is setup, the actual test suite names are not really readable nor unique, nor are the names saved to the result files.

• I would flip columns for trunk and the current commit so they are consistent with the before/after format used in other places where we are reporting comparisons.

This is actually in line with how the data is saved under the hood and presented on the command line when running tests.

See https://github.com/WordPress/gutenberg/actions/runs/8989145786/job/24691572268?pr=61450#step:5:104 for example.

• Using the full commit hash for the header gets pretty long, possibly truncate to 7 characters, which is the default length for most short style SHA-1 hashes or go with the generic "before" and "after" like we do elsewhere.

I was thinking about this but this is not guaranteed to be a commit hash. It can also be a branch name or a tag name.

Again, the output is the same on command line as well.

[youknowriad]

TIL about the job summaries, really nice and avoid digging into the logs. The discoverability is increased a lot without adding huge distracting comments to the PR.

Now I wonder if there's a way to make this a little bit more visible in the main PR page or something (a link or something).

[youknowriad]

@WordPress/gutenberg-core We should look at adding this summary to the Gutenberg job as well.

[tyxla]

also cc @jsnajdr who has been looking at reducing performance test noise and we were also discussing surfacing the results to devs in PRs.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: swissspidy <[email protected]>
Co-authored-by: joemcgill <[email protected]>
Co-authored-by: youknowriad <[email protected]>
Co-authored-by: desrosj <[email protected]>
Co-authored-by: Mamaduka <[email protected]>
Co-authored-by: tyxla <[email protected]>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[youknowriad]

This is a great improvement. Thanks

[Mamaduka]

Ideally we'd prepare & reset the environment like core does (disable cron, disable external HTTP requests, flush object cache). But I don't know where to start 🤔 The setup is quite different from core's.

Can you share where the core setup handles this?

[swissspidy]

Ideally we'd prepare & reset the environment like core does (disable cron, disable external HTTP requests, flush object cache). But I don't know where to start 🤔 The setup is quite different from core's.

Can you share where the core setup handles this?

In test-performance.yml

[desrosj]

Thanks for this, @swissspidy!

For anyone with more ideas about how to use job summaries, Trac-56150 is an overarching ticket for proposing new ways to utilize this feature in our workflows managed in WordPress/wordpress-develop.