Content suggestion: Documenting testing strategies

[dontcallmedom]

(Not a very specific proposal yet, but more a fuzzy idea that would benefit from refining before making an actual RFC.)

What is the new suggestion?

Have a structured approach to documenting how to set up and run tests for Web technologies that need specific approaches, e.g. because of hardware integration à la WebNFC, WebBluetooth, Media Capture, WebXR, WebAuthn, or because they expose a different execution environment (*Workers, shaders in WebGL/WebGPU, WebAssembly, …)

Why is it important or useful?

A reasonable reading of the MDN Web Testing Report would be that it any friction that exists in the process of setting up and running tests make it more likely it won't get done or will get done poorly. MDN may have a role to play there by making the topic more prominent and spreading the word about practical approaches.

[chrisdavidmills]

This is a really good idea @dontcallmedom . I could imagine:

1. An optional section being added to API landing pages titled something like "Testing strategies", which will explan high-level approaches to testing that API, what is needed, etc.
2. A standard guide type called "Testing the ", which would be created as a child page of the main API landing page, and contain further information on more testing strategy specifics, if the information was too long to go in a subsection on the page itself.
3. These sections/pages only being included in cases where the API requires some kind of special testing setup, or there are specific gotchas to know about. Cases where the answer isn't just "load your page in the browser you want to test it in".

[wbamberg]

@dontcallmedom and I had a call today about this, and here are my notes:

---

Overall scope

The initial proposal is about testing strategies for specific Web APIs. But maybe it's worth expanding this out to how MDN documents testing in general. As such it might have a research component:

• what testing documentation do we currently have on MDN? (notably https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing, anything else?)
• assessing the needs implied by the MDN testing report: https://insights.developer.mozilla.org/#web-testing
• designing documentation to fill the gaps

The other strand is about testing strategies for specific Web APIs that have particular testing requirements. Here we might (as Chris suggested: #3917 (comment)) have a "Testing the ... API" guide under the overview page of such APIs.

• it would be very helpful to work with domain experts here. Would love a model where OWD could figure out which docs are needed at a high level and work with various domain experts for various APIs, helping them navigate MDN and help write stuff.
• domain experts could be found through the organizations associated with OWD SC members.

How much detail should we go into for testing strategies?

There's a big range which makes a big difference to the amount of work it is.

• maybe best to experiment here: pick an API, write an outline, get feedback. Start with a smaller guide in a first phase.

Which APIs are promising candidates?

• Web Authentication: there's a proposed WebAuthN Adoption CG (https://www.w3.org/community/blog/2020/04/21/proposed-group-webauthn-adoption-community-group/) which would be very interested in helping to have testing docs on MDN. Also the FIDO alliance (https://fidoalliance.org/)

• WebXR/Immersive Web: strong community push for adoption

• WebRTC: also strong desire to reduce barriers to adoption.

Accessibility testing

Another possible item in here is accessibility testing, which would be very much in our remit, and for which we could certainly get help from OWD allies. This seems like a third possible strand, alongside (1)"overall Web testing strategies" and (2)"testing strategies for specific APIs".

Research project?

Should we consider for Q3, for this to be a research project that determines what testing-related content we are missing on MDN and prioritizes it?

[hamishwillee]

Slightly hesitant to comment on this one. Here's my gut reaction anyway (perhaps worth moving this whole issue to a discussion):

• Yes we need to invest in this. Testing is one of the pillars of software development and it is clearly under-served.
• The cross browser testing stuff is a huge problem all on its own. If it were me I would "pick an API, write an outline, get feedback. Start with a smaller guide in a first phase.".
• I would run a separate project for MDN.
• The whole thing needs to be very iterative. We probably want a constrained scope even on the research report.

And some impressions from my last 6 months ...

• MDN mostly collates client side testing tools running in a particular (firefox) browser.
  • It does talk about linters etc. But it doesn't cover the big picture "cross browser" compatibility issues highlighted by the mdn testing report.
  • I think maybe there is a missing page on types of testing, that then leads to finding the tools you need to do different things.
    • We have useful scattered resources that could be better marshalled. For example we might better "surface" features like: https://developer.mozilla.org/en-US/docs/Tools/Page_Inspector/How_to/Examine_and_edit_CSS#browser_compat_warnings
    • we also jump into topics like "understanding client side frameworks" from top level testing topic. Perhaps not focused enough on the end goal.
  • It took me a fair while in the job (and pointers from Daniel Beck) to discover some of the online tools you can use for cross-browser testing. You might argue that once BCD is in place others should not need these. IMO worth documenting.
  • What would be really nice is a service that can parse your codebase and it spits out a BCD report on what you actually used.
• Whenever I get to a problem case that requires extra hardware I do not have I am blocked. So real problems being highlighted.