[RFC] Realtime component coverage addon

[shilman]

Summary

Use Vitest to provide component code coverage stats in realtime in Storybook's UI

Problem Statement

Code coverage shows how a test suite exercises the lines/branches of code. It's typically accessible via a report, showing the overall "test health" of your project. But when developing a component in isolation, it's helpful to know its code coverage so you can create tests with a rapid feedback loop. This proposal aims to provide realtime code coverage in Storybook's UI that updates as you modify the component or its stories.

Non-goals

Full project coverage is also helpful but challenging to compute in realtime for large projects. Producing a standards-compliant project coverage report in a test run is also a useful project, but it is a separate one.

Implementation

I propose an addon that shows a component's coverage when that component is selected in Storybook's UI:

When the component or its stories are updated, that coverage should also update in real time. This can be achieved if we define "component coverage" as the coverage that comes explicitly from the stories associated with that component. Consider a component Button with stories defined in Button.stories.tsx. Updating Button's component coverage would consist of running all the stories in Button.stories.tsx through a coverage tool like Vitest. This approximation would ignore coverage from stories for higher-level components that include Button. But it's an approximation that should be computable almost instantly.

In addition to displaying coverage as a single metric in the tab, the addon should also show the spans of Component code that are covered by those tests:

We can implement all this on top of the Vitest integration by selectively calling Vite and using its Node API on specific story files.

Prior Art

Storybook has an existing coverage addon. It focuses on project coverage, does not update in real time, and requires some configuration. However, it could (and probably should!) be the basis for this project.

In addition, many code coverage tools are out there with different visualizations. Some notable ones:

• Wallaby
• Vitest
• Codecov
• Istanbul/nyc

Deliverables

1. Addon that displays component coverage in its addon tab and annotated component source in its addon panel.
2. Mechanism for watching files and invoking Vitest on a story file when the file or its component is edited. This might be part of Vitest's watch mode or a separate mechanism
3. Mechanism for reporting the results from Vitest back to the addon

Risks

1. This could result in Storybook using more computing resources and lead to a bad experience for users on slow machines
2. Depending on the detailed implementation, this could conflict with Vitest-based functional testing
3. The "component coverage" approximation is not useful. Users feel that they need coverage from higher-level components.

Unresolved Questions

• Is this compact with the Vitest browser and mode, or do we need to use happy-dom? If the latter, does it open us up to many additional corner cases?
• Can we use Vitest watch mode, or do we need a separate mechanism to watch for file changes and invoke Vitest separately?

Alternatives considered / Abandoned Ideas

No response

[donaldpipowitch]

Stupid question, is the real time feature a "must" or a "can"? To phrase differently: for a deployed storybook instance, would it also be possible to load an existing coverage report and embed it there?

[shilman]

For context we've put this idea on the back burner. In 8.5 we shipped a non-realtime version that just builds a typical istanbul report.

In CI you'd run vitest and can configure that to produce a coverage report also. We haven't documented best practices there but now that you bring it up we probably should. Cc @kylegach

[donaldpipowitch]

Cool, thank you!