Core: Prevent stories lookup in node_modules by valentinpalkovic · Pull Request #25214 · storybookjs/storybook

valentinpalkovic · 2023-12-13T12:45:32Z

What I did

A glob pattern like
"../packages/**/*.stories.@(js|jsx|mjs|ts|tsx)" will look up stories even in submodule node_module folders like ../packages/a/node_modules/**/*.stories.*

A couple of places were affected where we don't exclude node_modules

export-order-loader hasn't excluded node_modules
csf-plugin was running on all stories, even on stories in node_module folders
the dynamic imports took node_module stories also into account. This only happens now, if node_modules is explicitly part of the stories glob pattern.

Related PRs

Core: Fix indexing errors by excluding node_modules stories #22873

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

stories
unit tests
integration tests
end-to-end tests

Manual testing

This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!

Documentation

Add or update documentation reflecting your changes
If you are deprecating/removing a feature, make sure to update
MIGRATION.MD

Checklist for Maintainers

When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli/src/sandbox-templates.ts
Make sure this PR contains one of the labels below:
Available labels
- bug: Internal changes that fixes incorrect behavior.
- maintenance: User-facing maintenance tasks.
- dependencies: Upgrading (sometimes downgrading) dependencies.
- build: Internal-facing build tooling & test updates. Will not show up in release changelog.
- cleanup: Minor cleanup style change. Will not show up in release changelog.
- documentation: Documentation only changes. Will not show up in release changelog.
- feature request: Introducing a new feature.
- BREAKING CHANGE: Changes that break compatibility in some way with current major version.
- other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

tmeasday

I think this makes sense, and could be a big win. But what if someone does actually want to include stories from node_modules? Is that just not allowed?

tmeasday · 2023-12-13T23:30:31Z

+function adjustRegexToExcludeNodeModules(originalRegex: RegExp) {
+  const originalRegexString = originalRegex.source;
+  const startsWithCaret = originalRegexString.startsWith('^');
+  const excludeNodeModulesPattern = startsWithCaret ? '(?!.*node_modules)' : '^(?!.*node_modules)';
+
+  // Combine the new exclusion pattern with the original regex
+  const adjustedRegexString = startsWithCaret
+    ? `^${excludeNodeModulesPattern}${originalRegexString.substring(1)}`
+    : excludeNodeModulesPattern + originalRegexString;
+
+  // Create and return the new regex
+  return new RegExp(adjustedRegexString);
+}


Would it make sense to instead of doing this adjustment here to do it in normalizeStoriesEntry?

storybook/code/lib/core-common/src/utils/normalize-stories.ts

Line 35 in c98bddd

export const normalizeStoriesEntry = (

I guess I'm just wondering where else we use the stories entry to lookup or watch files? For instance, I think we do it here also:

storybook/code/lib/core-server/src/utils/watch-story-specifiers.ts

Lines 26 to 31 in c98bddd

export function watchStorySpecifiers(

specifiers: NormalizedStoriesSpecifier[],

options: { workingDir: Path },

onInvalidate: (specifier: NormalizedStoriesSpecifier, path: Path, removed: boolean) => void

) {

// See https://www.npmjs.com/package/watchpack for full options.

So given your comment below, we might not want to entirely disregard stories from node_modules for indexing / watching the index. Or at least if we did we'd want to commit to stories from the node_modules not being supported at all in SB.

I think we should definitely consider that (for perf reasons). I can imagine that setting up file watchers recursively in node_modules as watch-story-specificers may very well be doing could be very costly. At the least we could feature flag it so it's off by default.

I would say we do that work separate to this PR, but the thing is if we were to adjust the stories entry, we wouldn't need to make the other changes that are in this PR.

Would it make sense to instead of doing this adjustment here to do it in normalizeStoriesEntry?

storybook/code/lib/core-common/src/utils/normalize-stories.ts

Line 35 in c98bddd

export const normalizeStoriesEntry = (

I guess I'm just wondering where else we use the stories entry to lookup or watch files? For instance, I think we do it here also:

storybook/code/lib/core-server/src/utils/watch-story-specifiers.ts

Lines 26 to 31 in c98bddd

export function watchStorySpecifiers(

specifiers: NormalizedStoriesSpecifier[],

options: { workingDir: Path },

onInvalidate: (specifier: NormalizedStoriesSpecifier, path: Path, removed: boolean) => void

) {

// See https://www.npmjs.com/package/watchpack for full options.

@tmeasday If you don't mind, I would like to pair on this since I am not sure how easily achievable this is. normalizeStoriesEntry doesn't deal with regex, but with globs and having excluding patterns in a glob string is not that easy. We could change normalizeStoriesEntry completely to not return a glob string for files, but instead a e.g. picomatch matcher, which can be configured to ignore certain patterns: https://github.com/micromatch/picomatch. But this would require major refactorings.

valentinpalkovic · 2023-12-14T10:16:58Z

I think this makes sense, and could be a big win. But what if someone does actually want to include stories from node_modules? Is that just not allowed?

@tmeasday Currently, if someone wants to include stories from node_modules, they can, but no transformations would be applied to it. The babel-loader/swc-loaders currently ignore node_modules and the CSF-plugin or the export-order-loader plugin. I think it is okay if babel/swc doesn't touch the file since my assumption is that if something is provided via node_modules, it is already transformed and consumable.

If the Stories are transformed into CJS, the export-order-loader plugin can't do much about it. So, I wouldn't configure it to transform node_modules as well.

The CSF-plugin requires source files, for example, to extract JSDOC comments or the original source code. This also doesn't make much sense for precompiled code from node_modules.

Other than that, Storybook will include these files in the index, but as mentioned, with potentially limited functionality (Source Code view is compiled code, JSDOC comments are not available anymore, and so on...)

Hence, supporting stories from node_modules is a tricky Endeavor which I personally wouldn't like to support.

Prevent Stories lookup in node_modules

a7cbe99

valentinpalkovic added bug csf builder-webpack5 ci:daily Run the CI jobs that normally run in the daily job. labels Dec 13, 2023

valentinpalkovic requested a review from shilman December 13, 2023 12:45

valentinpalkovic self-assigned this Dec 13, 2023

valentinpalkovic requested a review from tmeasday December 13, 2023 12:46

valentinpalkovic marked this pull request as ready for review December 13, 2023 13:46

valentinpalkovic requested a review from ndelangen as a code owner December 13, 2023 13:46

tmeasday reviewed Dec 13, 2023

View reviewed changes

ndelangen approved these changes Dec 21, 2023

View reviewed changes

Merge branch 'next' into valentin/fix-monorepo-stories-lookup

569175b

valentinpalkovic merged commit 91f1bb8 into next Jan 11, 2024

valentinpalkovic deleted the valentin/fix-monorepo-stories-lookup branch January 11, 2024 13:18

github-actions Bot mentioned this pull request Jan 11, 2024

Release: Prerelease 8.0.0-alpha.10 #25546

Merged

27 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Core: Prevent stories lookup in node_modules#25214

Core: Prevent stories lookup in node_modules#25214
valentinpalkovic merged 2 commits into
nextfrom
valentin/fix-monorepo-stories-lookup

valentinpalkovic commented Dec 13, 2023 •

edited

Loading

Uh oh!

tmeasday left a comment

Uh oh!

tmeasday Dec 13, 2023

Uh oh!

tmeasday Dec 15, 2023

Uh oh!

valentinpalkovic Dec 27, 2023

Uh oh!

valentinpalkovic commented Dec 14, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	export function watchStorySpecifiers(
	specifiers: NormalizedStoriesSpecifier[],
	options: { workingDir: Path },
	onInvalidate: (specifier: NormalizedStoriesSpecifier, path: Path, removed: boolean) => void
	) {
	// See https://www.npmjs.com/package/watchpack for full options.

Uh oh!

Conversation

valentinpalkovic commented Dec 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What I did

Related PRs

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

Manual testing

Documentation

Checklist for Maintainers

🦋 Canary release

Uh oh!

tmeasday left a comment

Choose a reason for hiding this comment

Uh oh!

tmeasday Dec 13, 2023

Choose a reason for hiding this comment

Uh oh!

tmeasday Dec 15, 2023

Choose a reason for hiding this comment

Uh oh!

valentinpalkovic Dec 27, 2023

Choose a reason for hiding this comment

Uh oh!

valentinpalkovic commented Dec 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

valentinpalkovic commented Dec 13, 2023 •

edited

Loading

valentinpalkovic commented Dec 14, 2023 •

edited

Loading