The same file is being linted multiple times during a compilation, causing very long build times

[decademoon]

• Operating System: macOS 11.1
• Node Version: 15.6.0
• NPM Version: yarn berry using PnP
• webpack Version: 4.46.0
• eslint-webpack-plugin Version: 2.4.3

Expected Behavior

Each module file should be linted only once.

Actual Behavior

The same module file is being linted multiple times unnecessarily which drastically increases the build time.

Code

// webpack.config.js
new ESLintPlugin({
  threads: true,
  extensions: ['js', 'vue'],
})

How Do We Reproduce?

The same module file will be linted each time it is required with a different resource query string. This code strips off the query string:

https://github.com/webpack-contrib/eslint-webpack-plugin/blob/f819203302d26a1beadb66ca42270bfa92c65927/src/index.js#L82-L95

In my case I'm using vue-loader which loads .vue files. It does some trickery behind the scenes such that when you require a .vue module it will translate that to multiple requires to load all the different parts of the file:

// This...
import Comp from './comp.vue';

// ...is converted into something like this
import Comp from './comp.vue?type=js';
import './comp.vue?type=css';

eslint-webpack-plugin lints the comp.vue file twice in this case. I did some logging and some of my modules are being linted 5 times, maybe more.

I did a quick test with this modification:

const seen = new Set();

 const processModule = (module) => {
   if (module.resource) {
     const file = module.resource.split('?')[0];
  
     if (
       file &&
       !seen.has(file) &&
       micromatch.isMatch(file, wanted) &&
       !micromatch.isMatch(file, exclude)
     ) { 
       // Queue file for linting.
       seen.add(file);
       lint(file);
     }
  }
};

and my initial build time reduced from 300s (!) down to 100s.

If I run yarn eslint . in my project directory it takes 44s but it seems eslint-webpack-plugin takes much longer to do essentially the same thing.

I tried setting the cache option to true (if such an option exists), thinking it would skip linting files that have already been linted so far in the compilation, but that made no difference.

[watjurk]

Hi, recently I've been working a lot with this plugin, and I think I have a solution to this issue but I don't know what are your thoughts on this:

change the module retrieving hook:
https://github.com/webpack-contrib/eslint-webpack-plugin/blob/cef4f7473707fb3f069ec44c54b5ed2d27d931f8/src/index.js#L114

to:

compilation.hooks.finishModules.tap(ESLINT_PLUGIN, processModules) // function processModules needs to be written

now we retrieve all modules at once, so we can filter them after striping resource query:

function processModules(modules) {
  const resources = modules.map((module) => module.resource.split('?')[0]);
  const resourcesUnique = new Set(resources);
  // TODO: process resourcesUnique
}

now when we are sure that every file if processed only once, we can remove all logic involving resultStorage:
https://github.com/webpack-contrib/eslint-webpack-plugin/blob/f819203302d26a1beadb66ca42270bfa92c65927/src/linter.js#L21
https://github.com/webpack-contrib/eslint-webpack-plugin/blob/f819203302d26a1beadb66ca42270bfa92c65927/src/linter.js#L291-L297

as far as I understand it was created to filter the results:
https://github.com/webpack-contrib/eslint-webpack-plugin/blob/f819203302d26a1beadb66ca42270bfa92c65927/src/linter.js#L79-L83

[alexander-akait]

Don't think we should keep cache for compiler, we have modules, i.e. we can use module as key for weak map

[watjurk]

I'm not sure if I understand your point. My idea is to completely remove everything involving our cache (called resultStorage)

[alexander-akait]

@watjurk When you change one file, it will invalidate whole cache, it is not effective

[watjurk]

This plugin as it is right now (in my understanding) doesn't use resultStorage as cache, every time lint (in src/linter.js) is called every file from files is linted again no matters what crossRunResultStorage says