feat: Introduce ecosystem tests for popular third-party plugins #127
Conversation
| 2. Run a lint command (i.e. `npx eslint .`) in that directory | ||
| 3. Assert that the lint command passed with 0 lint reports. | ||
|
|
||
| This will all be runnable locally with a `package.json` script like `npm run test:ecosystem --plugin unicorn`. |
There was a problem hiding this comment.
| This will all be runnable locally with a `package.json` script like `npm run test:ecosystem --plugin unicorn`. | |
| This will all be runnable locally with a `package.json` script like `npm run test:ecosystem --plugin eslint-plugin-unicorn`. |
The CI job uses eslint-plugin-unicorn.
| In the case of a breakage being discovered, this RFC proposes the following process: | ||
|
|
||
| 1. An ESLint team member should file a bug report on the plugin's repository -if it doesn't yet exist-, as well as an issue on `eslint/eslint` linking to that bug report | ||
| 2. If the issue isn't resolved within two weeks: | ||
| 1. The plugin will be removed from ESLint's ecosystem CI job | ||
| 2. An ESLint team member should file a followup issue to re-add it once the breakage is fixed |
There was a problem hiding this comment.
Is this the process for pull requests that cause breakages in a third party plugin? If so, I think it's not necessary to file an extra issue in the eslint repo. The pull request seems the best place to keep track of the problem since it cannot be merged until the problem is fixed in the plugin or the ecosystem test is disabled.
It's not clear though how breakages will be handled if they occur in the main branch for some reason.
| Automation could be added for at least the filing of issues on plugin failures. | ||
| That does not seem worth the time expenditure given how rarely plugins are expected to fail. | ||
| Is that accurate? |
There was a problem hiding this comment.
I agree this automation isn't necessary, and we'll probably want to double-check the PR and/or the testing before filing an issue.
| However, to be safe, this RFC proposes adding a CI job in three steps: | ||
|
|
||
| 1. On a branch that and updated from `main` several times a week | ||
| 2. On the `main` branch only | ||
| 3. On all branches, alongside existing CI jobs |
There was a problem hiding this comment.
I think the point is to catch the problem before merging a PR, so just 2. and 3., both from the start, makes the most sense to me.
There was a problem hiding this comment.
As for 3., I think "all branches" -> "all PRs targeting the main branch" would better clarify it (assuming that was the intent).
|
That none of the At least one of eg I also note that the selection criteria is similar to those outlined in the suggested |
Summary
Adding an CI job to the
eslint/eslintrepo that checks changes against a small selection of third-party plugins.Related Issues
eslint/eslint#19139