Build: Add agent scan to detect automated accounts#34293
Conversation
|
View your CI Pipeline Execution ↗ for commit 1ff1d08
☁️ Nx Cloud last updated this comment at |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a new GitHub Actions workflow Changes
Sequence Diagram(s)sequenceDiagram
rect rgba(200,200,255,0.5)
participant PR as Pull Request
end
rect rgba(200,255,200,0.5)
participant GH as GitHub Actions
end
rect rgba(255,200,200,0.5)
participant Agent as Agentscan Action
end
rect rgba(255,255,200,0.5)
participant Cache as Cache
end
rect rgba(200,255,255,0.5)
participant API as GitHub API
end
PR->>GH: opens/reopens PR (pull_request_target)
GH->>Cache: restore `.agentscan-cache` (key: github.actor)
GH->>Agent: run Agentscan with inputs + cache-path
Agent-->>GH: outputs (classification, community-flagged)
GH->>API: if community-flagged != 'true' -> run `agent-scan-label-pr.mjs` to add label
GH->>API: if community-flagged == 'true' -> run `agent-scan-close-pr.mjs` to close PR
GH->>Cache: save `.agentscan-cache`
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
.github/workflows/agent-scan.yml (2)
58-70: Consider adding a comment explaining why the PR was closed.When closing community-flagged PRs, adding a comment would help the contributor understand why their PR was closed and what recourse they have if it's a false positive.
💬 Proposed fix to add explanatory comment before closing
- name: Close community flagged accounts if: steps.agentscan.outputs.community-flagged == 'true' uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0 with: script: | const prNumber = context.payload.pull_request.number; + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: prNumber, + body: '🚫 This PR has been automatically closed because the account has been community-flagged as automated/suspicious. If you believe this is a mistake, please reach out to the maintainers.', + }); + await github.rest.pulls.update({ owner: context.repo.owner, repo: context.repo.repo, pull_number: prNumber, state: 'closed', });🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/agent-scan.yml around lines 58 - 70, Add an explanatory comment before closing the PR in the "Close community flagged accounts" step: use the same prNumber/owner/repo values to call github.rest.issues.createComment with a concise body explaining the reason for closure (community-flagged), links or steps for appeal if it's a false positive, and any contact or next steps, then proceed to call github.rest.pulls.update (state: 'closed') as currently implemented.
21-27: Consider adding version comments to SHA-pinned actions for maintainability.Lines 38 and 60 include version comments (e.g.,
# v8.0.0), but the checkout, cache, and agentscan actions lack them. Adding version comments helps maintainers quickly identify which version is pinned and when updates are needed.📝 Suggested version comments
- - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v4.x.x - name: Cache AgentScan analysis - uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 + uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v4.x.x🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/agent-scan.yml around lines 21 - 27, The workflow uses SHA-pinned actions (actions/checkout@de0fac2e..., actions/cache@6682284..., and the AgentScan action) without human-readable version comments; update the YAML to add trailing comments (e.g., "# v2.7.0" or appropriate semver) next to those pinned SHAs so maintainers can quickly see the intended major/minor version for actions/checkout, actions/cache and the AgentScan action; confirm the semver used matches the upstream release corresponding to each SHA before adding the comment.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/agent-scan.yml:
- Line 37: The conditional uses contains(...) to test
steps.agentscan.outputs.community-flagged which can match substrings like "not
true"; change both checks to use strict equality against the literal string
'true' instead of contains — e.g., replace
!contains(steps.agentscan.outputs.community-flagged, 'true') with
steps.agentscan.outputs.community-flagged == 'true' (or the inverted strict
check as needed) so both occurrences (the condition with fromJSON(...) && ...
and the later line 59 usage) consistently and explicitly compare
steps.agentscan.outputs.community-flagged to 'true'.
- Around line 36-57: The createComment call currently posts an empty body
because the script's body array is empty; update the
github.rest.issues.createComment invocation in the actions/github-script step to
pass a non-empty message (use the prNumber and the classification variable) —
e.g., construct a clear comment string explaining the PR was flagged (include
the ${classification}), instruct maintainers and contributors to review and to
ping if this is a false positive, and replace the empty body array in the
createComment call so the workflow posts that meaningful message instead of a
blank comment.
---
Nitpick comments:
In @.github/workflows/agent-scan.yml:
- Around line 58-70: Add an explanatory comment before closing the PR in the
"Close community flagged accounts" step: use the same prNumber/owner/repo values
to call github.rest.issues.createComment with a concise body explaining the
reason for closure (community-flagged), links or steps for appeal if it's a
false positive, and any contact or next steps, then proceed to call
github.rest.pulls.update (state: 'closed') as currently implemented.
- Around line 21-27: The workflow uses SHA-pinned actions
(actions/checkout@de0fac2e..., actions/cache@6682284..., and the AgentScan
action) without human-readable version comments; update the YAML to add trailing
comments (e.g., "# v2.7.0" or appropriate semver) next to those pinned SHAs so
maintainers can quickly see the intended major/minor version for
actions/checkout, actions/cache and the AgentScan action; confirm the semver
used matches the upstream release corresponding to each SHA before adding the
comment.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 95d61e1a-1b95-4d11-892a-680f1818b608
📒 Files selected for processing (1)
.github/workflows/agent-scan.yml
Package BenchmarksCommit: The following packages have significant changes to their size or dependencies:
|
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 50 | 50 | 0 |
| Self size | 20.54 MB | 20.52 MB | 🎉 -21 KB 🎉 |
| Dependency size | 16.56 MB | 16.56 MB | 🎉 -95 B 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/cli
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 184 | 184 | 0 |
| Self size | 782 KB | 782 KB | 🎉 -7 B 🎉 |
| Dependency size | 68.20 MB | 68.18 MB | 🎉 -20 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
@storybook/codemod
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 177 | 177 | 0 |
| Self size | 32 KB | 32 KB | 🚨 +34 B 🚨 |
| Dependency size | 66.73 MB | 66.71 MB | 🎉 -21 KB 🎉 |
| Bundle Size Analyzer | Link | Link |
create-storybook
| Before | After | Difference | |
|---|---|---|---|
| Dependency count | 51 | 51 | 0 |
| Self size | 1.04 MB | 1.04 MB | 🚨 +1 KB 🚨 |
| Dependency size | 37.10 MB | 37.07 MB | 🎉 -21 KB 🎉 |
| Bundle Size Analyzer | node | node |
| id: agentscan | ||
| uses: MatteoGabriele/agentscan-action@a584774dd15cabe6df4c6ab45fc43514a3b56b2d | ||
| with: | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} |
There was a problem hiding this comment.
Same security considerations as #34013, we should ideally make a dedicated API token
- with exactly the right privileges so it can't be abused more than necessary
- and unique to ensure any breach allows us to trace back which workflow was abused
I have yet to create a token for my own PR, whoever figures it out first can ping the other :) Jeppe shared a solution that I believe is mentioned on my PR.
There was a problem hiding this comment.
@Sidnioulz I think secrets.GITHUB_TOKEN is a unique generated token per job, it's already unique.
| uses: MatteoGabriele/agentscan-action@a584774dd15cabe6df4c6ab45fc43514a3b56b2d | ||
| with: | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| skip-members: "dependabot[bot],AriPerkkio,huang-julien,MichaelArestad,yannbf,vanessayuenn,jonniebigodes,Sidnioulz,kasperpeulen,valentinpalkovic,github-actions[bot],ndelangen,shilman,JReinhold,ghengeveld,storybook-bot,kylegach" |
There was a problem hiding this comment.
- Can we use a team identifier instead? This will age poorly 😅
- What is the rationale behind skipping ourselves? Perf / cost? If the agent scan job ever also detects agent-made PRs made by human accounts, I think we'll want to flag ourselves and hold ourselves to account to (e.g. if this informs the number of requested approvals)
There was a problem hiding this comment.
Agent scan scans by accounts activity, not PR content.
It's goal is to detect highly automated accounts so it probably doesn't makes sense to scan us. Otherwise if someone's flagged, all hist future PRs will be flagged too 😂
There was a problem hiding this comment.
Thanks for clarifying :) Let's see if it's not too painful to add group support to agent-scan, otherwise feel free to resolve this.
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (5)
.github/scripts/agent-scan-close-pr.mjs (2)
8-8: Add defensive null check forpull_requestpayload.If the workflow context is somehow missing the
pull_requestobject (e.g., misconfigured trigger), accessing.numberwill throw an unclearTypeError. Adding a guard provides a clearer error message.🛡️ Proposed defensive check
const octokit = github.getOctokit(token); + const pullRequest = github.context.payload.pull_request; + if (!pullRequest) { + throw new Error('This script must be run in a pull_request context'); + } - const prNumber = github.context.payload.pull_request.number; + const prNumber = pullRequest.number;🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/scripts/agent-scan-close-pr.mjs at line 8, Add a defensive null check around github.context.payload.pull_request before using it to derive prNumber: validate that github.context && github.context.payload && github.context.payload.pull_request exist and if not, log or throw a clear error explaining the missing pull_request payload (instead of letting the TypeError occur) and exit gracefully; then safely assign prNumber from github.context.payload.pull_request.number. Reference the variable prNumber and the object github.context.payload.pull_request when implementing this guard.
10-14: Consider adding a label or comment explaining why the PR was closed.The PR is closed silently without any record of why. Per the earlier discussion between maintainers, adding an
agent-scan:community-flaggedlabel before closing would provide a clear audit trail and help contributors understand the closure reason.📝 Proposed fix to add label before closing
+ await octokit.rest.issues.addLabels({ + ...github.context.repo, + issue_number: prNumber, + labels: ['agent-scan:community-flagged'], + }); + await octokit.rest.pulls.update({ ...github.context.repo, pull_number: prNumber, state: 'closed', });🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/scripts/agent-scan-close-pr.mjs around lines 10 - 14, Before calling octokit.rest.pulls.update to close the PR, add a label and/or comment explaining the reason; use the existing prNumber and github.context.repo values to call octokit.rest.issues.addLabels (or octokit.rest.issues.createComment to add a text explanation) to apply the "agent-scan:community-flagged" label and a short explanatory comment, then proceed with the pull update call that sets state: 'closed' so the closure is recorded with an audit trail..github/workflows/agent-scan.yml (2)
35-35: Hardcoded skip-members list will require manual maintenance.This list of usernames will need updates whenever team membership changes. As noted in past discussion, consider whether a GitHub team identifier could be used instead to reduce maintenance burden.
If a team-based approach isn't feasible with the current agentscan-action, consider adding a comment documenting when/how to update this list.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/agent-scan.yml at line 35, Replace the hardcoded skip-members CSV in the agentscan-action configuration with a less brittle mechanism: either switch the skip-members value to reference a GitHub team identifier (if agentscan-action supports team names/IDs) or, if that isn’t supported, add an in-file comment adjacent to the skip-members line documenting the canonical owner(s) responsible for updates and the process/timestamp to update the list; update the entry referencing the exact key "skip-members" so reviewers can find and change behavior in the action configuration.
44-48: Consider adding a label before closing for audit trail.When closing PRs from community-flagged accounts, there's no record explaining why. Adding an
agent-scan:community-flaggedlabel (as discussed in past reviews) would provide transparency and help contributors understand the closure reason.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/agent-scan.yml around lines 44 - 48, Update the "Close community flagged accounts" step to add an `agent-scan:community-flagged` label before running the close script: call the GitHub API or gh CLI within that step to add the label to the target PR (using the same INPUT_TOKEN env) and only then invoke node .github/scripts/agent-scan-close-pr.mjs; ensure the label name matches `agent-scan:community-flagged` and that the step handles API errors so closure still proceeds or logs failures appropriately..github/scripts/agent-scan-label-pr.mjs (1)
15-15: Add defensive null check forpull_requestpayload.Same issue as in
agent-scan-close-pr.mjs— accessing.numberwithout a guard can produce an unclear error if the context is malformed.🛡️ Proposed defensive check
const octokit = github.getOctokit(token); - const prNumber = github.context.payload.pull_request.number; + const pullRequest = github.context.payload.pull_request; + if (!pullRequest) { + throw new Error('This script must be run in a pull_request context'); + } + const prNumber = pullRequest.number;🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/scripts/agent-scan-label-pr.mjs at line 15, The pull request accesses github.context.payload.pull_request.number directly via the prNumber variable which can throw if payload.pull_request is null; update the code that defines prNumber to defensively check github.context.payload and github.context.payload.pull_request exist (e.g., guard with an if/early return or throw a clear error) before reading .number, and ensure the failure path logs a helpful message and exits/returns so downstream code (using prNumber) won't run with undefined.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/scripts/agent-scan-label-pr.mjs:
- Around line 4-8: The CLASSIFICATION_MAP constant uses the wrong key
'automated' so lookups for the agentscan-action output 'automation' fail; update
CLASSIFICATION_MAP (symbol name CLASSIFICATION_MAP) so the key is 'automation'
mapping to 'automated' (i.e., 'automation': 'automated') instead of the current
'automated' key; after updating, ensure any code that reads the incoming
classification (the variable used where CLASSIFICATION_MAP is indexed) still
uses the same lookup (+ fallback) and then also verify the workflow condition
that mentions 'suspicious' is valid (remove or adjust it if agentscan-action
never emits 'suspicious').
In @.github/workflows/agent-scan.yml:
- Around line 38-43: The workflow and labeling script disagree on classification
names: the action emits "automation" but CLASSIFICATION_MAP in
agent-scan-label-pr.mjs expects "automated", and the workflow condition checks
for "suspicious" which is not emitted; fix by adding a "automation": "automated"
entry to CLASSIFICATION_MAP in agent-scan-label-pr.mjs (so
CLASSIFICATION_MAP["automation"] -> "automated") and update the workflow
condition used in the step (the contains(...) array) to check for "automation"
instead of "suspicious" so the condition and labeling script use the same
classification token.
---
Nitpick comments:
In @.github/scripts/agent-scan-close-pr.mjs:
- Line 8: Add a defensive null check around github.context.payload.pull_request
before using it to derive prNumber: validate that github.context &&
github.context.payload && github.context.payload.pull_request exist and if not,
log or throw a clear error explaining the missing pull_request payload (instead
of letting the TypeError occur) and exit gracefully; then safely assign prNumber
from github.context.payload.pull_request.number. Reference the variable prNumber
and the object github.context.payload.pull_request when implementing this guard.
- Around line 10-14: Before calling octokit.rest.pulls.update to close the PR,
add a label and/or comment explaining the reason; use the existing prNumber and
github.context.repo values to call octokit.rest.issues.addLabels (or
octokit.rest.issues.createComment to add a text explanation) to apply the
"agent-scan:community-flagged" label and a short explanatory comment, then
proceed with the pull update call that sets state: 'closed' so the closure is
recorded with an audit trail.
In @.github/scripts/agent-scan-label-pr.mjs:
- Line 15: The pull request accesses github.context.payload.pull_request.number
directly via the prNumber variable which can throw if payload.pull_request is
null; update the code that defines prNumber to defensively check
github.context.payload and github.context.payload.pull_request exist (e.g.,
guard with an if/early return or throw a clear error) before reading .number,
and ensure the failure path logs a helpful message and exits/returns so
downstream code (using prNumber) won't run with undefined.
In @.github/workflows/agent-scan.yml:
- Line 35: Replace the hardcoded skip-members CSV in the agentscan-action
configuration with a less brittle mechanism: either switch the skip-members
value to reference a GitHub team identifier (if agentscan-action supports team
names/IDs) or, if that isn’t supported, add an in-file comment adjacent to the
skip-members line documenting the canonical owner(s) responsible for updates and
the process/timestamp to update the list; update the entry referencing the exact
key "skip-members" so reviewers can find and change behavior in the action
configuration.
- Around line 44-48: Update the "Close community flagged accounts" step to add
an `agent-scan:community-flagged` label before running the close script: call
the GitHub API or gh CLI within that step to add the label to the target PR
(using the same INPUT_TOKEN env) and only then invoke node
.github/scripts/agent-scan-close-pr.mjs; ensure the label name matches
`agent-scan:community-flagged` and that the step handles API errors so closure
still proceeds or logs failures appropriately.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 92f8490c-6daf-4f9f-939b-0db72161b81c
📒 Files selected for processing (4)
.github/scripts/agent-scan-close-pr.mjs.github/scripts/agent-scan-label-pr.mjs.github/scripts/package.json.github/workflows/agent-scan.yml
✅ Files skipped from review due to trivial changes (1)
- .github/scripts/package.json
There was a problem hiding this comment.
♻️ Duplicate comments (1)
.github/scripts/agent-scan-label-pr.mjs (1)
5-9:⚠️ Potential issue | 🟡 MinorClarify the classification comment to match actual input/output terms.
Line 7 says
automated, but the incoming classification value isautomationand it maps toautomated. Updating this comment will avoid future confusion.✏️ Suggested comment fix
/** * agent scan classification rename * - organic -> human - * - automated + * - automation -> automated * - mixed */🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/scripts/agent-scan-label-pr.mjs around lines 5 - 9, Update the top-of-file classification comment in .github/scripts/agent-scan-label-pr.mjs to reflect the actual input/output terms: change the list to show "human" (was organic), and explicitly note that incoming value "automation" maps to output "automated" (so display as "automation -> automated"), plus "mixed"; this clarifies the mapping used by the script.
🧹 Nitpick comments (1)
.github/scripts/agent-scan-label-pr.mjs (1)
1-31: Add a short AGENTS.md note for this new CI gate.Since this introduces automated account scanning/labeling behavior for PRs, contributor/maintainer guidance should briefly document what
agent-scan:*labels mean and expected maintainer actions.Based on learnings: Keep
AGENTS.mdup to date when Storybook's architecture, tooling, workflows, or contributor guidance changes.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/scripts/agent-scan-label-pr.mjs around lines 1 - 31, Add a short AGENTS.md documenting the new CI gate: describe the meaning of agent-scan:* labels produced by the .github/scripts/agent-scan-label-pr.mjs script (reference CLASSIFICATION_MAP and the addLabels usage that emits labels like agent-scan:human, agent-scan:automated, agent-scan:mixed), list expected maintainer actions when each label appears (e.g., review for sensitive scans, verify automation, or investigate mixed results), and include guidance on keeping AGENTS.md updated when agent detection or workflows change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In @.github/scripts/agent-scan-label-pr.mjs:
- Around line 5-9: Update the top-of-file classification comment in
.github/scripts/agent-scan-label-pr.mjs to reflect the actual input/output
terms: change the list to show "human" (was organic), and explicitly note that
incoming value "automation" maps to output "automated" (so display as
"automation -> automated"), plus "mixed"; this clarifies the mapping used by the
script.
---
Nitpick comments:
In @.github/scripts/agent-scan-label-pr.mjs:
- Around line 1-31: Add a short AGENTS.md documenting the new CI gate: describe
the meaning of agent-scan:* labels produced by the
.github/scripts/agent-scan-label-pr.mjs script (reference CLASSIFICATION_MAP and
the addLabels usage that emits labels like agent-scan:human,
agent-scan:automated, agent-scan:mixed), list expected maintainer actions when
each label appears (e.g., review for sensitive scans, verify automation, or
investigate mixed results), and include guidance on keeping AGENTS.md updated
when agent detection or workflows change.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f3cc3c12-5405-4884-b2f6-7a3b0a06cd6c
📒 Files selected for processing (3)
.github/scripts/agent-scan-close-pr.mjs.github/scripts/agent-scan-label-pr.mjs.github/workflows/agent-scan.yml
✅ Files skipped from review due to trivial changes (1)
- .github/workflows/agent-scan.yml
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/scripts/agent-scan-close-pr.mjs
Closes #
What I did
This PR adds AgentScan into our workflow for newly created PRs.
Checklist for Contributors
Testing
The changes in this PR are covered in the following automated tests:
Manual testing
Caution
This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!
Documentation
MIGRATION.MD
Checklist for Maintainers
When this PR is ready for testing, make sure to add
ci:normal,ci:mergedorci:dailyGH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found incode/lib/cli-storybook/src/sandbox-templates.tsMake sure this PR contains one of the labels below:
Available labels
bug: Internal changes that fixes incorrect behavior.maintenance: User-facing maintenance tasks.dependencies: Upgrading (sometimes downgrading) dependencies.build: Internal-facing build tooling & test updates. Will not show up in release changelog.cleanup: Minor cleanup style change. Will not show up in release changelog.documentation: Documentation only changes. Will not show up in release changelog.feature request: Introducing a new feature.BREAKING CHANGE: Changes that break compatibility in some way with current major version.other: Changes that don't fit in the above categories.🦋 Canary release
This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the
@storybookjs/coreteam here.core team members can create a canary release here or locally with
gh workflow run --repo storybookjs/storybook publish.yml --field pr=<PR_NUMBER>Summary by CodeRabbit