New RFC: AI generated Code rules via AGENTS.md

[edmundmiller]

Have you read the RFC docs?

• Yes, I have read and understood the RFC docs

Summary

As generated code becomes easier and easier to create, what is our policy in nf-core?

Let's put cursor rules, CLAUDE.md files, etc., in our repos. I don't think we should have agents running constantly or that nf-core should be paying for that. I don't want people to have to keep their siloed rules files, regenerate these rules every time they want to contribute, or generate rules that aren't accurate and misguide their AI-generated code, further wasting the community's time and resources.

Example PR: nf-core/tools#3678

I'm hoping to keep this not in any ethical or views on it, more as a how are we going to cope with this as an org from a code quality standpoint and quality of nf-core as an organization.

Champion

@edmundmiller

Background & Motivation

Accepting generated PRs is going to come in

I personally want more agent-generated code; I want to review it first. We lack humans who understand the codebase in lots of repos(tools, website, ops) that we can leverage the generated code to tackle long-standing issues.

We're seeing lots of generated code coming in and are utilizing it. The principle is to keep PRs small and concise, so that humans can review the code and aren't overwhelmed by the extreme scope.

Again, the goal is not that nf-core will start running entirely on agents and whatnot. It's more that I would rather have more documentation and rules for people who are using AI to contribute to nf-core, improve their experience, and collaborate and iterate on those rules versus us just cutting that off and not adding any of the markdown files that are not changing any of the code base itself, just on the principle that we are being inundated with slop.

Unless we want to block PRs as AI-generated, which I think will just discourage more people from contributing to nf-core, we should just improve their experience, and maybe we will have less slop and better code contributed.

Against Argument

nf-core/tools#3676 (comment)

nf-core/tools#3678 (comment)

I don't see how generated code can fix the problem with lack of people having insight into the tools repo.

I totally agree with the latter point, but we can only get more people having (some) insight but having them write code. and generated PRs are a lot trickier to review then human generated code, because they hide the experience level of the person opening the PR, so it is difficult to triage and make it more difficult to guide the people through the PR.

I already see them popping up here and on the website and they cost more effort on my side than the give features/contributors

Goals

• Add markdown files to guide AI-generated code so it better adheres to nf-core standards
• Prevent wasted time from off-the-mark AI-generated code PRs coming in
• Close long-standing issues that we're not able to find the resources to complete in nf-core.

Non-Goals

• Move away from the goal of being "A global community effort to collect a curated set of open‑source analysis pipelines built using Nextflow."
• Run AI Models to generate code
• Allow AI-generated code to get merged without being reviewed by humans

References

• Lint for version captures in modules tools#3676 (comment)
• Initialize Claude Code tools#3678 (comment)

[nf-core-bot]

RFC approval status: 🕐 Pending

RFC has approvals from 2/7 required @core-team quorum.

Review Status	Core Team members
✅ Approved	@maxulysse, @mribeirodantas
❌ Rejected	@mashehu, @jfy133, @kenibrewer
🕐 Pending	@ewels, @JoseEspinosa, @mirpedrol, @FriederikeHanssen, @edmundmiller, @sateeshperi, @FranBonath, @christopher-hakkaart, @nvnieuwk

[maxulysse]

/approve

[mashehu]

/reject You can read my arguments in the "Against Arguments" section.

TL;DR These files will not solve our review bottle neck just make it worse.

More specific to general policy: providing these kind of files, just invites more drive-by PRs (speaking for the infra repos here)

[edmundmiller]

More specific to general policy: providing these kinds of files just invites more drive-by PRs (speaking for the infra repos here)

Okay, let's go with that for the sake of the argument. These drive-by PRs are going to come regardless, though, whether we invite them or not.

How do you think we should filter them? Should we just reject all of them? Will that drive community members away?

And is there a split in your mind between the infra repos and the pipeline repos?

[mashehu]

Okay, let's go with that for the sake of the argument. These drive-by PRs are going to come regardless, though, whether we invite them or not.
How do you think we should filter them? Should we just reject all of them? Will that drive community members away?

I try to give them a fair chance, if they are not too sloppy. But I am definitely less motivated to help along if I don't see a lot of engagement.
in my experience these haven't come too much from community members and none of them stuck around afterwards.

And is there a split in your mind between the infra repos and the pipeline repos?

I am definitely stricter for tools, because of the crucial role it plays for us. Less so with stats for example. For pipelines: I would leave it up to the maintainers of each pipeline, but I don't think we should put it in the template for example.

[jfy133]

As an AI skeptic, I'm not familiar with what you mean by rules or CLAUDE.md ... Could you expand more on what all of these are?

My instinct is we have our specs already for modules/subworkflows/pipelines/test data, and we should write something for configs.

We just review against those... If AI slop doesn't meet them then 🤷, I don't think the source of the code matters practically (ethically is another thing but not on scope of this). Unless I don't understand what these files are what you are referring to?

[jfy133]

Question above aimed at @edmundmiller (mobile sucks sometimes)

[mashehu]

See for an example file: https://github.com/nf-core/tools/pull/3678/files

The idea behind these files are that the help the model to get more context specific (but as usual it is a bit of black magic with unclear results on what works and what doesn't)

[jfy133]

But that looks to me like useful docs for humans too? Why make it AI specific?

And would you have to have one markdown file per AI LLM?

If so seems to me like a lot of clutter when it would be useful for humans and thus integrated into our proper documentation...

Currently leaning towards reject because we should document properly for everyone and not discriminate against humans 😬😏

[christopher-hakkaart]

IMO, AI will be used with or without things like CLAUDE files. I'd advocate for adding the files but supplementing them with opinionated documentation about how they should be used.

[jfy133]

Is CLAUDE.md meant to be a standard file format/name or something?

If not, I would say having a page on the website which describes our 'policy' for AI code and include the contents of the file in the example PR would be fine.

I agree these sort of things are inevitable despite my reservations about it. But I don't think we should invest time into highly refined things for each different LLM. Our existing review system should prevent slop.

I think what we should invest more time on is improving/writing developer documentation on development etiquette first - this is a bigger time sink for us than AI related stuff (that I've seen 0 myself).