Add defensive CLAUDE.md, making AI (claude code) help enforce our AI PR rules#118681
Add defensive CLAUDE.md, making AI (claude code) help enforce our AI PR rules#118681Ivorforce wants to merge 1 commit into
CLAUDE.md, making AI (claude code) help enforce our AI PR rules#118681Conversation
Cataclysm DDA tried having an AGENTS.md that forbade AIs from offering C++ code. It doesn't work. AIs ignore instructions that are in conflict (user saying I want code and AGENTS saying no code allowed) |
Introduces a repository-local CLAUDE.md that summarizes Godot's contribution rules and gates AI-assisted edits behind a five-item acceptance checklist the assistant must recite verbatim before modifying any file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5c74dcc to
cd23e06
Compare
You can see i tested the file above. It works, at least for claude code. |
|
@Ivorforce Yes I saw that you tested this particular file for Claude and that it seems to work, that is why I quoted the bit about "tooling for other AIs" (because based on CDDA's repeated attempts, that's where things get hairy, and the assumption that only Claude is used will soon be false |
|
I really, really, really dislike the idea of introducing a dedicated file at the project root for this. The files in the project root of the default branch are the very first that will be shown to any user opening the repo, so the idea of painting that first impression with an LLM-focused file just rubs me the wrong way. Not to mention that this isn't a one-size-fits-all solution, with many LLMs only recognizing their own configuration1; many tools will inevitably ignore this file outright. If this must be introduced—something I'm not convinced is necessary—I'd sooner have it done in a way that doesn't effectively stain our project. For instance: given running But as is? Nah. I respect the intent behind this PR, but I cannot in good conscience condone nor approve its integration. Footnotes |
That's an interesting idea. IIRC at least claude code automatically reloads Edit: One caveat; I think some AIs never compile the code they write so they'd never see the |
Sadly it seems that placing them* at the project root is the only way for them to be respected across the entire repo (apparently you can place them in a subfolder but then it will only apply to AI doing task on said subfolder
|
|
I tested As for @Repiteo: I tested your proposal (generating the file), and found that AI by default doesn't compile and test the changes. It will happily make the edits without ever running |
CLAUDE.md with AI-contributor acceptance gateCLAUDE.md, making AI (claude code) help enforce our AI PR rules
|
After some more testing, it seems that as long as it's not a request where an agent has to make any sort of decisions on its own (i.e. the request is simple and direct), then it ignores it. This would be stuff like "make all the definitions in this function Additionally, If I say something vague like, "make this class better", it explicitly pushes back on that, citing the rules in |
I agree with this sentiment, especially when we consider:
Imagine the impression a user would have when greeted by 5+ LLM-focused files. As a human-centric change, would updating the |
It already is in the pull request template. People attempting to contribute AI code haven't cared so far, and by your question, I'm assuming you have also so far ignored it 😅 |
I meant adding the checklist directly into the PR template. I do appreciate that, ironically, people doing nothing but reading AI prompts still do not want to read. But perhaps if it were directly in there it might make a difference. 🤷 One can dream I suppose.
It does look like I missed it, my bad 😓. In my defence, I had read the contributing guidelines before making my first PR October 2025, but I had not read them since you added the checklist in February 2026. |
Ah, right. We have an open PR that is suggesting to do something similar: #118624 It could use some healthy discussion too :)
No worries. You're far from the only one. Links in the pull request template seem to have a tendency to get ignored, which is part of the reason our current AI PR mitigation technique (inform the user & hope for compliance) hasn't worked out great so far. |
|
I'm converting this to draft since this change is very controversial. The number of undisclosed AI assisted PRs is still growing, though. I too would like to avoid this kind of change if we can, but that really hinges on how bad the AI situation gets. Let's hope that other solutions emerge we can choose instead. |
|
The crux of the issue is that the ones shovelling heaps of LLM are the least likely to read anything. I apologise if this wastes your time but I wanted to bring it to your attention in case it helps: Zig's code of conduct, is strict, no exceptions. Shovellers still litter the issue tracker, but the maintainers just close it with a link to their policy. I couldn't find any shovelled PRs being closed in the last two weeks... so maybe it has an effect? I didn't look too thoroughly. 🤷 I would happily support an equally strict policy here but I won't push it. That said, one reason I brought it up is that we allow LLMs for translations I found Zig's approach rather interesting:
Personally I see this as completely fair, after all, if you require an LLM for translation, you're in no better position than anyone else to verify the translation was correct. |
This comment was marked as abuse.
This comment was marked as abuse.
Agreed, and it's the very thought that led to this PR: inserting the text into the one place that makes it's impossible to ignore. While musing about other project's AI policies is interesting (I've seen a bunch already, and they end up quite different), Godot has to find its own. |
|
Hi, looks like there was already some empirical testing above, but just wanted to confirm as an outsider who's never contributed to Godot and saw this PR linked by another project considering the same. I used the Claude Code harness with these prompts to see if it would lose attention on the CLAUDE.md instructions after a few turns:
After ~50K tokens used on initial questions/exploration, the CLAUDE.md still worked as intended, as after the last prompt the LLM used its question tool to output: Successfully recalling the AI gate instructions and the rule about not changing vendored modules. I'm not the type of person who would just leave a vibe-coded drive-by PR, the above was just a somewhat silly test, but I thought that maybe an outsider's validation that the actual contents of the file are working well could be useful! |



Note
AI Disclosure: The
CLAUDE.mdfile was written by AI, as per my instructions. The PR description is written by me, by hand.We're seeing more and more AI PRs popping up. Many AI contributors are unaware of our rules and guidelines. Some even let the AI run loose and don't double-check its contributions. This is a violation of our guidelines, and wastes our time with reviews of unmoderated (by the author) slop.
Thought process / proposed fix
I am proposing to bring the AI on our side, so to speak, and have it take part in enforcing our rules. In this PR, this is done for claude code.
The aim is notably not to help the AI to code, or understand the codebase - that is the responsibility of the AI and/or AI contributor. This PR (almost entirely) adds failsafes that help us, the Godot community, maintain a workable engine for everyone.
Concepts covered in
CLAUDE.md:Effectiveness of this change
Before this change, the AI happily jumps on the task, no questions asked:
Before this PR
After this change, the AI consistently (10/10 times) gates the changes with an 'acceptance of terms' question.
It also asks the user to provide an existing issue, and proposes to benchmark.
After this PR
I think it's a clear improvement of the status quo.
Caveats
CLAUDE.md. We may need more tooling like this, for other AIs. Or perhaps they will converge and we can offer a single one that is compatible with all AIs. (Edit:AGENTS.mddoesn't work)