Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 29 additions & 121 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,45 +1,26 @@
<div align="center">
![vibe-mod — write the rules in plain English, the AI builds the mod config; natural language in, mod config out](https://raw.githubusercontent.com/Two-Weeks-Team/vibe-mod/main/assets/banner.jpg)

<img src="./assets/banner.jpg" alt="vibe-mod — you write the rules in plain English, the AI builds the mod config. Natural language in, mod config out." width="880">
**Write a moderation rule in plain English. It runs deterministically — shadow-tested first, with one-click undo.**

<strong>Write a moderation rule in plain English.<br>It runs deterministically — shadow-tested first, with one-click undo.</strong>
[![CI](https://github.com/Two-Weeks-Team/vibe-mod/actions/workflows/ci.yml/badge.svg)](https://github.com/Two-Weeks-Team/vibe-mod/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE) [![Reddit App Directory](https://img.shields.io/badge/Reddit%20App%20Directory-Live-FF4500?logo=reddit&logoColor=white)](https://developers.reddit.com/apps/vibe-mod) · Built on [Reddit Devvit](https://developers.reddit.com/)

[![CI](https://github.com/Two-Weeks-Team/vibe-mod/actions/workflows/ci.yml/badge.svg)](https://github.com/Two-Weeks-Team/vibe-mod/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)
[![Reddit App Directory](https://img.shields.io/badge/Reddit%20App%20Directory-Live-FF4500?logo=reddit&logoColor=white)](https://developers.reddit.com/apps/vibe-mod)
&nbsp;·&nbsp; Built on [Reddit Devvit](https://developers.reddit.com/)

</div>

> A moderator types _"Send to mod queue any post under 50 characters from accounts less than 7 days old."_
> vibe-mod turns that sentence into a real rule, runs it in **24-hour shadow mode** (logging what it
> _would_ do, acting on nothing), shows a **preview** against recent posts, and keeps **30-day undo** on
> every action it takes. The AI is used **only when you write a rule** — never on your community's posts.
> A moderator types _"Send to mod queue any post under 50 characters from accounts less than 7 days old."_ vibe-mod turns that sentence into a real rule, runs it in **24-hour shadow mode** (logging what it _would_ do, acting on nothing), shows a **preview** against recent posts, and keeps **30-day undo** on every action it takes. The AI is used **only when you write a rule** — never on your community's posts.

## What it does

Type a moderation rule in plain English — _"remove posts whose title is ALL CAPS"_, _"send links from
accounts under 7 days old to the mod queue"_ — and vibe-mod turns it into a real, working rule. It shows
you exactly what the rule will do, runs it quietly for 24 hours first, and lets you undo any action for 30
days. **No YAML, no regex, no code.**
Type a moderation rule in plain English — _"remove posts whose title is ALL CAPS"_, _"send links from accounts under 7 days old to the mod queue"_ — and vibe-mod turns it into a real, working rule. It shows you exactly what the rule will do, runs it quietly for 24 hours first, and lets you undo any action for 30 days. **No YAML, no regex, no code.**

The AI only ever reads the sentence _you_ type — never your community's posts, comments, or usernames.
Once it has written the rule, the AI is done: every check after that is plain, deterministic logic, so the
same post always gets the same decision, and there's **zero AI cost per post**.
The AI only ever reads the sentence _you_ type — never your community's posts, comments, or usernames. Once it has written the rule, the AI is done: every check after that is plain, deterministic logic, so the same post always gets the same decision, and there's **zero AI cost per post**.

## How to use it

1. Install **vibe-mod** on your subreddit from the [App Directory](https://developers.reddit.com/apps/vibe-mod).
2. **Mod Tools → "vibe-mod: Compose rule"** → type your rule → **Compile + Preview**. (If your sentence is
ambiguous, vibe-mod asks a quick clarifying question instead of guessing.)
2. **Mod Tools → "vibe-mod: Compose rule"** → type your rule → **Compile + Preview**. (If your sentence is ambiguous, vibe-mod asks a quick clarifying question instead of guessing.)
3. Review the **dry-run preview** — which of your recent posts the rule would have caught. Nothing happens yet.
4. **Activate** it. The rule runs in **24-hour shadow mode** (just logging what it would do), then goes
live automatically. Watch those decisions under _"vibe-mod: View rules + log"_.
5. If it ever acts on something you disagree with, open that item's `⋯` menu → **"vibe-mod: Undo this
action"** (available for 30 days).
4. **Activate** it. The rule runs in **24-hour shadow mode** (just logging what it would do), then goes live automatically. Watch those decisions under _"vibe-mod: View rules + log"_.
5. If it ever acts on something you disagree with, open that item's `⋯` menu → **"vibe-mod: Undo this action"** (available for 30 days).

Six starter rules are seeded as drafts on install so you have something to look at, and the mod team gets a
one-time welcome message with a 3-step start guide.
Six starter rules are seeded as drafts on install so you have something to look at, and the mod team gets a one-time welcome message with a 3-step start guide.

## Settings you can tune (per subreddit)

Expand All @@ -52,115 +33,42 @@ No OpenAI key and no billing — **vibe-mod covers the AI cost**, up to 50 rule
## Why a new rule can't hurt your community

- 🕒 **24-hour shadow mode** — every new rule only _logs_ what it would do for a full day before it can act.
- 👀 **Dry-run preview** — see exactly which recent posts a rule catches _before_ you turn it on.
- 👀 **Dry-run preview** — see exactly which of your recent posts a rule catches _before_ you turn it on.
- ↩️ **30-day undo** — every action vibe-mod takes is reversible with one click.
- 🛑 **Guarded actions** — `report` / `flair` / `lock` / `modqueue` / `remove` are allowed, but
`ban` / `mute` / `permaban` / `approve` stay blocked unless you explicitly tick a checkbox.
- 🧠 **No AI on your content** — the model only sees the sentence you typed, runs once per rule (never per
post), and never reads posts, comments, or usernames.
- 🛑 **Guarded actions** — `report` / `flair` / `lock` / `modqueue` / `remove` are allowed, but `ban` / `mute` / `permaban` / `approve` stay blocked unless you explicitly tick a checkbox.
- 🧠 **No AI on your content** — the model only sees the sentence you typed, runs once per rule (never per post), and never reads posts, comments, or usernames.

## How it compares to AutoModerator

vibe-mod is **not** an AutoMod natural-language wrapper, and **not** an AI that reads your subreddit and
decides things:
vibe-mod is **not** an AutoMod natural-language wrapper, and **not** an AI that reads your subreddit and decides things:

| | **vibe-mod** | **AutoModerator** | **Generic "AI moderation"** |
| --- | --- | --- | --- |
| **Authoring** | plain English + a preview | YAML + regex, no preview | varies |
| **When the AI runs** | once, when you write the rule | n/a | on every post/comment |
| **Per-post cost** | **$0** | $0 | per-post token cost |
| **New-rule safety** | **24h shadow**, then auto-promotes | live immediately | usually live immediately |
| **Undo** | **per-action, 30-day, one click** | none built-in | rare |
| **Sees your content?** | only the sentence you typed | n/a | yes, sent to the model |
- **Authoring** — one plain-English sentence with a dry-run preview, instead of hand-editing YAML + regex with no preview.
- **When the AI runs** — once, at rule-edit time. (AutoMod uses no AI; "generic AI moderation" bots call a model on _every_ post.)
- **Per-post cost** — **$0**: the model already ran at edit time. (Runtime AI bots pay per post and hit rate limits.)
- **New-rule safety** — 24-hour shadow mode + dry-run preview + 30-day undo, vs live-on-save with no undo.
- **Sees your content?** — the model only ever sees your typed sentence, never posts, comments, or usernames.

The idea: **AI is great at turning intent into a rule, and bad at applying rules consistently.** vibe-mod
uses it only for the first part and uses plain, repeatable logic for the rest.
The idea: **AI is great at turning intent into a rule, and bad at applying rules consistently.** vibe-mod uses it only for the first part and plain, deterministic TypeScript for the rest.

## Fetch domains

This app makes outbound requests to exactly one external domain:

- **`api.openai.com`** — used **only** when a moderator clicks "Compile" to turn a plain-English sentence
into a structured rule. It does **not** run on posts or comments, and **Reddit content (post/comment
bodies, usernames) is never sent** — only the moderator's own typed sentence plus a fixed system prompt.
- **`api.openai.com`** — used **only** when a moderator clicks "Compile" to turn a plain-English sentence into a structured rule. It does **not** run on posts or comments, and **Reddit content (post/comment bodies, usernames) is never sent** — only the moderator's own typed sentence plus a fixed system prompt.

## Permissions

- `reddit` (scope `moderator`) — to take moderation actions (report / flair / lock / modqueue / remove;
ban / mute / permaban / approve only with an explicit checkbox) and to send the one-time welcome message.
- `reddit` (scope `moderator`) — to take moderation actions (report / flair / lock / modqueue / remove; ban / mute / permaban / approve only with an explicit checkbox) and to send the one-time welcome message.
- `redis` — to store your compiled rules, the audit log, undo tokens, and quota counters.
- `http` (`api.openai.com`) — to compile English rules into structured rules, as above.

<details>
<summary><strong>🔧 How it works (architecture)</strong></summary>

The single load-bearing idea: **build-time AI, runtime determinism.** The model runs exactly once per rule
edit; runtime evaluation is plain TypeScript — no network, no model, fully reproducible.

```
Moderator types a rule
│ (only the moderator's sentence is sent — never Reddit content)
OpenAI gpt-5.4-mini ──► JSON ──► Zod strict parse + action whitelist ──► rules:draft (Redis)
(build-time only) (reject if invalid)
│ dry-run preview / activate
rules:active (Redis)
Reddit triggers (onPostSubmit / onCommentSubmit / onPostReport / onCommentReport / onPostFlairUpdate)
Deterministic evaluator (pure TS, 0 network, 0 LLM) ──► builds a "fact bag" from the item + author + state
Action executor ──► shadow? log only : live? act + write 30-day undo token + audit entry
Scheduler: audit retention (daily) · dry-run replay · shadow-promote check (15 min) · rate-limit breaker (5 min)
```

Guarantees that hold **by construction** (verify in code):

| What | Value | Verify |
| --- | --- | --- |
| LLM calls per post/comment at runtime | **0** (pure-TS evaluator, no network) | [`src/server/evaluator.ts`](./src/server/evaluator.ts) |
| LLM calls per rule | exactly **1**, at edit time | [`src/server/routes/compose.ts`](./src/server/routes/compose.ts) |
| New-rule blast radius for first 24h | **0 live actions** (shadow default on) | `shadow: true` in [`rule-schema.ts`](./src/shared/rule-schema.ts) |
| Live action reversibility | **100% for 30 days** (per-action undo) | [`src/server/executor.ts`](./src/server/executor.ts) |
| Reddit content sent to the LLM | **none** (only the mod's typed sentence) | _Fetch domains_, above |

- **Runtime:** Devvit Web app (Hono server, `@devvit/web`); state in Devvit Redis, scoped per install.
- Multi-rule conflicts are surfaced as a read-only preview in _"vibe-mod: View rules + log"_ (see
[`docs/conflict-handling.md`](./docs/conflict-handling.md)).

</details>

<details>
<summary><strong>💻 For developers</strong></summary>

```bash
npm install # installs deps + git hooks (npm ci does NOT work here — esbuild EBADPLATFORM)
npm run typecheck # tsc --noEmit
npm test # 236 tests (1 skipped); npm run test:devvit for the @devvit/test harness
npm run acceptance # G1..G4 exit gates
npm run doctor # pre-deploy preflight (devvit.json integrity, fetch-domain↔permissions, route parity)
npm run build # tsc --noEmit && vite build → dist/server/index.cjs (CJS server bundle)
npm run openai:smoketest # real OpenAI API (needs OPENAI_API_KEY in .env) — model comparison table
npm run dev # = devvit playtest (needs `devvit login` + `devvit upload` first)
```

| Path | What |
| --- | --- |
| `src/shared/{rule-schema,system-prompt,starter-rules}.ts` | Zod v4 strict schema · gpt-5.4 prompt + few-shot · 6 seed rules |
| `src/server/{evaluator,fact-bag,executor,devvit-helpers}.ts` | deterministic evaluator · fact bag · action executor + audit + undo · `@devvit/web` adapters |
| `src/server/index.ts` + `src/server/routes/*` | Hono entry (re-exports `app`) + menu / form / trigger / scheduler route modules |
| `scripts/{acceptance,devvit-doctor,replay,openai-smoketest}.ts` | the `npm run` tooling |
| `test/` + `vitest.devvit.config.ts` | reusable in-memory Devvit testkit + the official `@devvit/test` config |
| `docs/devvit-setup-guide.md` | how to take this repo to a published Devvit app (wizard → upload → settings → playtest → publish) |
| `assets/icon.png` | the 1024² App Directory icon (`marketingAssets.icon` in `devvit.json`) |

The Devvit runtime (routing/RPC) is verified by `devvit playtest`; everything else is covered by the test
suite + an `npm run acceptance` gate. CI runs lint → format → `tsc` → tests → `@devvit/test` → acceptance →
`vite build` → "server bundle loads" smoke.

</details>
## How it works

The one idea: **build-time AI, runtime determinism.** The model runs once, when you write a rule, turning your sentence into a deterministic rule; every runtime decision after that is plain TypeScript — no model, no network, reproducible, and free. The full architecture (trigger + scheduler map, the deterministic evaluator, and the guarantees that hold by construction) is in **[docs/architecture.md](./docs/architecture.md)**.

## For developers

The test suite (236 tests), the in-memory Devvit testkit, the `npm run acceptance` gate, the pre-deploy `doctor`, and the rest of the `npm run` tooling are documented in **[docs/for-developers.md](./docs/for-developers.md)**. CI runs lint → format → `tsc` → tests → `@devvit/test` → acceptance → `vite build` on every push.

## Privacy & Terms

Expand Down
51 changes: 51 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# vibe-mod — architecture

The single load-bearing idea: **build-time AI, runtime determinism.** The model runs exactly once per
rule edit; runtime evaluation is plain TypeScript — no network, no model, fully reproducible.

```
Moderator types a rule
│ (only the moderator's sentence is sent — never Reddit content)
OpenAI gpt-5.4-mini ──► JSON ──► Zod strict parse + action whitelist ──► rules:draft (Redis)
(build-time only) (reject if invalid)
│ dry-run preview / activate
rules:active (Redis)
Reddit triggers (onPostSubmit / onCommentSubmit / onPostReport / onCommentReport / onPostFlairUpdate)
Deterministic evaluator (pure TS, 0 network, 0 LLM) ──► fact bag from the item + author + sub state
Action executor ──► shadow? log only : live? act + write 30-day undo token + audit entry
Scheduler: audit retention (daily) · dry-run replay · shadow-promote check (15 min) · rate-limit breaker (5 min)
```
Comment on lines +6 to +23
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

펜스 코드블록에 언어 지정이 빠져 markdownlint(MD040) 경고가 발생합니다.

``` 대신 예: ```text로 언어를 명시해 주세요.

수정 예시
-```
+```text
 Moderator types a rule
         │  (only the moderator's sentence is sent — never Reddit content)
         ▼
 OpenAI gpt-5.4-mini   ──►  JSON  ──►  Zod strict parse + action whitelist  ──►  rules:draft (Redis)
@@
 Scheduler: audit retention (daily) · dry-run replay · shadow-promote check (15 min) · rate-limit breaker (5 min)
-```
+```
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)

[warning] 6-6: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/architecture.md` around lines 6 - 23, The fenced code block in
docs/architecture.md (the multi-line ASCII diagram wrapped by triple backticks)
is missing a language tag causing markdownlint MD040; update that specific
fenced block (the diagram starting "Moderator types a rule" and ending with
"rate-limit breaker (5 min)") to include a language specifier like "text" (e.g.,
change ``` to ```text) so the linter recognizes it as a non-code/text block.


## Guarantees that hold by construction

| What | Value | Verify |
| --- | --- | --- |
| LLM calls per post/comment at runtime | **0** (pure-TS evaluator, no network) | [`src/server/evaluator.ts`](../src/server/evaluator.ts) |
| LLM calls per rule | exactly **1**, at edit time | [`src/server/routes/compose.ts`](../src/server/routes/compose.ts) |
| New-rule blast radius for first 24h | **0 live actions** (shadow default on) | `shadow: true` in [`rule-schema.ts`](../src/shared/rule-schema.ts) |
| Live action reversibility | **100% for 30 days** (per-action undo) | [`src/server/executor.ts`](../src/server/executor.ts) |
| Reddit content sent to the LLM | **none** (only the mod's typed sentence) | README → _Fetch domains_ |
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This reference to the README section is currently plain text. Converting it to a Markdown link improves the documentation's usability by allowing readers to navigate directly to the detailed explanation of fetch domains, especially since other entries in this table are already linked to their respective source files.

Suggested change
| Reddit content sent to the LLM | **none** (only the mod's typed sentence) | README → _Fetch domains_ |
| Reddit content sent to the LLM | **none** (only the mod's typed sentence) | [README → _Fetch domains_](../README.md#fetch-domains) |


## Runtime

- **Devvit Web app** (Hono server, `@devvit/web`); all state in Devvit Redis, scoped per installation:
`rules:active`, `rules:draft`, `audit`, `rollback:<actionId>`, plus daily-quota counters.
- The evaluator builds a **fact bag** from the triggering item + author account + subreddit-scoped state,
then evaluates the rule's boolean tree over a closed set of fact paths. Zero network, zero model.
- The executor, in shadow, only logs; live, it acts and writes a 30-day undo token + an audit entry.
- Multi-rule conflicts are surfaced as a read-only preview in _"vibe-mod: View rules + log"_
(see [`conflict-handling.md`](./conflict-handling.md)).

## Tested without Devvit's runtime

A 236-test suite (1 skipped): unit + route tests (`app.fetch()` against Devvit/OpenAI doubles) +
property-based tests, the official [`@devvit/test`](https://www.npmjs.com/package/@devvit/test) harness
for the executor, an `npm run acceptance` gate (G1–G4), and an `npm run replay` event replayer. The Devvit
runtime itself (routing, payload injection, RPC) is verified by `devvit playtest`. See
[`for-developers.md`](./for-developers.md).
28 changes: 28 additions & 0 deletions docs/for-developers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# vibe-mod — for developers

```bash
npm install # installs deps + git hooks (npm ci does NOT work here — esbuild EBADPLATFORM)
npm run typecheck # tsc --noEmit
npm test # 236 tests (1 skipped); npm run test:devvit for the @devvit/test harness
npm run acceptance # G1..G4 exit gates
npm run doctor # pre-deploy preflight (devvit.json integrity, fetch-domain↔permissions, route parity)
npm run build # tsc --noEmit && vite build → dist/server/index.cjs (CJS server bundle)
npm run openai:smoketest # real OpenAI API (needs OPENAI_API_KEY in .env) — model comparison table
npm run dev # = devvit playtest (needs `devvit login` + `devvit upload` first)
```

| Path | What |
| --- | --- |
| `src/shared/{rule-schema,system-prompt,starter-rules}.ts` | Zod v4 strict schema · gpt-5.4 prompt + few-shot · 6 seed rules |
| `src/server/{evaluator,fact-bag,executor,devvit-helpers}.ts` | deterministic evaluator · fact bag · action executor + audit + undo · `@devvit/web` adapters |
| `src/server/index.ts` + `src/server/routes/*` | Hono entry (re-exports `app`) + menu / form / trigger / scheduler route modules |
| `scripts/{acceptance,devvit-doctor,replay,openai-smoketest}.ts` | the `npm run` tooling |
| `test/` + `vitest.devvit.config.ts` | reusable in-memory Devvit testkit + the official `@devvit/test` config |
| `docs/devvit-setup-guide.md` | how to take this repo to a published Devvit app (wizard → upload → settings → playtest → publish) |
| `docs/architecture.md` | build-time-AI / runtime-determinism design + the by-construction guarantees |
Comment on lines +21 to +22
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

These documentation paths are currently plain text. Since these files reside in the same directory, converting them to relative Markdown links would improve navigability within the documentation, maintaining consistency with how for-developers.md is linked from the architecture document.

Suggested change
| `docs/devvit-setup-guide.md` | how to take this repo to a published Devvit app (wizard → upload → settings → playtest → publish) |
| `docs/architecture.md` | build-time-AI / runtime-determinism design + the by-construction guarantees |
| [docs/devvit-setup-guide.md](./devvit-setup-guide.md) | how to take this repo to a published Devvit app (wizard → upload → settings → playtest → publish) |
| [docs/architecture.md](./architecture.md) | build-time-AI / runtime-determinism design + the by-construction guarantees |

| `assets/icon.png` | the 1024² App Directory icon (`marketingAssets.icon` in `devvit.json`) |

The Devvit runtime (routing/RPC) is verified by `devvit playtest`; everything else is covered by the test
suite + the acceptance gate. CI (`.github/workflows/ci.yml`) runs lint (0 warnings) → format check →
`tsc` → tests (coverage) → `@devvit/test` → acceptance → `vite build` → "server bundle loads" smoke on
every push. Dependabot groups `@devvit/*` updates into one weekly PR.