Skip to content
Merged
34 changes: 16 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<h1 align="center">Corgi</h1>

<p align="center">
A community-governed Bluesky feed where subscribers vote on the ranking algorithm.
Community-shaped Bluesky ranking with inspectable policy and receipts.
</p>

[![Deploy](https://github.com/andrewnordstrom-eng/bluesky-community-feed/actions/workflows/deploy.yml/badge.svg)](https://github.com/andrewnordstrom-eng/bluesky-community-feed/actions/workflows/deploy.yml)
Expand All @@ -16,41 +16,39 @@
[![Node >= 20](https://img.shields.io/badge/Node-%3E%3D20-339933)](https://nodejs.org/)
[![TypeScript](https://img.shields.io/badge/TypeScript-6-3178C6)](https://www.typescriptlang.org/)

Corgi is a production Bluesky custom feed where subscribers democratically vote on ranking weights and content rules. The backend applies those decisions in a transparent, auditable scoring pipeline. Every ranking decision is decomposed, stored, and explainable.
Corgi Commons is a production Bluesky custom feed shaped through a limited governance pilot. Approved participants can vote on five global ranking signals, topic priorities that shape relevance, and content rules that shape eligibility. Closed results are reviewed and approved before the complete policy is applied and the feed is rescored. Bluesky shows the ordered posts; Corgi exposes the policy and available ranking receipts.

> **Research question:** Can communities meaningfully govern their own recommendation algorithms?

![Governance dashboard screenshot](docs/screenshots/dashboard.png)

**Live production snapshot:** on 2026-07-07 at 03:00 UTC, `GET /api/transparency/stats` reported epoch 2 active with 3,348 scored posts, 3,007 unique authors, and 0 votes in the current epoch. See [the dated anonymized metrics packet](docs/lab/2026-07-07-recsys-live-metrics-packet.md) for commands, caveats, and counterfactual receipts.

---

## Try It

**No coding required** — just a Bluesky account.
**No coding required.** Anyone can use the anonymous demo; a Bluesky account is needed only to save or subscribe to the feed.

1. **Add Corgi's feed** — Search "Community Governed" in Bluesky's Feeds tab, or open:
[View feed on Bluesky](https://bsky.app/profile/corgi-network.bsky.social/feed/community-gov)
1. **Open Corgi Commons** — view or subscribe to the public feed without a Corgi account:
[Open Corgi Commons on Bluesky](https://bsky.app/profile/corgi-network.bsky.social/feed/community-gov)

2. **Vote on the algorithm** — Sign in with your Bluesky handle + [app password](https://bsky.app/settings/app-passwords):
[feed.corgi.network/vote](https://feed.corgi.network/vote)
2. **Try isolated shadow governance** — propose a policy, combine it with 24 scripted deterministic voter archetypes, and inspect how a frozen comparison corpus reranks without changing production:
[feed.corgi.network/demo](https://feed.corgi.network/demo)

3. **See the transparency dashboard** — View current weights, vote history, and score breakdowns:
[feed.corgi.network/dashboard](https://feed.corgi.network/dashboard)
3. **Request pilot access** — production governance participation is currently available through an approved waitlist:
[feed.corgi.network/start](https://feed.corgi.network/start)

---

## Features

**Governance**
- Community voting on ranking components: starts with recency, engagement, bridging, source diversity, relevance — the registry accepts arbitrary additions (see [Contributing a scoring component](docs/contributing-scoring-components.md))
- Content-rule voting with include/exclude keywords
- Epoch-based governance lifecycle with trimmed-mean aggregation
- Ballots can cover five global ranking signals, topic priorities, and include/exclude content rules
- Include rules act as an allowlist, excludes take precedence, and rule adoption requires at least 30% support among content-rule ballots
- Scheduled or manual rounds with configurable voting windows, arithmetic mean below 10 ballots, 10% trimmed mean at 10 or more, results review, operator approval, and post-approval rescoring
- Append-only audit log (DB-enforced, no edits or deletes)

**Scoring & Transparency**
- Full score decomposition persisted per post per epoch — raw, weight, weighted for every registered component, in a normalized long table
- Score decomposition persisted per post per epoch — raw, weight, and weighted contribution for every registered component, in a normalized long table
- Transparency endpoints: per-post explanations, counterfactual analysis, feed-level statistics
- Redis-backed feed serving with snapshot cursors (<50ms response time)

Expand All @@ -61,7 +59,7 @@ Corgi is a production Bluesky custom feed where subscribers democratically vote
- Interaction analytics: scroll depth, engagement attribution, keyword performance

**Research Mode**
- Private feed gating: approved participant list for controlled studies
- Approved governance-participant list for the limited pilot; viewing Corgi Commons and using the shadow demo stay public
- Research consent flow with IRB-ready architecture
- Anonymized data export (deterministic hashing, correlatable across tables)
- Legal framework: Terms of Service, Privacy Policy, research consent separation
Expand All @@ -75,7 +73,7 @@ Corgi is a production Bluesky custom feed where subscribers democratically vote
- Research data export: votes, scores, engagement, epochs, audit log (CSV/JSON)

**Engineering**
- 500+ automated tests (unit, integration, stress) with PR-gated CI
- Automated unit, integration, contract, and stress tests with PR-gated CI
- Genuinely pluggable scoring components: implement the `ScoringComponent` interface from `@corgi/feed-sdk`, register, ship. No schema migration required. See the [contribution guide](docs/contributing-scoring-components.md) and the [civility example](examples/civility-component/) for an end-to-end walk-through. ADR-0001 covers the design.
- Pre-commit hooks (husky + lint-staged + tsc)
- Dependabot for automated dependency updates
Expand Down Expand Up @@ -212,7 +210,7 @@ Five components, each normalized to 0.0–1.0:
| Source Diversity | Author variety | Diminishing returns per author (1.0 → 0.7 → 0.5 → 0.3) |
| Relevance | Topic match × community preference | Weighted dot product of topic vectors and governance weights |

**Final score:** `total = Σ(component_raw × governance_weight)` where weights sum to 1.0 and are set by community vote.
**Component score:** `total = Σ(component_raw × approved_signal_weight)` where the five global signal weights sum to 1.0. Topic preferences affect the relevance component only; eligibility rules and publication adjustments are applied separately.

All 15 numeric values (5× raw, weight, weighted) are persisted per post per epoch for full auditability.

Expand Down
13 changes: 7 additions & 6 deletions docs/PRD.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@
Status: canonical product brief
Owner: bluesky-feed
Service class: production_service
Last updated: 2026-04-05
Last updated: 2026-07-13

## Mission

Prove that a real social feed can be community-governed without becoming opaque,
slow, or unserious. Subscribers should be able to influence ranking behavior and
see exactly how those choices affect the feed.
Prove that a real social feed can be community-shaped without becoming opaque,
slow, or unserious. Approved pilot participants can propose ranking policy while
anyone can inspect what changed and why.

## Problem

Expand All @@ -19,8 +19,9 @@ algorithm, and cannot audit the consequences of policy changes.

## Outcomes

- Subscribers can vote on ranking weights and content rules through a usable
governance surface.
- Approved pilot participants can vote on five global signal weights, topic
priorities, and content rules through a usable governance surface. Anyone can
inspect the public feed or use the isolated shadow demo without joining the pilot.
- The ranking pipeline persists full score decomposition so every result is
explainable — in a normalized long table that supports any-N components,
not a 5-column fossilized schema.
Expand Down
93 changes: 37 additions & 56 deletions docs/SYSTEM_OVERVIEW.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,13 @@
# Community-Governed Bluesky Feed Generator
# Corgi Commons Feed Generator

## What This Is

A Bluesky custom feed where **subscribers democratically vote on algorithm parameters**. Instead of one person deciding how posts are ranked, the community does through Polis-style deliberation.
A production Bluesky custom feed with inspectable, community-shaped ranking. Corgi Commons is public to view and subscribe to; production governance participation is currently an approved waitlist pilot.

- Users subscribe to the feed in their Bluesky app
- Subscribers vote on how much weight each scoring component should have
- Votes are aggregated using trimmed mean (removes outliers)
- The feed algorithm uses the community-chosen weights

**No one has built this before.** This is the first governance-weighted custom feed on AT Protocol.
- Anyone can view Corgi Commons in Bluesky and use the isolated shadow demo
- Approved pilot participants can submit signal weights, topic priorities, and content rules
- Fewer than 10 ballots use an arithmetic mean; 10 or more use a 10% trimmed mean
- Closed results require review and operator approval before the complete policy is applied and the feed is rescored

---

Expand Down Expand Up @@ -49,7 +47,7 @@ A Bluesky custom feed where **subscribers democratically vote on algorithm param
2. **Scoring Pipeline** (every 5 minutes) → Calculates 5 component scores per post, applies governance weights, stores full decomposition
3. **Top 1000 posts** → Pushed to Redis sorted set (`feed:current`)
4. **getFeedSkeleton** → Bluesky app requests feed, we read from Redis and return post URIs
5. **Users vote** → Weights aggregated via trimmed mean → Next epoch uses new weights
5. **Approved participants vote** → Ballots aggregate after the configurable window → Results review and operator approval → Complete policy application → Rescore

---

Expand Down Expand Up @@ -86,49 +84,42 @@ Example with default weights:

### How Voting Works

1. **Eligibility**: Only feed subscribers can vote (tracked when they request the feed)
2. **Weight Distribution**: Voters set their preferred weight for each component (must sum to 100%)
3. **One Vote Per Epoch**: Each subscriber gets one vote per governance epoch (can update)
4. **Audit Trail**: All votes logged to append-only audit table
1. **Eligibility**: Production voting is limited to approved waitlist participants during the pilot
2. **Three ballot channels**: Five global signal weights, topic priorities that shape relevance, and include/exclude content rules
3. **One ballot per epoch**: An approved participant can update their ballot while voting is open
4. **Audit trail**: Governance and operator actions are recorded in the append-only audit table

### Trimmed Mean Aggregation

To prevent manipulation by outliers:

1. Collect all votes for the epoch
1. Collect all ballots for the epoch
2. For each component independently:
- Sort values
- Remove top 10% and bottom 10%
- Calculate mean of remaining values
- With fewer than 10 ballots, calculate the arithmetic mean
- With 10 or more, sort values and remove the top and bottom 10%
- Calculate the mean of the remaining values
3. Normalize result to sum to exactly 1.0

Minimum 10 votes required before trimming kicks in.
Content-rule keywords use a separate threshold: at least 30% support among ballots that submit content rules. Include rules act as an allowlist and excludes take precedence.

### Epoch Lifecycle

```
┌──────────────┐ votes ≥ MIN_VOTES ┌──────────────┐
│ ACTIVE │ ─────────────────────▶ │ VOTING │
│ (current) │ │ (closing) │
└──────────────┘ └──────┬───────┘
│
┌────────────────────────────────────────┘
│ aggregate votes, create new epoch
▼
┌──────────────┐ ┌──────────────┐
│ ACTIVE │ ◀──────────────────────│ CLOSED │
│ (new epoch) │ │ (archived) │
└──────────────┘ └──────────────┘
```
1. A vote is scheduled or opened manually with a configurable voting window.
2. The window closes and ballots are aggregated into proposed signals, topics, and content rules.
3. Results enter review; an operator approves or rejects the complete proposal.
4. Approval applies all three policy channels to the active epoch, returns it to the running phase, and queues a durable rescore.
5. Bluesky receives the resulting ordered post URIs; Corgi exposes policy and receipt metadata.

---

## Database Schema (Key Tables)

### posts

Stores all ingested Bluesky posts with `deleted` flag for soft deletes.

### post_scores (GOLDEN RULE)

Stores **decomposed scores** - 15 columns per post per epoch:
- 5 raw scores (recency_score, engagement_score, etc.)
- 5 weights from governance epoch
Expand All @@ -137,18 +128,23 @@ Stores **decomposed scores** - 15 columns per post per epoch:
- epoch_id (critical for measuring governance impact)

### governance_epochs

Current and historical governance periods with weight distribution.

### governance_votes
Individual subscriber votes (one per voter per epoch).

Individual approved-participant ballots (one per voter per epoch, updatable while open).

### governance_audit_log

**Append-only** record of all governance actions (trust anchor).

### subscribers
Tracks who uses the feed and is eligible to vote.

Tracks feed subscriptions and participant state; pilot voting eligibility is separately allowlisted.
Comment thread
AndrewNordstrom marked this conversation as resolved.

### jetstream_cursor

Persists cursor every 1000 events to resume without gaps.

---
Expand All @@ -160,38 +156,23 @@ The React frontend provides:
- **Current Weights**: Radar chart showing active epoch's weight distribution
- **Feed Statistics**: Posts scored, unique authors, average bridging
- **Epoch History**: Timeline of all governance epochs with weight changes
- **Post Explanations**: Why any post has its current rank (score breakdown)
- **Post Explanations**: Score and ranking provenance when a Corgi receipt is available
- **Audit Log**: Complete history of governance actions

---

## Current Limitations

1. **All of Bluesky**: Currently ingests everything, not scoped to topic/community
2. **No Moderation Layer**: Relies on Bluesky's own moderation
3. **Relevance**: Uses keyword + embedding classification at ingestion time; governance weights determine per-topic relevance at scoring time
4. **Single Feed**: One governance epoch affects one feed
1. **Pilot governance access**: Viewing and the shadow demo are public; production voting is allowlisted
2. **Single public governed feed**: Corgi Commons is the production feed; multi-community governance is not yet public
3. **Relevance model**: Curated keyword topic classification is primary; sentence-embedding similarity is optional behind configuration
4. **Receipt coverage**: A receipt requires persisted score provenance for the post and epoch

---

## Future: Collection Governance

Currently users vote on **ranking** (how to order posts). Future extension: vote on **collection** (what to include).

### Potential Collection Rules

- **Topic Keywords**: Include posts mentioning specific terms
- **Source Allowlists/Blocklists**: Include/exclude specific accounts
- **Content Type Filters**: Text only, images allowed, no reposts
- **Language Filters**: English only, multilingual, etc.
- **Engagement Thresholds**: Minimum likes to be considered

### Implementation Path
## Content-Rule Governance

1. Add `collection_rules` JSONB column to `governance_epochs`
2. Create voting UI for rule proposals
3. Jetstream handlers check rules before inserting posts
4. Separate epochs for collection rules vs. ranking weights
Content rules are part of the current ballot. Include-keyword rules form an allowlist, exclude-keyword rules take precedence, and adopted rules are applied with the approved signal and topic policy before rescoring. The public shadow demo exposes only bounded exclusion-rule mechanics for this release.

---

Expand Down
Loading
Loading