Repository navigation
feat(devpass): add model census survey + reward - #3169
Merged
Merged
Conversation
Yearly DevPass-only "Model Census" asking which coding models are worth their cost, based on verified usage: - model_survey_response table (per user+model+year, value/quality/ speed scores, recommend, use case, usage snapshot) - /model-survey eligibility + submit routes; the org's first entry each year grants a free current-tier Reset Pass ($0 dev_plan_reset_pass_reward transaction, audit-logged) - /public/model-survey/results with a 5-response anonymity threshold - usage-triggered census reminder dialog in the dashboard shell with a 14-day snooze cookie, survey form at /dashboard/survey - public results registry at /data/[year] with Dataset JSON-LD - seeded census data, API spec tests, and Playwright e2e for apps/code (pnpm --filter code test:e2e) Claude-Session: https://claude.ai/code/session_019bW8g9tRS3vXnUjezwoJZN
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a quarterly model survey with database storage, authenticated eligibility and submission APIs, Reset Pass rewards, public anonymized census results, dashboard survey flows, year-specific census pages, seed data, and Playwright coverage. ChangesModel census
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Developer
participant Dashboard
participant SurveyAPI
participant Database
participant CensusPage
Developer->>Dashboard: Open dashboard
Dashboard->>SurveyAPI: GET /model-survey/eligibility
SurveyAPI->>Database: Read usage and survey responses
Database-->>SurveyAPI: Eligible models and reward state
SurveyAPI-->>Dashboard: Eligibility payload
Dashboard->>SurveyAPI: POST /model-survey
SurveyAPI->>Database: Store response and grant quarterly reward
Database-->>SurveyAPI: Stored response and reward result
SurveyAPI-->>Dashboard: Submission result
CensusPage->>SurveyAPI: GET /public/model-survey/results?year=year
SurveyAPI->>Database: Aggregate anonymized responses
Database-->>SurveyAPI: Published census metrics
SurveyAPI-->>CensusPage: Yearly census results
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (4)
apps/code/src/app/dashboard/DashboardShell.tsx (1)
72-79: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueNav item year is computed once at module load, not per render.
resourceNavItemsis a module-scope constant, sonew Date().getUTCFullYear()is evaluated once when the module is first loaded, not on each render. A user who keeps a tab open across the UTC year boundary will see the "Model census" link keep pointing at the previous year until a full page reload.♻️ Proposed fix: compute the href at render time
-const resourceNavItems: Array<{ - label: string; - href: Route; - icon: typeof BarChart3; -}> = [ - { label: "Coding models", href: "/coding-models" as Route, icon: Code }, - { - label: "Model census", - href: `/data/${new Date().getUTCFullYear()}` as Route, - icon: Stamp, - }, -]; +const staticResourceNavItems: Array<{ + label: string; + href: Route; + icon: typeof BarChart3; +}> = [{ label: "Coding models", href: "/coding-models" as Route, icon: Code }];Then inside
DashboardShell, buildresourceNavItemswithuseMemo(() => [...staticResourceNavItems, { label: "Model census", href: \/data/${new Date().getUTCFullYear()}` as Route, icon: Stamp }], [])`.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/code/src/app/dashboard/DashboardShell.tsx` around lines 72 - 79, Move the dynamic “Model census” year calculation out of the module-scope resourceNavItems constant and compute it inside DashboardShell during rendering, using useMemo with the existing static navigation items as the base. Ensure the href reflects the current UTC year when the component renders.apps/code/e2e/census.pw.ts (1)
34-36: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winReplace fixed
waitForTimeout(3000)sleeps with a direct polling assertion.Both sites use a hardcoded 3s sleep before asserting the dialog stays hidden.
expect(...).not.toBeVisible()already polls internally, so the sleep just adds fixed latency and can still flake under slower CI if the negative assertion runs before the (500-900ms) dialog logic would have fired again on a regression.
apps/code/e2e/census.pw.ts#L34-L36: droppage.waitForTimeout(3000)and letawait expect(dialog).not.toBeVisible({ timeout: 5000 })do the waiting afterpage.reload().apps/code/e2e/census.pw.ts#L73-L75: same change afterpage.goto("/dashboard")— drop the sleep and pass an explicittimeoutto thenot.toBeVisible()assertion instead.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/code/e2e/census.pw.ts` around lines 34 - 36, Replace the fixed 3-second sleeps at apps/code/e2e/census.pw.ts:34-36 and apps/code/e2e/census.pw.ts:73-75 with direct polling assertions: remove each page.waitForTimeout call and set the corresponding dialog not.toBeVisible assertion timeout to 5000ms after page.reload() and page.goto("/dashboard"), respectively.packages/db/src/schema.ts (2)
590-609: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSame enum literal sets are independently redefined in three places. The
primaryUseCase(7 values) and tier ("lite"/"pro"/"max") sets have one root cause — no shared source of truth — but are copy-pasted across the DB schema and two API route files, risking silent drift.
packages/db/src/schema.ts#L590-L609: export theprimaryUseCaseand tier literal arrays as named constants (e.g.MODEL_SURVEY_USE_CASES,DEV_PLAN_TIERS) from this file (or a small shared module) instead of inlining them only here.apps/api/src/routes/model-survey.ts#L31-L41: builduseCaseEnum/tierEnumfrom the shared constants instead of re-listing the literals.apps/api/src/routes/public-model-survey.ts#L14-L22: builduseCaseEnumfrom the same shared constant instead of a third independent copy.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/schema.ts` around lines 590 - 609, Create and export shared primary-use-case and development-plan-tier literal arrays from the schema definitions around primaryUseCase and devPlanTier, then use those constants to build the enums in model-survey.ts and public-model-survey.ts instead of duplicating literals. Update packages/db/src/schema.ts lines 590-609 to define the constants, apps/api/src/routes/model-survey.ts lines 31-41 to consume both shared constants, and apps/api/src/routes/public-model-survey.ts lines 14-22 to consume the shared use-case constant.
610-617: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winConsider a DB-level guardrail for the one-reward-per-org-per-year invariant.
The single-reward-per-org-per-year rule is currently enforced only by the
SELECT ... FOR UPDATElock inmodel-survey.ts. A partial unique index would make this invariant hold even if a future code path (e.g. a backfill script, a refactor) skips the lock.🛡️ Optional defense-in-depth index
index("model_survey_response_organization_id_idx").on(table.organizationId), + uniqueIndex("model_survey_response_org_year_reward_unique") + .on(table.organizationId, table.year) + .where(sql`${table.rewardTier} IS NOT NULL`),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/schema.ts` around lines 610 - 617, Add a database-level partial unique index in the table definition callback for model survey responses, keyed by organizationId and year and restricted to rows representing a reward. Use the schema’s existing reward-presence column or predicate, and generate the corresponding migration so the one-reward-per-organization-per-year invariant is enforced independently of the SELECT ... FOR UPDATE path in model-survey.ts.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/api/src/routes/public-model-survey.ts`:
- Around line 99-146: Apply the anonymity threshold to the per-use-case buckets
built in useCasesByModel before returning them from the models mapping. Omit or
aggregate buckets whose counts are below the chosen threshold, reusing
MIN_PUBLIC_RESPONSES where appropriate, and ensure the returned useCases cannot
expose low-count individual responses while preserving published model totals.
In `@apps/code/src/app/dashboard/survey/SurveyClient.tsx`:
- Around line 229-236: Update rateAnother to reset every controlled survey
field, including valueScore, qualityScore, speedScore, wouldRecommend, and
primaryUseCase, alongside modelId and comment. Use the form’s empty/default
values so the next model starts without selections from the previous submission.
In `@apps/code/src/lib/model-survey.ts`:
- Around line 22-29: Replace the raw fetch and manual ModelSurveyResults
assertion in the model survey results helper with the applicable generated typed
API client, passing the client into the helper if necessary. Preserve the year
query parameter, revalidation behavior, null return for unsuccessful responses,
and typed result handling.
- Around line 17-33: Update fetchModelSurveyResults so fetch exceptions and
non-OK responses are not converted to null, which CensusPage interprets as an
empty registry. Remove the failure-to-null handling in fetchModelSurveyResults
and propagate the error to Next’s error handling, while preserving the
successful JSON result path.
In `@packages/db/src/seed.ts`:
- Around line 1865-1873: The seed flow must create the census respondent users
and organizations before inserting survey responses. In the setup around
censusRespondents and the EXTRA_USERS/EXTRA_ORGS upserts, move
bulkInsert(tables.modelSurveyResponse, censusResponses) to run after those
upserts while preserving the existing response data.
---
Nitpick comments:
In `@apps/code/e2e/census.pw.ts`:
- Around line 34-36: Replace the fixed 3-second sleeps at
apps/code/e2e/census.pw.ts:34-36 and apps/code/e2e/census.pw.ts:73-75 with
direct polling assertions: remove each page.waitForTimeout call and set the
corresponding dialog not.toBeVisible assertion timeout to 5000ms after
page.reload() and page.goto("/dashboard"), respectively.
In `@apps/code/src/app/dashboard/DashboardShell.tsx`:
- Around line 72-79: Move the dynamic “Model census” year calculation out of the
module-scope resourceNavItems constant and compute it inside DashboardShell
during rendering, using useMemo with the existing static navigation items as the
base. Ensure the href reflects the current UTC year when the component renders.
In `@packages/db/src/schema.ts`:
- Around line 590-609: Create and export shared primary-use-case and
development-plan-tier literal arrays from the schema definitions around
primaryUseCase and devPlanTier, then use those constants to build the enums in
model-survey.ts and public-model-survey.ts instead of duplicating literals.
Update packages/db/src/schema.ts lines 590-609 to define the constants,
apps/api/src/routes/model-survey.ts lines 31-41 to consume both shared
constants, and apps/api/src/routes/public-model-survey.ts lines 14-22 to consume
the shared use-case constant.
- Around line 610-617: Add a database-level partial unique index in the table
definition callback for model survey responses, keyed by organizationId and year
and restricted to rows representing a reward. Use the schema’s existing
reward-presence column or predicate, and generate the corresponding migration so
the one-reward-per-organization-per-year invariant is enforced independently of
the SELECT ... FOR UPDATE path in model-survey.ts.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: b65fffc9-60f8-4e08-97da-b304fad8f2d8
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (27)
apps/api/src/index.tsapps/api/src/routes/index.tsapps/api/src/routes/model-survey.spec.tsapps/api/src/routes/model-survey.tsapps/api/src/routes/organization.tsapps/api/src/routes/public-model-survey.tsapps/api/src/utils/devpass-filter.tsapps/code/.gitignoreapps/code/e2e/census.pw.tsapps/code/package.jsonapps/code/playwright.config.tsapps/code/src/app/dashboard/DashboardShell.tsxapps/code/src/app/dashboard/components/SurveyReminderDialog.tsxapps/code/src/app/dashboard/survey/SurveyClient.tsxapps/code/src/app/dashboard/survey/page.tsxapps/code/src/app/data/[year]/page.tsxapps/code/src/app/data/page.tsxapps/code/src/app/sitemap.tsapps/code/src/lib/cookies.tsapps/code/src/lib/model-survey.tsapps/code/tsconfig.jsonpackages/db/migrations/1784664094_quick_mercury.sqlpackages/db/migrations/meta/1784664094_snapshot.jsonpackages/db/migrations/meta/_journal.jsonpackages/db/src/relations.tspackages/db/src/schema.tspackages/db/src/seed.ts
Comment on lines
+17
to
+33
| export async function fetchModelSurveyResults( | ||
| year: number, | ||
| ): Promise<ModelSurveyResults | null> { | ||
| const config = getConfig(); | ||
| try { | ||
| const res = await fetch( | ||
| `${config.apiBackendUrl}/public/model-survey/results?year=${year}`, | ||
| { next: { revalidate: 300 } }, | ||
| ); | ||
| if (!res.ok) { | ||
| return null; | ||
| } | ||
| return (await res.json()) as ModelSurveyResults; | ||
| } catch { | ||
| return null; | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not convert API failures into an empty census.
CensusPage treats null as an empty registry, so a timeout or non-2xx response is published as “Registry open” rather than an unavailable result. Let the error reach Next’s error handling (or return an explicit unavailable state).
Proposed fix
export async function fetchModelSurveyResults(
year: number,
): Promise<ModelSurveyResults | null> {
const config = getConfig();
- try {
- const res = await fetch(
- `${config.apiBackendUrl}/public/model-survey/results?year=${year}`,
- { next: { revalidate: 300 } },
- );
- if (!res.ok) {
- return null;
- }
- return (await res.json()) as ModelSurveyResults;
- } catch {
- return null;
+ const res = await fetch(
+ `${config.apiBackendUrl}/public/model-survey/results?year=${year}`,
+ { next: { revalidate: 300 } },
+ );
+ if (!res.ok) {
+ throw new Error(`Model census request failed: ${res.status}`);
}
+ return (await res.json()) as ModelSurveyResults;
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| export async function fetchModelSurveyResults( | |
| year: number, | |
| ): Promise<ModelSurveyResults | null> { | |
| const config = getConfig(); | |
| try { | |
| const res = await fetch( | |
| `${config.apiBackendUrl}/public/model-survey/results?year=${year}`, | |
| { next: { revalidate: 300 } }, | |
| ); | |
| if (!res.ok) { | |
| return null; | |
| } | |
| return (await res.json()) as ModelSurveyResults; | |
| } catch { | |
| return null; | |
| } | |
| } | |
| export async function fetchModelSurveyResults( | |
| year: number, | |
| ): Promise<ModelSurveyResults | null> { | |
| const config = getConfig(); | |
| const res = await fetch( | |
| `${config.apiBackendUrl}/public/model-survey/results?year=${year}`, | |
| { next: { revalidate: 300 } }, | |
| ); | |
| if (!res.ok) { | |
| throw new Error(`Model census request failed: ${res.status}`); | |
| } | |
| return (await res.json()) as ModelSurveyResults; | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/code/src/lib/model-survey.ts` around lines 17 - 33, Update
fetchModelSurveyResults so fetch exceptions and non-OK responses are not
converted to null, which CensusPage interprets as an empty registry. Remove the
failure-to-null handling in fetchModelSurveyResults and propagate the error to
Next’s error handling, while preserving the successful JSON result path.
Comment is now mandatory on model-survey submissions (API validation + form), so every census response carries qualitative feedback. Claude-Session: https://claude.ai/code/session_019bW8g9tRS3vXnUjezwoJZN
The census block referenced EXTRA_USERS rows before they were inserted, so seeding a fresh database (CI) failed on the user FK. Claude-Session: https://claude.ai/code/session_019bW8g9tRS3vXnUjezwoJZN
…rvey # Conflicts: # pnpm-lock.yaml
- suppress single-respondent use-case buckets in public results - add partial unique index backing one-reward-per-org-per-year - share survey enum literals from @llmgateway/db - fully reset the census form when rating another model - distinguish registry-offline from registry-empty on /data/[year] - compute the census nav year at render time Claude-Session: https://claude.ai/code/session_019bW8g9tRS3vXnUjezwoJZN
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/api/src/routes/public-model-survey.ts`:
- Around line 22-26: Update the top-level total calculations in the survey
aggregation flow to include only responses associated with published model IDs
that meet the five-response threshold. Ensure totalResponses, totalRespondents,
and totalModelsRated are all derived from that filtered set, or return
suppressed totals when no model qualifies; preserve the existing per-use-case
bucket suppression.
In `@packages/db/migrations/1784744600_overjoyed_war_machine.sql`:
- Line 1: Update the migration statement for model_survey_response to create the
unique index concurrently, and configure the migration so this statement runs
outside Drizzle Kit’s default transaction boundary. Preserve the existing index
name, columns, and reward_tier predicate.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 3c8c4811-05a5-4e1b-989e-596a41a48488
📒 Files selected for processing (10)
apps/api/src/routes/model-survey.spec.tsapps/api/src/routes/model-survey.tsapps/api/src/routes/public-model-survey.tsapps/code/src/app/dashboard/DashboardShell.tsxapps/code/src/app/dashboard/survey/SurveyClient.tsxapps/code/src/app/data/[year]/page.tsxpackages/db/migrations/1784744600_overjoyed_war_machine.sqlpackages/db/migrations/meta/1784744600_snapshot.jsonpackages/db/migrations/meta/_journal.jsonpackages/db/src/schema.ts
🚧 Files skipped from review as they are similar to previous changes (6)
- packages/db/src/schema.ts
- apps/code/src/app/dashboard/DashboardShell.tsx
- apps/api/src/routes/model-survey.spec.ts
- apps/api/src/routes/model-survey.ts
- apps/code/src/app/dashboard/survey/SurveyClient.tsx
- apps/code/src/app/data/[year]/page.tsx
Responses and the free Reset Pass reward are now scoped to (year, quarter): members can re-rate models each quarter (trend data) and earn one pass per org per wave. The public /data/[year] report still aggregates the full year. Migration regenerated as a single artifact since the table never shipped. Claude-Session: https://claude.ai/code/session_019bW8g9tRS3vXnUjezwoJZN
Stat tiles now count only registry-published models, so totals always match the table and sub-threshold models never surface. Claude-Session: https://claude.ai/code/session_019bW8g9tRS3vXnUjezwoJZN
Replace the raw fetch in fetchModelSurveyResults with a shared createPublicServerApiClient (openapi-fetch, no cookie read) so the /data/[year] page keeps its typed contract and stays static/ISR. Claude-Session: https://claude.ai/code/session_019bW8g9tRS3vXnUjezwoJZN
…rvey # Conflicts: # apps/api/src/routes/organization.ts # apps/api/src/utils/devpass-filter.ts # packages/db/src/schema.ts
…feat/devpass-model-survey
…rvey # Conflicts: # pnpm-lock.yaml
smakosh
enabled auto-merge
July 22, 2026 21:07
smakosh
added this pull request to the merge queue
Jul 22, 2026
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The DevPass Model Census — a quarterly-wave, DevPass-only survey answering one question: which coding models are actually worth the money? Usage-triggered reminder → 1-minute survey → free Reset Pass → public data report.
The flow
/dashboard/survey): passport-styled form — value-for-money, output quality, speed (1–5 stamp scores), APPROVED/DENIED recommend stamps, primary use case, required field notes. Members can rate every model they qualify for; each response snapshots the qualifying request count.devPlanResetPasses<Tier>increment as a purchase, recorded as a $0dev_plan_reset_pass_rewardtransaction (new type) +dev_plan.reset_pass_rewardaudit event. Redeemable through the existing redeem flow, refund-ineligible, excluded from invoices, revenue metrics, andpaidTransactionTypes./data/[year], e.g./data/2026): "The 2026 DevPass Model Census" — registry ranked by value score with quality/speed meters, recommend %, stat tiles, methodology, Dataset JSON-LD, sitemap entry. Fed byGET /public/model-survey/results.Design decisions (deviations/additions beyond the original ask)
projectHourlyModelStatswith the model-ratings name normalization) — the published data can honestly claim every rating is backed by real usage.FOR UPDATEon the org row inside the insert transaction).year,quarter) columns +/data/[year]route (yearly report aggregates all four waves) — 2027 needs zero code.model_rating(richer dimensions, DevPass-only, usage-window-verified). Cross-posting the quality score to public model pages is a possible follow-up.dev_plan_reset_passso reset-pass revenue analytics don't count free grants.model_survey_prompt_shown/dismissed/accepted,model_survey_completed(client) +model_survey_submitted(server).apps/code/e2e,*.pw.tsso root vitest globs never pick them up):pnpm --filter code test:e2eagainst a freshly seeded local stack.Testing
apps/api/src/routes/model-survey.spec.ts— 14 specs: eligibility gating (auth, non-devpass, inactive plan, threshold, window, ordering), submit + reward grant (org counter, $0 transaction), no double reward, 409 dupes, tier-bound grant, public aggregation + anonymity threshold + year scoping. ✅pnpm build(all apps) ✅,pnpm format✅https://claude.ai/code/session_019bW8g9tRS3vXnUjezwoJZN
Summary by CodeRabbit
New Features
Bug Fixes
Tests