Skip to content

feat: analytics#4068

Merged
chronark merged 96 commits intomainfrom
feat/analytics
Nov 4, 2025
Merged

feat: analytics#4068
chronark merged 96 commits intomainfrom
feat/analytics

Conversation

@Flo4604
Copy link
Member

@Flo4604 Flo4604 commented Oct 6, 2025

What does this PR do?

This does too many things, but mainly:

  • Extends our cache package to Many Operations (SWRMany GetMany SetMany) and so on, also adds a tests for this

  • Clickhouse user "management"

    • Create or update an existing user in clickhouse and give it permission to tables, set RLS quota and settings profile.
    • Generates and encrypt the password in vault for later usage
    • New table to store limits usernames and encrypted password
  • Seed analytics data but in go

    • Basically what our pnpm local does but insert the keys/identities into the db too and uses our bufferer.
  • 2 New permissions for reading analytics api.*.read_analytics and api.<api_id>.read_analytics.

  • Error docs + analytics docs

    • We now automatically generate mdx files for our go fault codes except gateway + update docs.json with them.
    • Adds docs for the new analytics errors + docs for how to use the analaytics (mostly claude written blame me if its wrong)
    • New engineering doc how us letting users query our clickhouse works
  • New endpoint that lets the user query clickhouse with his user and with that

    • New openapi error specs
    • tests for the endpoint that are WIP
    • Adds a connecting manager that holds the cache for the http connection + password.
  • Adds a clickhouse query parser which does things

    • Is this a valid written query in the first place?
    • Rename table aliases e.g key_verifications => key_verifications_raw_v2
    • Is the query using the tables its allowed to?
    • Is it using any functions that we did not allow yet?
    • Injects a given workspace id to all queries
    • Injects "Security" filters like a apiId so we can query only the rows the API key has perms for.
    • Supports "virtual" columns, which are columns that don't actually exist in clickhouse but we wish they would,
      • We collect all the values of the query for those virtual columns and the config takes in resolvers that basically maps queryValues => what should we search with in this case its externalId => identity_id and apiId => key_auth_id
    • Result transformation, its nice if we can select stuff but doesnt help if we throw the user a key_auth_id or identity_id so we collect the results and rewrite what they actually need.
    • Rewrites any limit statements to whats is allowed via config/struct
  • Regenerated the sql schema for our ts client

  • Regenerated our protobuf generation stuff

Fixes #3562

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Chore (refactoring code, technical debt, workflow improvements)
  • Enhancement (small improvements)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How should this be tested?

(edited by @chronark)

make up

Then,

pnpm local to trigger the build and generate new permissions

go run . create-clickhouse-user \
--database-primary="unkey:password@tcp(127.0.0.1:3306)/unkey?parseTime=true&interpolateParams=true" \
--clickhouse-url="clickhouse://default:password@localhost:9000?secure=false" \
--vault-s3-url="http://127.0.0.1:3902" \
--vault-s3-bucket="vault" \
--vault-s3-access-key-id="minio_root_user" \
--vault-s3-access-key-secret="minio_root_password" \
--vault-master-keys="Ch9rZWtfMmdqMFBJdVhac1NSa0ZhNE5mOWlLSnBHenFPENTt7an5MRogENt9Si6wms4pQ2XIvqNSIgNpaBenJmXgcInhu6Nfv2U=" \
--workspace-id="REPLACE_ME"
go run . dev seed verifications \
    --api-id REPLACE_ME \
    --num-verifications 1000000

You can use your favorite curl/httpie/gui and call the endpoint

curl -X POST http://127.0.0.1:7070/v2/analytics.getVerifications \
    -H "Authorization: Bearer <ROOT_KEY>" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg query "YOUR_MULTILINE_QUERY" '{query: $query}')" | jq

So please go trough the docs and run the queries from there or write your owns and test if it works

Checklist

Required

  • Filled out the "How to test" section in this PR
  • Read Contributing Guide
  • Self-reviewed my own code
  • Commented on my code in hard-to-understand areas
  • Ran pnpm build
  • Ran pnpm fmt
  • Checked for warnings, there are none
  • Removed all console.logs
  • Merged the latest changes from main onto my branch with git pull origin main
  • My changes don't cause any responsiveness issues

Appreciated

  • [] If a UI change was made: Added a screen recording or screenshots to this PR
  • Updated the Unkey Docs if changes were necessary

@changeset-bot
Copy link

changeset-bot bot commented Oct 6, 2025

⚠️ No Changeset found

Latest commit: 1ca4e74

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@vercel
Copy link

vercel bot commented Oct 6, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
dashboard Ready Ready Preview Comment Nov 4, 2025 4:57pm
1 Skipped Deployment
Project Deployment Preview Comments Updated (UTC)
engineering Ignored Ignored Preview Nov 4, 2025 4:57pm

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Oct 6, 2025

📝 Walkthrough

Walkthrough

Adds a new analytics feature: POST /v2/analytics.getVerifications endpoint with SQL parser/enforcement, per-workspace ClickHouse connection manager and provisioning, RBAC read_analytics permissions, cache multi-key primitives, DB schema/sqlc updates, CLI/tools, extensive docs, and tests.

Changes

Cohort / File(s) Summary
API route & handler
go/apps/api/routes/v2_analytics_get_verifications/handler.go, go/apps/api/routes/register.go, go/apps/api/run.go
New POST /v2/analytics.getVerifications handler, route registration and runtime wiring; request parsing, RBAC checks, workspace ClickHouse connection lookup, query execution and response shaping.
OpenAPI & spec
go/apps/api/openapi/..., go/apps/api/openapi/spec/paths/v2/analytics/getVerifications/*, go/apps/api/openapi/openapi-split.yaml, go/apps/api/openapi/openapi-generated.yaml
Added analytics operation, request/response schemas, response-data shape, error response schemas, tag/server entries and public exposure.
SQL parser & AST rewrites
go/pkg/clickhouse/query-parser/* (types.go, parser.go, tables.go, filter.go, validation.go, limits.go, cte.go, tests)
New ClickHouse SQL parser: SELECT-only enforcement, function whitelist, table aliasing/allowlist, CTE handling, workspace/security filter injection, LIMIT enforcement, and associated tests.
ClickHouse client & error handling
go/pkg/clickhouse/client.go, go/pkg/clickhouse/interface.go, go/pkg/clickhouse/errors.go, go/pkg/clickhouse/noop.go, go/pkg/clickhouse/user.go
Client extensions (QueryToMaps, Exec, Ping, Close), ClickHouse error classification/wrapping, noop implementations, user provisioning for grants/RLS/quotas and default allowed tables.
Analytics connection manager
go/internal/services/analytics/connection_manager.go, go/apps/api/run.go, go/apps/api/routes/services.go
Per-workspace ConnectionManager with SWR settings cache, Vault decryption for credentials, cached ClickHouse connections and liveness checks; Noop manager fallback.
RBAC & frontend permissions
go/pkg/rbac/permissions.go, apps/dashboard/.../permissions/permissions.ts, apps/dashboard/.../components/.../permissions.ts
Added read_analytics action and corresponding workspace- and API-level permission entries surfaced to dashboard payloads.
Cache multi-key primitives & middleware
go/pkg/cache/*, go/pkg/cache/clustering/cluster_cache.go, go/pkg/cache/middleware/tracing.go, go/pkg/cache/many_test.go, go/pkg/cache/noop.go
Added bulk cache ops: GetMany/SetMany/SetNullMany/SWRMany with batching, revalidation, cluster invalidation, tracing middleware and tests.
DB schema, sqlc queries & generated models
go/pkg/db/schema.sql, go/pkg/db/queries/*, go/pkg/db/*_generated.go, internal/db/src/schema/clickhouse_workspace_settings.ts
New clickhouse_workspace_settings table, multiple sqlc queries, generated types and bulk-insert helpers for settings, key_auth, identities and related operations.
DB bulk helpers & querier changes
go/pkg/db/querier_bulk_generated.go, go/pkg/db/querier_generated.go
Added bulk insert methods and new querier methods for ClickHouse workspace settings, key_auth, identities and related queries.
Error codes & middleware mapping
go/pkg/codes/*, go/pkg/zen/middleware_errors.go, apps/docs/errors/*
Added analytics-related URNs/constants, mapped new errors to HTTP statuses (400/412/422/429/503) and added multiple error documentation pages.
Docs & examples
apps/docs/analytics/*, apps/docs/apis/features/analytics.mdx, apps/docs/docs.json, apps/docs/errors/*
New analytics docs: Getting Started, Overview, Query Examples, Schema Reference, Query Restrictions; docs.json navigation updates and many error docs.
Tests (handler, parser, cache, clickhouse)
go/apps/api/routes/v2_analytics_get_verifications/*_test.go, go/pkg/clickhouse/query-parser/*_test.go, go/pkg/cache/many_test.go, go/pkg/clickhouse/key_verifications_test.go
Extensive tests covering route statuses (200/400/401/403/404/412/422/429/503), parser validation, cache multi-key semantics, and ClickHouse verification behavior.
Runtime config, CLI & harness
go/apps/api/config.go, go/cmd/api/main.go, go/cmd/create-clickhouse-user/*, go/cmd/dev/*, go/main.go, go/pkg/testutil/http.go
New ClickhouseAnalyticsURL config/flag; CLI to provision ClickHouse users; harness wiring and SetupAnalytics helpers; dev/seed CLI registration.
ClickHouse schema & seeding tooling
go/cmd/dev/seed/verifications.go, go/pkg/clickhouse/key_verifications_test.go
Seeder for verifications data, deterministic external_id mapping and verification-related test changes.
Infrastructure & build
deployment/docker-compose.yaml, go/go.mod, go/Makefile, go/pkg/db/generate.go, go/apps/api/openapi/generate.go, apps/docs/package.json
Docker-compose env updates, added clickhouse-sql-parser dependency, updated go generate to go run for tools, Makefile tweaks, and docs tooling version bump.
Dashboard UI small additions
apps/dashboard/app/(app)/[workspaceSlug]/apis/[apiId]/settings/components/copy-key-space-id.tsx, .../settings-client.tsx
Added CopyKeySpaceId component and integrated it into API settings UI.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant APIServer as API Server
    participant Keys as Key Service
    participant Parser as SQL Parser
    participant ConnMgr as Analytics ConnMgr
    participant CH as ClickHouse

    Client->>APIServer: POST /v2/analytics.getVerifications { query }
    APIServer->>Keys: validate root key, fetch permissions
    Keys-->>APIServer: permissions (api.*.read_analytics / api.<id>.read_analytics)
    APIServer->>Parser: Parse & rewrite query (SELECT-only, functions, tables)
    alt parser error
        Parser-->>APIServer: error (400)
        APIServer-->>Client: 400 Bad Request
    else
        Parser-->>APIServer: rewritten SQL + injected security filters
        APIServer->>ConnMgr: GetConnection(workspaceID)
        alt not configured / conn fail
            ConnMgr-->>APIServer: 412/503
            APIServer-->>Client: 412/503
        else
            ConnMgr-->>APIServer: ClickHouse client
            APIServer->>CH: Execute rewritten SQL
            alt resource/quota error
                CH-->>APIServer: 422/429
                APIServer-->>Client: 422/429
            else
                CH-->>APIServer: rows ([]map[string]any)
                APIServer-->>Client: 200 { meta, data: rows }
            end
        end
    end
Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Areas needing extra attention:

  • go/pkg/clickhouse/query-parser/* — AST rewrites, function/table whitelist, LIMIT injection, CTE handling, and comprehensive tests.
  • go/internal/services/analytics/connection_manager.go & go/pkg/clickhouse/user.go — Vault decryption, DSN/template usage, connection caching, liveness, concurrency and fault wrapping.
  • go/apps/api/routes/v2_analytics_get_verifications/handler.go — RBAC evaluation, permission→key_space mapping, security filters, and error→HTTP mapping.
  • go/pkg/cache/* — SWRMany batching, inflight deduplication, concurrency and correctness of multi-key flows.
  • DB schema and sqlc-generated files — migration correctness, IN-clause expansion, bulk-insert implementations and use in provisioning/seeding.
  • go/pkg/clickhouse/errors.go — error classification and mapping to user-facing fault codes/messages.

Possibly related PRs

Suggested labels

Core Team

Pre-merge checks and finishing touches

❌ Failed checks (1 warning, 1 inconclusive)
Check name Status Explanation Resolution
Out of Scope Changes check ⚠️ Warning The PR introduces significant out-of-scope changes: cache package extensions, ClickHouse user management, analytics seeding in Go, protobuf regeneration, and TypeScript schema updates are beyond the scope of implementing a single endpoint. Consider splitting into focused PRs: (1) analytics endpoint implementation, (2) cache enhancements, (3) user management and seeding infrastructure.
Title check ❓ Inconclusive The title 'feat: analytics' is vague and generic, using a broad term without describing the main deliverable or specific functionality being added. Consider a more specific title highlighting the primary feature, e.g., 'feat: add analytics query endpoint with ClickHouse integration'.
✅ Passed checks (2 passed)
Check name Status Explanation
Description check ✅ Passed The PR description is comprehensive and detailed, covering all major changes, testing instructions, and checklist completion despite not strictly following the template format.
Linked Issues check ✅ Passed The PR implements all coding requirements for #3562: implements the /v2/analytics.getVerifications endpoint, provides server-side handling and validation, and integrates analytics querying returning verification data per the v2 API contract.
✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/analytics

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 972eb87 and fa2bf01.

📒 Files selected for processing (4)
  • apps/docs/apis/features/analytics.mdx (2 hunks)
  • apps/docs/docs.json (8 hunks)
  • apps/docs/errors/user/bad_request/invalid_analytics_query.mdx (1 hunks)
  • apps/docs/errors/user/bad_request/invalid_analytics_table.mdx (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • apps/docs/errors/user/bad_request/invalid_analytics_table.mdx
  • apps/docs/errors/user/bad_request/invalid_analytics_query.mdx
🧰 Additional context used
🧠 Learnings (5)
📓 Common learnings
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 4190
File: go/internal/services/keys/verifier.go:51-53
Timestamp: 2025-10-30T15:10:52.743Z
Learning: PR #4190 for unkeyed/unkey is focused solely on database schema and query changes for identity-based credits. It adds IdentityCredits and KeyCredits fields to structs and queries, but does not implement the priority enforcement logic in the usagelimiter. The logic implementation is intentionally deferred to a later PR in the stack.
📚 Learning: 2025-02-24T13:35:00.816Z
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 2910
File: apps/engineering/content/design/components/form/form-input.variants.tsx:38-38
Timestamp: 2025-02-24T13:35:00.816Z
Learning: In the Unkey codebase, example values in documentation and demo components intentionally use themed fake data (e.g., Middle Earth references) rather than generic placeholder text.

Applied to files:

  • apps/docs/docs.json
📚 Learning: 2025-08-20T11:41:36.718Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3810
File: go/apps/api/routes/v2_ratelimit_limit/handler.go:54-56
Timestamp: 2025-08-20T11:41:36.718Z
Learning: In the Unkey codebase, the X-Unkey-Metrics: disabled header is used by the v1 API when replaying ratelimit requests to the v2 API (go/apps/api/routes/v2_ratelimit_limit/handler.go) to prevent double billing/logging to ClickHouse. This is a legitimate internal service-to-service communication pattern for API migration scenarios, not intended for external client usage.

Applied to files:

  • apps/docs/apis/features/analytics.mdx
📚 Learning: 2025-08-17T22:54:06.787Z
Learnt from: perkinsjr
Repo: unkeyed/unkey PR: 3798
File: apps/dashboard/app/(app)/apis/[apiId]/_overview/hooks/use-spent-credits.ts:118-121
Timestamp: 2025-08-17T22:54:06.787Z
Learning: In the Unkey dashboard, real-time data updates are prioritized over caching optimizations. The 10-second refetch interval for spent credits and similar metrics is intentional and used consistently across the codebase to provide real-time dashboard functionality.

Applied to files:

  • apps/docs/apis/features/analytics.mdx
📚 Learning: 2025-09-24T18:57:34.843Z
Learnt from: mcstepp
Repo: unkeyed/unkey PR: 4010
File: QUICKSTART-DEPLOY.md:17-17
Timestamp: 2025-09-24T18:57:34.843Z
Learning: In the Unkey deployment platform, API key environment variables use component-specific naming but share the same secret value: UNKEY_API_KEY for the ctrl service (validator), API_KEY for the CLI client, and CTRL_API_KEY for the dashboard client. The ctrl service acts as the source of truth for validation.

Applied to files:

  • apps/docs/apis/features/analytics.mdx
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
  • GitHub Check: Test Dashboard / Test Dashboard
  • GitHub Check: Test Packages / Test
  • GitHub Check: Build / Build
  • GitHub Check: Test API / API Test Local
  • GitHub Check: Test Go API Local / Test
  • GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (7)
apps/docs/apis/features/analytics.mdx (2)

8-10: New v2 analytics availability note is clear and well-positioned.

The Note component effectively sets user expectations about the v2 analytics feature being opt-in and not yet publicly available. This is consistent with the private-beta approach described in the PR objectives.


78-80: Tags querying description is clearer and more actionable.

The consolidated statement ("Tags can be used to filter and aggregate data in your analytics") is more concise than prior phrasing and directly communicates the capability users have.

apps/docs/docs.json (5)

3-41: Navbar and branding configuration additions are well-structured.

The new navbar, footer, and background color configuration follow standard Mintlify patterns and enhance the documentation site's visual identity and navigation. Social links (GitHub, X) and external links (Blog, Discord, Dashboard) are properly configured.


172-181: Analytics section correctly hidden from public navigation.

Marking the analytics documentation group with "hidden": true properly gates the private-beta feature from public visibility. The restructured pages array aligns with the PR objectives of adding analytics documentation.


257-282: User error documentation is comprehensive and well-organized.

The new subgrouping structure (Bad Request, Too Many Requests, Unprocessable Entity) improves error taxonomy and makes navigation intuitive. Analytics-specific errors (invalid_analytics_function, invalid_analytics_query_type, invalid_analytics_table, query_quota_exceeded) are properly included.


174-180: Verify that the analytics/query-restrictions page file exists.

Line 179 references "analytics/query-restrictions", but this file is not documented as created in the PR objectives or AI summary. The confirmed analytics documentation files are: overview, getting-started, query-examples, and schema-reference. Query-restrictions is not mentioned. Please confirm this file exists or remove the reference to avoid a broken documentation link.


450-453: Styling and theme configuration follows Mintlify standards.

The addition of "styling": { "codeblocks": "system" } and "theme": "maple" applies consistent theming across the documentation site.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@Flo4604
Copy link
Member Author

Flo4604 commented Oct 6, 2025

@coderabbitai review

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Oct 6, 2025

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
go/apps/api/routes/v2_analytics_get_verifications/422_test.go (1)

63-68: Avoid fixed 10 s sleep in this test.

A hard sleep adds 10 s to every run and still risks flaking if buffering ever takes longer. Please poll until the condition is met (or times out) so the test fails fast when ready.

-	// Wait for data to be buffered and flushed to ClickHouse
-	time.Sleep(10 * time.Second)
-
-	// Query should fail with 422 due to max_memory_usage limit being exceeded
-	res := testutil.CallRoute[Request, Response](h, route, headers, req)
-	require.Equal(t, 422, res.Status)
+	var res testutil.TestResponse[Response]
+	require.Eventually(t, func() bool {
+		res = testutil.CallRoute[Request, Response](h, route, headers, req)
+		return res.Status == http.StatusUnprocessableEntity
+	}, 30*time.Second, 500*time.Millisecond)
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c83b88e and 58af255.

📒 Files selected for processing (16)
  • apps/engineering/content/docs/architecture/services/analytics.mdx (1 hunks)
  • go/apps/api/routes/v2_analytics_get_verifications/422_test.go (1 hunks)
  • go/apps/api/routes/v2_analytics_get_verifications/503_test.go (1 hunks)
  • go/cmd/create-clickhouse-user/main.go (1 hunks)
  • go/pkg/clickhouse/user.go (1 hunks)
  • go/pkg/db/bulk_clickhouse_workspace_settings_insert.sql_generated.go (1 hunks)
  • go/pkg/db/clickhouse_workspace_settings_find_by_workspace_id.sql_generated.go (1 hunks)
  • go/pkg/db/clickhouse_workspace_settings_insert.sql_generated.go (1 hunks)
  • go/pkg/db/clickhouse_workspace_settings_update_limits.sql_generated.go (1 hunks)
  • go/pkg/db/models_generated.go (1 hunks)
  • go/pkg/db/querier_generated.go (7 hunks)
  • go/pkg/db/queries/clickhouse_workspace_settings_insert.sql (1 hunks)
  • go/pkg/db/queries/clickhouse_workspace_settings_update_limits.sql (1 hunks)
  • go/pkg/db/schema.sql (1 hunks)
  • go/pkg/testutil/http.go (4 hunks)
  • internal/db/src/schema/clickhouse_workspace_settings.ts (1 hunks)
✅ Files skipped from review due to trivial changes (1)
  • apps/engineering/content/docs/architecture/services/analytics.mdx
🚧 Files skipped from review as they are similar to previous changes (7)
  • go/pkg/db/models_generated.go
  • go/apps/api/routes/v2_analytics_get_verifications/503_test.go
  • go/pkg/db/queries/clickhouse_workspace_settings_update_limits.sql
  • go/pkg/db/queries/clickhouse_workspace_settings_insert.sql
  • go/pkg/db/bulk_clickhouse_workspace_settings_insert.sql_generated.go
  • go/pkg/db/clickhouse_workspace_settings_find_by_workspace_id.sql_generated.go
  • go/pkg/db/clickhouse_workspace_settings_insert.sql_generated.go
🧰 Additional context used
🧠 Learnings (12)
📓 Common learnings
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 4190
File: go/internal/services/keys/verifier.go:51-53
Timestamp: 2025-10-30T15:10:52.743Z
Learning: PR #4190 for unkeyed/unkey is focused solely on database schema and query changes for identity-based credits. It adds IdentityCredits and KeyCredits fields to structs and queries, but does not implement the priority enforcement logic in the usagelimiter. The logic implementation is intentionally deferred to a later PR in the stack.
📚 Learning: 2025-04-22T14:40:51.459Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 3161
File: go/pkg/clickhouse/schema/databases/001_verifications/002_raw_key_verifications_v1.sql:31-33
Timestamp: 2025-04-22T14:40:51.459Z
Learning: The ClickHouse table schemas in the codebase mirror the production environment and cannot be modified directly in PRs without careful migration planning.

Applied to files:

  • internal/db/src/schema/clickhouse_workspace_settings.ts
  • go/pkg/db/schema.sql
📚 Learning: 2024-10-04T20:44:38.489Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 2180
File: apps/dashboard/lib/constants/workspace-navigations.tsx:56-118
Timestamp: 2024-10-04T20:44:38.489Z
Learning: When typing the `workspace` parameter in functions like `createWorkspaceNavigation`, prefer importing the `Workspace` type from the database module and picking the necessary keys (e.g., `features`) instead of redefining the interface.

Applied to files:

  • internal/db/src/schema/clickhouse_workspace_settings.ts
📚 Learning: 2025-09-12T08:01:20.792Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3944
File: go/pkg/db/acme_challenge_update_verified_with_expiry.sql_generated.go:31-39
Timestamp: 2025-09-12T08:01:20.792Z
Learning: Do not review or suggest changes to files with sql_generated.go suffix or other files marked as auto-generated (containing "Code generated by" comments), as these are generated by tools like sqlc and changes would be overwritten on regeneration.

Applied to files:

  • go/pkg/db/clickhouse_workspace_settings_update_limits.sql_generated.go
📚 Learning: 2025-07-16T15:38:53.491Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3606
File: go/pkg/db/replica.go:8-11
Timestamp: 2025-07-16T15:38:53.491Z
Learning: For debugging database replica usage in go/pkg/db/replica.go, it's acceptable to mark QueryRowContext operations as "success" even though SQL errors only surface during row.Scan() calls. The timing metrics are the primary concern for debugging replica performance patterns.

Applied to files:

  • go/pkg/testutil/http.go
📚 Learning: 2025-08-14T16:25:48.167Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3785
File: go/apps/api/routes/v2_keys_reroll_key/401_test.go:52-61
Timestamp: 2025-08-14T16:25:48.167Z
Learning: User Flo4604 requested creation of a GitHub issue to track converting all test files to use table-driven test patterns as a broader codebase improvement, following the suggestion made during review of go/apps/api/routes/v2_keys_reroll_key/401_test.go.

Applied to files:

  • go/apps/api/routes/v2_analytics_get_verifications/422_test.go
📚 Learning: 2025-04-08T09:34:24.576Z
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 2872
File: apps/dashboard/lib/trpc/routers/ratelimit/createNamespace.ts:36-39
Timestamp: 2025-04-08T09:34:24.576Z
Learning: In the Unkey dashboard, when making database queries involving workspaces, use `ctx.workspace.id` directly instead of fetching the workspace separately for better performance and security.

Applied to files:

  • go/pkg/db/querier_generated.go
📚 Learning: 2025-01-31T13:50:45.004Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 2825
File: apps/dashboard/app/(app)/logs/components/table/logs-table.tsx:100-118
Timestamp: 2025-01-31T13:50:45.004Z
Learning: Sensitive data in logs (passwords, tokens, etc.) is masked before being written to Clickhouse, ensuring secure storage of log data.

Applied to files:

  • go/cmd/create-clickhouse-user/main.go
📚 Learning: 2025-02-27T14:35:15.251Z
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 2918
File: apps/dashboard/lib/trpc/routers/api/overview-api-search.ts:0-0
Timestamp: 2025-02-27T14:35:15.251Z
Learning: When using Drizzle ORM with LIKE queries, use the sql`` tagged template syntax (e.g., sql`${table.name} LIKE ${`%${input.query}%`}`) instead of direct string interpolation to prevent SQL injection attacks.

Applied to files:

  • go/cmd/create-clickhouse-user/main.go
📚 Learning: 2025-09-15T17:40:51.536Z
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 3973
File: go/pkg/db/schema.sql:307-308
Timestamp: 2025-09-15T17:40:51.536Z
Learning: The environments table in the Unkey codebase is not being used in production yet, so schema changes to it don't require complex migration sequences to handle existing data or concurrent usage.

Applied to files:

  • go/cmd/create-clickhouse-user/main.go
  • go/pkg/db/schema.sql
📚 Learning: 2025-04-22T14:43:11.724Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 3161
File: go/pkg/clickhouse/schema/databases/002_ratelimits/006_ratelimits_per_day_v1.sql:1-13
Timestamp: 2025-04-22T14:43:11.724Z
Learning: In the unkey project, the SQL files in clickhouse/schema/databases represent the current production schema and shouldn't be modified directly in PRs. Schema changes require dedicated migration scripts.

Applied to files:

  • go/pkg/db/schema.sql
📚 Learning: 2025-10-21T09:45:47.560Z
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 4107
File: go/pkg/clickhouse/key_verifications_test.go:20-20
Timestamp: 2025-10-21T09:45:47.560Z
Learning: In ClickHouse tests (e.g., go/pkg/clickhouse/key_verifications_test.go), parallel execution with t.Parallel() is safe when tests use workspaceID-based isolation. Each test generates a unique workspaceID (via uid.New(uid.WorkspacePrefix)), and all database operations are scoped to that workspaceID, providing logical isolation even when tests share a single ClickHouse instance.

Applied to files:

  • go/pkg/db/schema.sql
🧬 Code graph analysis (5)
internal/db/src/schema/clickhouse_workspace_settings.ts (1)
internal/db/src/schema/workspaces.ts (1)
  • workspaces (17-105)
go/pkg/testutil/http.go (8)
go/internal/services/caches/caches.go (2)
  • Caches (18-42)
  • New (145-266)
go/pkg/testutil/containers/containers.go (1)
  • ClickHouse (130-132)
go/internal/services/analytics/connection_manager.go (3)
  • ConnectionManager (22-24)
  • NewConnectionManager (47-85)
  • ConnectionManagerConfig (37-44)
go/pkg/testutil/seed/seed.go (2)
  • Seeder (31-36)
  • New (39-46)
go/gen/proto/vault/v1/service.pb.go (3)
  • EncryptRequest (104-110)
  • EncryptRequest (123-123)
  • EncryptRequest (138-140)
go/pkg/clickhouse/user.go (2)
  • UserConfig (11-28)
  • DefaultAllowedTables (143-152)
go/pkg/db/clickhouse_workspace_settings_insert.sql_generated.go (1)
  • InsertClickhouseWorkspaceSettingsParams (42-54)
go/gen/proto/vault/v1/object.pb.go (3)
  • Encrypted (252-264)
  • Encrypted (277-277)
  • Encrypted (292-294)
go/apps/api/routes/v2_analytics_get_verifications/422_test.go (5)
go/pkg/testutil/http.go (3)
  • NewHarness (59-205)
  • WithMaxQueryMemoryBytes (270-274)
  • CallRoute (383-417)
go/pkg/testutil/seed/seed.go (1)
  • CreateApiRequest (84-92)
go/pkg/clickhouse/schema/requests.go (1)
  • KeyVerificationRequestV1 (66-92)
go/pkg/uid/uid.go (2)
  • RequestPrefix (19-19)
  • KeyPrefix (16-16)
go/apps/api/routes/v2_analytics_get_verifications/handler.go (3)
  • Handler (49-56)
  • Request (26-26)
  • Response (27-27)
go/pkg/db/querier_generated.go (9)
go/pkg/partition/db/database.go (1)
  • DBTX (10-10)
go/pkg/db/models_generated.go (2)
  • ClickhouseWorkspaceSetting (569-581)
  • Identity (643-652)
go/pkg/db/identity_find_many.sql_generated.go (1)
  • FindIdentitiesParams (21-25)
go/pkg/db/api_find_key_auth_by_ids.sql_generated.go (2)
  • FindKeyAuthsByIdsParams (23-26)
  • FindKeyAuthsByIdsRow (28-31)
go/pkg/db/api_find_key_auth_by_key_auth_ids.sql_generated.go (2)
  • FindKeyAuthsByKeyAuthIdsParams (23-26)
  • FindKeyAuthsByKeyAuthIdsRow (28-31)
go/pkg/db/key_auth_get_by_id.sql_generated.go (1)
  • GetKeyAuthByIDRow (26-33)
go/pkg/db/clickhouse_workspace_settings_insert.sql_generated.go (1)
  • InsertClickhouseWorkspaceSettingsParams (42-54)
go/pkg/db/key_auth_insert.sql_generated.go (1)
  • InsertKeyAuthParams (31-37)
go/pkg/db/clickhouse_workspace_settings_update_limits.sql_generated.go (1)
  • UpdateClickhouseWorkspaceSettingsLimitsParams (26-35)
go/cmd/create-clickhouse-user/main.go (8)
go/pkg/cli/flag.go (6)
  • Flag (20-27)
  • String (419-451)
  • Required (298-317)
  • StringSlice (612-644)
  • Int (532-569)
  • Int64 (647-684)
go/pkg/cli/command.go (1)
  • Action (21-21)
go/pkg/clickhouse/client.go (1)
  • Config (37-44)
go/apps/api/config.go (2)
  • Config (23-105)
  • S3Config (16-21)
go/pkg/vault/storage/interface.go (1)
  • Storage (15-30)
go/pkg/db/clickhouse_workspace_settings_insert.sql_generated.go (1)
  • InsertClickhouseWorkspaceSettingsParams (42-54)
go/pkg/db/clickhouse_workspace_settings_update_limits.sql_generated.go (1)
  • UpdateClickhouseWorkspaceSettingsLimitsParams (26-35)
go/pkg/clickhouse/user.go (1)
  • UserConfig (11-28)
🔇 Additional comments (1)
go/pkg/db/querier_generated.go (1)

151-155: LGTM! Generated code aligns with PR objectives.

The newly added interface methods support the analytics feature implementation:

  • ClickHouse workspace settings management (FindClickhouseWorkspaceSettingsByWorkspaceID, InsertClickhouseWorkspaceSettings, UpdateClickhouseWorkspaceSettingsLimits)
  • Identity batch operations (FindIdentities)
  • Key auth lookups (FindKeyAuthsByIds, FindKeyAuthsByKeyAuthIds, GetKeyAuthByID, InsertKeyAuth)

All methods follow consistent naming conventions and parameter patterns. Since this is sqlc-generated code, any necessary adjustments should be made to the source SQL queries rather than this file.

Also applies to: 273-280, 344-363, 872-884, 1003-1031, 1199-1216, 1818-1830

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

♻️ Duplicate comments (5)
go/Makefile (1)

37-38: Pin the buf CLI version in go run (duplicate of past concern).

Both lines now invoke go run github.com/bufbuild/buf/cmd/buf without an @version specifier, which causes non-reproducible fetches and pollutes go.sum. This reintroduces the issue flagged in the previous review.

Please pin a specific buf release (e.g., @vX.Y.Z) or use a BUF_VERSION Make variable for both invocations so make generate remains deterministic.

Apply this diff to pin the buf version:

-	go run github.com/bufbuild/buf/cmd/buf generate --template ./buf.gen.connect.yaml --clean --path "./proto/cache" --path "./proto/ctrl" --path "./proto/krane" --path "./proto/partition" --path "./proto/vault"
-	go run github.com/bufbuild/buf/cmd/buf generate --template ./buf.gen.restate.yaml --path "./proto/hydra"
+	go run github.com/bufbuild/buf/cmd/buf@v1.32.2 generate --template ./buf.gen.connect.yaml --clean --path "./proto/cache" --path "./proto/ctrl" --path "./proto/krane" --path "./proto/partition" --path "./proto/vault"
+	go run github.com/bufbuild/buf/cmd/buf@v1.32.2 generate --template ./buf.gen.restate.yaml --path "./proto/hydra"

(Replace v1.32.2 with the appropriate buf version your project uses.)

go/pkg/codes/generate.go (2)

253-264: URN parsing drops every error code.

The URN parsing assumes parts[0] == "err" but real URNs start with urn:err:..., causing all error codes to be skipped. No MDX files will be generated.

Apply this fix to handle both formats:

 		// Parse URN to get file path
-		// Example: err:user:bad_request:client_closed_request -> user/bad_request/client_closed_request.mdx
+		// Example: urn:err:user:bad_request:client_closed_request -> user/bad_request/client_closed_request.mdx
 		parts := strings.Split(errCode.URN, ":")
-		if len(parts) < 4 || parts[0] != "err" {
+		
+		// Handle both "err:..." and "urn:err:..." formats
+		startIdx := 0
+		if len(parts) > 0 && parts[0] == "urn" {
+			startIdx = 1
+		}
+		if len(parts) < startIdx+4 || parts[startIdx] != "err" {
 			continue
 		}
 
 		// Build file path from URN parts (skip "err:" prefix)
-		pathParts := parts[1 : len(parts)-1]
+		pathParts := parts[startIdx+1 : len(parts)-1]
 		fileName := parts[len(parts)-1] + ".mdx"
 		filePath := filepath.Join(append([]string{baseDocsPath}, append(pathParts, fileName)...)...)

317-326: URN parsing drops every error code (same bug).

Same parsing bug as in generateMissingMDXFiles - assumes parts[0] == "err" when URNs start with urn:err:.... The validPaths map will be incomplete, preventing obsolete file detection.

Apply the same fix as above:

 		parts := strings.Split(errCode.URN, ":")
-		if len(parts) < 4 || parts[0] != "err" {
+		
+		// Handle both "err:..." and "urn:err:..." formats
+		startIdx := 0
+		if len(parts) > 0 && parts[0] == "urn" {
+			startIdx = 1
+		}
+		if len(parts) < startIdx+4 || parts[startIdx] != "err" {
 			continue
 		}
 
 		// Build file path from URN
-		pathParts := parts[1 : len(parts)-1]
+		pathParts := parts[startIdx+1 : len(parts)-1]
 		fileName := parts[len(parts)-1] + ".mdx"
 		filePath := filepath.Join(append([]string{baseDocsPath}, append(pathParts, fileName)...)...)
deployment/docker-compose.yaml (1)

114-125: Critical: Fix Kafka advertised listeners for Docker network.

The Kafka service upgrade is correct, but Line 121 configures KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092, which advertises the loopback address. This prevents other containers (like apiv2 at line 94) from connecting to Kafka within the Docker network. This issue was previously flagged but remains unresolved.

Apply this diff:

-      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
go/apps/api/openapi/openapi-generated.yaml (1)

2029-2067: Fix unsatisfiable allOf: BaseError blocks BadRequestErrorDetails.errors

BadRequestErrorDetails extends BaseError via allOf but BaseError has additionalProperties: false, so the added required property errors is currently disallowed, making the schema impossible to satisfy. Quick fix: declare errors (optional) on BaseError, and keep it required only for BadRequestErrorDetails.

Apply this minimal diff:

 components:
   schemas:
     BaseError:
       properties:
         detail:
           description: A human-readable explanation specific to this occurrence of the problem. This provides detailed information about what went wrong and potential remediation steps. The message is intended to be helpful for developers troubleshooting the issue.
           example: Property foo is required but is missing.
           type: string
         status:
           description: HTTP status code that corresponds to this error. This will match the status code in the HTTP response. Common codes include `400` (Bad Request), `401` (Unauthorized), `403` (Forbidden), `404` (Not Found), `409` (Conflict), and `500` (Internal Server Error).
           example: 404
           format: int
           type: integer
         title:
           description: A short, human-readable summary of the problem type. This remains constant from occurrence to occurrence of the same problem and should be used for programmatic handling.
           example: Not Found
           type: string
         type:
           description: A URI reference that identifies the problem type. This provides a stable identifier for the error that can be used for documentation lookups and programmatic error handling. When followed, this URI should provide human-readable documentation for the problem type.
           example: https://unkey.com/docs/errors/unkey/resource/not_found
           type: string
+        errors:
+          description: List of individual validation errors that occurred in the request (present on 400 responses).
+          type: array
+          items:
+            $ref: "#/components/schemas/ValidationError"
       required:
         - title
         - detail
         - status
         - type
       type: object
       additionalProperties: false
🧹 Nitpick comments (7)
go/pkg/codes/generate.go (2)

238-303: Extract duplicated URN parsing logic.

The URN parsing logic (splitting, validating, extracting path parts) is duplicated across all three functions. Consider extracting it into a helper function to improve maintainability.

Example helper:

// parseErrorURN parses an error URN and returns the path parts and filename,
// or an error if the URN format is invalid.
func parseErrorURN(urn string) (pathParts []string, fileName string, err error) {
	parts := strings.Split(urn, ":")
	
	// Handle both "err:..." and "urn:err:..." formats
	startIdx := 0
	if len(parts) > 0 && parts[0] == "urn" {
		startIdx = 1
	}
	if len(parts) < startIdx+4 || parts[startIdx] != "err" {
		return nil, "", fmt.Errorf("invalid URN format: %s", urn)
	}
	
	pathParts = parts[startIdx+1 : len(parts)-1]
	fileName = parts[len(parts)-1]
	return pathParts, fileName, nil
}

Also applies to: 305-369, 371-576


401-418: Type assertions without safety checks.

Lines 401, 412, and 418 perform type assertions without checking the ok value, which could panic if the docs.json structure is unexpected. While acceptable for a build-time tool (panics help catch structure changes early), adding safety checks would make debugging easier.

Example for line 401:

-		dd := dropdown.(map[string]interface{})
+		dd, ok := dropdown.(map[string]interface{})
+		if !ok {
+			continue
+		}
go/pkg/zen/middleware_errors.go (1)

304-316: Verify response type naming.

The analytics connection failure correctly returns 503 status, but it uses InternalServerErrorResponse as the response type. While this works functionally (since the status field is set to 503), the type name suggests it's for 500 errors. Consider whether a dedicated ServiceUnavailableErrorResponse type should be added to the OpenAPI schema for semantic clarity, or verify this reuse is intentional.

go/apps/api/openapi/openapi-generated.yaml (3)

97-108: Constrain query input and update example to current table alias

Add sane length bounds to reduce abuse and align with backend limits; update example to the public alias the parser supports (no version suffix).

 V2AnalyticsGetVerificationsRequestBody:
   type: object
   required:
     - query
   properties:
     query:
       type: string
       description: |
         SQL query to execute against your analytics data.
         Only SELECT queries are allowed.
-      example: "SELECT COUNT(*) as total FROM key_verifications_v1 WHERE outcome = 'VALID' AND time >= now() - INTERVAL 7 DAY"
+      minLength: 7
+      maxLength: 20000
+      example: "SELECT COUNT(*) AS total FROM key_verifications WHERE outcome = 'VALID' AND time >= now() - INTERVAL 7 DAY"

Please confirm the canonical user-facing table/alias (e.g., key_verifications) so the example matches the parser’s rewrite rules. Based on learnings.


3331-3399: Document required permissions and confirm endpoint visibility

This endpoint omits a “Required Permissions” section while other endpoints include it. Add explicit RBAC docs for api.*.read_analytics and api.<api_id>.read_analytics. Also, it's marked x-hidden and x-speakeasy-ignore — verify this is intentional for the published bundle.

 /v2/analytics.getVerifications:
   post:
     description: |
       Execute custom SQL queries against your key verification analytics.
       For complete documentation including available tables, columns, data types, query examples, see the schema reference in the API documentation.
+      
+      Required Permissions
+      
+      Your root key must have one of:
+      - `api.*.read_analytics`
+      - `api.<api_id>.read_analytics`
@@
-    x-hidden: true
-    x-speakeasy-ignore: true
+    # TODO: Set to false/omit when ready to publish in docs/SDKs
+    x-hidden: true
+    x-speakeasy-ignore: true

If the endpoint should be public at launch, drop the hidden flags in the generated bundle/template.


2094-2108: Optional: tighten result row schema for better DX

You can keep items as open objects but hint common fields to improve SDK typings (non-breaking).

 V2AnalyticsGetVerificationsResponseData:
   type: array
   description: Array of verification rows returned by the query. Fields vary based on the SQL SELECT clause.
   items:
-    type: object
-    additionalProperties: true
-    description: Dynamic row with fields determined by the query. Can include any combination of fields like time, outcome, count, key_id, etc.
+    type: object
+    additionalProperties: true
+    properties:
+      time:
+        type: integer
+        format: int64
+        description: Unix ms timestamp (when selected)
+      outcome:
+        type: string
+        description: Verification outcome (when selected)
+      count:
+        type: integer
+        format: int64
+        description: Aggregated row count (when selected)
+      key_id:
+        type: string
+        description: Key identifier (when selected)
+    description: Dynamic row with fields determined by the query; common fields are hinted for tooling only.
go/main.go (1)

8-11: Optional: gate dev-only commands behind build tags

If you want to exclude dev commands from release builds, split files and use tags.

Example:

- dev "github.com/unkeyed/unkey/go/cmd/dev"
+//go:build dev
+// +build dev
+dev "github.com/unkeyed/unkey/go/cmd/dev"

And wrap the Commands entry in a dev-tagged file:

//go:build dev
package main

func init() {
    app.Commands = append(app.Commands, dev.Cmd)
}
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 58af255 and f0cb0c6.

⛔ Files ignored due to path filters (1)
  • go/go.sum is excluded by !**/*.sum
📒 Files selected for processing (25)
  • apps/docs/analytics/getting-started.mdx (1 hunks)
  • apps/docs/docs.json (8 hunks)
  • apps/docs/errors/user/bad_request/invalid_analytics_query.mdx (1 hunks)
  • apps/docs/errors/user/unprocessable_entity/query_rows_limit_exceeded.mdx (1 hunks)
  • deployment/docker-compose.yaml (3 hunks)
  • go/Makefile (2 hunks)
  • go/apps/api/config.go (1 hunks)
  • go/apps/api/openapi/gen.go (4 hunks)
  • go/apps/api/openapi/openapi-generated.yaml (6 hunks)
  • go/apps/api/openapi/spec/paths/v2/analytics/getVerifications/index.yaml (1 hunks)
  • go/apps/api/routes/v2_analytics_get_verifications/handler.go (1 hunks)
  • go/apps/api/run.go (2 hunks)
  • go/cmd/api/main.go (2 hunks)
  • go/go.mod (6 hunks)
  • go/internal/services/caches/caches.go (2 hunks)
  • go/main.go (2 hunks)
  • go/pkg/cache/cache.go (4 hunks)
  • go/pkg/cache/clustering/cluster_cache.go (3 hunks)
  • go/pkg/cache/interface.go (1 hunks)
  • go/pkg/cache/middleware/tracing.go (4 hunks)
  • go/pkg/clickhouse/key_verifications_test.go (3 hunks)
  • go/pkg/clickhouse/query-parser/cte.go (1 hunks)
  • go/pkg/codes/generate.go (7 hunks)
  • go/pkg/testutil/http.go (4 hunks)
  • go/pkg/zen/middleware_errors.go (5 hunks)
🚧 Files skipped from review as they are similar to previous changes (6)
  • go/cmd/api/main.go
  • apps/docs/errors/user/unprocessable_entity/query_rows_limit_exceeded.mdx
  • apps/docs/analytics/getting-started.mdx
  • go/apps/api/config.go
  • apps/docs/errors/user/bad_request/invalid_analytics_query.mdx
  • go/apps/api/openapi/spec/paths/v2/analytics/getVerifications/index.yaml
🧰 Additional context used
🧠 Learnings (25)
📓 Common learnings
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 4190
File: go/internal/services/keys/verifier.go:51-53
Timestamp: 2025-10-30T15:10:52.743Z
Learning: PR #4190 for unkeyed/unkey is focused solely on database schema and query changes for identity-based credits. It adds IdentityCredits and KeyCredits fields to structs and queries, but does not implement the priority enforcement logic in the usagelimiter. The logic implementation is intentionally deferred to a later PR in the stack.
📚 Learning: 2025-09-15T19:53:28.487Z
Learnt from: mcstepp
Repo: unkeyed/unkey PR: 3952
File: go/proto/ctrl/v1/routing.proto:0-0
Timestamp: 2025-09-15T19:53:28.487Z
Learning: In the Unkey codebase, authentication/authorization errors intentionally return NOT_FOUND error codes instead of distinct auth error codes (like FORBIDDEN or UNAUTHORIZED) for security reasons. This prevents attackers from distinguishing between "resource doesn't exist" and "you don't have permission to access this resource", avoiding information disclosure about workspace existence and access boundaries.

Applied to files:

  • go/pkg/zen/middleware_errors.go
📚 Learning: 2025-07-15T14:25:05.608Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 3560
File: go/apps/api/routes/v2_keys_create_key/handler.go:353-466
Timestamp: 2025-07-15T14:25:05.608Z
Learning: In the Unkey codebase, input validation for API endpoints is handled at the OpenAPI schema layer, which validates request fields like permission slugs (pattern: "^[a-zA-Z0-9_]+$", length: 1-100 characters) before requests reach the handler code. This validation occurs during the zen.BindBody call in handlers.

Applied to files:

  • go/pkg/zen/middleware_errors.go
📚 Learning: 2025-03-19T09:25:59.751Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 2955
File: go/apps/api/routes/v2_identities_create_identity/handler.go:162-202
Timestamp: 2025-03-19T09:25:59.751Z
Learning: In the Unkey codebase, input validation for API endpoints is primarily handled through OpenAPI schema validation, which occurs before requests reach the handler code. For example, in the identities.createIdentity endpoint, minimum values for ratelimit duration and limit are defined in the OpenAPI schema rather than duplicating these checks in the handler.

Applied to files:

  • go/pkg/zen/middleware_errors.go
📚 Learning: 2025-08-08T15:09:01.312Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3753
File: go/apps/api/openapi/config.yaml:9-10
Timestamp: 2025-08-08T15:09:01.312Z
Learning: Repo unkeyed/unkey: For go/apps/api/openapi, oapi-codegen is used and does not support OpenAPI 3.1 union types like [T, "null"]; an overlay step is required to downconvert to 3.0-style nullable before code generation.

Applied to files:

  • go/apps/api/openapi/gen.go
  • go/go.mod
📚 Learning: 2025-08-08T15:09:01.312Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3753
File: go/apps/api/openapi/config.yaml:9-10
Timestamp: 2025-08-08T15:09:01.312Z
Learning: Repo unkeyed/unkey: In go/apps/api/openapi, oapi-codegen doesn’t support OpenAPI 3.1 union nullability; overlay.yaml must be applied before codegen. The overlay key in oapi-codegen config isn’t supported—use a pre-step (programmatic or CLI) to merge overlay into the bundled spec, then run oapi-codegen.

Applied to files:

  • go/apps/api/openapi/gen.go
📚 Learning: 2025-08-21T15:54:45.198Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 3825
File: go/internal/services/usagelimiter/limit.go:38-0
Timestamp: 2025-08-21T15:54:45.198Z
Learning: In go/internal/services/usagelimiter/limit.go, the UpdateKeyCreditsDecrement operation cannot be safely wrapped with db.WithRetry due to the lack of idempotency mechanisms in the current tech stack. Retrying this non-idempotent write operation risks double-charging users if the first attempt commits but the client sees a transient error.

Applied to files:

  • go/apps/api/openapi/gen.go
📚 Learning: 2025-07-16T15:38:53.491Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3606
File: go/pkg/db/replica.go:8-11
Timestamp: 2025-07-16T15:38:53.491Z
Learning: For debugging database replica usage in go/pkg/db/replica.go, it's acceptable to mark QueryRowContext operations as "success" even though SQL errors only surface during row.Scan() calls. The timing metrics are the primary concern for debugging replica performance patterns.

Applied to files:

  • go/pkg/testutil/http.go
📚 Learning: 2025-07-22T09:02:12.495Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 3638
File: deployment/docker-compose.yaml:81-94
Timestamp: 2025-07-22T09:02:12.495Z
Learning: The docker-compose.yaml file in deployment/ is specifically for development environments, not production. Kafka and other service configurations in this file should be optimized for development convenience rather than production security/hardening.

Applied to files:

  • deployment/docker-compose.yaml
  • go/Makefile
📚 Learning: 2025-09-01T02:33:43.791Z
Learnt from: imeyer
Repo: unkeyed/unkey PR: 3899
File: go/buf.gen.yaml:0-0
Timestamp: 2025-09-01T02:33:43.791Z
Learning: In the unkeyed/unkey repository, buf commands are executed from the `go/` directory where the `buf.yaml` file is located. This means the `out: gen` configuration in `go/buf.gen.yaml` generates files to `go/gen/` relative to the repository root, which aligns with the expected directory structure.

Applied to files:

  • go/Makefile
  • go/go.mod
📚 Learning: 2025-09-01T01:57:42.227Z
Learnt from: imeyer
Repo: unkeyed/unkey PR: 3899
File: go/proto/metald/v1/metald.proto:5-9
Timestamp: 2025-09-01T01:57:42.227Z
Learning: In the unkeyed/unkey repository, buf is configured to properly resolve metald proto imports like "metald/v1/vm.proto" without needing the full "go/proto/" prefix. The buf lint command `buf lint --path proto/metald` passes successfully with these relative import paths.

Applied to files:

  • go/Makefile
  • go/go.mod
📚 Learning: 2025-10-15T10:12:40.810Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 4098
File: go/proto/ctrl/v1/deployment.proto:33-36
Timestamp: 2025-10-15T10:12:40.810Z
Learning: In the Unkey codebase proto files (ctrl/v1/build.proto, ctrl/v1/deployment.proto, hydra/v1/deployment.proto), use `dockerfile_path` (not `docker_file_path`) for consistency in generated Go field names.

Applied to files:

  • go/Makefile
📚 Learning: 2025-09-12T08:01:20.792Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3944
File: go/pkg/db/acme_challenge_update_verified_with_expiry.sql_generated.go:31-39
Timestamp: 2025-09-12T08:01:20.792Z
Learning: Do not review or suggest changes to files with sql_generated.go suffix or other files marked as auto-generated (containing "Code generated by" comments), as these are generated by tools like sqlc and changes would be overwritten on regeneration.

Applied to files:

  • go/go.mod
📚 Learning: 2025-07-15T14:59:30.212Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 3560
File: go/deploy/metald/internal/database/repository.go:0-0
Timestamp: 2025-07-15T14:59:30.212Z
Learning: go/deploy/metald cannot currently import helpers from go/pkg/db because it is not yet part of the main Go module; avoid suggesting such imports until the modules are unified.

Applied to files:

  • go/go.mod
📚 Learning: 2025-04-08T09:34:24.576Z
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 2872
File: apps/dashboard/lib/trpc/routers/ratelimit/createNamespace.ts:36-39
Timestamp: 2025-04-08T09:34:24.576Z
Learning: In the Unkey dashboard, when making database queries involving workspaces, use `ctx.workspace.id` directly instead of fetching the workspace separately for better performance and security.

Applied to files:

  • go/apps/api/routes/v2_analytics_get_verifications/handler.go
📚 Learning: 2025-04-08T09:34:24.576Z
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 2872
File: apps/dashboard/lib/trpc/routers/ratelimit/createNamespace.ts:36-39
Timestamp: 2025-04-08T09:34:24.576Z
Learning: When querying or updating namespaces in the Unkey dashboard, always scope the operations to the current workspace using `eq(table.workspaceId, ctx.workspace.id)` to prevent cross-workspace access.

Applied to files:

  • go/apps/api/routes/v2_analytics_get_verifications/handler.go
📚 Learning: 2025-09-01T08:29:10.199Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3895
File: go/pkg/db/key_list_live_by_auth_id.sql_generated.go:98-105
Timestamp: 2025-09-01T08:29:10.199Z
Learning: In Unkey's ListLiveKeysByKeyAuthID query, adding ka.workspace_id = k.workspace_id constraint negatively impacts index performance. Workspace validation is handled upstream at the API level before the query is called, making the additional constraint unnecessary.

Applied to files:

  • go/apps/api/routes/v2_analytics_get_verifications/handler.go
📚 Learning: 2025-09-22T18:44:56.279Z
Learnt from: perkinsjr
Repo: unkeyed/unkey PR: 4009
File: apps/dashboard/app/(app)/[workspace]/apis/[apiId]/_overview/components/table/components/log-details/index.tsx:4-5
Timestamp: 2025-09-22T18:44:56.279Z
Learning: In the Unkey dashboard, the workspace hook (useWorkspace) provides security validation by checking database access and user authorization to the workspace, with 10-minute caching for performance. Using URL params (useParams) for workspace slug would bypass this security validation and allow unauthorized access attempts. Always use the workspace hook for workspace-scoped navigation and handle loading states properly rather than switching to URL parameters.

Applied to files:

  • go/apps/api/routes/v2_analytics_get_verifications/handler.go
📚 Learning: 2025-09-15T20:45:05.696Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3952
File: go/apps/ctrl/services/routing/service.go:69-91
Timestamp: 2025-09-15T20:45:05.696Z
Learning: In Unkey's routing service, gateway lookups should be workspace-scoped using FindGatewayByHostnameAndWorkspace instead of hostname-only queries to prevent cross-tenant access issues.

Applied to files:

  • go/apps/api/routes/v2_analytics_get_verifications/handler.go
📚 Learning: 2025-10-30T15:10:52.743Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 4190
File: go/internal/services/keys/verifier.go:51-53
Timestamp: 2025-10-30T15:10:52.743Z
Learning: PR #4190 for unkeyed/unkey is focused solely on database schema and query changes for identity-based credits. It adds IdentityCredits and KeyCredits fields to structs and queries, but does not implement the priority enforcement logic in the usagelimiter. The logic implementation is intentionally deferred to a later PR in the stack.

Applied to files:

  • go/pkg/clickhouse/key_verifications_test.go
📚 Learning: 2025-10-21T09:45:47.560Z
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 4107
File: go/pkg/clickhouse/key_verifications_test.go:20-20
Timestamp: 2025-10-21T09:45:47.560Z
Learning: In ClickHouse tests (e.g., go/pkg/clickhouse/key_verifications_test.go), parallel execution with t.Parallel() is safe when tests use workspaceID-based isolation. Each test generates a unique workspaceID (via uid.New(uid.WorkspacePrefix)), and all database operations are scoped to that workspaceID, providing logical isolation even when tests share a single ClickHouse instance.

Applied to files:

  • go/pkg/clickhouse/key_verifications_test.go
📚 Learning: 2025-08-27T13:48:54.016Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3841
File: go/pkg/db/queries/key_find_for_verification.sql:15-15
Timestamp: 2025-08-27T13:48:54.016Z
Learning: The pending_migration_id field in FindKeyForVerification is only used internally by get_migrated.go for migration logic and doesn't need to be exposed in KeyVerifier struct or API response DTOs since it's an internal implementation detail.

Applied to files:

  • go/pkg/clickhouse/key_verifications_test.go
📚 Learning: 2025-07-03T05:58:10.699Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3421
File: go/apps/api/openapi/openapi.yaml:196-200
Timestamp: 2025-07-03T05:58:10.699Z
Learning: In the Unkey codebase, OpenAPI 3.1 is used, which allows sibling keys (such as `description`) alongside `$ref` in schema objects. Do not flag this as an error in future reviews.

Applied to files:

  • go/apps/api/openapi/openapi-generated.yaml
📚 Learning: 2025-09-12T18:11:33.481Z
Learnt from: mcstepp
Repo: unkeyed/unkey PR: 3952
File: apps/dashboard/lib/trpc/routers/deploy/rollback.ts:23-24
Timestamp: 2025-09-12T18:11:33.481Z
Learning: In apps/dashboard/lib/trpc/routers/deploy/rollback.ts, the CTRL_URL environment variable should fail fast with a clear error message if missing in non-development environments, rather than defaulting to localhost which can mask production configuration issues.

Applied to files:

  • go/apps/api/openapi/openapi-generated.yaml
📚 Learning: 2025-02-24T13:35:00.816Z
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 2910
File: apps/engineering/content/design/components/form/form-input.variants.tsx:38-38
Timestamp: 2025-02-24T13:35:00.816Z
Learning: In the Unkey codebase, example values in documentation and demo components intentionally use themed fake data (e.g., Middle Earth references) rather than generic placeholder text.

Applied to files:

  • apps/docs/docs.json
🧬 Code graph analysis (13)
go/internal/services/caches/caches.go (7)
go/pkg/cache/interface.go (1)
  • Cache (7-49)
go/pkg/db/models_generated.go (1)
  • ClickhouseWorkspaceSetting (569-581)
go/pkg/cache/scoped_key.go (3)
  • ScopedKey (49-62)
  • ScopedKeyToString (81-81)
  • ScopedKeyFromString (82-82)
go/pkg/db/api_find_key_auth_by_key_auth_ids.sql_generated.go (1)
  • FindKeyAuthsByKeyAuthIdsRow (28-31)
go/pkg/db/api_find_key_auth_by_ids.sql_generated.go (1)
  • FindKeyAuthsByIdsRow (28-31)
go/pkg/cache/cache.go (1)
  • Config (34-51)
go/pkg/cache/middleware/tracing.go (1)
  • WithTracing (16-18)
go/apps/api/run.go (4)
go/internal/services/analytics/connection_manager.go (3)
  • NewNoopConnectionManager (192-194)
  • NewConnectionManager (47-85)
  • ConnectionManagerConfig (37-44)
go/apps/api/routes/register.go (1)
  • Register (62-581)
go/apps/api/routes/services.go (1)
  • Services (17-30)
go/internal/services/caches/caches.go (1)
  • Caches (21-49)
go/pkg/zen/middleware_errors.go (2)
go/pkg/codes/constants_gen.go (12)
  • UnkeyDataErrorsKeySpaceNotFound (93-93)
  • UserErrorsBadRequestInvalidAnalyticsQuery (24-24)
  • UserErrorsBadRequestInvalidAnalyticsTable (26-26)
  • UserErrorsBadRequestInvalidAnalyticsFunction (28-28)
  • UserErrorsBadRequestInvalidAnalyticsQueryType (30-30)
  • UserErrorsUnprocessableEntityQueryExecutionTimeout (35-35)
  • UserErrorsUnprocessableEntityQueryMemoryLimitExceeded (37-37)
  • UserErrorsUnprocessableEntityQueryRowsLimitExceeded (39-39)
  • UserErrorsTooManyRequestsQueryQuotaExceeded (44-44)
  • UnkeyDataErrorsAnalyticsNotConfigured (141-141)
  • UnkeyAppErrorsPreconditionPreconditionFailed (171-171)
  • UnkeyDataErrorsAnalyticsConnectionFailed (143-143)
go/apps/api/openapi/gen.go (8)
  • BadRequestErrorResponse (65-71)
  • Meta (279-282)
  • BadRequestErrorDetails (47-62)
  • ValidationError (2015-2029)
  • UnprocessableEntityErrorResponse (500-506)
  • BaseError (74-86)
  • TooManyRequestsErrorResponse (471-477)
  • InternalServerErrorResponse (181-187)
go/apps/api/openapi/gen.go (1)
go/pkg/codes/unkey_data.go (1)
  • Data (112-165)
go/pkg/testutil/http.go (5)
apps/agent/services/vault/service.go (2)
  • Service (20-30)
  • New (39-74)
go/internal/services/analytics/connection_manager.go (3)
  • ConnectionManager (22-24)
  • NewConnectionManager (47-85)
  • ConnectionManagerConfig (37-44)
go/gen/proto/vault/v1/service.pb.go (3)
  • EncryptRequest (104-110)
  • EncryptRequest (123-123)
  • EncryptRequest (138-140)
go/pkg/clickhouse/user.go (2)
  • UserConfig (11-28)
  • DefaultAllowedTables (143-152)
go/pkg/db/clickhouse_workspace_settings_insert.sql_generated.go (1)
  • InsertClickhouseWorkspaceSettingsParams (42-54)
go/pkg/cache/clustering/cluster_cache.go (1)
go/pkg/cache/interface.go (2)
  • CacheHit (55-55)
  • Op (66-66)
go/pkg/clickhouse/query-parser/cte.go (1)
go/pkg/clickhouse/query-parser/types.go (1)
  • Parser (23-27)
go/pkg/codes/generate.go (2)
go/pkg/codes/user_request.go (1)
  • User (53-72)
go/pkg/codes/unkey_data.go (1)
  • Data (112-165)
go/pkg/cache/cache.go (2)
go/pkg/cache/interface.go (8)
  • CacheHit (55-55)
  • Miss (63-63)
  • Hit (61-61)
  • Null (59-59)
  • Op (66-66)
  • WriteValue (72-72)
  • WriteNull (75-75)
  • Noop (70-70)
go/pkg/prometheus/metrics/cache.go (1)
  • CacheRevalidations (103-112)
go/main.go (3)
go/cmd/create-clickhouse-user/main.go (1)
  • Cmd (20-65)
go/cmd/dev/main.go (1)
  • Cmd (8-15)
go/cmd/dev/seed/run.go (1)
  • Cmd (7-13)
go/apps/api/routes/v2_analytics_get_verifications/handler.go (16)
go/apps/api/openapi/gen.go (4)
  • V2AnalyticsGetVerificationsRequestBody (535-539)
  • V2AnalyticsGetVerificationsResponseBody (542-548)
  • V2AnalyticsGetVerificationsResponseData (551-551)
  • Meta (279-282)
go/pkg/clickhouse/interface.go (1)
  • ClickHouse (59-68)
go/internal/services/analytics/connection_manager.go (1)
  • ConnectionManager (22-24)
go/internal/services/caches/caches.go (2)
  • Caches (21-49)
  • Config (62-74)
go/pkg/zen/request_util.go (1)
  • BindBody (10-23)
go/pkg/clickhouse/query-parser/types.go (2)
  • Config (14-20)
  • SecurityFilter (8-11)
go/pkg/rbac/permissions.go (5)
  • Limit (77-77)
  • Tuple (178-187)
  • ResourceType (11-11)
  • Api (21-21)
  • ReadAnalytics (70-70)
go/internal/services/keys/options.go (1)
  • WithPermissions (47-52)
go/pkg/clickhouse/errors.go (1)
  • WrapClickHouseError (219-253)
go/pkg/cache/scoped_key.go (1)
  • ScopedKey (49-62)
go/pkg/db/api_find_key_auth_by_ids.sql_generated.go (2)
  • FindKeyAuthsByIdsRow (28-31)
  • FindKeyAuthsByIdsParams (23-26)
go/pkg/array/map.go (1)
  • Map (23-29)
go/pkg/cache/interface.go (3)
  • Key (51-53)
  • Null (59-59)
  • CacheHit (55-55)
go/pkg/array/reduce.go (1)
  • Reduce (28-34)
go/internal/services/caches/op.go (1)
  • DefaultFindFirstOp (9-22)
go/pkg/db/api_find_key_auth_by_key_auth_ids.sql_generated.go (2)
  • FindKeyAuthsByKeyAuthIdsRow (28-31)
  • FindKeyAuthsByKeyAuthIdsParams (23-26)
go/pkg/cache/middleware/tracing.go (1)
go/pkg/cache/interface.go (3)
  • CacheHit (55-55)
  • Miss (63-63)
  • Op (66-66)
go/pkg/clickhouse/key_verifications_test.go (4)
go/pkg/array/random.go (1)
  • Random (23-29)
go/pkg/clickhouse/schema/types.go (1)
  • KeyVerificationV2 (6-19)
go/pkg/uid/uid.go (1)
  • RequestPrefix (19-19)
go/pkg/array/reduce.go (1)
  • Reduce (28-34)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)
  • GitHub Check: Test Dashboard / Test Dashboard
  • GitHub Check: Test Go API Local / Test
  • GitHub Check: Test API / API Test Local
  • GitHub Check: Build / Build
  • GitHub Check: Test Packages / Test
🔇 Additional comments (26)
go/Makefile (1)

27-27: Good: --wait flag ensures services are ready before proceeding.

Adding --wait to the docker compose command is a helpful improvement for the development workflow, ensuring all services are fully available before downstream steps execute.

apps/docs/docs.json (3)

171-181: Analytics integration looks solid.

The new analytics pages (overview, getting-started, query-examples, schema-reference, query-restrictions) are properly integrated into the Features navigation, and the analytics error pages are correctly categorized under Data errors and the new User Errors groupings. Aligns well with the PR's new /v2/analytics.getVerifications endpoint.


257-281: User Errors reorganization is clean.

Breaking user-facing errors into Bad Request, Too Many Requests, and Unprocessable Entity groups is sensible and the analytics-specific errors (invalid_analytics_query, invalid_analytics_table, invalid_analytics_function) map correctly to 400-level codes. Query quota/timeout/memory/rows limit errors properly categorized.


172-172: Clarify the explicit "hidden": false on Analytics.

This is the only navigation group with an explicit "hidden": false field (line 172). If this is intentional to keep it visible by default, that's fine, but if it's residual from an earlier iteration, it can be removed since false is the Mintlify default.

go/go.mod (3)

42-42: Clarify the sqlc version downgrade.

The sqlc dependency was downgraded from v1.30.0 to v1.29.0. This is unusual and the PR description does not explain the reason. Confirm that:

  1. This downgrade is intentional and not accidental.
  2. There is no compatibility issue with v1.30.0.
  3. Any code generation artifacts (e.g., *_generated.go files) remain compatible.

Please provide context for why sqlc was downgraded. If this is a workaround for a bug in v1.30.0, document it inline as you've done for other replace blocks.


334-338: Verify necessity of replace blocks for go-yit and pingcap/tidb/pkg/parser.

Two replace blocks have been added to work around YAML parsing and SQLC engine errors. These are workarounds for upstream issues. Confirm:

  1. These pinned versions are the minimal necessary to resolve the issues.
  2. There is a plan to remove these replaces when upstream issues are resolved.
  3. The specific commit hashes or pseudo-versions chosen are correct and don't introduce their own issues.

Consider adding tracking comments or links to upstream issues in the replace directives to aid future maintenance.


11-11: Document the SQLC downgrade reason; ClickHouse SQL parser is safe to add.

SQLC v1.30.0 is mainly a bug fix release with a security fix for output plugins, but includes BREAKING changes for MySQL type mappings (int16 for SMALLINT/YEAR, int8 for TINYINT). Add a comment in go.mod explaining why the downgrade from v1.30.0 to v1.29.0 was necessary. The new ClickHouse SQL parser dependency is safe—no known vulnerabilities and compatible with the existing ClickHouse Go driver v2.40.1.

go/pkg/clickhouse/query-parser/cte.go (2)

7-26: Excellent documentation addressing previous feedback.

The comprehensive comment effectively explains what CTEs are, why tracking is necessary, and provides a clear example. This directly addresses the past review feedback requesting clarification.


27-41: The concern about nil map panic is unfounded.

The cteNames field is properly initialized in the NewParser constructor with make(map[string]bool) at parser.go line 16. The code is safe—no nil panic risk exists.

Likely an incorrect or invalid review comment.

go/pkg/codes/generate.go (4)

53-84: LGTM! Clean end-to-end MDX generation workflow.

The main function correctly collects error codes from all domains and orchestrates the MDX generation, cleanup, and docs.json update with appropriate error handling.


86-92: LGTM! Well-defined metadata structure.

The ErrorCodeInfo struct appropriately captures all necessary information for MDX and docs.json generation.


157-182: LGTM! Clean signature update.

The processErrorDomain function correctly aggregates error codes from all categories and returns them for MDX generation.


186-236: LGTM! Proper metadata extraction and collection.

The processCategory function correctly extracts descriptions from comments and builds ErrorCodeInfo entries for each error code.

go/pkg/cache/interface.go (1)

12-36: LGTM! Clean interface extension for bulk operations.

The multi-key methods follow consistent naming conventions and signatures. The documentation is clear, and the method signatures properly handle batch operations by using slices for input keys and maps for returning results indexed by key.

go/pkg/zen/middleware_errors.go (6)

40-40: LGTM! Correct NotFound error mapping.

Adding KeySpaceNotFound to the NotFound error block is appropriate and follows the existing pattern for resource-not-found errors.


61-61: Good clarification.

The updated comment distinguishes general validation errors from the new analytics-specific Bad Request errors, improving code organization.


81-97: LGTM! Correct Bad Request mapping for analytics queries.

The analytics query validation errors correctly return 400 status with BadRequestErrorResponse. The error codes align with the constants defined in go/pkg/codes/constants_gen.go.


99-113: LGTM! Appropriate Unprocessable Entity mapping.

Query resource limit errors (timeout, memory, rows) correctly return 422 status, distinguishing them from malformed query errors (400). The response type matches the OpenAPI schema.


115-127: LGTM! Correct rate limiting error mapping.

The quota exceeded error appropriately returns 429 status with TooManyRequestsErrorResponse. Note: PR objectives mention removing max-rows-to-read quota as a follow-up item.


262-263: LGTM! Appropriate precondition mapping.

Adding AnalyticsNotConfigured to the Precondition Failed block is correct—analytics must be configured before use.

deployment/docker-compose.yaml (4)

74-75: LGTM! Correct dependency addition.

Adding s3 as a dependency with health check for the apiv2 service is appropriate, as the analytics feature requires vault access for storing encrypted ClickHouse credentials.


89-93: LGTM! Consistent environment variable naming.

The vault-related environment variables now follow the UNKEY_VAULT_* naming convention, improving consistency across the configuration.


95-95: LGTM! Correct analytics URL configuration.

The UNKEY_CLICKHOUSE_ANALYTICS_URL correctly points to the ClickHouse HTTP interface using the Docker network service name and standard port.


126-131: LGTM! Appropriate healthcheck configuration.

The Kafka healthcheck using kafka-topics.sh --list is a good way to verify the broker is operational, and the timing parameters are reasonable.

go/apps/api/openapi/openapi-generated.yaml (1)

170-187: 422 description may conflict with “remove max-rows-to-read quota” follow-up

You mention “scans too many rows” here. Chronark’s follow-up asked to remove max-rows-to-read quota. Ensure the implementation and docs match; if that quota was removed, adjust this description accordingly.

go/main.go (1)

36-38: CLI wiring LGTM; confirm dev command exposure in releases

clickhouseUser.Cmd and dev.Cmd are correctly registered. Please confirm the dev tooling should ship in production binaries; if not, gate via build tags or an env flag.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
go/pkg/testutil/http.go (1)

256-290: Consider adding option builders for all configurable fields

The options pattern is well-implemented, but option builders are only provided for 4 out of 6 configurable fields. Consider adding:

  • WithQuotaDurationSeconds
  • WithMaxQueryExecutionTime

This would provide test authors full control over analytics quota configuration when needed for edge-case testing.

Add the missing option builders:

+func WithQuotaDurationSeconds(seconds int32) SetupAnalyticsOption {
+	return func(c *setupAnalyticsConfig) {
+		c.QuotaDurationSeconds = seconds
+	}
+}
+
+func WithMaxQueryExecutionTime(seconds int32) SetupAnalyticsOption {
+	return func(c *setupAnalyticsConfig) {
+		c.MaxQueryExecutionTime = seconds
+	}
+}
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b737958 and b7ae657.

📒 Files selected for processing (8)
  • apps/docs/analytics/query-examples.mdx (1 hunks)
  • go/apps/api/routes/v2_analytics_get_verifications/422_test.go (1 hunks)
  • go/internal/services/caches/caches.go (2 hunks)
  • go/pkg/clickhouse/key_verifications_test.go (4 hunks)
  • go/pkg/clickhouse/query-parser/cte.go (1 hunks)
  • go/pkg/clickhouse/user.go (1 hunks)
  • go/pkg/testutil/http.go (4 hunks)
  • go/pkg/zen/middleware_errors.go (5 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • go/pkg/clickhouse/query-parser/cte.go
🧰 Additional context used
🧠 Learnings (13)
📓 Common learnings
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 4190
File: go/internal/services/keys/verifier.go:51-53
Timestamp: 2025-10-30T15:10:52.743Z
Learning: PR #4190 for unkeyed/unkey is focused solely on database schema and query changes for identity-based credits. It adds IdentityCredits and KeyCredits fields to structs and queries, but does not implement the priority enforcement logic in the usagelimiter. The logic implementation is intentionally deferred to a later PR in the stack.
📚 Learning: 2025-04-08T09:34:24.576Z
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 2872
File: apps/dashboard/lib/trpc/routers/ratelimit/createNamespace.ts:36-39
Timestamp: 2025-04-08T09:34:24.576Z
Learning: In the Unkey dashboard, when making database queries involving workspaces, use `ctx.workspace.id` directly instead of fetching the workspace separately for better performance and security.

Applied to files:

  • go/pkg/clickhouse/user.go
📚 Learning: 2025-10-21T09:45:47.560Z
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 4107
File: go/pkg/clickhouse/key_verifications_test.go:20-20
Timestamp: 2025-10-21T09:45:47.560Z
Learning: In ClickHouse tests (e.g., go/pkg/clickhouse/key_verifications_test.go), parallel execution with t.Parallel() is safe when tests use workspaceID-based isolation. Each test generates a unique workspaceID (via uid.New(uid.WorkspacePrefix)), and all database operations are scoped to that workspaceID, providing logical isolation even when tests share a single ClickHouse instance.

Applied to files:

  • go/pkg/clickhouse/user.go
  • go/pkg/clickhouse/key_verifications_test.go
📚 Learning: 2025-04-08T09:34:24.576Z
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 2872
File: apps/dashboard/lib/trpc/routers/ratelimit/createNamespace.ts:36-39
Timestamp: 2025-04-08T09:34:24.576Z
Learning: When querying or updating namespaces in the Unkey dashboard, always scope the operations to the current workspace using `eq(table.workspaceId, ctx.workspace.id)` to prevent cross-workspace access.

Applied to files:

  • go/pkg/clickhouse/user.go
📚 Learning: 2025-09-22T18:44:56.279Z
Learnt from: perkinsjr
Repo: unkeyed/unkey PR: 4009
File: apps/dashboard/app/(app)/[workspace]/apis/[apiId]/_overview/components/table/components/log-details/index.tsx:4-5
Timestamp: 2025-09-22T18:44:56.279Z
Learning: In the Unkey dashboard, the workspace hook (useWorkspace) provides security validation by checking database access and user authorization to the workspace, with 10-minute caching for performance. Using URL params (useParams) for workspace slug would bypass this security validation and allow unauthorized access attempts. Always use the workspace hook for workspace-scoped navigation and handle loading states properly rather than switching to URL parameters.

Applied to files:

  • go/pkg/clickhouse/user.go
📚 Learning: 2025-10-30T15:10:52.743Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 4190
File: go/internal/services/keys/verifier.go:51-53
Timestamp: 2025-10-30T15:10:52.743Z
Learning: PR #4190 for unkeyed/unkey is focused solely on database schema and query changes for identity-based credits. It adds IdentityCredits and KeyCredits fields to structs and queries, but does not implement the priority enforcement logic in the usagelimiter. The logic implementation is intentionally deferred to a later PR in the stack.

Applied to files:

  • go/pkg/clickhouse/user.go
  • go/pkg/clickhouse/key_verifications_test.go
📚 Learning: 2025-08-14T16:25:48.167Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3785
File: go/apps/api/routes/v2_keys_reroll_key/401_test.go:52-61
Timestamp: 2025-08-14T16:25:48.167Z
Learning: User Flo4604 requested creation of a GitHub issue to track converting all test files to use table-driven test patterns as a broader codebase improvement, following the suggestion made during review of go/apps/api/routes/v2_keys_reroll_key/401_test.go.

Applied to files:

  • go/pkg/clickhouse/user.go
  • go/pkg/clickhouse/key_verifications_test.go
  • go/apps/api/routes/v2_analytics_get_verifications/422_test.go
📚 Learning: 2025-07-15T14:25:05.608Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 3560
File: go/apps/api/routes/v2_keys_create_key/handler.go:353-466
Timestamp: 2025-07-15T14:25:05.608Z
Learning: In the Unkey codebase, input validation for API endpoints is handled at the OpenAPI schema layer, which validates request fields like permission slugs (pattern: "^[a-zA-Z0-9_]+$", length: 1-100 characters) before requests reach the handler code. This validation occurs during the zen.BindBody call in handlers.

Applied to files:

  • go/pkg/clickhouse/user.go
  • go/pkg/zen/middleware_errors.go
📚 Learning: 2025-08-27T13:48:54.016Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3841
File: go/pkg/db/queries/key_find_for_verification.sql:15-15
Timestamp: 2025-08-27T13:48:54.016Z
Learning: The pending_migration_id field in FindKeyForVerification is only used internally by get_migrated.go for migration logic and doesn't need to be exposed in KeyVerifier struct or API response DTOs since it's an internal implementation detail.

Applied to files:

  • go/pkg/clickhouse/key_verifications_test.go
📚 Learning: 2025-07-16T15:38:53.491Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3606
File: go/pkg/db/replica.go:8-11
Timestamp: 2025-07-16T15:38:53.491Z
Learning: For debugging database replica usage in go/pkg/db/replica.go, it's acceptable to mark QueryRowContext operations as "success" even though SQL errors only surface during row.Scan() calls. The timing metrics are the primary concern for debugging replica performance patterns.

Applied to files:

  • go/pkg/clickhouse/key_verifications_test.go
  • go/pkg/testutil/http.go
📚 Learning: 2025-09-15T19:53:28.487Z
Learnt from: mcstepp
Repo: unkeyed/unkey PR: 3952
File: go/proto/ctrl/v1/routing.proto:0-0
Timestamp: 2025-09-15T19:53:28.487Z
Learning: In the Unkey codebase, authentication/authorization errors intentionally return NOT_FOUND error codes instead of distinct auth error codes (like FORBIDDEN or UNAUTHORIZED) for security reasons. This prevents attackers from distinguishing between "resource doesn't exist" and "you don't have permission to access this resource", avoiding information disclosure about workspace existence and access boundaries.

Applied to files:

  • go/pkg/zen/middleware_errors.go
📚 Learning: 2025-03-19T09:25:59.751Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 2955
File: go/apps/api/routes/v2_identities_create_identity/handler.go:162-202
Timestamp: 2025-03-19T09:25:59.751Z
Learning: In the Unkey codebase, input validation for API endpoints is primarily handled through OpenAPI schema validation, which occurs before requests reach the handler code. For example, in the identities.createIdentity endpoint, minimum values for ratelimit duration and limit are defined in the OpenAPI schema rather than duplicating these checks in the handler.

Applied to files:

  • go/pkg/zen/middleware_errors.go
📚 Learning: 2025-09-12T13:25:41.849Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 3958
File: internal/clickhouse/src/requests.ts:7-8
Timestamp: 2025-09-12T13:25:41.849Z
Learning: In the ClickHouse v2 table migration, the location field was renamed from `colo` to `region` across all tables including `default.api_requests_raw_v2`. The zod insert schemas must be updated accordingly to avoid runtime insert failures.

Applied to files:

  • apps/docs/analytics/query-examples.mdx
🧬 Code graph analysis (5)
go/internal/services/caches/caches.go (7)
go/pkg/db/models_generated.go (1)
  • ClickhouseWorkspaceSetting (569-581)
go/pkg/cache/scoped_key.go (3)
  • ScopedKey (49-62)
  • ScopedKeyToString (81-81)
  • ScopedKeyFromString (82-82)
go/pkg/db/api_find_key_auth_by_key_auth_ids.sql_generated.go (1)
  • FindKeyAuthsByKeyAuthIdsRow (28-31)
go/pkg/db/api_find_key_auth_by_ids.sql_generated.go (1)
  • FindKeyAuthsByIdsRow (28-31)
go/pkg/cache/clustering/cluster_cache.go (1)
  • Config (34-58)
go/pkg/cache/cache.go (1)
  • Config (34-51)
go/pkg/cache/middleware/tracing.go (1)
  • WithTracing (16-18)
go/pkg/clickhouse/key_verifications_test.go (4)
go/pkg/clickhouse/client.go (1)
  • New (62-208)
go/pkg/array/random.go (1)
  • Random (23-29)
go/pkg/clickhouse/schema/types.go (1)
  • KeyVerificationV2 (6-19)
go/pkg/array/reduce.go (1)
  • Reduce (28-34)
go/pkg/testutil/http.go (6)
go/internal/services/caches/caches.go (2)
  • Caches (21-49)
  • New (166-313)
go/internal/services/analytics/connection_manager.go (3)
  • ConnectionManager (22-24)
  • NewConnectionManager (47-85)
  • ConnectionManagerConfig (37-44)
go/gen/proto/vault/v1/service.pb.go (3)
  • EncryptRequest (104-110)
  • EncryptRequest (123-123)
  • EncryptRequest (138-140)
go/pkg/clickhouse/user.go (1)
  • UserConfig (22-39)
go/pkg/db/clickhouse_workspace_settings_insert.sql_generated.go (1)
  • InsertClickhouseWorkspaceSettingsParams (42-54)
go/gen/proto/vault/v1/object.pb.go (3)
  • Encrypted (252-264)
  • Encrypted (277-277)
  • Encrypted (292-294)
go/pkg/zen/middleware_errors.go (2)
go/pkg/codes/constants_gen.go (12)
  • UnkeyDataErrorsKeySpaceNotFound (93-93)
  • UserErrorsBadRequestInvalidAnalyticsQuery (24-24)
  • UserErrorsBadRequestInvalidAnalyticsTable (26-26)
  • UserErrorsBadRequestInvalidAnalyticsFunction (28-28)
  • UserErrorsBadRequestInvalidAnalyticsQueryType (30-30)
  • UserErrorsUnprocessableEntityQueryExecutionTimeout (35-35)
  • UserErrorsUnprocessableEntityQueryMemoryLimitExceeded (37-37)
  • UserErrorsUnprocessableEntityQueryRowsLimitExceeded (39-39)
  • UserErrorsTooManyRequestsQueryQuotaExceeded (44-44)
  • UnkeyDataErrorsAnalyticsNotConfigured (141-141)
  • UnkeyAppErrorsPreconditionPreconditionFailed (171-171)
  • UnkeyDataErrorsAnalyticsConnectionFailed (143-143)
go/apps/api/openapi/gen.go (8)
  • BadRequestErrorResponse (65-71)
  • Meta (279-282)
  • BadRequestErrorDetails (47-62)
  • ValidationError (2015-2029)
  • UnprocessableEntityErrorResponse (500-506)
  • BaseError (74-86)
  • TooManyRequestsErrorResponse (471-477)
  • InternalServerErrorResponse (181-187)
go/apps/api/routes/v2_analytics_get_verifications/422_test.go (6)
go/pkg/testutil/http.go (3)
  • NewHarness (59-208)
  • WithMaxQueryMemoryBytes (273-277)
  • CallRoute (385-419)
go/pkg/testutil/seed/seed.go (1)
  • CreateApiRequest (84-92)
go/pkg/clickhouse/interface.go (1)
  • ClickHouse (59-68)
go/pkg/clickhouse/schema/requests.go (1)
  • KeyVerificationRequestV1 (66-92)
go/pkg/uid/uid.go (2)
  • RequestPrefix (19-19)
  • KeyPrefix (16-16)
go/apps/api/routes/v2_analytics_get_verifications/handler.go (3)
  • Handler (49-56)
  • Request (26-26)
  • Response (27-27)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
  • GitHub Check: Test Packages / Test
  • GitHub Check: Test Go API Local / Test
  • GitHub Check: Test Dashboard / Test Dashboard
  • GitHub Check: Test API / API Test Local
  • GitHub Check: Build / Build
  • GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (20)
apps/docs/analytics/query-examples.mdx (2)

628-657: ✅ Moving average example table reference fixed.

The previous critical inconsistency between readable SQL (line 642) and JSON format (line 653) has been resolved. Both now correctly reference key_verifications_per_day_v1, ensuring users copying either format will get working queries.


1-732: Well‑structured analytics documentation with consistent examples.

The file provides comprehensive, well‑organized ClickHouse SQL examples across all key analytics scenarios. Each example clearly shows both readable multi-line and single-line JSON formats for API requests. All ~28 query examples maintain consistency between formats, use correct table names with _v1 versioning, and include appropriate time filters.

The documentation aligns well with the new /v2/analytics.getVerifications endpoint and should effectively guide users in writing analytics queries.

go/apps/api/routes/v2_analytics_get_verifications/422_test.go (1)

30-42: Fix the invalid range loop.

Line 30 uses for i := range 50_000, which is illegal because Go’s range only works on strings, arrays, slices, maps, or channels—this code will not compile. Please switch to a standard counted loop.

-	for i := range 50_000 {
+	for i := 0; i < 50_000; i++ {
⛔ Skipped due to learnings
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3785
File: go/apps/api/routes/v2_keys_reroll_key/401_test.go:52-61
Timestamp: 2025-08-14T16:25:48.167Z
Learning: User Flo4604 requested creation of a GitHub issue to track converting all test files to use table-driven test patterns as a broader codebase improvement, following the suggestion made during review of go/apps/api/routes/v2_keys_reroll_key/401_test.go.
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3800
File: go/apps/api/integration/multi_node_usagelimiting/run.go:107-126
Timestamp: 2025-08-19T08:57:31.793Z
Learning: Go 1.22+ supports "range over integers" syntax: `for range N` iterates N times, and `for i := range N` iterates with i from 0 to N-1. This is valid Go syntax and should not be flagged as a compilation error.
Learnt from: imeyer
Repo: unkeyed/unkey PR: 3733
File: go/pkg/repeat/every_test.go:383-383
Timestamp: 2025-08-05T15:19:50.563Z
Learning: Go 1.24 introduced testing.B.Loop method which simplifies benchmark functions by handling iteration internally, ensuring setup/cleanup runs only once per benchmark count, and preventing compiler optimizations that could affect benchmark accuracy.
Learnt from: ogzhanolguncu
Repo: unkeyed/unkey PR: 4107
File: go/pkg/clickhouse/key_verifications_test.go:20-20
Timestamp: 2025-10-21T09:45:47.560Z
Learning: In ClickHouse tests (e.g., go/pkg/clickhouse/key_verifications_test.go), parallel execution with t.Parallel() is safe when tests use workspaceID-based isolation. Each test generates a unique workspaceID (via uid.New(uid.WorkspacePrefix)), and all database operations are scoped to that workspaceID, providing logical isolation even when tests share a single ClickHouse instance.
go/pkg/zen/middleware_errors.go (4)

40-40: LGTM - KeySpaceNotFound error mapping

The addition of UnkeyDataErrorsKeySpaceNotFound to the Not Found error group follows the established pattern and is appropriate for the analytics feature.


61-97: LGTM - Analytics query validation error handling

The split into "General validation" and "Query validation" categories improves code organization. All analytics query validation errors correctly return 400 Bad Request with consistent error response structures.


99-113: LGTM - Query resource limit error handling

Using 422 Unprocessable Entity for query resource limits (timeout, memory, rows) is semantically appropriate and follows HTTP best practices.


115-127: LGTM - Query quota exceeded error handling

Using 429 Too Many Requests for quota exceeded is the correct HTTP status code for rate limiting scenarios.

go/internal/services/caches/caches.go (5)

34-44: LGTM - New cache field declarations

The three new cache fields are well-documented with clear descriptions of their key-value types and purposes. The choice of key types (string for workspace-scoped settings, ScopedKey for workspace+resource pairs) is appropriate for each use case.


248-264: LGTM - ClickhouseSetting cache initialization

The cache configuration is appropriate with 1-minute freshness for workspace settings that don't change frequently. Error handling is correctly implemented.


266-283: LGTM - KeyAuthToApiRow cache initialization

The 10-minute freshness window is appropriate for key_auth to API mappings that are relatively stable. Proper use of ScopedKey converters and error handling.


285-302: LGTM - ApiToKeyAuthRow cache initialization with error handling

The cache initialization follows the established pattern with proper error handling at lines 300-302. The configuration mirrors KeyAuthToApiRow appropriately.


308-310: LGTM - Cache wiring with tracing

The new caches are correctly wrapped with tracing middleware and added to the returned Caches struct, maintaining consistency with existing cache initialization.

go/pkg/clickhouse/user.go (5)

11-19: LGTM - Secure identifier validation patterns

The regex patterns properly restrict identifiers to alphanumeric characters and underscores, with proper anchoring (^ and $) to prevent SQL injection attacks.


21-39: LGTM - UserConfig structure

The struct is well-organized with clear separation between per-window quotas and per-query limits. All necessary configuration fields are present for comprehensive user provisioning.


41-62: LGTM - Comprehensive identifier validation

This validation function effectively addresses SQL injection concerns by ensuring all identifiers (username, workspace ID, table names) match safe patterns before being used in SQL statements. The descriptive error messages aid in debugging invalid configurations.


64-171: ConfigureUser implementation is secure with proper validation

The method implements comprehensive user provisioning with proper security controls:

  1. SQL injection prevention: All identifiers are validated via validateIdentifiers before use
  2. Password security: Password is bound as a parameter (line 77), not interpolated
  3. Idempotent operations: Uses IF NOT EXISTS and OR REPLACE for safe re-runs
  4. Proper error handling: Errors are wrapped with context and returned appropriately
  5. Operational logging: Progress is logged at appropriate levels

The workspace_id in the ROW POLICY USING clause (line 106) is string-interpolated but is validated first. This appears to be a ClickHouse limitation—ROW POLICY USING clauses require compile-time constant expressions and don't support parameterization like WHERE clauses do.

Note: Line 124's comment documenting why max_result_rows is intentionally omitted from the quota is excellent—it explains that per-window limits are too restrictive for analytics while per-query limits still apply via the settings profile.


173-183: LGTM - Default allowed tables

The default table list covers the necessary analytics tables with appropriate granularity levels (raw, minute, hour, day, month aggregations).

go/pkg/testutil/http.go (3)

14-15: LGTM - Analytics testing infrastructure additions

The new imports and AnalyticsConnectionManager field properly extend the test harness to support analytics testing scenarios.

Also applies to: 55-55


164-173: LGTM - Analytics connection manager initialization

The connection manager is properly initialized with the correct ClickHouse DSN from containers.ClickHouse(t) (line 170), which resolves the previous concern about hard-coded localhost addresses. All required dependencies are correctly wired.


291-350: LGTM - SetupAnalytics implementation

The method correctly orchestrates analytics setup for testing:

  1. Sensible defaults: 10M rows, 1GB memory, reasonable query/time limits
  2. Password security: Test password is encrypted via Vault before storage
  3. Complete provisioning: Configures both ClickHouse (user, permissions, quotas) and MySQL (credential storage)
  4. Proper error handling: Uses require.NoError to fail fast on setup issues

The hard-coded test password is appropriate for test environments.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
go/apps/api/openapi/openapi-generated.yaml (1)

2047-2058: BadRequest details still unsatisfiable due to BaseError additionalProperties: false

BadRequestErrorDetails uses allOf to extend BaseError with errors, but BaseError disallows additional properties, so errors can never be present. Fix one of:

  • Relax BaseError by allowing additional properties, or
  • Stop using allOf and define a concrete schema that includes the base fields plus errors.

Minimal change:

         BaseError:
@@
-            additionalProperties: false
+            additionalProperties: true

Also applies to: 2060-2085

🧹 Nitpick comments (4)
go/apps/api/openapi/openapi-generated.yaml (4)

3348-3423: Document required permissions and confirm visibility

  • Add a “Required Permissions” section to match other endpoints and the new RBAC: api.*.read_analytics or api.<api_id>.read_analytics.
  • Confirm x-hidden: true is intentional; otherwise this won’t appear in published SDK/docs.

Example addition in description:

             description: |
                 Execute custom SQL queries against your key verification analytics.
                 For complete documentation including available tables, columns, data types, query examples, see the schema reference in the API documentation.
+                
+                Required Permissions
+                - `api.*.read_analytics`
+                - `api.<api_id>.read_analytics`

97-108: Consider basic validation on query size

Add minLength/maxLength to query (e.g., minLength: 1, maxLength: 20000) to fail fast on pathological payloads and align with backend limits.


2113-2126: Clarify timestamp units in example rows

If time is epoch milliseconds (as example suggests), state it explicitly to avoid SDK/doc ambiguity.


187-205: 429 docs: mention Retry-After header

Add guidance that responses may include Retry-After and clients should honor it. Helps interoperable backoff behavior.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b7ae657 and 77aa422.

📒 Files selected for processing (4)
  • go/apps/api/openapi/gen.go (4 hunks)
  • go/apps/api/openapi/openapi-generated.yaml (6 hunks)
  • go/apps/api/openapi/spec/paths/v2/analytics/getVerifications/index.yaml (1 hunks)
  • go/pkg/zen/middleware_errors.go (5 hunks)
🧰 Additional context used
🧠 Learnings (10)
📓 Common learnings
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 4190
File: go/internal/services/keys/verifier.go:51-53
Timestamp: 2025-10-30T15:10:52.743Z
Learning: PR #4190 for unkeyed/unkey is focused solely on database schema and query changes for identity-based credits. It adds IdentityCredits and KeyCredits fields to structs and queries, but does not implement the priority enforcement logic in the usagelimiter. The logic implementation is intentionally deferred to a later PR in the stack.
📚 Learning: 2025-09-15T19:53:28.487Z
Learnt from: mcstepp
Repo: unkeyed/unkey PR: 3952
File: go/proto/ctrl/v1/routing.proto:0-0
Timestamp: 2025-09-15T19:53:28.487Z
Learning: In the Unkey codebase, authentication/authorization errors intentionally return NOT_FOUND error codes instead of distinct auth error codes (like FORBIDDEN or UNAUTHORIZED) for security reasons. This prevents attackers from distinguishing between "resource doesn't exist" and "you don't have permission to access this resource", avoiding information disclosure about workspace existence and access boundaries.

Applied to files:

  • go/pkg/zen/middleware_errors.go
  • go/apps/api/openapi/openapi-generated.yaml
📚 Learning: 2025-07-15T14:25:05.608Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 3560
File: go/apps/api/routes/v2_keys_create_key/handler.go:353-466
Timestamp: 2025-07-15T14:25:05.608Z
Learning: In the Unkey codebase, input validation for API endpoints is handled at the OpenAPI schema layer, which validates request fields like permission slugs (pattern: "^[a-zA-Z0-9_]+$", length: 1-100 characters) before requests reach the handler code. This validation occurs during the zen.BindBody call in handlers.

Applied to files:

  • go/pkg/zen/middleware_errors.go
📚 Learning: 2025-03-19T09:25:59.751Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 2955
File: go/apps/api/routes/v2_identities_create_identity/handler.go:162-202
Timestamp: 2025-03-19T09:25:59.751Z
Learning: In the Unkey codebase, input validation for API endpoints is primarily handled through OpenAPI schema validation, which occurs before requests reach the handler code. For example, in the identities.createIdentity endpoint, minimum values for ratelimit duration and limit are defined in the OpenAPI schema rather than duplicating these checks in the handler.

Applied to files:

  • go/pkg/zen/middleware_errors.go
📚 Learning: 2025-08-08T15:09:01.312Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3753
File: go/apps/api/openapi/config.yaml:9-10
Timestamp: 2025-08-08T15:09:01.312Z
Learning: Repo unkeyed/unkey: For go/apps/api/openapi, oapi-codegen is used and does not support OpenAPI 3.1 union types like [T, "null"]; an overlay step is required to downconvert to 3.0-style nullable before code generation.

Applied to files:

  • go/apps/api/openapi/gen.go
📚 Learning: 2025-08-08T15:09:01.312Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3753
File: go/apps/api/openapi/config.yaml:9-10
Timestamp: 2025-08-08T15:09:01.312Z
Learning: Repo unkeyed/unkey: In go/apps/api/openapi, oapi-codegen doesn’t support OpenAPI 3.1 union nullability; overlay.yaml must be applied before codegen. The overlay key in oapi-codegen config isn’t supported—use a pre-step (programmatic or CLI) to merge overlay into the bundled spec, then run oapi-codegen.

Applied to files:

  • go/apps/api/openapi/gen.go
📚 Learning: 2025-08-08T15:09:01.312Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3753
File: go/apps/api/openapi/config.yaml:9-10
Timestamp: 2025-08-08T15:09:01.312Z
Learning: Repo unkeyed/unkey: In go/apps/api/openapi, overlays are required to downconvert 3.1 union nullability to 3.0-style nullable before running oapi-codegen; config.yaml’s output-options.overlay is not recognized by oapi-codegen, so overlays must be applied in a pre-step (programmatic or CLI) prior to codegen.

Applied to files:

  • go/apps/api/openapi/gen.go
📚 Learning: 2025-08-21T15:54:45.198Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 3825
File: go/internal/services/usagelimiter/limit.go:38-0
Timestamp: 2025-08-21T15:54:45.198Z
Learning: In go/internal/services/usagelimiter/limit.go, the UpdateKeyCreditsDecrement operation cannot be safely wrapped with db.WithRetry due to the lack of idempotency mechanisms in the current tech stack. Retrying this non-idempotent write operation risks double-charging users if the first attempt commits but the client sees a transient error.

Applied to files:

  • go/apps/api/openapi/gen.go
📚 Learning: 2025-07-03T05:58:10.699Z
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 3421
File: go/apps/api/openapi/openapi.yaml:196-200
Timestamp: 2025-07-03T05:58:10.699Z
Learning: In the Unkey codebase, OpenAPI 3.1 is used, which allows sibling keys (such as `description`) alongside `$ref` in schema objects. Do not flag this as an error in future reviews.

Applied to files:

  • go/apps/api/openapi/openapi-generated.yaml
📚 Learning: 2025-09-12T18:11:33.481Z
Learnt from: mcstepp
Repo: unkeyed/unkey PR: 3952
File: apps/dashboard/lib/trpc/routers/deploy/rollback.ts:23-24
Timestamp: 2025-09-12T18:11:33.481Z
Learning: In apps/dashboard/lib/trpc/routers/deploy/rollback.ts, the CTRL_URL environment variable should fail fast with a clear error message if missing in non-development environments, rather than defaulting to localhost which can mask production configuration issues.

Applied to files:

  • go/apps/api/openapi/openapi-generated.yaml
🧬 Code graph analysis (1)
go/pkg/zen/middleware_errors.go (2)
go/pkg/codes/constants_gen.go (12)
  • UnkeyDataErrorsKeySpaceNotFound (93-93)
  • UserErrorsBadRequestInvalidAnalyticsQuery (24-24)
  • UserErrorsBadRequestInvalidAnalyticsTable (26-26)
  • UserErrorsBadRequestInvalidAnalyticsFunction (28-28)
  • UserErrorsBadRequestInvalidAnalyticsQueryType (30-30)
  • UserErrorsUnprocessableEntityQueryExecutionTimeout (35-35)
  • UserErrorsUnprocessableEntityQueryMemoryLimitExceeded (37-37)
  • UserErrorsUnprocessableEntityQueryRowsLimitExceeded (39-39)
  • UserErrorsTooManyRequestsQueryQuotaExceeded (44-44)
  • UnkeyDataErrorsAnalyticsNotConfigured (141-141)
  • UnkeyAppErrorsPreconditionPreconditionFailed (171-171)
  • UnkeyDataErrorsAnalyticsConnectionFailed (143-143)
go/apps/api/openapi/gen.go (8)
  • BadRequestErrorResponse (65-71)
  • Meta (279-282)
  • BadRequestErrorDetails (47-62)
  • ValidationError (2030-2044)
  • UnprocessableEntityErrorResponse (515-521)
  • BaseError (74-86)
  • TooManyRequestsErrorResponse (486-492)
  • ServiceUnavailableErrorResponse (471-477)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)
  • GitHub Check: Test API / API Test Local
  • GitHub Check: Test Packages / Test
  • GitHub Check: Build / Build
  • GitHub Check: Test Go API Local / Test
  • GitHub Check: Test Dashboard / Test Dashboard
🔇 Additional comments (15)
go/apps/api/openapi/spec/paths/v2/analytics/getVerifications/index.yaml (4)

1-11: LGTM!

The endpoint configuration is properly structured with appropriate security (rootKey), visibility controls (hidden from docs and SDK generation), and operation naming.


12-15: LGTM!

The summary and description are clear and appropriately guide users to consult the schema reference documentation for detailed information about available tables, columns, and query examples.


16-28: LGTM!

The request body and success response are properly configured with appropriate schema references and clear descriptions.


29-76: LGTM!

The error responses comprehensively cover all expected error scenarios for analytics queries, including query validation (400), resource limits (422), quota enforcement (429), and connectivity issues (503). Each error response has clear descriptions that help users understand and troubleshoot issues.

go/pkg/zen/middleware_errors.go (6)

40-40: LGTM!

Adding KeySpaceNotFound to the Not Found error group is appropriate and follows the existing pattern for data-related not found errors.


61-97: LGTM!

The comment update clarifies that the first Bad Request block handles general validation, while the new analytics query validation block (lines 81-97) appropriately handles query-specific errors. The new error codes comprehensively cover query validation scenarios (invalid queries, tables, functions, and query types) and return the correct 400 Bad Request response.


99-113: LGTM!

The Unprocessable Entity (422) error handler appropriately covers query resource limit violations (timeout, memory, rows). The semantic choice of 422 is correct here since the query syntax is valid but the operation cannot be completed due to resource constraints.


115-127: LGTM!

The Too Many Requests (429) error handler correctly handles query quota violations, using the appropriate HTTP status code and response type for rate limiting scenarios.


262-263: LGTM!

Adding AnalyticsNotConfigured to the Precondition Failed (412) error group is semantically correct, as it indicates the analytics feature must be configured before it can be used.


304-323: LGTM!

The Service Unavailable (503) error handler correctly addresses analytics connection failures. The code now properly uses ServiceUnavailableErrorResponse (line 313), which addresses the concern raised in the previous review. The additional logging for this error type is appropriate given the operational significance of service availability issues.

go/apps/api/openapi/gen.go (3)

464-521: LGTM!

The new error response types (ServiceUnavailableErrorResponse, TooManyRequestsErrorResponse, and UnprocessableEntityErrorResponse) are properly structured and consistent with existing error response types. The comprehensive documentation comments provide clear guidance on when each error occurs and how to handle it.


549-566: LGTM!

The analytics verification types are well-designed. The use of []map[string]interface{} for V2AnalyticsGetVerificationsResponseData is appropriate since the returned fields vary based on the SQL SELECT clause. The comments clearly explain this flexible behavior.


2083-2084: LGTM!

The request body alias follows the established pattern used throughout the codebase for JSON content type request bodies.

go/apps/api/openapi/openapi-generated.yaml (2)

152-169: LGTM: NotFound error shape and semantics

Schema and description align with the repo’s NOT_FOUND-on-auth pattern to reduce information leakage.


6213-6214: LGTM: analytics tag added

Good categorization for the new surface.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 77aa422 and d77d3e2.

📒 Files selected for processing (1)
  • deployment/docker-compose.yaml (4 hunks)
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: Flo4604
Repo: unkeyed/unkey PR: 4190
File: go/internal/services/keys/verifier.go:51-53
Timestamp: 2025-10-30T15:10:52.743Z
Learning: PR #4190 for unkeyed/unkey is focused solely on database schema and query changes for identity-based credits. It adds IdentityCredits and KeyCredits fields to structs and queries, but does not implement the priority enforcement logic in the usagelimiter. The logic implementation is intentionally deferred to a later PR in the stack.
📚 Learning: 2025-07-22T09:02:12.495Z
Learnt from: chronark
Repo: unkeyed/unkey PR: 3638
File: deployment/docker-compose.yaml:81-94
Timestamp: 2025-07-22T09:02:12.495Z
Learning: The docker-compose.yaml file in deployment/ is specifically for development environments, not production. Kafka and other service configurations in this file should be optimized for development convenience rather than production security/hardening.

Applied to files:

  • deployment/docker-compose.yaml
🪛 Gitleaks (8.28.0)
deployment/docker-compose.yaml

[high] 93-93: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.

(generic-api-key)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
  • GitHub Check: Test Go API Local / Test
  • GitHub Check: Build / Build
  • GitHub Check: Test API / API Test Local
  • GitHub Check: Test Dashboard / Test Dashboard
  • GitHub Check: Test Packages / Test
  • GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (3)
deployment/docker-compose.yaml (3)

74-75: Addition of s3 service health dependency looks good.

The apiv2 service now depends on s3 being healthy, which aligns with the analytics/vault integration changes in this PR. No concerns with this change.


89-92: Environment variable naming is now consistent with UNKEY_ prefix convention.

The renaming from VAULT_S3_* to UNKEY_VAULT_S3_* improves consistency across the configuration. The new UNKEY_CLICKHOUSE_ANALYTICS_URL aligns with the analytics feature additions. This looks good, aside from the secret handling concern flagged separately.

Also applies to: 95-95


89-95: No security issue: This is a local development file with shared example credentials.

The secret value is not a real credential but an example value that appears identically across multiple development files (docker-compose.yaml, api.yaml, gw.yaml, ctrl.yaml), indicating it's an intentional test/placeholder. All surrounding environment variables use generic test credentials (password, test-token-123, minio_root_user, etc.), and the file is committed to git—not excluded by .gitignore. This is consistent with the repo's established practice of using hardcoded generic credentials in local development environments for convenience.

Likely an incorrect or invalid review comment.

Comment on lines +114 to +116
image: bufbuild/bufstream:0.4.4
network_mode: host
command: ["serve", "--inmemory"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Critical: Kafka network_mode: host breaks service discovery for apiv2.

The change from the previous Kafka configuration to network_mode: host with the simplified Bufstream setup creates a networking problem. The apiv2 service (line 63) uses the default bridge network (networks: - default), but Kafka now operates in the host's network namespace. When apiv2 tries to connect to kafka:9092 (line 94), the DNS name kafka will not resolve because:

  • network_mode: host bypasses Docker's internal DNS and container networking
  • apiv2 in the default network cannot reach services in host mode

This will cause connection failures. Either:

  1. Add Kafka back to the networks: - default section and expose ports, or
  2. Change all dependent services (apiv2, etc.) to also use network_mode: host and update connection strings to localhost:9092

The previous configuration (despite the advertised listeners issue flagged in past reviews) did work for inter-container communication. Verify this change is intentional and test inter-service connectivity before merging.


🏁 Script executed:

#!/bin/bash
# Extract the relevant sections of the docker-compose.yaml file
echo "=== Lines 60-70 (apiv2 service) ==="
sed -n '60,70p' deployment/docker-compose.yaml

echo ""
echo "=== Lines 88-100 (Kafka service environment) ==="
sed -n '88,100p' deployment/docker-compose.yaml

echo ""
echo "=== Lines 114-120 (Kafka service image/network/command) ==="
sed -n '114,120p' deployment/docker-compose.yaml

echo ""
echo "=== Lines 450-465 (volumes section) ==="
sed -n '450,465p' deployment/docker-compose.yaml

echo ""
echo "=== Check if kafka service has networks defined ==="
sed -n '105,125p' deployment/docker-compose.yaml

Length of output: 1872


🌐 Web query:

Docker Compose network_mode host service DNS resolution other containers

💡 Result:

Short answer: A service using network_mode: host does NOT join Docker Compose networks and therefore is not visible to Docker's embedded DNS — it cannot resolve other services by their Compose service names. Use one of these fixes:

  • Don’t use host mode; use a user-defined bridge network so service names resolve. [1][2]
  • Have the host-mode container talk to other services via published host ports (localhost:PORT) or the other containers' host IPs. [2]
  • Add static entries (extra_hosts or /etc/hosts) or run an external DNS reachable from the host network and point the container's DNS to it. [1][3]
  • If you must mix host-mode and bridge-mode, expose needed services on the host (publish ports) or use an external network/DNS service for name resolution. [1][3]

References: Docker Compose networking (user networks & service discovery). [1] network_mode: host behavior and caveats. [2] Community explanations showing host-mode containers can’t use Docker embedded DNS for other containers. [3]


Critical: Kafka network_mode: host breaks service discovery for apiv2.

The change to network_mode: host (line 116) creates a confirmed networking problem. Services using network_mode: host do not join Docker Compose networks and are therefore invisible to Docker's embedded DNS, making them unreachable by service name.

The apiv2 service (lines 61-62) uses the default bridge network and attempts to connect to UNKEY_KAFKA_BROKERS: "kafka:9092" (line 95). This DNS name will not resolve because Kafka operates in the host network namespace, not Docker's internal network.

This will cause connection failures. Choose one approach:

  1. Remove network_mode: host from Kafka and add it to a Docker network, or
  2. Change all dependent services to also use network_mode: host and update connection strings to localhost:9092

Additional issues:

  • kafka_data volume is defined (line 455) but not mounted in the Kafka service—either mount it or remove the definition
  • Line 91 contains a base64-encoded secret flagged by security scanning; verify this should be committed in the dev environment
🤖 Prompt for AI Agents
In deployment/docker-compose.yaml around lines 114-116, the Kafka service is set
to network_mode: host which prevents it from joining Docker Compose networks and
breaks DNS-based service discovery for apiv2; fix by removing network_mode: host
and attaching Kafka to the same compose network as apiv2 (or alternatively make
all dependent services use host networking and update broker addresses to
localhost:9092), ensure the kafka_data volume defined at line 455 is actually
mounted into the Kafka service or remove the unused volume definition, and
verify whether the base64 secret on line 91 should be committed in the dev repo
(rotate or move to env/secret manager if not appropriate).

clickhouse:
clickhouse-keeper:
s3:
kafka_data:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Unused volume definition: kafka_data is defined but not mounted.

Line 459 adds kafka_data: to the volumes section, but the Kafka service (lines 112–117) does not mount this volume. Either:

  • Add volumes: - kafka_data:/path/to/data to the Kafka service if data persistence is intended, or
  • Remove the unused volume definition

With the current --inmemory command flag, persistent storage is not needed, so removing the unused volume is likely correct.

🤖 Prompt for AI Agents
In deployment/docker-compose.yaml around line 459, the volume kafka_data is
declared but never mounted by any service; either remove this unused volume
entry or attach it to the Kafka service if persistence is required. Given the
compose uses the --inmemory flag, delete the kafka_data: entry from the volumes
section to avoid dead configuration, or alternatively add a volumes: -
kafka_data:/var/lib/kafka (or your Kafka data path) under the Kafka service if
you intend to persist data.

@chronark chronark merged commit cb51841 into main Nov 4, 2025
20 of 21 checks passed
@chronark chronark deleted the feat/analytics branch November 4, 2025 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement /v2/analytics.getVerifications

3 participants