fix(clickhouse): improve latest keys used queries for high volume (150M +)

[mcstepp]

What does this PR do?

Fixes query timeouts on the API Requests page for users with extreme verification volumes (150M+ requests every 12 hours).

Problem

Users with high API traffic experienced 30+ second query timeouts when loading the verification logs table. The dashboard became completely unusable for monitoring API activity.

Root Cause: The ClickHouse query scanned all 150M verification rows within the time window, aggregated thousands of keys, then limited to 50 results. This meant 99.5% of aggregation work was wasted on every page load.

Solution

Implemented a materialized view optimization that provides 100-1000x performance improvement:

New Materialized View (keys_last_used_v1): Pre-aggregates the latest verification per key

• Stores one row per unique key instead of scanning 150M rows
• Uses AggregatingMergeTree for automatic aggregation during merges
• Automatically populated from new inserts with 90-day TTL

Dual Query Strategy:

• Fast Path (Recent Data): Queries keys_last_used_v1 materialized view when no explicit time filters are set and data is recent (within 5 minutes). Returns keys by most recent activity.
• Accurate Path (Time Windows): When users set explicit time filters, queries key_verifications_per_hour_v2 (pre-aggregated) + key_verifications_raw_v2 (recent) to find ALL keys with ANY activity in the specified time window, not just recently active keys.

Hybrid Data Source:

• Uses key_verifications_per_hour_v2 for historical pre-aggregated data across the full time range
• Uses key_verifications_raw_v2 for all raw data in the time range
• Unions both sources to ensure complete coverage, then aggregates to find top keys by last activity
• Result: Scans hourly aggregates + raw data only for the specified window instead of full 150M row scan

Key Difference:

• MV path: "Show me the 50 most recently used keys" (fast, but may miss keys active in time window) (last X timeframe)
• Historical path: "Show me all keys with activity in this time window" (accurate, uses pre-aggregated + raw data)

Changes

ClickHouse Schema:

• pkg/clickhouse/schema/024_keys_last_used_v1.sql - New materialized view and target table
• pkg/clickhouse/migrations/20260129000000.sql - Migration to create the materialized view

Query Logic:

• web/internal/clickhouse/src/keys/keys.ts - Rewritten query to use materialized view + hourly aggregates
• web/internal/clickhouse/src/latest_verifications.ts - Updated to use new materialized view
• web/internal/clickhouse/src/logs.ts - Query optimizations
• web/internal/clickhouse/src/verifications.ts - Query optimizations

Dependencies

Required:

• ClickHouse migration must be applied before deploying code changes
• Existing key_verifications_per_hour_v2 table (already in production)

No Breaking Changes:

• API interface remains unchanged
• Backward compatible with existing queries
• Materialized view adds <1% insert overhead

Fixes #4118

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Chore (refactoring code, technical debt, workflow improvements)
• Enhancement (small improvements)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How should this be tested?

Tested locally and in vercel preview environments with 150M verifications. Local clickhouse docker couldn't do more than 100M, so schema and query logic was tested locally, and time outs with higher volume was tested with clickhouse preview/vercel preview, using sentry errors for timeout confirmation.

Before: Timing out with 75M verification

After: 150M verifications loaded in under half of the timeout duration.

Since we're not backfilling this table, latest key activity should be extremely fast since there's less rows to scan in the new MV.

See screenshots for verification. I updated table names, so CH preview cluster will not have accurate data until populated.

Historical Timeframe:

Last 2 weeks:

Checklist

Required

• Filled out the "How to test" section in this PR
• Read Contributing Guide
• Self-reviewed my own code
• Commented on my code in hard-to-understand areas
• Ran pnpm build
• Ran pnpm fmt
• Ran make fmt on /go directory
• Checked for warnings, there are none
• Removed all console.logs
• Merged the latest changes from main onto my branch with git pull origin main
• My changes don't cause any responsiveness issues

Appreciated

• If a UI change was made: Added a screen recording or screenshots to this PR
• Updated the Unkey Docs if changes were necessary

[linear]

ENG-2129 Fix data fetching so users with large volumes can see them

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
dashboard	Ready	Preview, Comment	Feb 12, 2026 4:50pm

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
engineering	Ignored	Preview	Feb 12, 2026 4:50pm

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a ClickHouse AggregatingMergeTree table and materialized view for per-key latest verification, a client flag to choose MV vs time-frame lookup, refactors queries to use PREWHERE/WHERE separation, and updates TRPC and client hooks to pass and respect the new flag.

Changes

Cohort / File(s)	Summary
ClickHouse migrations & schema pkg/clickhouse/migrations/20260129000000.sql, pkg/clickhouse/schema/024_keys_last_used_v1.sql	Add default.key_last_used_v1 (AggregatingMergeTree with SimpleAggregateFunction columns, TTL 90 days, index_granularity=8192) and default.key_last_used_mv_v1 materialized view aggregating key_verifications_raw_v2 by workspace/key/identity into the table.
Keys overview query logic (server-side) web/internal/clickhouse/src/keys/keys.ts, web/apps/dashboard/lib/trpc/routers/api/keys/query-overview-logs/index.ts	Introduce useTimeFrameFilter param and branching: MV-backed fast path (reads key_last_used_v1) vs time-frame path (hourly + raw aggregation); switch to last_time-based cursors and produce combined counts/outcomes. Add .meta({ skipBatch: true }) to TRPC procedure.
Client hook & schema web/apps/dashboard/app/(app)/[workspaceSlug]/apis/[apiId]/_overview/components/table/hooks/use-logs-query.ts, web/apps/dashboard/app/(app)/[workspaceSlug]/apis/[apiId]/_overview/components/table/query-logs.schema.ts	Detect explicit time-frame filter in hook, pass useTimeFrameFilter in payload, add useTimeFrameFilter: z.boolean().optional() to payload schema, and include it in memo deps.
PREWHERE/WHERE refactor — logs & verifications web/internal/clickhouse/src/logs.ts, web/internal/clickhouse/src/verifications.ts, web/internal/clickhouse/src/latest_verifications.ts	Separate predicates into PREWHERE for indexed columns (workspace_id, time, key_space_id, key_id) and WHERE for non-indexed filters (tags, outcomes, requestIds, hosts, methods, statusCodes, path); apply across total_count and data queries.
Client post-processing & minor transforms web/internal/clickhouse/src/keys/keys.ts (client-side transforms)	Transform outcome_counts arrays into outcome_counts objects; ensure ordering/pagination uses last_time/last_request_id/last_tags from top_keys; adjust cursor logic for last_time-based pagination.

Sequence Diagram(s)

sequenceDiagram
    participant Client as Dashboard Client
    participant TRPC as TRPC Router
    participant CH as ClickHouse
    participant MV as Materialized View
    participant Raw as Raw Verifications

    Client->>TRPC: queryKeysOverviewLogs(useTimeFrameFilter, filters)
    TRPC->>CH: Execute SQL (PREWHERE workspace_id,time + WHERE other filters)
    Note over CH: top_keys CTE chooses source
    Note over CH: - MV path reads `key_last_used_v1` (populated by MV)
    Note over CH: - Time-frame path computes from hourly + raw tables
    Raw->>MV: INSERTs into `key_verifications_raw_v2`
    MV->>CH: Aggregate inserts into `key_last_used_v1`
    CH->>CH: Compute counts, outcome distributions, join with top_keys
    CH->>TRPC: Return paginated top keys with last_time, request_id, tags, counts
    TRPC->>Client: Respond with transformed results (outcome_counts object)

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	Title clearly describes the main change: improving queries for high-volume (150M+) ClickHouse operations.
Description check	✅ Passed	Description is comprehensive and follows template structure with detailed problem/solution explanation, testing approach, and checklist items completed.
Linked Issues check	✅ Passed	PR meets all objectives from issue #4118: implements batch-optimized queries and pagination for large volumes (150M+ verifications), eliminating timeouts through materialized views and hybrid data sources.
Out of Scope Changes check	✅ Passed	All changes directly support the core objective: ClickHouse schema additions, query rewrites for performance optimization, and supporting API parameter additions to enable dual-path query strategy.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch ENG-2129-high-volume-improvements

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[perkinsjr]

This seems to work with a large amount of data.

Will let someone like Andreas comment on the SQL implementation.

👍

[coderabbitai]

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

web/internal/clickhouse/src/logs.ts (1)

84-143: ⚠️ Potential issue | 🟠 Major

Normalize nullable array parameters to empty arrays before executing the ClickHouse query.

The Zod schema allows null for requestIds, hosts, excludeHosts, methods, and statusCodes (all defined as .nullable()). However, ClickHouse does not support Nullable(Array(T)) types—it will reject the parameter binding at runtime if null is passed. The current code directly spreads args into parameters without normalization, so any null value will cause a type mismatch error.

Normalize these parameters to empty arrays before query execution. The SQL pattern length({param: Array(...)}) > 0 is correct for detecting active filters; it just requires that null is replaced with [].

Suggested normalization

-    const parameters: Record<string, any> = { ...args };
+    const parameters: Record<string, any> = {
+      ...args,
+      requestIds: args.requestIds ?? [],
+      hosts: args.hosts ?? [],
+      excludeHosts: args.excludeHosts ?? [],
+      methods: args.methods ?? [],
+      statusCodes: args.statusCodes ?? [],
+    };

🤖 Fix all issues with AI agents

In `@pkg/clickhouse/migrations/20260129000000.sql`:
- Around line 11-13: The ORDER BY for the AggregatingMergeTree is incorrect:
change the ORDER BY clause that currently lists (`workspace_id`, `key_space_id`,
`key_id`, `time`) so it exactly matches the materialized view's GROUP BY
(`workspace_id`, `key_space_id`, `key_id`, `identity_id`); remove `time` (a
SimpleAggregateFunction) from ORDER BY and include `identity_id` instead so the
AggregatingMergeTree can merge/aggregate rows correctly.

In `@pkg/clickhouse/schema/024_keys_last_used_v1.sql`:
- Around line 9-20: The AggregatingMergeTree table `keys_last_used_v1` currently
orders by (`workspace_id`, `key_space_id`, `key_id`) which allows rows with
different `identity_id` to be merged, corrupting the SimpleAggregateFunction
results; update the table definition so the ORDER BY includes `identity_id`
(i.e., ORDER BY (`workspace_id`, `key_space_id`, `key_id`, `identity_id`)) to
ensure rows are only merged when the identity matches and preserve correct "last
used" aggregation for the `time`, `request_id`, `outcome`, and `tags` columns.

In `@web/internal/clickhouse/src/keys/keys.ts`:
- Around line 262-279: The top_keys CTE is incorrectly filtering using the
global last_time from the materialized view keys_last_used_v1 (via max(time) as
last_time), which excludes keys that were active inside the requested historical
window but had later activity; change the aggregation to compute a time-bound
per-hour/per-record max using a conditional aggregate (e.g., replace max(time)
as last_time with a maxIf(time, time BETWEEN {startTime: UInt64} AND {endTime:
UInt64}) or equivalent) and use that conditional value in the HAVING/ORDER
clauses so top_keys correctly reflects activity within the requested
startTime/endTime range; update any references to last_time, last_request_id,
and last_tags in the top_keys CTE to use corresponding time-bound aggregates
(e.g., anyLast* variants scoped to the same time condition) so the CTE no longer
relies on the global materialized-view timestamp.
- Around line 281-349: The query currently returns keys from top_keys even when
outcome/tag filters yield zero events; fix this by excluding zero-count keys
when filters are active: when either (${outcomeCondition}) or (${tagConditions})
is non-empty, change the final FROM to join only keys that have counts (replace
"FROM top_keys t" with "FROM top_keys t INNER JOIN aggregated_counts a ON
t.key_id = a.key_id" and use a.valid_count/a.error_count directly instead of
COALESCE), otherwise keep the original LEFT behavior; alternatively implement
the exclusion in aggregated_counts using HAVING sum(count) > 0 when filters are
applied so keys with zero matching events are omitted.

[coderabbitai]

Actionable comments posted: 3

🤖 Fix all issues with AI agents

In `@web/internal/clickhouse/src/keys/keys.ts`:
- Around line 249-257: The linter flags unnecessary template literals in the
pagination cursor build: update the cursorCondition assignment in the block that
checks args.cursorTime (the code manipulating cursorCondition based on
timeDirection) to use regular string literals instead of backtick template
strings—replace `AND t.last_time > {cursorTime: Nullable(UInt64)}` and `AND
t.last_time < {cursorTime: Nullable(UInt64)}` with normal quoted strings so
cursorCondition is set with plain string values.
- Around line 316-335: The original topKeysCTE path (used when
useTimeFrameFilter is false) is missing the cursor pagination filter: update the
topKeysCTE that queries default.keys_last_used_v1 (the block that defines
top_keys AS ...) to alias the table (e.g. AS t) and inject the existing
cursorCondition into its WHERE clause when args.cursorTime is present so
pagination uses t.last_time (or the same column name used by cursorCondition)
and respects timeDirection; ensure you preserve keyIdConditions, timeDirection
and LIMIT {limit: Int} and do not add a GROUP BY.
- Around line 398-419: When useTimeFrameFilter=false the original path leaves
keys from top_keys that have no matching aggregated_counts (producing zero
counts via LEFT JOIN+COALESCE) when outcome/tag filters are active; change the
original logic to mirror the new path by either (A) making the join between
top_keys and aggregated_counts (and/or outcome_counts) an INNER JOIN in the
original branch (e.g., in the keys_for_pagination/top_keys selection) so only
keys with matching events remain, or (B) add a HAVING clause on the aggregated
counts (e.g., HAVING COALESCE(a.valid_count,0)+COALESCE(a.error_count,0) > 0)
before ordering/pagination; update references to keys_with_usage, top_keys,
aggregated_counts, hourly_counts and recent_counts accordingly.

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

web/internal/clickhouse/src/logs.ts (1)

460-460: ⚠️ Potential issue | 🔴 Critical

Missing INTERVALS.threeDays definition causes runtime error.

getThreeDayLogsTimeseries references INTERVALS.threeDays, but this key doesn't exist in the INTERVALS object (lines 227-293). This will cause a runtime error when invoked.

🐛 Proposed fix: Add the missing interval definition

   day: {
     table: "default.api_requests_per_day_v2",
     step: "DAY",
     stepSize: 1,
   },
+  threeDays: {
+    table: "default.api_requests_per_day_v2",
+    step: "DAYS",
+    stepSize: 3,
+  },
   week: {
     table: "default.api_requests_per_day_v2",
     step: "DAYS",
     stepSize: 7,
   },

🤖 Fix all issues with AI agents

In `@web/internal/clickhouse/src/keys/keys.ts`:
- Around line 396-417: The query returns keys with both valid_count and
error_count equal to 0; update the main SELECT that builds results from
top_keys/aggregated_counts/outcome_counts to filter these out at the SQL level
by adding a HAVING clause (e.g., HAVING valid_count > 0 OR error_count > 0)
after the GROUP BY but before ORDER BY (${orderByClause}), so top_keys rows with
zero total requests are excluded for the rolling-window path; ensure the change
targets the query that defines top_keys / aggregated_counts and preserves the
existing COALESCE semantics for valid_count and error_count.

🧹 Nitpick comments (4)

web/internal/clickhouse/src/logs.ts (1)

47-48: Coding guideline violation: any type usage.

The Record<string, any> type violates the coding guidelines which state "Never compromise type safety: No any". Consider using a more specific type.

♻️ Suggested refactor

-    // biome-ignore lint/suspicious/noExplicitAny: Safe to use any here
-    const parameters: Record<string, any> = { ...args };
+    const parameters: Record<string, string | number | null | Array<string | number>> = { ...args };

As per coding guidelines: "Never compromise type safety: No any, no ! (non-null assertion), no as Type"

web/apps/dashboard/lib/trpc/routers/api/keys/query-overview-logs/index.ts (1)

46-57: Inconsistent nullish handling between ClickHouse and database queries.

Lines 56-57 correctly use ?? (nullish coalescing) for names and identities passed to ClickHouse, but lines 84-85 still use || (logical OR) for the same fields passed to queryApiKeys. This inconsistency could cause different behavior for empty arrays.

♻️ Suggested fix for consistency

     const { keys } = await queryApiKeys({
       apiId: input.apiId,
       workspaceId: ctx.workspace.id,
       keyIds: keyIdsFromLogs.map((id) => ({ operator: "is", value: id })),
-      names: input.names || null,
-      identities: input.identities || null,
+      names: input.names ?? null,
+      identities: input.identities ?? null,
     });

web/internal/clickhouse/src/keys/keys.ts (2)

281-282: Redundant time filter condition.

Line 281 (AND time >= {startTime: UInt64}) is redundant since line 282 already uses AND time BETWEEN {startTime: UInt64} AND {endTime: UInt64}, which includes the >= startTime check.

♻️ Remove redundant condition

       WHERE workspace_id = {workspaceId: String}
           AND key_space_id = {keyspaceId: String}
           AND (${keyIdConditions})
-          AND time >= {startTime: UInt64}
           AND time BETWEEN {startTime: UInt64} AND {endTime: UInt64}

---

258-263: Verify the rolling window detection threshold.

The 5-minute threshold (now - 5 * 60 * 1000) determines whether to use the fast MV path vs. the accurate time-frame path. This assumes queries with endTime within 5 minutes of now are "rolling" queries that can use the MV's global latest data.

Consider documenting this threshold or making it configurable, as it directly affects query accuracy vs. performance trade-offs.