Skip to content

Fix analytics history and log token accounting #2904

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
diegosouzapw merged 4 commits into from
May 29, 2026
Merged

Fix analytics history and log token accounting #2904

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
79f1133
fix(analytics): union daily_usage_summary for data beyond raw retenti…
htkarabulut May 25, 2026
2b20a29
fix(logs): avoid double-counting cached input tokens
htkarabulut May 29, 2026
d7f0e4f
chore: merge release/v3.8.6 into fix/analytics-history-union-v386
diegosouzapw May 29, 2026
35ba423
chore: parametrize date inputs in SQL UNION CTE queries to prevent SQ…
diegosouzapw May 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
181 changes: 157 additions & 24 deletions src/app/api/usage/analytics/route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { NextResponse } from "next/server";
import { requireManagementAuth } from "@/lib/api/requireManagementAuth";
import { getApiKeys } from "@/lib/db/apiKeys";
import { getDbInstance } from "@/lib/db/core";
import { getUserDatabaseSettings } from "@/lib/db/databaseSettings";

function getRangeStartIso(range: string): string | null {
const end = new Date();
Expand Down Expand Up @@ -327,6 +328,14 @@ export async function GET(request: Request) {
}
}

// Compute the raw-data cutoff: rows older than this may have been rolled up to
// daily_usage_summary and deleted from usage_history.
const dbSettings = getUserDatabaseSettings();
const rawRetentionDays = dbSettings.aggregation?.rawDataRetentionDays ?? 30;
const rawCutoff = new Date();
rawCutoff.setDate(rawCutoff.getDate() - rawRetentionDays);
const rawCutoffIso = rawCutoff.toISOString();

const conditions = [];
const params: Record<string, string> = {};

Expand All @@ -351,6 +360,93 @@ export async function GET(request: Request) {

const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(" AND ")}` : "";

// Build a UNION data source that merges recent raw rows with aggregated history.
// daily_usage_summary rows are included only when the query window extends before rawCutoffIso.
// The api_key filter is intentionally NOT applied to daily_usage_summary (api_key not stored there).
const needsAggregated = !sinceIso || sinceIso < rawCutoffIso;

const rawConditions: string[] = [];
if (sinceIso) rawConditions.push("timestamp >= @since");
if (untilIso) rawConditions.push("timestamp <= @until");
if (apiKeyWhere) rawConditions.push(apiKeyWhere);
const rawWhere = rawConditions.length > 0 ? `WHERE ${rawConditions.join(" AND ")}` : "";

// Aggregated rows only span dates within the requested window (no api_key filter).
const aggConditions: string[] = [];
if (sinceIso) {
// Use date comparison on the summary's date column (YYYY-MM-DD).
const sinceDate = sinceIso.split("T")[0];
aggConditions.push("date >= @sinceDate");
params.sinceDate = sinceDate;
}
if (untilIso) {
const untilDate = untilIso.split("T")[0];
aggConditions.push("date <= @untilDate");
params.untilDate = untilDate;
}
const aggWhere = aggConditions.length > 0 ? `WHERE ${aggConditions.join(" AND ")}` : "";
Comment on lines +368 to +387

@gemini-code-assist gemini-code-assist Bot May 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-critical critical

SQL Injection & Data Leakage / Double Counting Vulnerability

  1. SQL Injection: Direct string interpolation of sinceIso and untilIso splits into the SQL query (date >= '${sinceDate}') creates a SQL injection vulnerability because these values originate from user-controlled query parameters (startDate and endDate).
  2. Double Counting & Data Leakage: Without a boundary check, the query will select overlapping rows from both usage_history and daily_usage_summary for the recent period. Furthermore, since daily_usage_summary does not store API key or connection details, any query filtered by API key will leak other keys' aggregated usage for the overlapping period.

Fix: Use parameter binding for the date parameters and restrict the queries using the rawCutoff boundary to ensure non-overlapping datasets.

    const rawConditions: string[] = [];
    if (needsAggregated) {
      rawConditions.push("timestamp >= @rawCutoff");
      params.rawCutoff = rawCutoffIso;
    } else if (sinceIso) {
      rawConditions.push("timestamp >= @since");
    }
    if (untilIso) rawConditions.push("timestamp <= @until");
    if (apiKeyWhere) rawConditions.push(apiKeyWhere);
    const rawWhere = rawConditions.length > 0 ? `WHERE ${rawConditions.join(" AND ")}` : "";

    // Aggregated rows only span dates within the requested window and strictly before the raw cutoff
    // to prevent double-counting and data leakage (since api_key/connection filters don't apply to daily_usage_summary).
    const aggConditions: string[] = [];
    if (sinceIso) {
      aggConditions.push("date >= @sinceDate");
      params.sinceDate = sinceIso.split("T")[0];
    }
    if (untilIso) {
      aggConditions.push("date <= @untilDate");
      params.untilDate = untilIso.split("T")[0];
    }
    aggConditions.push("date < @rawCutoffDate");
    params.rawCutoffDate = rawCutoffIso.split("T")[0];
    const aggWhere = aggConditions.length > 0 ? `WHERE ${aggConditions.join(" AND ")}` : "";
References
  1. Never write raw SQL in routes — always go through src/lib/db/ domain modules. (link)


// Unified source CTE: columns aligned to usage_history shape needed by analytics queries.
// Fields not available in daily_usage_summary default to 0/NULL.
const unifiedSource = needsAggregated
? `(
SELECT
timestamp,
provider,
model,
tokens_input,
tokens_output,
tokens_cache_read,
tokens_cache_creation,
tokens_reasoning,
service_tier,
success,
latency_ms,
connection_id,
api_key_id,
api_key_name,
combo_name,
requested_model
FROM usage_history
${rawWhere}
UNION ALL
SELECT
date || 'T12:00:00.000Z' as timestamp,
provider,
model,
total_input_tokens as tokens_input,
total_output_tokens as tokens_output,
0 as tokens_cache_read,
0 as tokens_cache_creation,
0 as tokens_reasoning,
'standard' as service_tier,
1 as success,
0 as latency_ms,
NULL as connection_id,
NULL as api_key_id,
NULL as api_key_name,
NULL as combo_name,
NULL as requested_model
FROM daily_usage_summary
${aggWhere}
)`
: `(SELECT
timestamp, provider, model,
tokens_input, tokens_output,
tokens_cache_read, tokens_cache_creation, tokens_reasoning,
service_tier, success, latency_ms,
connection_id, api_key_id, api_key_name,
combo_name, requested_model
FROM usage_history
${whereClause}
)`;

// When using the unified source the WHERE filters are already embedded inside.
// For the original whereClause-based queries that still reference usage_history directly
// (e.g. fallbackRow, accountRows) we keep them as-is since they need joins or
// columns only present in usage_history.
const unifiedWhere = ""; // no additional WHERE needed — filters embedded in unifiedSource

// Fetch pricing data for cost calculation (no rows loaded)
const { getPricing } = await import("@/lib/db/settings");
const rawPricingByProvider = (await getPricing()) as PricingByProvider;
Expand Down Expand Up @@ -383,8 +479,8 @@ export async function GET(request: Request) {
COALESCE(AVG(latency_ms), 0) as avgLatencyMs,
COALESCE(MIN(timestamp), '') as firstRequest,
COALESCE(MAX(timestamp), '') as lastRequest
FROM usage_history
${whereClause}
FROM ${unifiedSource} AS _u
${unifiedWhere}
`
)
.get(params) as Record<string, unknown>;
Expand All @@ -398,8 +494,8 @@ export async function GET(request: Request) {
COALESCE(SUM(tokens_input), 0) as promptTokens,
COALESCE(SUM(tokens_output), 0) as completionTokens,
COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens
FROM usage_history
${whereClause}
FROM ${unifiedSource} AS _u
${unifiedWhere}
GROUP BY DATE(timestamp)
ORDER BY date ASC
`
Expand All @@ -419,8 +515,8 @@ export async function GET(request: Request) {
COALESCE(SUM(tokens_cache_read), 0) as cacheReadTokens,
COALESCE(SUM(tokens_cache_creation), 0) as cacheCreationTokens,
COALESCE(SUM(tokens_reasoning), 0) as reasoningTokens
FROM usage_history
${whereClause}
FROM ${unifiedSource} AS _u
${unifiedWhere}
GROUP BY DATE(timestamp), LOWER(provider), LOWER(model), serviceTier
ORDER BY date ASC
`
Expand Down Expand Up @@ -478,8 +574,8 @@ export async function GET(request: Request) {
COALESCE(AVG(latency_ms), 0) as avgLatencyMs,
COALESCE(SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END), 0) as successfulRequests,
COALESCE(MAX(timestamp), '') as lastUsed
FROM usage_history
${whereClause}
FROM ${unifiedSource} AS _u
${unifiedWhere}
GROUP BY LOWER(model), LOWER(provider), serviceTier
ORDER BY requests DESC
`
Expand All @@ -498,8 +594,8 @@ export async function GET(request: Request) {
COALESCE(SUM(tokens_cache_read), 0) as cacheReadTokens,
COALESCE(SUM(tokens_cache_creation), 0) as cacheCreationTokens,
COALESCE(SUM(tokens_reasoning), 0) as reasoningTokens
FROM usage_history
${whereClause}
FROM ${unifiedSource} AS _u
${unifiedWhere}
GROUP BY LOWER(provider), LOWER(model), serviceTier
`
)
Expand All @@ -516,8 +612,8 @@ export async function GET(request: Request) {
COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens,
COALESCE(AVG(latency_ms), 0) as avgLatencyMs,
COALESCE(SUM(CASE WHEN success = 1 THEN 1 ELSE 0 END), 0) as successfulRequests
FROM usage_history
${whereClause}
FROM ${unifiedSource} AS _u
${unifiedWhere}
GROUP BY LOWER(provider)
ORDER BY requests DESC
`
Expand Down Expand Up @@ -608,9 +704,8 @@ export async function GET(request: Request) {
COALESCE(SUM(tokens_cache_creation), 0) as cacheCreationTokens,
COALESCE(SUM(tokens_reasoning), 0) as reasoningTokens,
COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens
FROM usage_history

${whereClause}
FROM ${unifiedSource} AS _u
${unifiedWhere}
GROUP BY serviceTier, LOWER(provider), LOWER(model)
`
)
Expand Down Expand Up @@ -661,8 +756,8 @@ export async function GET(request: Request) {
strftime('%w', timestamp) as dayOfWeek,
COUNT(*) as requests,
COALESCE(SUM(tokens_input + tokens_output), 0) as totalTokens
FROM usage_history
${whereClause}
FROM ${unifiedSource} AS _u
${unifiedWhere}
GROUP BY DATE(timestamp), strftime('%w', timestamp)
)
GROUP BY dayOfWeek
Expand Down Expand Up @@ -1116,19 +1211,58 @@ export async function GET(request: Request) {
}

const presetSinceIso = getRangeStartIso(presetRange);
const presetConditions = [];
const presetParams: Record<string, string> = {};

// Build unified source for preset cost queries (same UNION logic as main query).
const presetNeedsAggregated = !presetSinceIso || presetSinceIso < rawCutoffIso;

const presetRawConds: string[] = [];
if (presetSinceIso) {
presetConditions.push("timestamp >= @presetSince");
presetRawConds.push("timestamp >= @presetSince");
presetParams.presetSince = presetSinceIso;
}
if (apiKeyWhere) {
presetConditions.push(apiKeyWhere);
presetRawConds.push(apiKeyWhere);
Object.assign(presetParams, params);
}
const presetRawWhere =
presetRawConds.length > 0 ? `WHERE ${presetRawConds.join(" AND ")}` : "";

const presetWhere =
presetConditions.length > 0 ? `WHERE ${presetConditions.join(" AND ")}` : "";
const presetAggConds: string[] = [];
if (presetSinceIso) {
const presetSinceDate = presetSinceIso.split("T")[0];
presetAggConds.push("date >= @presetSinceDate");
presetParams.presetSinceDate = presetSinceDate;
}
const presetAggWhere =
presetAggConds.length > 0 ? `WHERE ${presetAggConds.join(" AND ")}` : "";
Comment on lines +1219 to +1238

@gemini-code-assist gemini-code-assist Bot May 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-critical critical

SQL Injection & Double Counting in Preset Query

This block suffers from the same SQL injection and double-counting vulnerabilities as the main query. Direct string interpolation is used for presetSinceIso splits, and there is no boundary check to prevent overlapping data between usage_history and daily_usage_summary.

Fix: Apply parameter binding and restrict the datasets using the rawCutoff boundary.

        const presetRawConds: string[] = [];
        if (presetNeedsAggregated) {
          presetRawConds.push("timestamp >= @presetRawCutoff");
          presetParams.presetRawCutoff = rawCutoffIso;
        } else if (presetSinceIso) {
          presetRawConds.push("timestamp >= @presetSince");
          presetParams.presetSince = presetSinceIso;
        }
        if (apiKeyWhere) {
          presetRawConds.push(apiKeyWhere);
          Object.assign(presetParams, params);
        }
        const presetRawWhere =
          presetRawConds.length > 0 ? `WHERE ${presetRawConds.join(" AND ")}` : "";

        const presetAggConds: string[] = [];
        if (presetSinceIso) {
          presetAggConds.push("date >= @presetSinceDate");
          presetParams.presetSinceDate = presetSinceIso.split("T")[0];
        }
        presetAggConds.push("date < @presetRawCutoffDate");
        presetParams.presetRawCutoffDate = rawCutoffIso.split("T")[0];
        const presetAggWhere =
          presetAggConds.length > 0 ? `WHERE ${presetAggConds.join(" AND ")}` : "";
References
  1. Never write raw SQL in routes — always go through src/lib/db/ domain modules. (link)


const presetUnifiedSource = presetNeedsAggregated
? `(
SELECT timestamp, provider, model, service_tier,
tokens_input, tokens_output,
tokens_cache_read, tokens_cache_creation, tokens_reasoning
FROM usage_history
${presetRawWhere}
UNION ALL
SELECT
date || 'T12:00:00.000Z' as timestamp,
provider, model,
'standard' as service_tier,
total_input_tokens as tokens_input,
total_output_tokens as tokens_output,
0 as tokens_cache_read,
0 as tokens_cache_creation,
0 as tokens_reasoning
FROM daily_usage_summary
${presetAggWhere}
)`
: `(SELECT timestamp, provider, model, service_tier,
tokens_input, tokens_output,
tokens_cache_read, tokens_cache_creation, tokens_reasoning
FROM usage_history
${presetRawWhere}
)`;

const presetModelRows = db
.prepare(
Expand All @@ -1142,8 +1276,7 @@ export async function GET(request: Request) {
COALESCE(SUM(tokens_cache_read), 0) as cacheReadTokens,
COALESCE(SUM(tokens_cache_creation), 0) as cacheCreationTokens,
COALESCE(SUM(tokens_reasoning), 0) as reasoningTokens
FROM usage_history
${presetWhere}
FROM ${presetUnifiedSource} AS _pu
GROUP BY LOWER(model), LOWER(provider), serviceTier
`
)
Expand Down
11 changes: 11 additions & 0 deletions src/lib/db/cleanup.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

import { getDbInstance } from "./core";
import { getUserDatabaseSettings } from "./databaseSettings";
import { rollupUsageHistoryBeforeDate } from "@/lib/usage/aggregateHistory";

interface CleanupResult {
deleted: number;
Expand Down Expand Up @@ -85,9 +86,19 @@ export async function cleanupUsageHistory(): Promise<CleanupResult> {
const cutoffDate = new Date();
cutoffDate.setDate(cutoffDate.getDate() - retentionDays);
const cutoffISO = cutoffDate.toISOString();
const cutoffDateStr = cutoffISO.split("T")[0];

const result: CleanupResult = { deleted: 0, errors: 0 };

try {
// Roll up rows that are about to be deleted into daily_usage_summary so that
// the analytics route can still surface historical data via the UNION query.
await rollupUsageHistoryBeforeDate(cutoffDateStr);
} catch (err: unknown) {
// Non-fatal: log but continue with deletion so cleanup still runs.
console.error("[Cleanup] Error rolling up usage_history before deletion:", err);
}
Comment on lines +93 to +100

@gemini-code-assist gemini-code-assist Bot May 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Data Loss Risk on Rollup Failure

If rollupUsageHistoryBeforeDate fails (e.g., due to database locks or disk issues), the cleanup process currently logs the error but still proceeds to execute DELETE FROM usage_history. This will result in permanent loss of raw usage data without any aggregated backup.

Furthermore, the catch block here is dead code because rollupUsageHistoryBeforeDate internally catches its own errors and returns an AggregationResult object with errors > 0 instead of throwing.

Fix: Check the returned rollupResult.errors and abort the deletion if any errors occurred.

  const rollupResult = await rollupUsageHistoryBeforeDate(cutoffDateStr);
  if (rollupResult.errors > 0) {
    console.error("[Cleanup] Aborting usage_history deletion because rollup failed.");
    result.errors += rollupResult.errors;
    return result;
  }


try {
const stmt = db.prepare("DELETE FROM usage_history WHERE timestamp < ?");
const runResult = stmt.run(cutoffISO);
Expand Down
61 changes: 60 additions & 1 deletion src/lib/usage/aggregateHistory.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
/**
* Aggregation utility functions for usage data summarization.
* Rolls up quota_snapshots into hourly and daily summary tables.
* Rolls up usage_history (and quota_snapshots) into daily summary tables.
*
* @module lib/usage/aggregateHistory
*/
Expand Down Expand Up @@ -130,6 +130,65 @@ export async function rollupHourlyQuota(
return result;
}

/**
* Roll up usage_history into daily_usage_summary before raw rows are deleted.
* This is the authoritative rollup — sourced from actual per-request token data,
* not from quota_snapshots. Should be called before cleanupUsageHistory() deletes rows.
*
* The ON CONFLICT clause uses SUM so re-running is additive-safe: if a date already
* has a partial rollup (e.g. from a previous partial cleanup), new rows accumulate.
*
* @param beforeDate - ISO date string (YYYY-MM-DD). Rows strictly before this date are rolled up.
* @returns Aggregation result with counts
*/
export async function rollupUsageHistoryBeforeDate(beforeDate: string): Promise<AggregationResult> {
const db = getDbInstance();

const result: AggregationResult = {
processed: 0,
inserted: 0,
errors: 0,
};

try {
const aggregateQuery = `
INSERT INTO daily_usage_summary (provider, model, date, total_requests, total_input_tokens, total_output_tokens, total_cost)
SELECT
LOWER(provider) as provider,
LOWER(model) as model,
DATE(timestamp) as date,
COUNT(*) as total_requests,
COALESCE(SUM(tokens_input), 0) as total_input_tokens,
COALESCE(SUM(tokens_output), 0) as total_output_tokens,
0.0 as total_cost
FROM usage_history
WHERE DATE(timestamp) < ?

@gemini-code-assist gemini-code-assist Bot May 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Performance Bottleneck: Index Bypassed in WHERE Clause

Using DATE(timestamp) < ? in the WHERE clause prevents SQLite from using any index on the timestamp column, forcing a full table scan of usage_history. Since usage_history can grow extremely large, this will cause severe performance degradation during cleanup.

Since beforeDate is a standard YYYY-MM-DD string, a direct string comparison timestamp < ? is perfectly correct and allows SQLite to perform an efficient index range scan.

Fix: Compare timestamp directly.

Suggested change
WHERE DATE(timestamp) < ?
WHERE timestamp < ?

AND provider IS NOT NULL AND provider != ''
AND model IS NOT NULL AND model != ''
GROUP BY LOWER(provider), LOWER(model), DATE(timestamp)
ON CONFLICT(provider, model, date) DO UPDATE SET
total_requests = daily_usage_summary.total_requests + excluded.total_requests,
total_input_tokens = daily_usage_summary.total_input_tokens + excluded.total_input_tokens,
total_output_tokens = daily_usage_summary.total_output_tokens + excluded.total_output_tokens
`;

const stmt = db.prepare(aggregateQuery);
const runResult = stmt.run(beforeDate);

result.processed = runResult.changes;
result.inserted = runResult.changes;

console.log(
`[Aggregation] usage_history rollup: ${result.inserted} rows for dates before ${beforeDate}`
);
} catch (err: any) {
console.error("[Aggregation] usage_history rollup error:", err);
result.errors++;
}

return result;
}

/**
* Get the cutoff date for raw data based on retention settings.
* Data older than this should be aggregated and cleaned up.
Expand Down
Loading