Skip to content

feat(logging): unify request log retention and artifacts #851

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rdself wants to merge 1 commit into from
Closed

feat(logging): unify request log retention and artifacts #851

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,25 @@

## [Unreleased]

> [!WARNING]
> **BREAKING CHANGE: request logging, retention, and logging environment variables have been redesigned.**
> On the first startup after upgrading, OmniRoute archives legacy request logs from `DATA_DIR/logs/`, legacy `DATA_DIR/call_logs/`, and `DATA_DIR/log.txt` into `DATA_DIR/log_archives/*.zip`, then removes the deprecated layout and switches to the new unified artifact format under `DATA_DIR/call_logs/`.

### ✨ New Features

- **Unified Request Log Artifacts:** Request logging now stores one SQLite index row plus one JSON artifact per request under `DATA_DIR/call_logs/`, with optional pipeline capture embedded in the same file.

### ⚠️ Breaking Changes

- **Request Log Layout:** Removed the old multi-file `DATA_DIR/logs/` request log sessions and the `DATA_DIR/log.txt` summary file. New requests are written as single JSON artifacts in `DATA_DIR/call_logs/YYYY-MM-DD/`.
- **Logging Environment Variables:** Replaced `LOG_*`, `ENABLE_REQUEST_LOGS`, `CALL_LOGS_MAX`, `CALL_LOG_PAYLOAD_MODE`, and `PROXY_LOG_MAX_ENTRIES` with the new `APP_LOG_*` and `CALL_LOG_RETENTION_DAYS` configuration model.
- **Pipeline Toggle Setting:** Replaced the legacy `detailed_logs_enabled` setting with `call_log_pipeline_enabled`. New pipeline details are embedded inside the request artifact instead of being stored as separate `request_detail_logs` records.

### 🛠️ Maintenance

- **Legacy Request Log Upgrade Backup:** Upgrades now archive old `data/logs/`, legacy `data/call_logs/`, and `data/log.txt` layouts into `DATA_DIR/log_archives/*.zip` before removing the deprecated structure.
- **Streaming Usage Persistence:** Streaming requests now write a single `usage_history` row on completion instead of emitting a duplicate in-progress usage row with empty status metadata.

---

## [3.3.9] - 2026-03-31
Expand Down
60 changes: 43 additions & 17 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,30 @@ _Your universal API proxy — one endpoint, 67+ providers, zero downtime. Now wi

---

## Breaking Change: Unified Logging Upgrade

> [!WARNING]
> **This release changes both the on-disk request log layout and the logging environment variables.**
>
> If you are upgrading an existing instance:
>
> - Request logs now live in `DATA_DIR/call_logs/YYYY-MM-DD/` as **one JSON artifact per request**.
> - The old `DATA_DIR/logs/` session folders and `DATA_DIR/log.txt` summary file are removed.
> - On the first startup after upgrading, OmniRoute creates a safety backup at `DATA_DIR/log_archives/*.zip` before removing the deprecated request log layout.
> - Legacy logging env vars such as `LOG_TO_FILE`, `LOG_FILE_PATH`, `LOG_MAX_FILE_SIZE`, `LOG_RETENTION_DAYS`, `LOG_LEVEL`, `LOG_FORMAT`, `ENABLE_REQUEST_LOGS`, `CALL_LOGS_MAX`, `CALL_LOG_PAYLOAD_MODE`, and `PROXY_LOG_MAX_ENTRIES` are no longer supported.
> - Use the new env model instead:
> - `APP_LOG_TO_FILE`
> - `APP_LOG_FILE_PATH`
> - `APP_LOG_MAX_FILE_SIZE`
> - `APP_LOG_RETENTION_DAYS`
> - `APP_LOG_LEVEL`
> - `APP_LOG_FORMAT`
> - `CALL_LOG_RETENTION_DAYS`
>
> For release details and upgrade notes, see the [CHANGELOG](CHANGELOG.md).

---

## 🆕 What's New in v3.0.0

> **Upgrading from v2.9.5?** — See the [full CHANGELOG](CHANGELOG.md#300--2026-03-22-release-candidate--not-yet-merged-to-main) for all changes.
Expand Down Expand Up @@ -1276,22 +1300,22 @@ OmniRoute v2.0 is built as an operational platform, not just a relay proxy.

### ☁️ Deployment & Platform

| Feature | What It Does |
| ----------------------------- | --------------------------------------------------------- |
| 🌐 **Deploy Anywhere** | Localhost, VPS, Docker, Cloud environments |
| 🚇 **Cloudflare Tunnel** 🆕 | One-click Quick Tunnel integration from the dashboard |
| 🔑 **API Key Model Filtering** | Native /v1/models response filtered via assigned Bearer context roles |
| ⚡ **Smart Cache Bypass** | Configurable TTL heuristics and forced refetch controls |
| 🔄 **Backup/Restore** | Export/import and disaster recovery flows |
| 🧙 **Onboarding Wizard** | First-run guided setup |
| 🔧 **CLI Tools Dashboard** | One-click setup for popular coding tools |
| 🎮 **Model Playground** | Test any provider/model/endpoint from the dashboard |
| 🔏 **CLI Fingerprint Toggle** | Per-provider fingerprint matching in Settings > Security |
| 🌐 **i18n (30 languages)** | Full dashboard + docs language support with RTL coverage |
| 🧹 **Clear All Models** | One-click model list clearing in provider details |
| 👁️ **Sidebar Controls** 🆕 | Hide components and integrations from Appearance Settings |
| 📋 **Issue Templates** | Standardized GitHub templates for bugs and features |
| 📂 **Custom Data Directory** | `DATA_DIR` override for storage location |
| Feature | What It Does |
| ------------------------------ | --------------------------------------------------------------------- |
| 🌐 **Deploy Anywhere** | Localhost, VPS, Docker, Cloud environments |
| 🚇 **Cloudflare Tunnel** 🆕 | One-click Quick Tunnel integration from the dashboard |
| 🔑 **API Key Model Filtering** | Native /v1/models response filtered via assigned Bearer context roles |
| ⚡ **Smart Cache Bypass** | Configurable TTL heuristics and forced refetch controls |
| 🔄 **Backup/Restore** | Export/import and disaster recovery flows |
| 🧙 **Onboarding Wizard** | First-run guided setup |
| 🔧 **CLI Tools Dashboard** | One-click setup for popular coding tools |
| 🎮 **Model Playground** | Test any provider/model/endpoint from the dashboard |
| 🔏 **CLI Fingerprint Toggle** | Per-provider fingerprint matching in Settings > Security |
| 🌐 **i18n (30 languages)** | Full dashboard + docs language support with RTL coverage |
| 🧹 **Clear All Models** | One-click model list clearing in provider details |
| 👁️ **Sidebar Controls** 🆕 | Hide components and integrations from Appearance Settings |
| 📋 **Issue Templates** | Standardized GitHub templates for bugs and features |
| 📂 **Custom Data Directory** | `DATA_DIR` override for storage location |

### Feature Deep Dive

Expand Down Expand Up @@ -1811,7 +1835,9 @@ opencode

**No request logs**

- Set `ENABLE_REQUEST_LOGS=true` in `.env`
- Request artifacts are written to `DATA_DIR/call_logs/` as one JSON file per request
- Enable pipeline capture from Dashboard → Logs → Request Logs if you need detailed per-stage payloads
- Set `APP_LOG_TO_FILE=true` if you also want application console logs in `logs/application/app.log`

**Connection test shows "Invalid" for OpenAI-compatible providers**

Expand Down
45 changes: 20 additions & 25 deletions open-sse/handlers/chatCore.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import {
import { HTTP_STATUS, PROVIDER_MAX_TOKENS } from "../config/constants.ts";
import { classifyProviderError, PROVIDER_ERROR_TYPES } from "../services/errorClassifier.ts";
import { updateProviderConnection } from "@/lib/db/providers";
import { isDetailedLoggingEnabled, saveRequestDetailLog } from "@/lib/db/detailedLogs";
import { isDetailedLoggingEnabled } from "@/lib/db/detailedLogs";
import { logAuditEvent } from "@/lib/compliance";
import { handleBypassRequest } from "../utils/bypassHandler.ts";
import {
Expand Down Expand Up @@ -533,6 +533,24 @@ export async function handleChatCore({
claudeCacheUsageMeta?: Record<string, unknown>;
}) => {
const callLogId = generateRequestId();
const pipelinePayloads = detailedLoggingEnabled ? reqLogger?.getPipelinePayloads?.() : null;

if (pipelinePayloads) {
Comment on lines 533 to +538

Copilot AI Mar 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

persistAttemptLogs references reqLogger before reqLogger is declared/initialized later in the same scope. This is fragile (TDZ risk if the function is ever invoked earlier) and commonly triggers TS “used before declaration” diagnostics depending on build settings. Consider declaring let reqLogger before persistAttemptLogs (assigning it later), or moving persistAttemptLogs below the reqLogger initialization / passing the pipeline payloads into persistAttemptLogs explicitly.

Copilot uses AI. Check for mistakes.
if (providerResponse !== undefined) {
pipelinePayloads.providerResponse = providerResponse as Record<string, unknown>;
}
if (clientResponse !== undefined) {
pipelinePayloads.clientResponse = clientResponse as Record<string, unknown>;
}
if (error) {
pipelinePayloads.error = {
...(typeof pipelinePayloads.error === "object" && pipelinePayloads.error
? (pipelinePayloads.error as Record<string, unknown>)
: {}),
message: error,
};
}
}
Comment on lines +536 to +553

@gemini-code-assist gemini-code-assist Bot Mar 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The pipelinePayloads object is only updated if it's non-null. If the request logger hasn't recorded any intermediate steps yet, getPipelinePayloads() returns null, causing the final provider/client responses and errors to be omitted from the pipeline capture even when detailed logging is enabled. This logic should be adjusted to ensure the final payloads are captured if detailed logging is active.

    let pipelinePayloads = detailedLoggingEnabled ? reqLogger?.getPipelinePayloads?.() : null;

    if (detailedLoggingEnabled) {
      if (providerResponse !== undefined || clientResponse !== undefined || error) {
        pipelinePayloads = pipelinePayloads || {};
        if (providerResponse !== undefined) {
          pipelinePayloads.providerResponse = providerResponse as Record<string, unknown>;
        }
        if (clientResponse !== undefined) {
          pipelinePayloads.clientResponse = clientResponse as Record<string, unknown>;
        }
        if (error) {
          pipelinePayloads.error = {
            ...(typeof pipelinePayloads.error === "object" && pipelinePayloads.error
              ? (pipelinePayloads.error as Record<string, unknown>)
              : {}),
            message: error,
          };
        }
      }
    }


saveCallLog({
id: callLogId,
Expand Down Expand Up @@ -565,31 +583,8 @@ export async function handleChatCore({
apiKeyId: apiKeyInfo?.id || null,
apiKeyName: apiKeyInfo?.name || null,
noLog: noLogEnabled,
pipelinePayloads,
}).catch(() => {});

if (!detailedLoggingEnabled) {
return;
}

try {
saveRequestDetailLog({
call_log_id: callLogId,
client_request: clientRawRequest?.body ?? body,
translated_request: providerRequest ?? null,
provider_response: providerResponse ?? null,
client_response: clientResponse ?? null,
provider,
model,
source_format: sourceFormat,
target_format: targetFormat,
duration_ms: Date.now() - startTime,
api_key_id: apiKeyInfo?.id || null,
no_log: noLogEnabled,
});
} catch (err) {
const errMessage = err instanceof Error ? err.message : String(err);
log?.debug?.("DETAIL_LOG", `Failed to save detailed log: ${errMessage}`);
}
};

// Primary path: merge client model id + alias target so config on either key applies; resolved
Expand Down
13 changes: 6 additions & 7 deletions open-sse/utils/logger.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
* Structured console logger utility for omniroute.
*
* Provides consistent, machine-parseable log output across the codebase.
* Supports two output formats controlled by LOG_FORMAT env var:
* Supports two output formats controlled by APP_LOG_FORMAT env var:
* - "text" (default): [LEVEL] [TAG] message {metadata}
* - "json": Single-line JSON objects for log aggregators
*
Expand All @@ -18,17 +18,16 @@
* reqLog.info("AUTH", "Token refreshed", { provider: "claude" });
*
* Environment variables:
* LOG_LEVEL — minimum level: debug | info | warn | error (default: info)
* LOG_FORMAT — output format: text | json (default: text)
* APP_LOG_LEVEL — minimum level: debug | info | warn | error (default: info)
* APP_LOG_FORMAT — output format: text | json (default: text)
*/
import { getAppLogFormat, getAppLogLevel } from "../../src/lib/logEnv";

const LEVELS = { debug: 0, info: 1, warn: 2, error: 3 };

const currentLevel =
LEVELS[((typeof process !== "undefined" && process.env?.LOG_LEVEL) || "info").toLowerCase()] ??
LEVELS.info;
const currentLevel = LEVELS[getAppLogLevel("info").toLowerCase()] ?? LEVELS.info;

const jsonFormat = typeof process !== "undefined" && process.env?.LOG_FORMAT === "json";
const jsonFormat = getAppLogFormat("text") === "json";

let requestCounter = 0;

Expand Down
Loading