feat: support ClickHouse log database

[Calcium-Ion]

⚠️ 提交说明 / PR Notice

Important

• 请提供人工撰写的简洁摘要，避免直接粘贴未经整理的 AI 输出。

📝 变更描述 / Description

本次变更为 LOG_SQL_DSN 增加 ClickHouse 日志库支持，主业务库 SQL_DSN 仍限制为 SQLite、MySQL 或 PostgreSQL，避免将 ClickHouse 用作事务型主库。

日志表在 ClickHouse 下改为手写 MergeTree 建表，按 created_at, request_id 排序，避免依赖 ClickHouse 不支持的自增 id。日志写入统一补齐 request_id，查询侧在 ClickHouse 分支使用 created_at DESC, request_id DESC 保持分页排序稳定，同时继续回填数字 id，保证前端接口兼容。

同时新增可选的 LOG_SQL_CLICKHOUSE_TTL_DAYS 配置：默认或设置为 0 时不自动删除日志；设置为正整数时通过 ClickHouse 表级 TTL 自动清理过期日志。docker-compose.yml 中补充了 ClickHouse 本地部署示例。

🚀 变更类型 / Type of change

• 🐛 Bug 修复 (Bug fix) - 请关联对应 Issue，避免将设计取舍、理解偏差或预期不一致直接归类为 bug
• ✨ 新功能 (New feature) - 重大特性建议先通过 Issue 沟通
• ⚡ 性能优化 / 重构 (Refactor)
• 📝 文档更新 (Documentation)

🔗 关联任务 / Related Issue

• Partially addresses 当前系统通用调用日志全量持久化入库无自动清理机制导致数据堆积数据几十个G #5342 by adding ClickHouse-backed log storage with optional TTL retention for high-volume usage logs.
• Partially addresses 增加自动删除日志功能，保持数据库清爽。 #238 by adding configurable automatic expiration for logs stored in ClickHouse.

✅ 提交前检查项 / Checklist

• 人工确认: 我已亲自整理并撰写此描述，没有直接粘贴未经处理的 AI 输出。
• 非重复提交: 我已搜索现有的 Issues 与 PRs，确认不是重复提交。
• Bug fix 说明: 若此 PR 标记为 Bug fix，我已提交或关联对应 Issue，且不会将设计取舍、预期不一致或理解偏差直接归类为 bug。
• 变更理解: 我已理解这些更改的工作原理及可能影响。
• 范围聚焦: 本 PR 未包含任何与当前任务无关的代码改动。
• 本地验证: 已在本地运行并通过测试或手动验证，维护者可以据此复核结果。
• 安全合规: 代码中无敏感凭据，且符合项目代码规范。

📸 运行证明 / Proof of Work

本地测试通过：

go test ./common ./model ./controller ./service

ok  	github.com/QuantumNous/new-api/common
ok  	github.com/QuantumNous/new-api/model
ok  	github.com/QuantumNous/new-api/controller
ok  	github.com/QuantumNous/new-api/service

Summary by CodeRabbit

• New Features
  • Added optional ClickHouse-backed logging with TTL configuration options.
• Refactor
  • Centralized database selection by “main” vs “log” role and database type (replacing per-database flags).
  • Standardized request ID generation across request handling and relay responses, using deterministic prefixes.
• Bug Fixes
  • Improved logging consistency (request-id initialization) plus more robust log aggregation, ordering, and ClickHouse-specific deletion behavior.
• Documentation / Config
  • Updated setup guidance and docker-compose logging examples.
• Tests
  • Removed the Waffo Pancake test suite.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4b8852d3-ea2a-4b71-9987-6b59c1af3268

📥 Commits

Reviewing files that changed from the base of the PR and between 188c465 and 7ec9cdd.

📒 Files selected for processing (1)

• model/log.go

---

Walkthrough

Replaces exported boolean database-flag variables (UsingSQLite, UsingPostgreSQL, UsingMySQL, LogSqlType) in common/database.go with a typed DatabaseType abstraction and centralized getter/setter functions. Adds ClickHouse as an optional separate log database with GORM driver integration, schema migration with configurable TTL, and ClickHouse-aware query paths in model/log.go. Centralizes request-ID generation into common.NewRequestId(). All call-sites across models, controllers, tests, and configuration are migrated to the new API.

Changes

DatabaseType Abstraction, ClickHouse Log DB, and Request-ID Centralization

Layer / File(s)	Summary
DatabaseType abstraction in common common/database.go, AGENTS.md, CLAUDE.md	Defines DatabaseType string type and constants for MySQL/SQLite/PostgreSQL/ClickHouse; replaces exported boolean flags with internal mainDatabaseType/logDatabaseType state and public MainDatabaseType, LogDatabaseType, SetMainDatabaseType, SetLogDatabaseType, SetDatabaseTypes, UsingMainDatabase, UsingLogDatabase functions. Documentation updated to match.
Centralized request-ID generation common/utils.go, middleware/request-id.go, relay/common/relay_info.go	Adds SHA-256-hashed build-path-derived prefix with random fallback to NewRequestId(); zero-pads nanoseconds in GetTimeString to 9 digits; removes inline ID construction from middleware and relay, replacing with common.NewRequestId().
DB initialization, chooseDB rewrite, and ClickHouse migration model/main.go, go.mod	Adds ClickHouse GORM driver dependency; rewrites chooseDB to return DatabaseType and block ClickHouse for the primary DB; updates InitDB/InitLogDB to call SetMainDatabaseType/SetLogDatabaseType; adds migrateLOGDB ClickHouse path creating the logs table with optional configurable TTL via ALTER TABLE; migrates all migration guards from legacy booleans to new predicates.
ClickHouse-aware log operations model/log.go	Adds ensureLogRequestId and createLog helpers ensuring RequestId is set before any insert; introduces clickHouseLogOrder/assignDisplayLogIds for ClickHouse query ordering and sequential ID normalization; updates GetLogByTokenId, GetAllLogs, GetUserLogs to branch on ClickHouse ordering; switches aggregations to COALESCE; adds ClickHouse-specific ALTER TABLE logs DELETE path in DeleteOldLog.
Model call-site predicate migration model/ability.go, model/channel.go, model/checkin.go, model/db_time.go, model/redemption.go, model/subscription.go, model/topup.go, model/usedata_rankings.go	Replaces all common.UsingSQLite, common.UsingPostgreSQL, common.UsingMySQL boolean accesses with common.UsingMainDatabase(common.DatabaseTypeX) calls across all database-type branch conditions.
Controller, test, and config updates controller/setup.go, controller/model_list_test.go, controller/token_test.go, model/task_cas_test.go, service/task_billing_test.go, service/waffo_pancake_test.go, docker-compose.yml	GetSetup derives the DatabaseType field directly from string(common.MainDatabaseType()); all test DB setup/teardown migrates to SetDatabaseTypes; waffo_pancake_test.go removed; docker-compose gains commented ClickHouse service, volume, and LOG_SQL_DSN environment variables.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• QuantumNous/new-api#2808: Introduces GetDBTimestamp() in the subscription billing system, which this PR updates to use the new common.UsingMainDatabase(...) predicate API.
• QuantumNous/new-api#4401: Extensively restructures controller/token_test.go including openTokenControllerTestDB, directly intersecting with this PR's test DB setup migration.
• QuantumNous/new-api#4935: Updates SQLite subscription-plan migration in model/main.go with database-type refactoring that aligns with this PR's common.UsingMainDatabase(...) gating changes.

Suggested labels

enhancement

Suggested reviewers

• seefs001

Poem

🐇 Hoppy news from the warren below,
Boolean flags? We've let those go!
DatabaseType typed and tidy and true,
ClickHouse now logs every query you do.
Request IDs hashed from the build path's name —
One NewRequestId() call, never the same! 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 13.64% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: support ClickHouse log database' directly and accurately summarizes the main objective of this PR—adding ClickHouse support as a log database backend.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch log-sql-dsn-clickhouse

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@model/log.go`:
- Around line 670-685: The ClickHouse deletion path in the if block checking
common.UsingLogDatabase(common.DatabaseTypeClickHouse) does not respect a limit
parameter when deleting old logs, unlike the non-ClickHouse branch. Modify the
LOG_DB.Exec call that executes the ALTER TABLE logs DELETE query to include a
LIMIT clause using the limit parameter that should be available as a function
parameter, preventing unbounded deletion operations and avoiding the enqueuing
of oversized mutations in ClickHouse.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 7dd470bb-8dff-420f-817b-790119e6b4b7

📥 Commits

Reviewing files that changed from the base of the PR and between 354d0fe and 188c465.

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

📒 Files selected for processing (24)

• AGENTS.md
• CLAUDE.md
• common/database.go
• common/utils.go
• controller/model_list_test.go
• controller/setup.go
• controller/token_test.go
• docker-compose.yml
• go.mod
• middleware/request-id.go
• model/ability.go
• model/channel.go
• model/checkin.go
• model/db_time.go
• model/log.go
• model/main.go
• model/redemption.go
• model/subscription.go
• model/task_cas_test.go
• model/topup.go
• model/usedata_rankings.go
• relay/common/relay_info.go
• service/task_billing_test.go
• service/waffo_pancake_test.go

💤 Files with no reviewable changes (1)

• service/waffo_pancake_test.go