diff --git a/docs-site/src/content/docs/guides/web-dashboard.md b/docs-site/src/content/docs/guides/web-dashboard.md index 730425fe0f0..739e6acefc6 100644 --- a/docs-site/src/content/docs/guides/web-dashboard.md +++ b/docs-site/src/content/docs/guides/web-dashboard.md @@ -39,7 +39,7 @@ bun run dev:gui | **Models** | Toggle native GPT and routed models, set provider allowlists and context caps, choose v1/base/v2, and configure the v2 thread limit. Configured providers stay visible as zero-model groups when discovery is off or returns no rows. | | **Logs** | Auto-refresh recent requests with tokens, requested effort and (when available) effective outbound effort, resolved model, provider, status, request id, duration, and error details. The detail view includes the exact reasoning wire field when the adapter emits one. Filter by opaque conversation/session id (when the client sends one) to total tokens and estimated list-price cost for the currently loaded Logs ring. | | **Usage / Debug** | Inspect token-usage coverage and trends, or enable opt-in provider transport and usage-extraction diagnostics. | -| **Storage** | Read-only CODEX_HOME disk breakdown (sessions, archives, DBs, attachments). Optional archived cleanup: preview the oldest N%, then quarantine to `CODEX_HOME/.trash` (default) or permanently delete behind an explicit checkbox. **Auto-cleanup policy** is opt-in and **default OFF** (`storageCleanupPolicy.enabled`); configure threshold/target/schedule/mode on the Storage page, or trigger **Run now**. Active sessions stay read-only. Cleanup is refused while Codex holds the newest/active `state_*.sqlite` locked. Quarantined files are **not** restorable from the dashboard — recover manually from `.trash//` using `manifest.json` if needed. | +| **Storage** | Read-only CODEX_HOME disk breakdown (sessions, archives, DBs, attachments). Optional archived cleanup: preview the oldest N%, then quarantine to `CODEX_HOME/.trash` (default) or permanently delete behind an explicit checkbox. **Auto-cleanup policy** is opt-in and **default OFF** (`storageCleanupPolicy.enabled`); configure threshold/target/schedule/mode on the Storage page, or trigger **Run now**. Quarantined entries can be restored from the Storage page (JSONL + threads). Active sessions stay read-only. Cleanup and restore are refused while Codex holds the newest/active `state_*.sqlite` locked. | | **Stop** | Gracefully stop the proxy and installed background service, restore native Codex, and exit (`POST /api/stop`). | ### Linking to a section diff --git a/docs-site/src/content/docs/ja/guides/web-dashboard.md b/docs-site/src/content/docs/ja/guides/web-dashboard.md index 737a4342f2a..2a253dc9c75 100644 --- a/docs-site/src/content/docs/ja/guides/web-dashboard.md +++ b/docs-site/src/content/docs/ja/guides/web-dashboard.md @@ -39,7 +39,7 @@ bun run dev:gui | **モデル** | ネイティブ GPT とルーティングモデルをオン/オフし、プロバイダー許可リストとコンテキスト上限、v1/base/v2、v2 スレッド数を設定します。 | | **ログ** | トークン、要求された強度と（利用可能な場合は）実際に送信された強度、実際のモデル、プロバイダー、状態、リクエスト ID、所要時間、エラー詳細を含む最近のリクエストを自動更新します。アダプターが reasoning パラメーターを送信した場合、詳細表示に正確な wire field も表示されます。 | | **使用量 / デバッグ** | トークン使用量の測定範囲と推移を見るか、オプションのプロバイダートランスポート/使用量抽出診断をオンにします。 | -| **ストレージ** | CODEX_HOME のディスク内訳（セッション、アーカイブ、DB、添付）を読み取り専用で表示。任意のアーカイブクリーンアップ: 最古 N% をプレビューし、既定では `CODEX_HOME/.trash` へ隔離、または明示チェックで完全削除。**自動クリーンアップ方針**はオプトインで**既定 OFF**（`storageCleanupPolicy.enabled`）。Storage ページでしきい値/目標/スケジュール/モードを設定するか **今すぐ実行**。アクティブセッションは読み取り専用。最新/アクティブな `state_*.sqlite` がロック中は拒否。隔離分はダッシュボードから復元不可 — 必要なら `.trash//` と `manifest.json` から手動復旧。 | +| **ストレージ** | CODEX_HOME のディスク内訳（セッション、アーカイブ、DB、添付）を読み取り専用で表示。任意のアーカイブクリーンアップ: 最古 N% をプレビューし、既定では `CODEX_HOME/.trash` へ隔離、または明示チェックで完全削除。**自動クリーンアップ方針**はオプトインで**既定 OFF**（`storageCleanupPolicy.enabled`）。Storage ページでしきい値/目標/スケジュール/モードを設定するか **今すぐ実行**。隔離エントリは Storage ページから復元可能（JSONL + スレッド）。アクティブセッションは読み取り専用。最新/アクティブな `state_*.sqlite` がロック中はクリーンアップと復元を拒否。 | | **停止** | プロキシとインストールされたバックグラウンドサービスを正常終了しネイティブ Codex を復元した後終了します(`POST /api/stop`)。 | ### セクションへのリンク diff --git a/docs-site/src/content/docs/ko/guides/web-dashboard.md b/docs-site/src/content/docs/ko/guides/web-dashboard.md index 6d7ca6baf78..8ab11d4e6d1 100644 --- a/docs-site/src/content/docs/ko/guides/web-dashboard.md +++ b/docs-site/src/content/docs/ko/guides/web-dashboard.md @@ -39,7 +39,7 @@ bun run dev:gui | **Models** | 네이티브 GPT와 라우팅 모델을 켜고 끄고, 프로바이더 allowlist와 컨텍스트 상한, v1/base/v2, v2 thread 수를 설정합니다. | | **Logs** | 토큰, 요청한 강도와 (사용 가능한 경우) 실제 전송 강도, 실제 모델, 프로바이더, 상태, 요청 id, 소요 시간, 오류 상세가 포함된 최근 요청을 자동 갱신합니다. 어댑터가 reasoning 매개변수를 전송한 경우 상세 보기에 정확한 wire field도 표시됩니다. 클라이언트가 보낸 불투명 대화/세션 id로 필터하면 현재 로드된 Logs 링의 토큰·추정 정가 합계를 볼 수 있습니다. | | **Usage / Debug** | 토큰 사용량의 측정 범위와 추이를 보거나, 선택적 프로바이더 전송/사용량 추출 진단을 켭니다. | -| **Storage** | CODEX_HOME 디스크 사용량(세션, 보관, DB, 첨부)을 읽기 전용으로 표시합니다. 선택적 보관 정리: 가장 오래된 N%를 미리본 뒤 기본으로 `CODEX_HOME/.trash`에 격리하거나, 명시 체크 후 영구 삭제합니다. **자동 정리 정책**은 opt-in이며 **기본 OFF**(`storageCleanupPolicy.enabled`)입니다. Storage 페이지에서 임계값/목표/일정/모드를 설정하거나 **지금 실행**하세요. 활성 세션은 읽기 전용입니다. Codex가 최신/활성 `state_*.sqlite`를 잠그면 거절합니다. 격리 파일은 대시보드에서 복원할 수 없습니다 — 필요하면 `.trash//`와 `manifest.json`으로 수동 복구하세요. | +| **Storage** | CODEX_HOME 디스크 사용량(세션, 보관, DB, 첨부)을 읽기 전용으로 표시합니다. 선택적 보관 정리: 가장 오래된 N%를 미리본 뒤 기본으로 `CODEX_HOME/.trash`에 격리하거나, 명시 체크 후 영구 삭제합니다. **자동 정리 정책**은 opt-in이며 **기본 OFF**(`storageCleanupPolicy.enabled`)입니다. Storage 페이지에서 임계값/목표/일정/모드를 설정하거나 **지금 실행**하세요. Storage 페이지에서 격리 항목을 복원할 수 있습니다(JSONL + 스레드). 활성 세션은 읽기 전용입니다. Codex가 최신/활성 `state_*.sqlite`를 잠그면 정리와 복원을 거절합니다. | | **Stop** | 프록시와 설치된 백그라운드 서비스를 정상 종료하고 네이티브 Codex를 복원한 뒤 끝냅니다(`POST /api/stop`). | ### 섹션으로 바로 가기 diff --git a/docs-site/src/content/docs/ru/guides/web-dashboard.md b/docs-site/src/content/docs/ru/guides/web-dashboard.md index fda9b25d2d0..70d8e9e22a1 100644 --- a/docs-site/src/content/docs/ru/guides/web-dashboard.md +++ b/docs-site/src/content/docs/ru/guides/web-dashboard.md @@ -39,7 +39,7 @@ bun run dev:gui | **Models** | Включение и отключение нативных GPT и маршрутизируемых моделей, настройка списков разрешённых провайдеров и лимитов контекста, выбор v1/base/v2 и настройка лимита потоков v2. | | **Logs** | Автообновляемый список недавних запросов: токены, запрошенный и, когда доступен, фактически отправленный уровень рассуждений, фактическая модель, провайдер, статус, id запроса, длительность и подробности ошибок. Если адаптер отправляет параметр рассуждений, в подробностях также отображается точное wire-поле. Можно фильтровать по непрозрачному id диалога/сессии (если клиент его передаёт) и суммировать токены и оценочную стоимость по прайс-листу в пределах загруженного кольца Logs. | | **Usage / Debug** | Просмотр покрытия и трендов расхода токенов либо включение опциональной диагностики транспорта провайдеров и извлечения данных об использовании. | -| **Storage** | Только чтение разбивки диска CODEX_HOME (сессии, архивы, БД, вложения). Опциональная очистка архива: предпросмотр самых старых N%, затем карантин в `CODEX_HOME/.trash` (по умолчанию) или безвозвратное удаление по явному флажку. **Политика автоочистки** — opt-in и **по умолчанию ВЫКЛ** (`storageCleanupPolicy.enabled`); порог/цель/расписание/режим на странице Storage или **Запустить сейчас**. Активные сессии только для чтения. Очистка отклоняется, пока Codex держит блокировку новейшего/активного `state_*.sqlite`. Из карантина **нельзя** восстановить через панель — вручную из `.trash//` и `manifest.json`. | +| **Storage** | Только чтение разбивки диска CODEX_HOME (сессии, архивы, БД, вложения). Опциональная очистка архива: предпросмотр самых старых N%, затем карантин в `CODEX_HOME/.trash` (по умолчанию) или безвозвратное удаление по явному флажку. **Политика автоочистки** — opt-in и **по умолчанию ВЫКЛ** (`storageCleanupPolicy.enabled`); порог/цель/расписание/режим на странице Storage или **Запустить сейчас**. Записи карантина можно восстановить со страницы Storage (JSONL + threads). Активные сессии только для чтения. Очистка и восстановление отклоняются, пока Codex держит блокировку новейшего/активного `state_*.sqlite`. | | **Stop** | Корректная остановка прокси и установленного фонового сервиса, восстановление нативного Codex и выход (`POST /api/stop`). | ### Ссылки на разделы diff --git a/docs-site/src/content/docs/zh-cn/guides/web-dashboard.md b/docs-site/src/content/docs/zh-cn/guides/web-dashboard.md index 4d675213140..d799eea5393 100644 --- a/docs-site/src/content/docs/zh-cn/guides/web-dashboard.md +++ b/docs-site/src/content/docs/zh-cn/guides/web-dashboard.md @@ -38,7 +38,7 @@ bun run dev:gui | **Models** | 开关原生 GPT 与路由模型，配置 provider allowlist、上下文上限、v1/base/v2 以及 v2 thread 数量。 | | **Logs** | 自动刷新近期请求，显示 token、请求强度以及（可用时）实际发送强度、实际模型、provider、状态、request id、耗时和错误详情。适配器发送 reasoning 参数时，详情中还会显示准确的 wire field。可按不透明会话/对话 ID（客户端提供时）筛选，并对当前已加载的 Logs 环形缓冲合计 token 与估算标价成本。 | | **Usage / Debug** | 查看 token usage 覆盖率与趋势，或启用可选的 provider transport 和 usage 提取诊断。 | -| **Storage** | 只读查看 CODEX_HOME 磁盘占用（会话、归档、数据库、附件）。可选归档清理：预览最旧 N%，默认隔离到 `CODEX_HOME/.trash`，或勾选后永久删除。**自动清理策略**为可选且**默认关闭**（`storageCleanupPolicy.enabled`）；可在 Storage 页配置阈值/目标/计划/模式，或点「立即运行」。活动会话保持只读。Codex 锁定最新/活动的 `state_*.sqlite` 时拒绝清理。隔离文件**不能**从仪表盘恢复——如需恢复，请根据 `manifest.json` 将文件从 `.trash//` 手动移回原位置。 | +| **Storage** | 只读查看 CODEX_HOME 磁盘占用（会话、归档、数据库、附件）。可选归档清理：预览最旧 N%，默认隔离到 `CODEX_HOME/.trash`，或勾选后永久删除。**自动清理策略**为可选且**默认关闭**（`storageCleanupPolicy.enabled`）；可在 Storage 页配置阈值/目标/计划/模式，或点「立即运行」。可在 Storage 页从隔离区恢复（JSONL + 线程）。活动会话保持只读。Codex 锁定最新/活动的 `state_*.sqlite` 时拒绝清理与恢复。 | | **Stop** | 优雅地停止代理和已安装的后台服务，恢复原生 Codex 并退出（`POST /api/stop`）。 | ### 链接到某个部分 diff --git a/gui/src/i18n/de.ts b/gui/src/i18n/de.ts index 435768a4c5a..d206e2c2d1e 100644 --- a/gui/src/i18n/de.ts +++ b/gui/src/i18n/de.ts @@ -901,7 +901,7 @@ export const de: Record = { "storage.cleanup.moreFiles": "…und {n} weitere", "storage.cleanup.permanent": "Dauerhaft löschen (ohne Quarantäne)", "storage.cleanup.permanentWarn": "Dauerhaftes Löschen kann nicht rückgängig gemacht werden.", - "storage.cleanup.quarantineNote": "Dateien wandern nach .trash unter CODEX_HOME. In dieser Version gibt es keine App-Wiederherstellung — behalte den Ordner oder verschiebe Dateien manuell anhand von manifest.json zurück.", + "storage.cleanup.quarantineNote": "Dateien wandern nach .trash unter CODEX_HOME. Du kannst sie im Quarantäne-Abschnitt darunter wiederherstellen.", "storage.cleanup.cancel": "Abbrechen", "storage.cleanup.confirmQuarantine": "In Quarantäne", "storage.cleanup.confirmPermanent": "Dauerhaft löschen", @@ -911,6 +911,7 @@ export const de: Record = { "storage.cleanup.cleanupFailed": "Bereinigung fehlgeschlagen.", "storage.cleanup.err.codex_busy": "Codex verwendet state.sqlite — beende Codex und versuche es erneut.", "storage.cleanup.err.stale_preview": "Archivdateien haben sich seit der Vorschau geändert — führe Vorschau erneut aus.", + "storage.cleanup.err.restore_pending_overlap": "Ausgewählte Archive überschneiden sich mit einer unvollständigen Wiederherstellung — zuerst Wiederherstellung abschließen oder erneut versuchen.", "storage.cleanup.err.referenced_history": "Ausgewählte Archive werden noch von Fork- oder paginierter Historie referenziert.", "storage.cleanup.err.invalid_digest": "Vorschaudigest fehlt oder ist ungültig.", "storage.cleanup.err.invalid_mode": "Modus muss quarantine oder permanent sein.", @@ -919,6 +920,36 @@ export const de: Record = { "storage.cleanup.err.db_reconcile_failed": "Codex-Statusdatenbank konnte nicht aktualisiert werden.", "storage.cleanup.err.cleanup_failed": "Bereinigung fehlgeschlagen.", + "storage.trash.title": "Quarantäne", + "storage.trash.help": "Archivierte Sitzungen in CODEX_HOME/.trash. Wiederherstellen legt JSONL-Dateien und Thread-Zeilen zurück.", + "storage.trash.empty": "Keine Quarantäne-Einträge.", + "storage.trash.loading": "Quarantäne wird geladen…", + "storage.trash.col.when": "Quarantäne seit", + "storage.trash.col.files": "Dateien", + "storage.trash.col.size": "Größe", + "storage.trash.col.mode": "Modus", + "storage.trash.col.id": "Eintrag", + "storage.trash.restore": "Wiederherstellen", + "storage.trash.confirmTitle": "Quarantäne-Eintrag wiederherstellen?", + "storage.trash.confirmBody": "{count} Datei(en) (~{size}) aus {id} zurück in archivierte Sitzungen legen.", + "storage.trash.cancel": "Abbrechen", + "storage.trash.confirmRestore": "Wiederherstellen", + "storage.trash.done": "{count} Datei(en) wiederhergestellt ({size}).", + "storage.trash.restoreFailed": "Wiederherstellung fehlgeschlagen.", + "storage.trash.listFailed": "Quarantäne-Einträge konnten nicht geladen werden.", + "storage.trash.mode.quarantine": "quarantine", + "storage.trash.mode.permanent": "permanent (unvollständig)", + "storage.trash.err.codex_busy": "Codex verwendet state.sqlite — beende Codex und versuche es erneut.", + "storage.trash.err.invalid_trash": "Trash-Eintrags-ID fehlt oder ist ungültig.", + "storage.trash.err.missing_trash": "Trash-Eintrag wurde nicht gefunden.", + "storage.trash.err.dest_exists": "Wiederherstellungsziel existiert bereits — entferne oder benenne die Archivdatei um und versuche es erneut.", + "storage.trash.err.fs_failed": "Dateisystem-Wiederherstellung fehlgeschlagen. Einige Dateien können bereits wiederhergestellt sein — prüfe archived_sessions und .trash.", + "storage.trash.err.storage_mutation_busy": "Eine andere Speicher-Bereinigung oder Wiederherstellung läuft — bitte kurz warten.", + "storage.trash.err.db_reconcile_failed": "Codex-Statusdatenbankzeilen konnten nicht wiederhergestellt werden.", + "storage.trash.err.restore_failed": "Wiederherstellung fehlgeschlagen.", + "storage.trash.err.restore_worker_timeout": "Wiederherstellung dauerte zu lange (über 10 Minuten) und wurde abgebrochen.", + "storage.trash.err.restore_worker_aborted": "Wiederherstellung wurde beim Herunterfahren abgebrochen.", + "storage.trash.err.restore_worker_failed": "Wiederherstellungs-Worker ist abgestürzt oder unerwartet fehlgeschlagen.", "storage.policy.title": "Automatische Bereinigungsrichtlinie", "storage.policy.help": "Optionale Stapelbereinigung, wenn archivierte Sitzungen einen Schwellwert überschreiten. Standardmäßig aus — wird nie automatisch aktiviert.", "storage.policy.loading": "Richtlinie wird geladen…", diff --git a/gui/src/i18n/en.ts b/gui/src/i18n/en.ts index 345ba36ef73..a2b07434d26 100644 --- a/gui/src/i18n/en.ts +++ b/gui/src/i18n/en.ts @@ -653,7 +653,7 @@ export const en = { "storage.cleanup.moreFiles": "…and {n} more", "storage.cleanup.permanent": "Delete permanently (skip quarantine)", "storage.cleanup.permanentWarn": "Permanent delete cannot be undone.", - "storage.cleanup.quarantineNote": "Files move to .trash under CODEX_HOME. There is no in-app restore in this release — keep the trash folder, or manually move files back using manifest.json.", + "storage.cleanup.quarantineNote": "Files move to .trash under CODEX_HOME. You can restore them from the Quarantine section below.", "storage.cleanup.cancel": "Cancel", "storage.cleanup.confirmQuarantine": "Quarantine", "storage.cleanup.confirmPermanent": "Delete permanently", @@ -663,6 +663,7 @@ export const en = { "storage.cleanup.cleanupFailed": "Cleanup failed.", "storage.cleanup.err.codex_busy": "Codex is using state.sqlite — try again after quitting Codex.", "storage.cleanup.err.stale_preview": "Archived files changed since preview — run Preview again.", + "storage.cleanup.err.restore_pending_overlap": "Selected archives overlap an incomplete trash restore — finish or retry restore first.", "storage.cleanup.err.referenced_history": "Selected archives are still referenced by forked or paginated history.", "storage.cleanup.err.invalid_digest": "Preview digest is missing or invalid.", "storage.cleanup.err.invalid_mode": "Cleanup mode must be quarantine or permanent.", @@ -671,6 +672,36 @@ export const en = { "storage.cleanup.err.db_reconcile_failed": "Could not update Codex state database.", "storage.cleanup.err.cleanup_failed": "Cleanup failed.", + "storage.trash.title": "Quarantine", + "storage.trash.help": "Archived sessions moved to CODEX_HOME/.trash. Restore puts JSONL files and thread rows back.", + "storage.trash.empty": "No quarantined entries.", + "storage.trash.loading": "Loading quarantine…", + "storage.trash.col.when": "Quarantined", + "storage.trash.col.files": "Files", + "storage.trash.col.size": "Size", + "storage.trash.col.mode": "Mode", + "storage.trash.col.id": "Entry", + "storage.trash.restore": "Restore", + "storage.trash.confirmTitle": "Restore quarantine entry?", + "storage.trash.confirmBody": "Restore {count} file(s) (~{size}) from {id} back to archived sessions.", + "storage.trash.cancel": "Cancel", + "storage.trash.confirmRestore": "Restore", + "storage.trash.done": "Restored {count} file(s) ({size}).", + "storage.trash.restoreFailed": "Restore failed.", + "storage.trash.listFailed": "Could not list quarantine entries.", + "storage.trash.mode.quarantine": "quarantine", + "storage.trash.mode.permanent": "permanent (incomplete)", + "storage.trash.err.codex_busy": "Codex is using state.sqlite — try again after quitting Codex.", + "storage.trash.err.invalid_trash": "Trash entry id is missing or invalid.", + "storage.trash.err.missing_trash": "Trash entry was not found.", + "storage.trash.err.dest_exists": "Restore destination already exists — remove or rename the archived file and retry.", + "storage.trash.err.fs_failed": "Filesystem restore failed. Some files may already be restored — check archived_sessions and .trash.", + "storage.trash.err.db_reconcile_failed": "Could not restore Codex state database rows.", + "storage.trash.err.storage_mutation_busy": "Another storage cleanup or restore is in progress — try again shortly.", + "storage.trash.err.restore_failed": "Restore failed.", + "storage.trash.err.restore_worker_timeout": "Restore took too long (over 10 minutes) and was stopped.", + "storage.trash.err.restore_worker_aborted": "Restore was cancelled during shutdown.", + "storage.trash.err.restore_worker_failed": "Restore worker crashed or failed unexpectedly.", "storage.policy.title": "Auto-cleanup policy", "storage.policy.help": "Optional batch cleanup when archived sessions exceed a threshold. Off by default — never enabled automatically.", "storage.policy.loading": "Loading policy…", diff --git a/gui/src/i18n/ja.ts b/gui/src/i18n/ja.ts index efc81243c91..d27de241526 100644 --- a/gui/src/i18n/ja.ts +++ b/gui/src/i18n/ja.ts @@ -620,7 +620,7 @@ export const ja: Record = { "storage.cleanup.moreFiles": "…ほか {n} 件", "storage.cleanup.permanent": "完全に削除する（隔離しない）", "storage.cleanup.permanentWarn": "完全削除は元に戻せません。", - "storage.cleanup.quarantineNote": "ファイルは CODEX_HOME 下の .trash へ移動します。この版にアプリ内復元はありません — ゴミ箱を残すか、manifest.json を使って手動で戻してください。", + "storage.cleanup.quarantineNote": "ファイルは CODEX_HOME 下の .trash へ移動します。下の隔離セクションから復元できます。", "storage.cleanup.cancel": "キャンセル", "storage.cleanup.confirmQuarantine": "隔離する", "storage.cleanup.confirmPermanent": "完全に削除", @@ -630,6 +630,7 @@ export const ja: Record = { "storage.cleanup.cleanupFailed": "クリーンアップに失敗しました。", "storage.cleanup.err.codex_busy": "Codex が state.sqlite を使用中です — Codex を終了して再試行してください。", "storage.cleanup.err.stale_preview": "プレビュー以降にアーカイブが変わりました — プレビューをやり直してください。", + "storage.cleanup.err.restore_pending_overlap": "選択したアーカイブは未完了の隔離復元と重なっています — 復元を完了するか再試行してください。", "storage.cleanup.err.referenced_history": "選択したアーカイブはフォークまたはページング履歴から参照されています。", "storage.cleanup.err.invalid_digest": "プレビューのダイジェストが無い、または無効です。", "storage.cleanup.err.invalid_mode": "モードは quarantine または permanent である必要があります。", @@ -638,6 +639,36 @@ export const ja: Record = { "storage.cleanup.err.db_reconcile_failed": "Codex の状態データベースを更新できませんでした。", "storage.cleanup.err.cleanup_failed": "クリーンアップに失敗しました。", + "storage.trash.title": "隔離", + "storage.trash.help": "CODEX_HOME/.trash へ移したアーカイブセッションです。復元すると JSONL とスレッド行が戻ります。", + "storage.trash.empty": "隔離エントリはありません。", + "storage.trash.loading": "隔離を読み込み中…", + "storage.trash.col.when": "隔離日時", + "storage.trash.col.files": "ファイル", + "storage.trash.col.size": "サイズ", + "storage.trash.col.mode": "モード", + "storage.trash.col.id": "エントリ", + "storage.trash.restore": "復元", + "storage.trash.confirmTitle": "隔離エントリを復元しますか？", + "storage.trash.confirmBody": "{id} から {count} 件（約 {size}）をアーカイブセッションへ戻します。", + "storage.trash.cancel": "キャンセル", + "storage.trash.confirmRestore": "復元", + "storage.trash.done": "{count} 件を復元しました（{size}）。", + "storage.trash.restoreFailed": "復元に失敗しました。", + "storage.trash.listFailed": "隔離一覧を取得できませんでした。", + "storage.trash.mode.quarantine": "隔離", + "storage.trash.mode.permanent": "完全削除（未完了）", + "storage.trash.err.codex_busy": "Codex が state.sqlite を使用中です — Codex を終了して再試行してください。", + "storage.trash.err.invalid_trash": "隔離エントリ ID が無い、または無効です。", + "storage.trash.err.missing_trash": "隔離エントリが見つかりません。", + "storage.trash.err.dest_exists": "復元先が既に存在します — アーカイブファイルを削除または改名して再試行してください。", + "storage.trash.err.fs_failed": "ファイルシステムの復元に失敗しました。一部は既に復元されている可能性があります — archived_sessions と .trash を確認してください。", + "storage.trash.err.storage_mutation_busy": "別のストレージクリーンアップまたは復元が進行中です — しばらくして再試行してください。", + "storage.trash.err.db_reconcile_failed": "Codex の状態データベース行を復元できませんでした。", + "storage.trash.err.restore_failed": "復元に失敗しました。", + "storage.trash.err.restore_worker_timeout": "復元が長時間（10 分超）かかったため停止しました。", + "storage.trash.err.restore_worker_aborted": "シャットダウン中に復元がキャンセルされました。", + "storage.trash.err.restore_worker_failed": "復元ワーカーがクラッシュまたは予期しないエラーで失敗しました。", "storage.policy.title": "自動クリーンアップ方針", "storage.policy.help": "アーカイブがしきい値を超えたときの任意の一括クリーンアップ。既定はオフ — 自動では有効になりません。", "storage.policy.loading": "方針を読み込み中…", diff --git a/gui/src/i18n/ko.ts b/gui/src/i18n/ko.ts index d44d0ab0ab0..1257596e93d 100644 --- a/gui/src/i18n/ko.ts +++ b/gui/src/i18n/ko.ts @@ -921,7 +921,7 @@ export const ko: Record = { "storage.cleanup.moreFiles": "…외 {n}개", "storage.cleanup.permanent": "영구 삭제(격리 건너뛰기)", "storage.cleanup.permanentWarn": "영구 삭제는 되돌릴 수 없습니다.", - "storage.cleanup.quarantineNote": "파일은 CODEX_HOME 아래 .trash로 이동합니다. 이 버전에는 앱 내 복원이 없습니다 — 휴지통을 유지하거나 manifest.json으로 수동으로 되돌리세요.", + "storage.cleanup.quarantineNote": "파일은 CODEX_HOME 아래 .trash로 이동합니다. 아래 격리 섹션에서 복원할 수 있습니다.", "storage.cleanup.cancel": "취소", "storage.cleanup.confirmQuarantine": "격리", "storage.cleanup.confirmPermanent": "영구 삭제", @@ -931,6 +931,7 @@ export const ko: Record = { "storage.cleanup.cleanupFailed": "정리에 실패했습니다.", "storage.cleanup.err.codex_busy": "Codex가 state.sqlite를 사용 중입니다 — Codex를 종료한 뒤 다시 시도하세요.", "storage.cleanup.err.stale_preview": "미리보기 이후 보관 파일이 변경되었습니다 — 미리보기를 다시 실행하세요.", + "storage.cleanup.err.restore_pending_overlap": "선택한 보관 파일이 미완료 휴지통 복원과 겹칩니다 — 복원을 완료하거나 다시 시도하세요.", "storage.cleanup.err.referenced_history": "선택한 보관본이 포크 또는 페이지 기록에서 아직 참조됩니다.", "storage.cleanup.err.invalid_digest": "미리보기 digest가 없거나 잘못되었습니다.", "storage.cleanup.err.invalid_mode": "모드는 quarantine 또는 permanent여야 합니다.", @@ -939,6 +940,36 @@ export const ko: Record = { "storage.cleanup.err.db_reconcile_failed": "Codex 상태 데이터베이스를 업데이트할 수 없습니다.", "storage.cleanup.err.cleanup_failed": "정리에 실패했습니다.", + "storage.trash.title": "격리", + "storage.trash.help": "CODEX_HOME/.trash로 옮긴 보관 세션입니다. 복원하면 JSONL과 스레드 행이 돌아갑니다.", + "storage.trash.empty": "격리된 항목이 없습니다.", + "storage.trash.loading": "격리 목록 불러오는 중…", + "storage.trash.col.when": "격리 시각", + "storage.trash.col.files": "파일", + "storage.trash.col.size": "크기", + "storage.trash.col.mode": "모드", + "storage.trash.col.id": "항목", + "storage.trash.restore": "복원", + "storage.trash.confirmTitle": "격리 항목을 복원할까요?", + "storage.trash.confirmBody": "{id}에서 파일 {count}개(약 {size})를 보관 세션으로 되돌립니다.", + "storage.trash.cancel": "취소", + "storage.trash.confirmRestore": "복원", + "storage.trash.done": "파일 {count}개를 복원했습니다({size}).", + "storage.trash.restoreFailed": "복원에 실패했습니다.", + "storage.trash.listFailed": "격리 목록을 불러오지 못했습니다.", + "storage.trash.mode.quarantine": "격리", + "storage.trash.mode.permanent": "영구(미완료)", + "storage.trash.err.codex_busy": "Codex가 state.sqlite를 사용 중입니다 — Codex를 종료한 뒤 다시 시도하세요.", + "storage.trash.err.invalid_trash": "격리 항목 ID가 없거나 잘못되었습니다.", + "storage.trash.err.missing_trash": "격리 항목을 찾을 수 없습니다.", + "storage.trash.err.dest_exists": "복원 대상이 이미 있습니다 — 보관 파일을 삭제하거나 이름을 바꾼 뒤 다시 시도하세요.", + "storage.trash.err.fs_failed": "파일 시스템 복원에 실패했습니다. 일부 파일이 이미 복원되었을 수 있습니다 — archived_sessions와 .trash를 확인하세요.", + "storage.trash.err.storage_mutation_busy": "다른 저장소 정리 또는 복원이 진행 중입니다 — 잠시 후 다시 시도하세요.", + "storage.trash.err.db_reconcile_failed": "Codex 상태 데이터베이스 행을 복원할 수 없습니다.", + "storage.trash.err.restore_failed": "복원에 실패했습니다.", + "storage.trash.err.restore_worker_timeout": "복원 시간이 너무 길어(10분 초과) 중단되었습니다.", + "storage.trash.err.restore_worker_aborted": "종료 중 복원이 취소되었습니다.", + "storage.trash.err.restore_worker_failed": "복원 워커가 충돌하거나 예기치 않게 실패했습니다.", "storage.policy.title": "자동 정리 정책", "storage.policy.help": "보관 세션이 임계값을 넘을 때 선택적으로 일괄 정리합니다. 기본은 꺼짐 — 자동으로 켜지지 않습니다.", "storage.policy.loading": "정책을 불러오는 중…", diff --git a/gui/src/i18n/ru.ts b/gui/src/i18n/ru.ts index 978ee28f5be..82efcb20d08 100644 --- a/gui/src/i18n/ru.ts +++ b/gui/src/i18n/ru.ts @@ -652,7 +652,7 @@ export const ru: Record = { "storage.cleanup.moreFiles": "…и ещё {n}", "storage.cleanup.permanent": "Удалить навсегда (без карантина)", "storage.cleanup.permanentWarn": "Безвозвратное удаление нельзя отменить.", - "storage.cleanup.quarantineNote": "Файлы перемещаются в .trash под CODEX_HOME. В этом выпуске нет восстановления в приложении — сохраняйте корзину или вручную верните файлы по manifest.json.", + "storage.cleanup.quarantineNote": "Файлы перемещаются в .trash под CODEX_HOME. Восстановить можно в разделе «Карантин» ниже.", "storage.cleanup.cancel": "Отмена", "storage.cleanup.confirmQuarantine": "В карантин", "storage.cleanup.confirmPermanent": "Удалить навсегда", @@ -662,6 +662,7 @@ export const ru: Record = { "storage.cleanup.cleanupFailed": "Не удалось выполнить очистку.", "storage.cleanup.err.codex_busy": "Codex использует state.sqlite — закройте Codex и повторите попытку.", "storage.cleanup.err.stale_preview": "Архивы изменились после предпросмотра — выполните предпросмотр снова.", + "storage.cleanup.err.restore_pending_overlap": "Выбранные архивы пересекаются с незавершённым восстановлением из корзины — завершите или повторите восстановление.", "storage.cleanup.err.referenced_history": "Выбранные архивы всё ещё ссылаются из forked или paginated history.", "storage.cleanup.err.invalid_digest": "Digest предпросмотра отсутствует или недействителен.", "storage.cleanup.err.invalid_mode": "Режим должен быть quarantine или permanent.", @@ -670,6 +671,36 @@ export const ru: Record = { "storage.cleanup.err.db_reconcile_failed": "Не удалось обновить базу состояния Codex.", "storage.cleanup.err.cleanup_failed": "Не удалось выполнить очистку.", + "storage.trash.title": "Карантин", + "storage.trash.help": "Архивные сессии в CODEX_HOME/.trash. Восстановление возвращает JSONL и строки потоков.", + "storage.trash.empty": "Нет записей в карантине.", + "storage.trash.loading": "Загрузка карантина…", + "storage.trash.col.when": "В карантине с", + "storage.trash.col.files": "Файлы", + "storage.trash.col.size": "Размер", + "storage.trash.col.mode": "Режим", + "storage.trash.col.id": "Запись", + "storage.trash.restore": "Восстановить", + "storage.trash.confirmTitle": "Восстановить запись карантина?", + "storage.trash.confirmBody": "Вернуть {count} файл(ов) (~{size}) из {id} в архивные сессии.", + "storage.trash.cancel": "Отмена", + "storage.trash.confirmRestore": "Восстановить", + "storage.trash.done": "Восстановлено {count} файл(ов) ({size}).", + "storage.trash.restoreFailed": "Не удалось восстановить.", + "storage.trash.listFailed": "Не удалось получить список карантина.", + "storage.trash.mode.quarantine": "карантин", + "storage.trash.mode.permanent": "permanent (незавершён)", + "storage.trash.err.codex_busy": "Codex использует state.sqlite — закройте Codex и повторите попытку.", + "storage.trash.err.invalid_trash": "Идентификатор записи корзины отсутствует или недействителен.", + "storage.trash.err.missing_trash": "Запись корзины не найдена.", + "storage.trash.err.dest_exists": "Цель восстановления уже существует — удалите или переименуйте архивный файл и повторите.", + "storage.trash.err.fs_failed": "Ошибка восстановления файлов. Часть файлов могла уже восстановиться — проверьте archived_sessions и .trash.", + "storage.trash.err.storage_mutation_busy": "Выполняется другая очистка или восстановление — повторите позже.", + "storage.trash.err.db_reconcile_failed": "Не удалось восстановить строки базы состояния Codex.", + "storage.trash.err.restore_failed": "Не удалось восстановить.", + "storage.trash.err.restore_worker_timeout": "Восстановление заняло слишком много времени (более 10 минут) и было остановлено.", + "storage.trash.err.restore_worker_aborted": "Восстановление отменено при завершении работы.", + "storage.trash.err.restore_worker_failed": "Worker восстановления завершился с ошибкой или аварийно.", "storage.policy.title": "Политика автоочистки", "storage.policy.help": "Необязательная пакетная очистка, когда архивные сессии превышают порог. По умолчанию выкл. — никогда не включается сама.", "storage.policy.loading": "Загрузка политики…", diff --git a/gui/src/i18n/zh.ts b/gui/src/i18n/zh.ts index b6d1b37aaf9..796724fc1d1 100644 --- a/gui/src/i18n/zh.ts +++ b/gui/src/i18n/zh.ts @@ -921,7 +921,7 @@ export const zh: Record = { "storage.cleanup.moreFiles": "…以及另外 {n} 个", "storage.cleanup.permanent": "永久删除（跳过隔离）", "storage.cleanup.permanentWarn": "永久删除无法撤销。", - "storage.cleanup.quarantineNote": "文件会移到 CODEX_HOME 下的 .trash。本版本没有应用内恢复——请保留该文件夹，或按 manifest.json 手动移回原位置。", + "storage.cleanup.quarantineNote": "文件会移到 CODEX_HOME 下的 .trash。可在下方「隔离区」恢复。", "storage.cleanup.cancel": "取消", "storage.cleanup.confirmQuarantine": "隔离", "storage.cleanup.confirmPermanent": "永久删除", @@ -931,6 +931,7 @@ export const zh: Record = { "storage.cleanup.cleanupFailed": "清理失败。", "storage.cleanup.err.codex_busy": "Codex 正在使用 state.sqlite — 请退出 Codex 后重试。", "storage.cleanup.err.stale_preview": "预览后归档文件已变化 — 请重新预览。", + "storage.cleanup.err.restore_pending_overlap": "所选归档与未完成的隔离区恢复重叠 — 请先完成或重试恢复。", "storage.cleanup.err.referenced_history": "所选归档仍被 fork 或分页历史引用。", "storage.cleanup.err.invalid_digest": "预览摘要缺失或无效。", "storage.cleanup.err.invalid_mode": "模式必须是 quarantine 或 permanent。", @@ -939,6 +940,36 @@ export const zh: Record = { "storage.cleanup.err.db_reconcile_failed": "无法更新 Codex 状态数据库。", "storage.cleanup.err.cleanup_failed": "清理失败。", + "storage.trash.title": "隔离区", + "storage.trash.help": "已移至 CODEX_HOME/.trash 的归档会话。恢复会把 JSONL 与线程行写回。", + "storage.trash.empty": "没有隔离条目。", + "storage.trash.loading": "正在加载隔离区…", + "storage.trash.col.when": "隔离时间", + "storage.trash.col.files": "文件", + "storage.trash.col.size": "大小", + "storage.trash.col.mode": "模式", + "storage.trash.col.id": "条目", + "storage.trash.restore": "恢复", + "storage.trash.confirmTitle": "恢复隔离条目？", + "storage.trash.confirmBody": "将 {count} 个文件（约 {size}）从 {id} 恢复到归档会话。", + "storage.trash.cancel": "取消", + "storage.trash.confirmRestore": "恢复", + "storage.trash.done": "已恢复 {count} 个文件（{size}）。", + "storage.trash.restoreFailed": "恢复失败。", + "storage.trash.listFailed": "无法列出隔离条目。", + "storage.trash.mode.quarantine": "隔离", + "storage.trash.mode.permanent": "永久（未完成）", + "storage.trash.err.codex_busy": "Codex 正在使用 state.sqlite — 请退出 Codex 后重试。", + "storage.trash.err.invalid_trash": "隔离条目 ID 缺失或无效。", + "storage.trash.err.missing_trash": "未找到隔离条目。", + "storage.trash.err.dest_exists": "恢复目标已存在 — 请删除或重命名归档文件后重试。", + "storage.trash.err.fs_failed": "文件系统恢复失败。部分文件可能已恢复 — 请检查 archived_sessions 与 .trash。", + "storage.trash.err.storage_mutation_busy": "另一项存储清理或恢复正在进行 — 请稍后再试。", + "storage.trash.err.db_reconcile_failed": "无法恢复 Codex 状态数据库行。", + "storage.trash.err.restore_failed": "恢复失败。", + "storage.trash.err.restore_worker_timeout": "恢复耗时过长（超过 10 分钟）已停止。", + "storage.trash.err.restore_worker_aborted": "关闭过程中恢复已取消。", + "storage.trash.err.restore_worker_failed": "恢复 worker 崩溃或意外失败。", "storage.policy.title": "自动清理策略", "storage.policy.help": "当归档大小超过阈值时可选批量清理。默认关闭——不会自动启用。", "storage.policy.loading": "正在加载策略…", diff --git a/gui/src/pages/Storage.tsx b/gui/src/pages/Storage.tsx index 57e63a35dd5..171172666a5 100644 --- a/gui/src/pages/Storage.tsx +++ b/gui/src/pages/Storage.tsx @@ -46,6 +46,56 @@ interface CleanupResult { message?: string; } +interface TrashEntry { + id: string; + epoch: string; + fileCount: number; + bytes: number; + quarantinedAt?: number; + mode?: "quarantine" | "permanent"; +} + +interface TrashList { + entries: TrashEntry[]; +} + +interface RestoreResult { + ok: boolean; + count: number; + bytes: number; + trashDir?: string; + error?: string; + message?: string; +} + +const GB = 1024 ** 3; + +interface CleanupPolicy { + enabled: boolean; + trigger: { archivedBytesOver: number }; + target: { reduceToBytes?: number; removeOldestPercent?: number }; + schedule: "startup" | "daily" | "weekly" | "manual"; + mode: "quarantine" | "permanent"; + lastRun?: { at: number; freedBytes: number; removed: number }; + nextRun?: number; + job?: { + status: "idle" | "running"; + reason?: string; + startedAt?: number; + finishedAt?: number; + lastError?: string; + lastOutcome?: { + ok: boolean; + skipped?: string; + deferred?: string; + error?: string; + mode?: string; + freedBytes?: number; + removed?: number; + }; + }; +} + const BUCKET_TKEYS: Record = { sessions: "storage.bucket.sessions", archived_sessions: "storage.bucket.archived_sessions", @@ -58,6 +108,21 @@ const BUCKET_TKEYS: Record = { const PRESETS = [10, 25, 50] as const; +const localizedCatch = (e: unknown, fallback: string): string => { + if (!(e instanceof Error)) return fallback; + const msg = e.message; + if ( + msg === "Failed to fetch" + || msg.includes("NetworkError") + || msg.includes("network error") + || msg.includes("JSON") + || msg.includes("Unexpected end of") + ) { + return fallback; + } + return msg || fallback; +}; + function bucketLabel(bucket: StorageBucket, t: TFn): string { const tkey = BUCKET_TKEYS[bucket.key]; return tkey ? t(tkey) : bucket.label; @@ -180,6 +245,7 @@ function ArchivedCleanupPanel({ switch (code) { case "codex_busy": return t("storage.cleanup.err.codex_busy"); case "stale_preview": return t("storage.cleanup.err.stale_preview"); + case "restore_pending_overlap": return t("storage.cleanup.err.restore_pending_overlap"); case "referenced_history": return t("storage.cleanup.err.referenced_history"); case "invalid_digest": return t("storage.cleanup.err.invalid_digest"); case "invalid_mode": return t("storage.cleanup.err.invalid_mode"); @@ -198,21 +264,6 @@ function ArchivedCleanupPanel({ percent: new Intl.NumberFormat(locale, { style: "percent", maximumFractionDigits: 0 }).format(value / 100), }); - const localizedCatch = (e: unknown, fallback: string): string => { - if (!(e instanceof Error)) return fallback; - const msg = e.message; - if ( - msg === "Failed to fetch" - || msg.includes("NetworkError") - || msg.includes("network error") - || msg.includes("JSON") - || msg.includes("Unexpected end of") - ) { - return fallback; - } - return msg || fallback; - }; - const runPreview = async () => { setBusy(true); setError(null); @@ -377,32 +428,239 @@ function ArchivedCleanupPanel({ ); } -const GB = 1024 ** 3; +function QuarantineTrashPanel({ + apiBase, + locale, + t, + onDone, + reloadToken, + onEntriesChange, +}: { + apiBase: string; + locale: Locale; + t: TFn; + onDone: () => void; + reloadToken: number; + onEntriesChange?: (entries: TrashEntry[]) => void; +}) { + const [entries, setEntries] = useState([]); + const [loading, setLoading] = useState(true); + const [busy, setBusy] = useState(false); + const [confirmEntry, setConfirmEntry] = useState(null); + const [status, setStatus] = useState(null); + const [error, setError] = useState(null); + const cancelRef = useRef(null); + const previousFocusRef = useRef(null); + const busyRef = useRef(false); + const loadGenerationRef = useRef(0); -interface CleanupPolicy { - enabled: boolean; - trigger: { archivedBytesOver: number }; - target: { reduceToBytes?: number; removeOldestPercent?: number }; - schedule: "startup" | "daily" | "weekly" | "manual"; - mode: "quarantine" | "permanent"; - lastRun?: { at: number; freedBytes: number; removed: number }; - nextRun?: number; - job?: { - status: "idle" | "running"; - reason?: string; - startedAt?: number; - finishedAt?: number; - lastError?: string; - lastOutcome?: { - ok: boolean; - skipped?: string; - deferred?: string; - error?: string; - mode?: string; - freedBytes?: number; - removed?: number; + useEffect(() => { + busyRef.current = busy; + }, [busy]); + + const closeConfirm = useCallback(() => setConfirmEntry(null), []); + + useEffect(() => { + if (!confirmEntry) return; + previousFocusRef.current = document.activeElement as HTMLElement | null; + cancelRef.current?.focus(); + const onKey = (e: KeyboardEvent) => { + if (e.key === "Escape" && !busyRef.current) closeConfirm(); }; + window.addEventListener("keydown", onKey); + return () => { + window.removeEventListener("keydown", onKey); + previousFocusRef.current?.focus(); + }; + }, [confirmEntry, closeConfirm]); + + const loadTrash = useCallback(async (signal?: AbortSignal) => { + const generation = ++loadGenerationRef.current; + setLoading(true); + try { + const res = await fetch(`${apiBase}/api/storage/trash`, { signal }); + const json = await res.json() as TrashList & { error?: string }; + if (signal?.aborted || generation !== loadGenerationRef.current) return; + if (!res.ok) throw new Error(json.error ?? "list_failed"); + const next = Array.isArray(json.entries) ? json.entries : []; + setEntries(next); + onEntriesChange?.(next); + setError(null); + } catch (e) { + if (signal?.aborted || generation !== loadGenerationRef.current) return; + if (e instanceof DOMException && e.name === "AbortError") return; + setEntries([]); + onEntriesChange?.([]); + setError(localizedCatch(e, t("storage.trash.listFailed"))); + } finally { + if (generation === loadGenerationRef.current) setLoading(false); + } + }, [apiBase, t, onEntriesChange]); + + useEffect(() => { + const controller = new AbortController(); + const timeout = window.setTimeout(() => { + void loadTrash(controller.signal); + }, 0); + return () => { + window.clearTimeout(timeout); + loadGenerationRef.current += 1; + controller.abort(); + }; + }, [loadTrash, reloadToken]); + + const mapRestoreError = (code: string | undefined, fallback?: string) => { + switch (code) { + case "codex_busy": return t("storage.trash.err.codex_busy"); + case "invalid_trash": return t("storage.trash.err.invalid_trash"); + case "missing_trash": return t("storage.trash.err.missing_trash"); + case "dest_exists": return t("storage.trash.err.dest_exists"); + case "fs_failed": return t("storage.trash.err.fs_failed"); + case "db_reconcile_failed": return t("storage.trash.err.db_reconcile_failed"); + case "storage_mutation_busy": return t("storage.trash.err.storage_mutation_busy"); + case "restore_failed": return t("storage.trash.err.restore_failed"); + case "restore_worker_timeout": return t("storage.trash.err.restore_worker_timeout"); + case "restore_worker_aborted": return t("storage.trash.err.restore_worker_aborted"); + case "restore_worker_failed": + return fallback ?? t("storage.trash.err.restore_worker_failed"); + default: return fallback ?? t("storage.trash.restoreFailed"); + } }; + + const runRestore = async () => { + if (!confirmEntry) return; + setBusy(true); + setError(null); + try { + const res = await fetch(`${apiBase}/api/storage/trash/restore`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ id: confirmEntry.id }), + }); + const json = await res.json() as RestoreResult; + if (!res.ok || !json.ok) { + throw new Error(mapRestoreError(json.error, json.message)); + } + closeConfirm(); + setStatus(t("storage.trash.done", { + count: String(json.count), + size: formatBytes(json.bytes, locale), + })); + onDone(); + } catch (e) { + setError(localizedCatch(e, t("storage.trash.restoreFailed"))); + } finally { + setBusy(false); + } + }; + + const formatWhen = (entry: TrashEntry) => { + const ms = entry.quarantinedAt ?? Number(entry.epoch.split("-")[0]); + if (!Number.isFinite(ms) || ms <= 0) return "—"; + return new Date(ms).toLocaleString(locale); + }; + + const modeLabel = (mode: TrashEntry["mode"]) => { + if (mode === "permanent") return t("storage.trash.mode.permanent"); + if (mode === "quarantine") return t("storage.trash.mode.quarantine"); + return "—"; + }; + + return ( +
+

{t("storage.trash.title")}

+

{t("storage.trash.help")}

+ + {status &&

{status}

} + {error && !confirmEntry &&

{error}

} + + {loading ? ( +

{t("storage.trash.loading")}

+ ) : entries.length === 0 ? ( +

{t("storage.trash.empty")}

+ ) : ( +
+ + + + + + + + + + + + {entries.map(entry => ( + + + + + + + + + ))} + +
{t("storage.trash.col.when")}{t("storage.trash.col.files")}{t("storage.trash.col.size")}{t("storage.trash.col.mode")}{t("storage.trash.col.id")} +
{formatWhen(entry)}{entry.fileCount}{formatBytes(entry.bytes, locale)}{modeLabel(entry.mode)}{entry.id} + +
+
+ )} + + {confirmEntry && ( +
!busy && closeConfirm()} + > +
e.stopPropagation()}> +

{t("storage.trash.confirmTitle")}

+

+ {t("storage.trash.confirmBody", { + count: String(confirmEntry.fileCount), + size: formatBytes(confirmEntry.bytes, locale), + id: confirmEntry.id, + })} +

+ {error &&

{error}

} +
+ + +
+
+
+ )} +
+ ); } function policyFieldsFromResponse(json: CleanupPolicy): CleanupPolicy { @@ -469,7 +727,6 @@ function AutoCleanupPolicyPanel({ useEffect(() => { const controller = new AbortController(); - // Defer so loadPolicy's setState is not synchronous inside the effect body. const timeout = window.setTimeout(() => { void loadPolicy(controller.signal); }, 0); @@ -555,7 +812,6 @@ function AutoCleanupPolicyPanel({ setError(null); setStatus(null); try { - // Persist current form values before running — abort if the form is invalid. const base = buildBody(); if (!base) { setError(t("storage.policy.invalid")); @@ -624,7 +880,6 @@ function AutoCleanupPolicyPanel({ outcome = job.lastOutcome; break; } - // Job finished so fast that startedAt advanced; accept matching finished marker. if (job.finishedAt && job.finishedAt >= startedAt && job.lastOutcome) { outcome = job.lastOutcome; break; @@ -849,6 +1104,9 @@ export default function Storage({ apiBase }: { apiBase: string }) { const { t, locale } = useI18n(); const [data, setData] = useState(null); const [loading, setLoading] = useState(true); + const [trashReloadToken, setTrashReloadToken] = useState(0); + // Stamp trash awareness with apiBase so a base change invalidates without an effect. + const [trashInfo, setTrashInfo] = useState({ apiBase, settled: false, hasEntries: false }); const loadGenerationRef = useRef(0); const fetchStorage = useCallback(async (signal?: AbortSignal) => { @@ -880,15 +1138,30 @@ export default function Storage({ apiBase }: { apiBase: string }) { }; }, [fetchStorage]); + const refreshAll = useCallback(() => { + void fetchStorage(); + setTrashReloadToken(n => n + 1); + }, [fetchStorage]); + + const onTrashEntriesChange = useCallback((entries: TrashEntry[]) => { + setTrashInfo({ apiBase, settled: true, hasEntries: entries.length > 0 }); + }, [apiBase]); + + const trashSettled = trashInfo.apiBase === apiBase && trashInfo.settled; + const trashHasEntries = trashInfo.apiBase === apiBase && trashInfo.hasEntries; const failed = !loading && (!data || data.error !== undefined); - const empty = !loading && !failed && data!.total.fileCount === 0; + const empty = !loading && !failed && data!.total.fileCount === 0 && trashSettled && !trashHasEntries; const archivedCount = data?.buckets.find(b => b.key === "archived_sessions")?.fileCount ?? 0; + const showBody = Boolean(data) && !failed; + // While storage is empty, keep the trash panel mounted until it reports so we + // do not flash the empty state over a non-empty quarantine. + const showTrashWhileSettling = showBody && (data!.total.fileCount > 0 || !trashSettled || trashHasEntries); return ( <>

{t("storage.title")}

-
@@ -905,30 +1178,44 @@ export default function Storage({ apiBase }: { apiBase: string }) { apiBase={apiBase} locale={locale} t={t} - onDone={() => void fetchStorage()} + onDone={() => refreshAll()} /> ) : ( <> -
-
{t("storage.card.total")}
{formatBytes(data!.total.bytes, locale)}
-
{t("storage.card.files")}
{data!.total.fileCount.toLocaleString(locale)}
-
{t("storage.card.home")}
{data!.codexHome}
-
- - + {data && data.total.fileCount > 0 && ( + <> +
+
{t("storage.card.total")}
{formatBytes(data.total.bytes, locale)}
+
{t("storage.card.files")}
{data.total.fileCount.toLocaleString(locale)}
+
{t("storage.card.home")}
{data.codexHome}
+
+ + + + )} void fetchStorage()} + onDone={() => refreshAll()} /> {archivedCount > 0 && ( void fetchStorage()} + onDone={() => refreshAll()} + /> + )} + {showTrashWhileSettling && ( + refreshAll()} /> )} diff --git a/src/codex/history-provider.ts b/src/codex/history-provider.ts index d08e68e6419..b333e0dbbd6 100644 --- a/src/codex/history-provider.ts +++ b/src/codex/history-provider.ts @@ -252,7 +252,7 @@ function parseSessionMetaLine(line: string): ParsedSessionMeta | null { * (codex-rs `apply_session_meta_from_item`). We base our patch on the most recent metadata so we * never resurrect a stale provider that a later app-written `session_meta` already changed. */ -function readLatestSessionMeta(path: string): ParsedSessionMeta | null { +export function readLatestSessionMeta(path: string): ParsedSessionMeta | null { const raw = readFileSync(path, "utf8"); const lines = raw.split("\n"); for (let i = lines.length - 1; i >= 0; i--) { @@ -265,6 +265,118 @@ function readLatestSessionMeta(path: string): ParsedSessionMeta | null { return null; } +/** + * Fields needed to re-insert a production-shaped `threads` row from a rollout JSONL when a + * Phase-2 quarantine predates full `satellite-backup.json` thread snapshots. + * + * Uses the same last-writer-wins `session_meta` fold as {@link readLatestSessionMeta}, plus the + * first user-message preview (codex-rs `list.rs` / `EventMsg::UserMessage` path). + */ +export interface RolloutThreadFields { + id: string; + modelProvider: string; + source: string; + firstUserMessage: string; + hasUserEvent: number; + cwd?: string; + historyMode?: string; + cliVersion?: string; +} + +function textFromContentParts(content: unknown): string | null { + if (typeof content === "string" && content.trim()) return content.trim(); + if (!Array.isArray(content)) return null; + const parts: string[] = []; + for (const part of content) { + if (!part || typeof part !== "object") continue; + const p = part as Record; + if (typeof p.text === "string" && p.text.trim()) parts.push(p.text.trim()); + else if (typeof p.input_text === "string" && p.input_text.trim()) parts.push(p.input_text.trim()); + } + const joined = parts.join("\n").trim(); + return joined || null; +} + +/** Extract the first user-message preview from a rollout line, or null. */ +function extractUserMessagePreview(line: string): string | null { + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch { + return null; + } + if (!parsed || typeof parsed !== "object") return null; + const record = parsed as { type?: unknown; payload?: unknown }; + const payload = record.payload; + if (!payload || typeof payload !== "object") return null; + const p = payload as Record; + + if (record.type === "event_msg") { + // codex-rs EventMsg::UserMessage — payload.type is "user_message" (or omitted in fixtures). + if (p.type === "user_message" || typeof p.message === "string") { + if (typeof p.message === "string" && p.message.trim()) return p.message.trim(); + const fromContent = textFromContentParts(p.content); + if (fromContent) return fromContent; + } + return null; + } + + if (record.type === "response_item") { + if (p.type === "message" && p.role === "user") { + return textFromContentParts(p.content); + } + } + return null; +} + +/** + * Reconstruct thread identity + listing fields from a staged/restored rollout JSONL. + * Returns null when the file is missing or has no parseable `session_meta`. + */ +export function readThreadFieldsFromRollout(path: string): RolloutThreadFields | null { + if (!path || !existsSync(path)) return null; + let raw: string; + try { + raw = readFileSync(path, "utf8"); + } catch { + return null; + } + const lines = raw.split("\n"); + let latest: ParsedSessionMeta | null = null; + let firstUserMessage = ""; + for (const line of lines) { + if (!line) continue; + if (line.includes("\"session_meta\"")) { + const meta = parseSessionMetaLine(line); + if (meta) latest = meta; + } + if (!firstUserMessage) { + const preview = extractUserMessagePreview(line); + if (preview) firstUserMessage = preview; + } + } + if (!latest) return null; + const payload = latest.record.payload; + const id = typeof payload.id === "string" ? payload.id : ""; + if (!id) return null; + const modelProvider = typeof payload.model_provider === "string" && payload.model_provider + ? payload.model_provider + : "openai"; + const source = typeof payload.source === "string" && payload.source + ? payload.source + : "cli"; + return { + id, + modelProvider, + source, + firstUserMessage, + hasUserEvent: firstUserMessage.trim() ? 1 : 0, + ...(typeof payload.cwd === "string" ? { cwd: payload.cwd } : {}), + ...(typeof payload.history_mode === "string" ? { historyMode: payload.history_mode } : {}), + ...(typeof payload.cli_version === "string" ? { cliVersion: payload.cli_version } : {}), + }; +} + /** * Make a thread's rollout reflect a provider/source change by APPENDING a new `session_meta` line, * rather than rewriting line 1. The appended line clones the latest metadata payload (so no field diff --git a/src/server/management/logs-usage-routes.ts b/src/server/management/logs-usage-routes.ts index 2d7d8ec68fc..e5270fc7dc9 100644 --- a/src/server/management/logs-usage-routes.ts +++ b/src/server/management/logs-usage-routes.ts @@ -34,7 +34,9 @@ import { primeCodexPoolQuotas } from "../../codex/auth-api"; import { DEFAULT_PROVIDER_CONTEXT_CAP, globalContextCapValue, providerContextCap, providerContextCaps, setAllProviderContextCaps, setGlobalContextCapValue, setProviderContextCap } from "../../providers/context-cap"; import { resolveCodexHomeDir } from "../../codex/home"; import { scanStorage } from "../../storage/scanner"; -import { executeArchivedCleanup, pickWireCleanupTestHooks, previewArchivedCleanup, type CleanupMode } from "../../storage/cleanup"; +import { executeArchivedCleanup, listTrashEntries, pickWireCleanupTestHooks, previewArchivedCleanup, type CleanupMode, type RestoreErrorCode } from "../../storage/cleanup"; +import { runArchivedCleanupJob } from "../../storage/cleanup-job"; +import { getRestoreTrashTestStreamResponse, runRestoreTrashEntryJob } from "../../storage/restore-job"; import { normalizeStorageCleanupPolicy, parseStorageCleanupPolicyInput, @@ -288,7 +290,7 @@ export async function handleLogsUsageRoutes(ctx: ManagementContext): Promise = { codex_busy: "Codex is using state.sqlite — try again after quitting Codex.", + storage_mutation_busy: "Another storage cleanup or restore is in progress — try again shortly.", stale_preview: "Archived files changed since preview — run Preview again.", + restore_pending_overlap: "Selected archives overlap an incomplete trash restore — finish or retry restore first.", referenced_history: "Selected archives are still referenced by forked or paginated history.", invalid_digest: "Preview digest is missing or invalid.", invalid_mode: "mode must be quarantine or permanent.", @@ -336,6 +344,99 @@ export async function handleLogsUsageRoutes(ctx: ManagementContext): Promise ({ + id, + epoch, + fileCount, + bytes, + ...(quarantinedAt !== undefined ? { quarantinedAt } : {}), + ...(mode ? { mode } : {}), + })), + }); + } catch { + return jsonResponse({ error: "trash_list_failed", entries: [] }, 500); + } + } + + if ( + process.env.OPENCODEX_CLEANUP_TEST_HOOKS === "1" && + url.pathname === "/api/storage/trash/restore/test-stream" && + req.method === "GET" + ) { + const stream = getRestoreTrashTestStreamResponse(); + if (stream) return stream; + return jsonResponse({ error: "not_available" }, 404); + } + + if (url.pathname === "/api/storage/trash/restore" && req.method === "POST") { + let body: { id?: unknown }; + try { body = await req.json(); } catch { return jsonResponse({ error: "invalid_json" }, 400); } + const id = typeof body?.id === "string" ? body.id : ""; + if (!id.trim()) { + return jsonResponse({ error: "invalid_trash", message: "Trash entry id is required." }, 400); + } + try { + const result = await runRestoreTrashEntryJob(id); + if (!result.ok) { + const status = + result.error === "codex_busy" + || result.error === "dest_exists" + || result.error === "storage_mutation_busy" + ? 409 + : result.error === "missing_trash" + ? 404 + : result.error === "invalid_trash" + ? 400 + : 500; + const messages: Record = { + invalid_trash: "Trash entry id is missing or invalid.", + missing_trash: "Trash entry was not found.", + codex_busy: "Codex is using state.sqlite — try again after quitting Codex.", + storage_mutation_busy: "Another storage cleanup or restore is in progress — try again shortly.", + dest_exists: "Restore destination already exists — remove or rename the archived file and retry.", + fs_failed: "Filesystem restore failed. Some files may already be restored — check archived_sessions and .trash.", + db_reconcile_failed: "Could not restore Codex state database rows.", + restore_failed: "Restore failed.", + restore_worker_timeout: "Restore took too long (over 10 minutes) and was stopped.", + restore_worker_aborted: "Restore was cancelled during shutdown.", + restore_worker_failed: "Restore worker crashed or failed unexpectedly.", + }; + const errorCode = result.error ?? "restore_failed"; + const baseMessage = messages[errorCode] ?? messages.restore_failed; + const message = + result.message && errorCode === "restore_worker_failed" + ? `${baseMessage} (${result.message})` + : baseMessage; + return jsonResponse({ + ok: false, + error: errorCode, + message, + count: result.count, + bytes: result.bytes, + restoredPaths: result.restoredPaths, + ...(result.trashDir ? { trashDir: result.trashDir } : {}), + }, status); + } + return jsonResponse({ + ok: true, + trashDir: result.trashDir, + count: result.count, + bytes: result.bytes, + restoredPaths: result.restoredPaths, + }); + } catch { + return jsonResponse({ + ok: false, + error: "restore_failed", + message: "Restore failed.", + }, 500); + } + } + if (url.pathname === "/api/storage/cleanup-policy/test-stream" && req.method === "GET") { const stream = getStorageCleanupPolicyTestStreamResponse(); if (stream) return stream; diff --git a/src/storage/cleanup-job.ts b/src/storage/cleanup-job.ts new file mode 100644 index 00000000000..147b3978e71 --- /dev/null +++ b/src/storage/cleanup-job.ts @@ -0,0 +1,57 @@ +/** + * Single-flight wrapper for manual archived cleanup (management API). + * + * Shares the CODEX_HOME storage-mutation coordinator with trash restore and + * (Phase 3) policy-driven cleanup. + */ +import { resolveCodexHomeDir } from "../codex/home"; +import { + executeArchivedCleanup, + type CleanupResult, + type ExecuteCleanupOptions, +} from "./cleanup"; +import { + resetStorageMutationCoordinatorForTests, + setStorageMutationCoordinatorTestHooks, + withStorageMutationSlot, + type StorageMutationCoordinatorTestHooks, +} from "./storage-mutation-coordinator"; + +export type CleanupJobTestHooks = StorageMutationCoordinatorTestHooks; + +export function setArchivedCleanupJobTestHooks(hooks: CleanupJobTestHooks | null): void { + setStorageMutationCoordinatorTestHooks(hooks); +} + +export function resetArchivedCleanupJobForTests(): void { + resetStorageMutationCoordinatorForTests(); +} + +/** + * Run archived cleanup under the shared storage-mutation gate. + * Returns immediately with `storage_mutation_busy` when restore or another + * cleanup is in flight for the same CODEX_HOME. + */ +export async function runArchivedCleanupJob( + options: ExecuteCleanupOptions, +): Promise { + const codexHome = options.codexHome ?? resolveCodexHomeDir(); + const mode = options.mode; + const percent = options.percent; + const result = await withStorageMutationSlot("cleanup", codexHome, () => + executeArchivedCleanup(options), + ); + if (result && typeof result === "object" && "ok" in result && result.ok === false + && "error" in result && result.error === "storage_mutation_busy") { + return { + ok: false, + mode, + percent, + count: 0, + bytes: 0, + removedPaths: [], + error: "storage_mutation_busy", + }; + } + return result as CleanupResult; +} diff --git a/src/storage/cleanup.ts b/src/storage/cleanup.ts index b8bb9885fef..8577d82a205 100644 --- a/src/storage/cleanup.ts +++ b/src/storage/cleanup.ts @@ -12,21 +12,28 @@ * satellite rows before staged files. Success never carries soft `dbWarning` / * `failedPaths`. */ -import { createHash } from "node:crypto"; +import { createHash, randomUUID } from "node:crypto"; import { + closeSync, existsSync, + fsyncSync, + linkSync, mkdirSync, + openSync, readdirSync, + readFileSync, renameSync, rmSync, statSync, unlinkSync, writeFileSync, + writeSync, chmodSync, } from "node:fs"; import { basename, isAbsolute, join, relative, resolve, sep } from "node:path"; import { Database } from "bun:sqlite"; import { resolveCodexHomeDir } from "../codex/home"; +import { readThreadFieldsFromRollout } from "../codex/history-provider"; export const ARCHIVED_SESSIONS_DIR = "archived_sessions"; export const TRASH_DIR = ".trash"; @@ -39,9 +46,11 @@ export type CleanupErrorCode = | "invalid_digest" | "stale_preview" | "codex_busy" + | "storage_mutation_busy" | "fs_failed" | "db_reconcile_failed" | "referenced_history" + | "restore_pending_overlap" | "cleanup_failed"; export interface ArchivedCandidate { @@ -327,24 +336,113 @@ export function selectOldestPercent(candidates: ArchivedCandidate[], percent: nu const pct = clampPercent(percent); if (pct <= 0 || candidates.length === 0) return []; if (pct >= 100) return [...candidates]; - const n = Math.max(1, Math.floor((candidates.length * pct) / 100)); + const n = percentSelectionTargetCount(candidates.length, pct); return candidates.slice(0, n); } +/** Count implied by percent selection over the full candidate list. */ +export function percentSelectionTargetCount(totalCount: number, percent: number): number { + const pct = clampPercent(percent); + if (pct <= 0 || totalCount === 0) return 0; + if (pct >= 100) return totalCount; + return Math.max(1, Math.floor((totalCount * pct) / 100)); +} + +function candidateOverlapsPendingRestore( + candidate: ArchivedCandidate, + pendingDestRels: ReadonlySet, +): boolean { + if (pendingDestRels.size === 0) return false; + for (const rel of candidate.physicalRelPaths) { + if (pendingDestRels.has(rel)) return true; + } + return pendingDestRels.has(candidate.relPath); +} + +/** Drop cleanup candidates whose physical paths overlap an in-progress restore. */ +export function filterCandidatesExcludingPendingRestore( + candidates: ArchivedCandidate[], + codexHome: string = resolveCodexHomeDir(), +): ArchivedCandidate[] { + const pendingDestRels = collectRestorePendingAcceptedDestRels(codexHome); + if (pendingDestRels.size === 0) return candidates; + return candidates.filter(c => !candidateOverlapsPendingRestore(c, pendingDestRels)); +} + +/** + * Oldest-first percent selection that skips pending-restore destinations without + * consuming the percent budget, backfilling with the next oldest safe candidates. + */ +export function selectOldestPercentSkippingPendingRestore( + candidates: ArchivedCandidate[], + percent: number, + codexHome: string = resolveCodexHomeDir(), +): ArchivedCandidate[] { + const target = percentSelectionTargetCount(candidates.length, percent); + if (target === 0) return []; + const pendingDestRels = collectRestorePendingAcceptedDestRels(codexHome); + const out: ArchivedCandidate[] = []; + for (const c of candidates) { + if (candidateOverlapsPendingRestore(c, pendingDestRels)) continue; + out.push(c); + if (out.length >= target) break; + } + return out; +} + +/** + * Reduce archived total toward `reduceToBytes` using oldest safe candidates only. + * Pending-restore destinations are skipped and do not count toward bytes freed. + */ +export function selectReduceToBytesSkippingPendingRestore( + candidates: ArchivedCandidate[], + reduceToBytes: number, + codexHome: string = resolveCodexHomeDir(), +): ArchivedCandidate[] { + if (!Number.isFinite(reduceToBytes) || reduceToBytes < 0) return []; + const total = candidates.reduce((sum, c) => sum + c.bytes, 0); + if (total <= reduceToBytes) return []; + const need = total - reduceToBytes; + const pendingDestRels = collectRestorePendingAcceptedDestRels(codexHome); + const out: ArchivedCandidate[] = []; + let freed = 0; + for (const c of candidates) { + if (candidateOverlapsPendingRestore(c, pendingDestRels)) continue; + out.push(c); + freed += c.bytes; + if (freed >= need) break; + } + return out; +} + +/** Accepted destination paths from every valid in-progress restore marker under `.trash`. */ +export function collectRestorePendingAcceptedDestRels(codexHome: string): Set { + const out = new Set(); + const trashRoot = join(codexHome, TRASH_DIR); + if (!existsSync(trashRoot)) return out; + for (const name of readdirSync(trashRoot)) { + if (!TRASH_EPOCH_DIR.test(name)) continue; + const read = readRestorePending(join(trashRoot, name)); + if (read.status !== "valid") continue; + for (const rel of read.state.acceptedDestRels) out.add(rel); + } + return out; +} + export function previewArchivedCleanup( percent: number, codexHome: string = resolveCodexHomeDir(), ): CleanupPreview { const all = listArchivedCandidates(codexHome); - const selected = selectOldestPercent(all, percent); + const safe = selectOldestPercentSkippingPendingRestore(all, percent, codexHome); const pct = clampPercent(percent); return { codexHome, percent: pct, - count: selected.length, - bytes: selected.reduce((sum, c) => sum + c.bytes, 0), - digest: computePreviewDigest(selected, pct), - candidates: selected, + count: safe.length, + bytes: safe.reduce((sum, c) => sum + c.bytes, 0), + digest: computePreviewDigest(safe, pct), + candidates: safe, }; } @@ -353,14 +451,14 @@ export function previewExactArchivedCleanup( candidates: ArchivedCandidate[], codexHome: string = resolveCodexHomeDir(), ): CleanupPreview { - const selected = [...candidates]; + const safe = filterCandidatesExcludingPendingRestore(candidates, codexHome); return { codexHome, percent: 0, - count: selected.length, - bytes: selected.reduce((sum, c) => sum + c.bytes, 0), - digest: computeExactPreviewDigest(selected), - candidates: selected, + count: safe.length, + bytes: safe.reduce((sum, c) => sum + c.bytes, 0), + digest: computeExactPreviewDigest(safe), + candidates: safe, }; } @@ -386,6 +484,13 @@ export function resolveExactArchivedCandidates( function openDbWritable(dbPath: string, busyTimeoutMs = 100): Database { const db = new Database(dbPath); + try { + // bun:sqlite exposes a binding-level timeout; set both so Windows lock waits + // honor the caller's budget (pragma alone has been flaky under CI contention). + (db as Database & { timeout?: number }).timeout = busyTimeoutMs; + } catch { + /* older bindings */ + } try { db.exec(`PRAGMA busy_timeout = ${busyTimeoutMs}`); } catch { @@ -601,6 +706,10 @@ type SqlRow = Record; interface SatelliteBackup { threadIds: string[]; + /** Full `threads` row images (SELECT *) captured under the state write lock. */ + threads?: SqlRow[]; + dynamicTools?: SqlRow[]; + spawnEdges?: SqlRow[]; logs?: { path: string; rows: SqlRow[] }; memories?: { path: string; @@ -618,6 +727,11 @@ interface SatelliteBackup { }; } +type SatelliteBackupRead = + | { status: "missing" } + | { status: "ok"; backup: SatelliteBackup } + | { status: "invalid" }; + interface ReconcileTestHooks { failAfterLogsMutation?: boolean; failAfterMemoriesMutation?: boolean; @@ -630,6 +744,30 @@ interface ReconcileTestHooks { } const SATELLITE_BACKUP_FILE = "satellite-backup.json"; +/** Marks an incomplete restore so retries can accept dest files and resume metadata. */ +const RESTORE_PENDING_FILE = "restore-pending.json"; + +type StagedFile = { from: string; to: string; relPath: string }; + +interface RestorePendingSections { + state: boolean; + logs: boolean; + memories: boolean; + goals: boolean; +} + +interface RestorePendingState { + version: 1; + filesRestored: true; + /** + * Planned CODEX_HOME-relative destinations for this restore attempt. + * Written before moves so a mid-loop failure can still accept placed dests + * on resume while finishing files that remain staged. + */ + acceptedDestRels: string[]; + /** Sections that still need reconciliation on retry. */ + pending: RestorePendingSections; +} function quoteIdent(name: string): string { return `"${name.replaceAll('"', '""')}"`; @@ -639,15 +777,132 @@ function selectRows(db: Database, sql: string, params: Array): return db.query>(sql).all(...params) as SqlRow[]; } -function insertRowsConflictIgnore(db: Database, table: string, rows: SqlRow[]): void { +function tableColumnNames(db: Database, table: string): Set { + if (!tableExists(db, table)) return new Set(); + const rows = db.query<{ name: string }, []>( + `PRAGMA table_info("${table.replaceAll('"', '""')}")`, + ).all(); + return new Set(rows.map(r => r.name)); +} + +/** Insert rows with ON CONFLICT DO NOTHING; returns only rows that were newly inserted. */ +function insertRowsConflictIgnore(db: Database, table: string, rows: SqlRow[]): SqlRow[] { + const inserted: SqlRow[] = []; + if (rows.length === 0) return inserted; + const allowed = tableColumnNames(db, table); for (const row of rows) { - const cols = Object.keys(row); + const cols = Object.keys(row).filter(c => allowed.has(c)); if (cols.length === 0) continue; - db.run( + const result = db.run( `INSERT INTO ${quoteIdent(table)} (${cols.map(quoteIdent).join(", ")}) VALUES (${cols.map(() => "?").join(", ")}) ON CONFLICT DO NOTHING`, cols.map(c => row[c] as string | number | bigint | null | Uint8Array), ); + if (result.changes > 0) inserted.push(row); + } + return inserted; +} + +/** Snapshot state-DB dependents that cleanup deletes with the thread rows. */ +function snapshotStateDependents( + db: Database, + threadIds: string[], +): Pick { + const out: Pick = {}; + if (threadIds.length === 0 || !tableExists(db, "threads")) return out; + + const threads: SqlRow[] = []; + for (const chunk of chunkIds(threadIds, SQLITE_ID_CHUNK * 2)) { + const placeholders = chunk.map(() => "?").join(","); + threads.push(...selectRows(db, `SELECT * FROM threads WHERE id IN (${placeholders})`, chunk)); } + out.threads = threads; + + if (tableExists(db, "thread_dynamic_tools")) { + const dynamicTools: SqlRow[] = []; + for (const chunk of chunkIds(threadIds, SQLITE_ID_CHUNK * 2)) { + const placeholders = chunk.map(() => "?").join(","); + dynamicTools.push(...selectRows( + db, + `SELECT * FROM thread_dynamic_tools WHERE thread_id IN (${placeholders})`, + chunk, + )); + } + out.dynamicTools = dynamicTools; + } + + if (tableExists(db, "thread_spawn_edges")) { + const spawnEdges: SqlRow[] = []; + for (const chunk of chunkIds(threadIds, SQLITE_ID_CHUNK)) { + const placeholders = chunk.map(() => "?").join(","); + spawnEdges.push(...selectRows( + db, + `SELECT * FROM thread_spawn_edges + WHERE parent_thread_id IN (${placeholders}) OR child_thread_id IN (${placeholders})`, + [...chunk, ...chunk], + )); + } + out.spawnEdges = spawnEdges; + } + + return out; +} + +/** Remap serialized absolute DB paths onto the newest DBs under the current Codex home. */ +function remapSatelliteBackupPaths( + backup: SatelliteBackup, + paths: RuntimeDbPaths, +): { ok: true; backup: SatelliteBackup } | { ok: false } { + const next: SatelliteBackup = { + threadIds: backup.threadIds, + ...(backup.threads ? { threads: backup.threads } : {}), + ...(backup.dynamicTools ? { dynamicTools: backup.dynamicTools } : {}), + ...(backup.spawnEdges ? { spawnEdges: backup.spawnEdges } : {}), + }; + if (backup.logs) { + if (!paths.logs) return { ok: false }; + next.logs = { ...backup.logs, path: paths.logs }; + } + if (backup.memories) { + if (!paths.memories) return { ok: false }; + next.memories = { ...backup.memories, path: paths.memories }; + } + if (backup.goals) { + if (!paths.goals) return { ok: false }; + next.goals = { ...backup.goals, path: paths.goals }; + } + return { ok: true, backup: next }; +} + +/** + * Same-volume move that never replaces an existing destination. + * + * `existsSync` + `renameSync` is TOCTOU: a live file created between the check + * and rename can be overwritten (Windows rename replaces files). Hard-link then + * unlink fails with EEXIST if `to` appears, which is what trash → archived_sessions + * restore needs. Callers under the same `CODEX_HOME` volume should not hit EXDEV. + */ +function renameNoReplace(from: string, to: string): void { + try { + linkSync(from, to); + } catch (error) { + const code = (error as NodeJS.ErrnoException | undefined)?.code; + // Hard links unavailable (rare FS) — refuse rather than clobber via rename. + if (code === "EXDEV" || code === "EPERM" || code === "ENOTSUP" || code === "EINVAL") { + throw Object.assign(new Error("rename_no_replace_unsupported"), { code, cause: error }); + } + throw error; + } + try { + unlinkSync(from); + } catch (error) { + // Roll back the hard link so we do not leave the file at both paths. + try { unlinkSync(to); } catch { /* best-effort */ } + throw error; + } +} + +function isExistError(error: unknown): boolean { + return (error as NodeJS.ErrnoException | undefined)?.code === "EEXIST"; } function updateRowFromSnapshot( @@ -739,6 +994,7 @@ interface SatelliteWriteLocks { function beginSatelliteWriteLocks( paths: RuntimeDbPaths, busyTimeoutMs: number, + only?: Partial>, ): SatelliteWriteLocks { const locks: SatelliteWriteLocks = {}; const order: Array<{ key: "logs" | "memories" | "goals"; path: string | null }> = [ @@ -748,6 +1004,7 @@ function beginSatelliteWriteLocks( ]; try { for (const { key, path } of order) { + if (only && !only[key]) continue; if (!path || !existsSync(path)) continue; const db = openDbWritable(path, busyTimeoutMs); try { @@ -982,11 +1239,17 @@ function deleteAndCommitSatellites( if (locks.memories && backup.memories) { deleteMemoriesInTx(locks.memories.db, backup.memories); if (backup.memories.consolidateTouched) { + // Capture under the write lock, but persist only after COMMIT+close. + // Holding BEGIN IMMEDIATE across writeFileSync lets Windows CI disk/AV + // latency stall the lock long enough for concurrent reopen hooks (and + // bun's default 5s test timeout) to hang — see PR #558 windows-latest. backup.memories.consolidatePostImage = readConsolidateGlobalJob(locks.memories.db); - writeSatelliteBackup(stageDir, backup); } commitSatelliteLock(locks.memories); locks.memories = undefined; + if (backup.memories.consolidateTouched) { + writeSatelliteBackup(stageDir, backup); + } if (hooks?.failAfterMemoriesMutation) throw new Error("test_fail_after_memories"); } if (locks.goals && backup.goals) { @@ -1152,6 +1415,10 @@ function reconcileDeletedThreads( satelliteLocks = beginSatelliteWriteLocks(paths, busyTimeoutMs); try { backup = snapshotSatelliteBackupInLocks(satelliteLocks, threadIds); + const stateDeps = snapshotStateDependents(stateDb, threadIds); + backup.threads = stateDeps.threads; + backup.dynamicTools = stateDeps.dynamicTools; + backup.spawnEdges = stateDeps.spawnEdges; try { if (hooks?.failSatelliteBackupWrite) { throw new Error("test_fail_satellite_backup_write"); @@ -1183,7 +1450,7 @@ function reconcileDeletedThreads( deleteThreadsAndDependents(stateDb, threadIds); if (hooks?.failBeforeStateCommit) throw new Error("test_fail_before_state_commit"); stateDb.exec("COMMIT"); - clearSatelliteBackup(stageDir); + // Keep satellite-backup.json for quarantine restore; permanent purge removes the stage. return { ok: true, threads }; } catch (error) { if (satelliteLocks) rollbackAllSatelliteLocks(satelliteLocks); @@ -1197,8 +1464,6 @@ function reconcileDeletedThreads( } } -type StagedFile = { from: string; to: string; relPath: string }; - function absFromRel(codexHome: string, relPath: string): string { if (relPath.includes("..") || isAbsolute(relPath) || /^[A-Za-z]:[\\/]/.test(relPath)) { throw new Error("invalid_rel_path"); @@ -1413,18 +1678,37 @@ export function executeArchivedCleanup(options: ExecuteCleanupOptions): CleanupR } let preview: CleanupPreview; + let unfilteredSelected: ArchivedCandidate[]; if (options.candidateRelPaths !== undefined) { const selected = resolveExactArchivedCandidates(options.candidateRelPaths, codexHome); if (selected === null) { return fail(mode, percent, "stale_preview"); } + unfilteredSelected = selected; preview = previewExactArchivedCleanup(selected, codexHome); } else { + const all = listArchivedCandidates(codexHome); + unfilteredSelected = selectOldestPercent(all, percent); preview = previewArchivedCleanup(percent, codexHome); } if (preview.digest.toLowerCase() !== options.digest.toLowerCase()) { + const pendingDestRels = collectRestorePendingAcceptedDestRels(codexHome); + const blocked = unfilteredSelected.filter(c => candidateOverlapsPendingRestore(c, pendingDestRels)); + const unfilteredDigest = options.candidateRelPaths !== undefined + ? computeExactPreviewDigest(unfilteredSelected) + : computePreviewDigest(unfilteredSelected, percent); + if ( + unfilteredDigest.toLowerCase() === options.digest.toLowerCase() + && blocked.length > 0 + ) { + return fail(mode, percent, "restore_pending_overlap"); + } return fail(mode, percent, "stale_preview"); } + const pendingDestRels = collectRestorePendingAcceptedDestRels(codexHome); + if (preview.candidates.some(c => candidateOverlapsPendingRestore(c, pendingDestRels))) { + return fail(mode, percent, "restore_pending_overlap"); + } if (preview.candidates.length === 0) { return { @@ -1568,9 +1852,12 @@ export function executeArchivedCleanup(options: ExecuteCleanupOptions): CleanupR digest: preview.digest, purgeIncomplete: true, purgedRelPaths: purge.purged.map(item => item.relPath), - entries: manifestEntries.filter(entry => - entry.physicalRelPaths.some(rel => survivingRelPaths.has(rel)), - ), + entries: manifestEntries + .map(entry => ({ + ...entry, + physicalRelPaths: entry.physicalRelPaths.filter(rel => survivingRelPaths.has(rel)), + })) + .filter(entry => entry.physicalRelPaths.length > 0), }, null, 2), ); } catch { /* best-effort: the pre-commit manifest is still on disk */ } @@ -1599,3 +1886,1046 @@ export function executeArchivedCleanup(options: ExecuteCleanupOptions): CleanupR removedPaths, }; } + +// --------------------------------------------------------------------------- +// Phase 2.1 — quarantine list + restore +// --------------------------------------------------------------------------- + +export type RestoreErrorCode = + | "invalid_trash" + | "missing_trash" + | "codex_busy" + | "storage_mutation_busy" + | "fs_failed" + | "db_reconcile_failed" + | "dest_exists" + | "restore_failed" + | "restore_worker_timeout" + | "restore_worker_aborted" + | "restore_worker_failed"; + +export interface TrashEntrySummary { + /** CODEX_HOME-relative path, e.g. `.trash/1700000000000`. */ + id: string; + /** Epoch directory name (may include collision suffix, e.g. `1700-1`). */ + epoch: string; + fileCount: number; + bytes: number; + quarantinedAt?: number; + mode?: CleanupMode; +} + +export interface RestoreResult { + ok: boolean; + trashDir?: string; + count: number; + bytes: number; + restoredPaths: string[]; + error?: RestoreErrorCode; + /** Optional operator-facing detail when the error code alone is insufficient. */ + message?: string; +} + +interface TrashManifest { + quarantinedAt?: number; + mode?: CleanupMode; + entries?: CleanupManifestEntry[]; +} + +/** Epoch dir names: digits, optionally `-N` from createExclusiveStageDir collision. */ +const TRASH_EPOCH_DIR = /^(\d+)(-\d+)?$/; + +/** + * Parse a trash `manifest.json` atomically. + * + * Any missing `entries` array, or any malformed entry / `physicalRelPaths` value / + * required field, rejects the **entire** manifest (returns null). Individual bad + * entries are never filtered out so a partial parse cannot silently drop evidence. + */ +function parseTrashManifest(raw: string): TrashManifest | null { + try { + const parsed = JSON.parse(raw) as unknown; + if (!parsed || typeof parsed !== "object") return null; + const o = parsed as Record; + if (!Array.isArray(o.entries)) return null; + + const entries: CleanupManifestEntry[] = []; + for (const e of o.entries) { + if (!e || typeof e !== "object" || Array.isArray(e)) return null; + const entry = e as Record; + if (typeof entry.relPath !== "string" || entry.relPath.length === 0) return null; + if (typeof entry.bytes !== "number" || !Number.isFinite(entry.bytes)) return null; + if (typeof entry.mtimeMs !== "number" || !Number.isFinite(entry.mtimeMs)) return null; + if (!Array.isArray(entry.physicalRelPaths) || entry.physicalRelPaths.length === 0) return null; + const physical: string[] = []; + for (const p of entry.physicalRelPaths) { + // Do not strip bad elements — one malformed path invalidates the whole manifest. + if (typeof p !== "string" || p.length === 0) return null; + physical.push(p); + } + if ("threadId" in entry && typeof entry.threadId !== "string") return null; + if ("rolloutPath" in entry && typeof entry.rolloutPath !== "string") return null; + if ( + "archived" in entry + && entry.archived !== null + && typeof entry.archived !== "number" + ) { + return null; + } + entries.push({ + relPath: entry.relPath, + bytes: entry.bytes, + mtimeMs: entry.mtimeMs, + physicalRelPaths: physical, + ...(typeof entry.threadId === "string" ? { threadId: entry.threadId } : {}), + ...(typeof entry.rolloutPath === "string" ? { rolloutPath: entry.rolloutPath } : {}), + ...(entry.archived === null || typeof entry.archived === "number" + ? { archived: entry.archived as number | null } + : {}), + }); + } + + const out: TrashManifest = { entries }; + if (typeof o.quarantinedAt === "number" && Number.isFinite(o.quarantinedAt)) { + out.quarantinedAt = o.quarantinedAt; + } + if (o.mode === "quarantine" || o.mode === "permanent") out.mode = o.mode; + return out; + } catch { + return null; + } +} + +/** + * Validate a trash entry id as a single `.trash/` segment under CODEX_HOME. + * Returns the absolute stage directory, or null when the id is unsafe / missing. + */ +export function resolveTrashStageDir( + trashId: string, + codexHome: string, +): { ok: true; stageDir: string; id: string } | { ok: false; error: RestoreErrorCode } { + const normalized = toForwardSlash(trashId.trim()).replace(/\/+$/, ""); + if (!normalized.startsWith(`${TRASH_DIR}/`)) return { ok: false, error: "invalid_trash" }; + const rest = normalized.slice(TRASH_DIR.length + 1); + if (!rest || rest.includes("/") || rest.includes("\\") || rest.includes("..")) { + return { ok: false, error: "invalid_trash" }; + } + if (!TRASH_EPOCH_DIR.test(rest)) return { ok: false, error: "invalid_trash" }; + let stageDir: string; + try { + stageDir = absFromRel(codexHome, `${TRASH_DIR}/${rest}`); + } catch { + return { ok: false, error: "invalid_trash" }; + } + if (!existsSync(stageDir)) return { ok: false, error: "missing_trash" }; + try { + if (!statSync(stageDir).isDirectory()) return { ok: false, error: "invalid_trash" }; + } catch { + return { ok: false, error: "missing_trash" }; + } + return { ok: true, stageDir, id: `${TRASH_DIR}/${rest}` }; +} + +function sumTrashEntryBytes(stageDir: string, manifest: TrashManifest | null): { + fileCount: number; + bytes: number; +} { + let fileCount = 0; + let bytes = 0; + let names: string[] = []; + try { + names = readdirSync(stageDir); + } catch { + return { fileCount: 0, bytes: 0 }; + } + for (const name of names) { + if ( + name === "manifest.json" + || name === SATELLITE_BACKUP_FILE + || name === RESTORE_PENDING_FILE + ) { + continue; + } + if (!isRolloutFileName(name)) continue; + try { + const st = statSync(join(stageDir, name)); + if (!st.isFile()) continue; + fileCount += 1; + bytes += st.size; + } catch { /* */ } + } + // Prefer live FS counts; fall back to manifest totals when the stage is empty of rollouts. + if (fileCount === 0 && manifest?.entries?.length) { + fileCount = manifest.entries.reduce((n, e) => n + Math.max(1, e.physicalRelPaths.length), 0); + bytes = manifest.entries.reduce((n, e) => n + (e.bytes || 0), 0); + } + return { fileCount, bytes }; +} + +/** List quarantine entries under `CODEX_HOME/.trash/` (relative ids only). */ +export function listTrashEntries( + codexHome: string = resolveCodexHomeDir(), +): TrashEntrySummary[] { + const trashRoot = join(codexHome, TRASH_DIR); + let names: string[] = []; + try { + names = readdirSync(trashRoot); + } catch { + return []; + } + const out: TrashEntrySummary[] = []; + for (const name of names) { + if (!TRASH_EPOCH_DIR.test(name)) continue; + const stageDir = join(trashRoot, name); + try { + if (!statSync(stageDir).isDirectory()) continue; + } catch { + continue; + } + let manifest: TrashManifest | null = null; + try { + manifest = parseTrashManifest(readFileSync(join(stageDir, "manifest.json"), "utf8")); + } catch { + manifest = null; + } + const { fileCount, bytes } = sumTrashEntryBytes(stageDir, manifest); + // Skip empty collision placeholders left behind without a manifest or rollouts. + if (fileCount === 0 && !manifest?.entries?.length) { + try { + if (!existsSync(join(stageDir, "manifest.json"))) continue; + } catch { + continue; + } + } + out.push({ + id: `${TRASH_DIR}/${name}`, + epoch: name, + fileCount, + bytes, + ...(manifest?.quarantinedAt !== undefined ? { quarantinedAt: manifest.quarantinedAt } : {}), + ...(manifest?.mode ? { mode: manifest.mode } : {}), + }); + } + out.sort((a, b) => { + const aq = a.quarantinedAt ?? (Number(a.epoch.split("-")[0]) || 0); + const bq = b.quarantinedAt ?? (Number(b.epoch.split("-")[0]) || 0); + return bq - aq || b.epoch.localeCompare(a.epoch); + }); + return out; +} + +function readSatelliteBackupFile(stageDir: string): SatelliteBackupRead { + const path = join(stageDir, SATELLITE_BACKUP_FILE); + if (!existsSync(path)) return { status: "missing" }; + try { + const raw = JSON.parse(readFileSync(path, "utf8")) as unknown; + if (!raw || typeof raw !== "object") return { status: "invalid" }; + const o = raw as SatelliteBackup; + if (!Array.isArray(o.threadIds)) return { status: "invalid" }; + return { status: "ok", backup: o }; + } catch { + // File exists but is truncated / malformed — distinct from a missing backup. + return { status: "invalid" }; + } +} + +function isSqlRowArray(value: unknown): value is SqlRow[] { + return Array.isArray(value) && value.every(row => row && typeof row === "object" && !Array.isArray(row)); +} + +/** True when a snapshotted thread row covers every NOT NULL column on the live schema. */ +function threadSnapshotCoversRequiredColumns(row: SqlRow, requiredCols: string[]): boolean { + for (const col of requiredCols) { + if (!(col in row) || row[col] === undefined) return false; + } + return true; +} + +function requiredThreadColumnNames(db: Database): string[] { + if (!tableExists(db, "threads")) return []; + const rows = db.query<{ name: string; notnull: number }, []>( + `PRAGMA table_info("threads")`, + ).all(); + return rows.filter(r => r.notnull === 1).map(r => r.name); +} + +/** + * Build a production-shaped thread row for schemas that predate full satellite snapshots. + * Prefer `readThreadFieldsFromRollout` (canonical history/session_meta path); fall back to + * the sparse manifest fields only when the live schema does not require model/source/message. + */ +function reconstructThreadRowFromRollout( + entry: CleanupManifestEntry, + rolloutAbsPath: string, + allowedCols: Set, + requiredCols: string[], +): SqlRow | null { + if (typeof entry.threadId !== "string" || typeof entry.rolloutPath !== "string") return null; + + const fields = readThreadFieldsFromRollout(rolloutAbsPath); + const row: SqlRow = { + id: entry.threadId, + rollout_path: entry.rolloutPath, + }; + + if (fields) { + // Prefer manifest thread id (binding) but keep rollout-derived listing fields. + if (allowedCols.has("model_provider")) row.model_provider = fields.modelProvider; + if (allowedCols.has("source")) row.source = fields.source; + if (allowedCols.has("first_user_message")) row.first_user_message = fields.firstUserMessage; + if (allowedCols.has("has_user_event")) row.has_user_event = fields.hasUserEvent; + if (allowedCols.has("cwd") && fields.cwd !== undefined) row.cwd = fields.cwd; + if (allowedCols.has("history_mode") && fields.historyMode !== undefined) { + row.history_mode = fields.historyMode; + } + if (allowedCols.has("cli_version") && fields.cliVersion !== undefined) { + row.cli_version = fields.cliVersion; + } + } + + if (allowedCols.has("archived")) { + row.archived = entry.archived ?? 1; + } + if (allowedCols.has("archived_at")) { + row.archived_at = null; + } + + // Fill remaining NOT NULL columns with safe empties when the rollout lacked them + // (e.g. fixture rollouts without a user turn still need first_user_message = ''). + for (const col of requiredCols) { + if (row[col] !== undefined) continue; + if (col === "id" || col === "rollout_path") continue; + if (col === "model_provider") row[col] = "openai"; + else if (col === "source") row[col] = "cli"; + else if (col === "first_user_message") row[col] = ""; + else if (col === "has_user_event") row[col] = 0; + else if (col === "archived") row[col] = entry.archived ?? 1; + else return null; // unknown required column we cannot invent + } + + // If the schema requires listing fields, refuse when the rollout was unreadable. + const needsSessionMeta = requiredCols.some( + c => c === "model_provider" || c === "source" || c === "first_user_message", + ); + if (needsSessionMeta && !fields) return null; + + return row; +} + +type RestorePendingRead = + | { status: "missing" } + | { status: "valid"; state: RestorePendingState } + | { status: "invalid" }; + +let _restorePendingSeq = 0; + +function parseRestorePendingState(raw: unknown): RestorePendingState | null { + if (!raw || typeof raw !== "object" || Array.isArray(raw)) return null; + const o = raw as Record; + if (o.version !== 1 || o.filesRestored !== true) return null; + if (!Array.isArray(o.acceptedDestRels)) return null; + const acceptedDestRels = o.acceptedDestRels.filter((r): r is string => typeof r === "string"); + if (acceptedDestRels.length !== o.acceptedDestRels.length) return null; + const pendingRaw = o.pending; + if (!pendingRaw || typeof pendingRaw !== "object" || Array.isArray(pendingRaw)) return null; + const p = pendingRaw as Record; + if ( + typeof p.state !== "boolean" + || typeof p.logs !== "boolean" + || typeof p.memories !== "boolean" + || typeof p.goals !== "boolean" + ) { + return null; + } + return { + version: 1, + filesRestored: true, + acceptedDestRels, + pending: { + state: p.state, + logs: p.logs, + memories: p.memories, + goals: p.goals, + }, + }; +} + +/** + * Distinguish a missing marker from a present-but-malformed one. An invalid marker + * must never be treated as a fresh restore (that would ignore already-moved files). + */ +function readRestorePending(stageDir: string): RestorePendingRead { + const path = join(stageDir, RESTORE_PENDING_FILE); + if (!existsSync(path)) return { status: "missing" }; + try { + const state = parseRestorePendingState(JSON.parse(readFileSync(path, "utf8")) as unknown); + if (!state) return { status: "invalid" }; + return { status: "valid", state }; + } catch { + return { status: "invalid" }; + } +} + +/** + * Atomically replace restore-pending.json: private temp in the stage, fsync, then rename. + * An interrupted update leaves the previous valid marker intact. + */ +function writeRestorePending( + stageDir: string, + state: RestorePendingState, + options?: { failBeforeRename?: boolean; failWrite?: boolean }, +): void { + if (options?.failWrite) throw new Error("test_fail_pending_write"); + const dest = join(stageDir, RESTORE_PENDING_FILE); + const tmp = join(stageDir, `${RESTORE_PENDING_FILE}.${process.pid}.${++_restorePendingSeq}.tmp`); + const payload = JSON.stringify(state); + const fd = openSync(tmp, "w", 0o600); + try { + writeSync(fd, payload, null, "utf8"); + fsyncSync(fd); + } catch (error) { + try { closeSync(fd); } catch { /* */ } + try { unlinkSync(tmp); } catch { /* */ } + throw error; + } + closeSync(fd); + chmodPrivatePath(tmp, 0o600); + if (options?.failBeforeRename) { + try { unlinkSync(tmp); } catch { /* */ } + throw new Error("test_fail_pending_rename"); + } + try { + renameSync(tmp, dest); + } catch (error) { + try { unlinkSync(tmp); } catch { /* */ } + throw error; + } +} + +function restoreThreadsFromManifest( + stateDbPath: string | null, + entries: CleanupManifestEntry[], + backup: SatelliteBackup | null, + busyTimeoutMs: number, + codexHome: string, +): { ok: true } | ReconcileErr { + const manifestThreadIds = entries + .map(e => e.threadId) + .filter((id): id is string => typeof id === "string"); + const backupThreadIds = backup?.threadIds ?? []; + const needsThreads = manifestThreadIds.length > 0 + || backupThreadIds.length > 0 + || Boolean(backup?.threads?.length); + + if (needsThreads && (!stateDbPath || !existsSync(stateDbPath))) { + return { ok: false, error: "db_reconcile_failed" }; + } + if (!stateDbPath || !existsSync(stateDbPath)) { + return { ok: true }; + } + + const result = withWritableDb(stateDbPath, busyTimeoutMs, db => { + if (!tableExists(db, "threads")) throw new Error("missing_threads_table"); + + const requiredCols = requiredThreadColumnNames(db); + const allowedCols = tableColumnNames(db, "threads"); + const snapshotThreads = backup?.threads && isSqlRowArray(backup.threads) + ? backup.threads + : []; + const completeSnapshots = snapshotThreads.filter(row => + threadSnapshotCoversRequiredColumns(row, requiredCols), + ); + const coveredIds = new Set( + completeSnapshots + .map(r => r.id) + .filter((id): id is string => typeof id === "string"), + ); + + // Legacy Phase-2 quarantine (no / incomplete satellite thread snapshots): reconstruct + // every required column from the restored rollout via the history-provider session path. + const toReconstruct = entries.filter( + e => typeof e.threadId === "string" + && typeof e.rolloutPath === "string" + && !coveredIds.has(e.threadId!), + ); + const reconstructed: SqlRow[] = []; + for (const entry of toReconstruct) { + let abs: string; + try { + abs = absFromRel(codexHome, entry.rolloutPath!); + } catch { + // Prefer the first restored physical path under archived_sessions. + const rel = entry.physicalRelPaths[0]; + if (!rel) throw new Error("missing_rollout_for_thread"); + abs = absFromRel(codexHome, rel); + } + // .jsonl.zst cannot be parsed here — require a plain .jsonl sibling or path. + if (abs.endsWith(ZST_SUFFIX)) { + const plain = abs.slice(0, -".zst".length); + if (existsSync(plain)) abs = plain; + } + const row = reconstructThreadRowFromRollout(entry, abs, allowedCols, requiredCols); + if (!row) throw new Error("thread_reconstruct_failed"); + reconstructed.push(row); + } + + if (completeSnapshots.length > 0) { + insertRowsConflictIgnore(db, "threads", completeSnapshots); + } + if (reconstructed.length > 0) { + insertRowsConflictIgnore(db, "threads", reconstructed); + } + + if (backup?.dynamicTools && isSqlRowArray(backup.dynamicTools) && tableExists(db, "thread_dynamic_tools")) { + insertRowsConflictIgnore(db, "thread_dynamic_tools", backup.dynamicTools); + } + if (backup?.spawnEdges && isSqlRowArray(backup.spawnEdges) && tableExists(db, "thread_spawn_edges")) { + insertRowsConflictIgnore(db, "thread_spawn_edges", backup.spawnEdges); + } + }); + if (!result.ok) return result; + return { ok: true }; +} + +function isSafeArchivedPhysicalRel(rel: string): boolean { + const normalized = toForwardSlash(rel); + if (!normalized.startsWith(`${ARCHIVED_SESSIONS_DIR}/`)) return false; + if (normalized.includes("..")) return false; + const rest = normalized.slice(ARCHIVED_SESSIONS_DIR.length + 1); + if (!rest || rest.includes("/")) return false; + return isRolloutFileName(rest); +} + +/** Test-only failure injection for restore atomicity regressions. */ +export interface RestoreTestHooks { + /** After state threads/dependents commit, before satellite commits. */ + failAfterStateCommit?: boolean; + /** After the first satellite DB commit (logs → memories → goals). */ + failAfterFirstSatelliteCommit?: boolean; + /** When the leftover staged-rollout completeness gate runs. */ + failAtLeftoverStageGate?: boolean; + /** Fail the initial restore-pending.json write (before any file moves). */ + failInitialPendingWrite?: boolean; + /** Fail a later pending update after the temp is written but before rename. */ + failPendingWriteBeforeRename?: boolean; + /** Crash immediately after file moves (marker already durable). */ + failAfterFileMoves?: boolean; + /** + * After this many successful rollout moves in the current attempt, throw. + * Exercises mid-loop failure with some dests placed and others still staged. + */ + failAfterMoveCount?: number; + /** Fail renaming the completed stage to a non-listable tombstone dir. */ + failStageTombstoneRename?: boolean; + /** After tombstone rename, skip best-effort tombstone delete (orphan is OK). */ + failTombstoneDelete?: boolean; + /** + * Test-only: spin-wait this many ms after rollout file moves, before DB + * reconcile, so cleanup can race an in-flight restore. + */ + holdAfterFileMovesMs?: number; +} + +/** + * Resume must not clear owed satellite work when the matching backup section is + * absent — fail closed per section instead. + */ +function failClosedSatelliteResume( + priorPending: RestorePendingState, + satelliteBackup: SatelliteBackup | null, +): RestoreErrorCode | null { + const owed = priorPending.pending; + if (!owed.logs && !owed.memories && !owed.goals) return null; + if (!satelliteBackup) return "db_reconcile_failed"; + if (owed.logs && !satelliteBackup.logs) return "db_reconcile_failed"; + if (owed.memories && !satelliteBackup.memories) return "db_reconcile_failed"; + if (owed.goals && !satelliteBackup.goals) return "db_reconcile_failed"; + return null; +} + +/** + * Successful restore finalization: rename the stage to a tombstone name that + * `listTrashEntries` ignores, then delete the tombstone best-effort. A failed + * rename leaves the original stage (and all evidence) intact for retry. + */ +function finalizeRestoredStage( + stageDir: string, + codexHome: string, + hooks?: Pick, +): boolean { + const trashRoot = join(codexHome, TRASH_DIR); + const epoch = basename(stageDir); + const tombstoneName = `.tombstone-${epoch}-${randomUUID()}`; + const tombstonePath = join(trashRoot, tombstoneName); + try { + if (hooks?.failStageTombstoneRename) throw new Error("test_fail_stage_tombstone_rename"); + renameSync(stageDir, tombstonePath); + } catch { + return false; + } + if (!hooks?.failTombstoneDelete) { + try { rmSync(tombstonePath, { recursive: true, force: true }); } catch { /* best-effort */ } + } + return true; +} + +/** + * Restore one quarantine entry: move JSONL back, re-insert threads (+ satellites + * when satellite-backup.json is present), then remove the trash directory. + * + * Late failures after files have moved never compensate metadata or restage. + * Instead they persist `restore-pending.json` (accepted dest paths + which + * state/logs/memories/goals sections still need work) atomically *before* any + * rollout move, then update it after each section so a retry can accept existing + * destinations and resume only missing metadata. + */ +export function restoreTrashEntry( + trashId: string, + options?: { + codexHome?: string; + busyTimeoutMs?: number; + _test?: RestoreTestHooks; + }, +): RestoreResult { + const codexHome = options?.codexHome ?? resolveCodexHomeDir(); + const busyTimeoutMs = options?.busyTimeoutMs ?? 100; + const hooks = options?._test; + + const resolved = resolveTrashStageDir(trashId, codexHome); + if (!resolved.ok) { + return { ok: false, count: 0, bytes: 0, restoredPaths: [], error: resolved.error }; + } + const { stageDir, id } = resolved; + + let manifestRaw: string; + try { + manifestRaw = readFileSync(join(stageDir, "manifest.json"), "utf8"); + } catch { + return { ok: false, trashDir: id, count: 0, bytes: 0, restoredPaths: [], error: "invalid_trash" }; + } + const manifest = parseTrashManifest(manifestRaw); + if (!manifest?.entries?.length) { + return { ok: false, trashDir: id, count: 0, bytes: 0, restoredPaths: [], error: "invalid_trash" }; + } + + const pendingRead = readRestorePending(stageDir); + if (pendingRead.status === "invalid") { + // Malformed marker means an incomplete restore may already have moved files; + // never treat it as a fresh restore. + return { ok: false, trashDir: id, count: 0, bytes: 0, restoredPaths: [], error: "fs_failed" }; + } + const priorPending = pendingRead.status === "valid" ? pendingRead.state : null; + const acceptedDest = new Set(priorPending?.acceptedDestRels ?? []); + + // Partial permanent purges may leave only a subset of physical files on disk — + // trim to survivors rather than failing the whole entry for a purged twin. + // Resume also treats already-restored accepted destinations as survivors. + const entries: CleanupManifestEntry[] = []; + for (const entry of manifest.entries) { + if (!entry.physicalRelPaths.every(isSafeArchivedPhysicalRel)) { + return { ok: false, trashDir: id, count: 0, bytes: 0, restoredPaths: [], error: "invalid_trash" }; + } + const surviving = entry.physicalRelPaths.filter(rel => { + if (existsSync(join(stageDir, basename(rel)))) return true; + if (!acceptedDest.has(rel)) return false; + try { + return existsSync(absFromRel(codexHome, rel)); + } catch { + return false; + } + }); + if (surviving.length === 0) { + return { ok: false, trashDir: id, count: 0, bytes: 0, restoredPaths: [], error: "fs_failed" }; + } + entries.push({ ...entry, physicalRelPaths: surviving }); + } + + const paths = discoverRuntimeDbPaths(codexHome); + const backupRead = readSatelliteBackupFile(stageDir); + if (backupRead.status === "invalid") { + return { + ok: false, + trashDir: id, + count: 0, + bytes: 0, + restoredPaths: [], + error: "db_reconcile_failed", + }; + } + + let satelliteBackup: SatelliteBackup | null = null; + if (backupRead.status === "ok") { + const remapped = remapSatelliteBackupPaths(backupRead.backup, paths); + if (!remapped.ok) { + return { + ok: false, + trashDir: id, + count: 0, + bytes: 0, + restoredPaths: [], + error: "db_reconcile_failed", + }; + } + satelliteBackup = remapped.backup; + } + + if (priorPending) { + const resumeErr = failClosedSatelliteResume(priorPending, satelliteBackup); + if (resumeErr) { + return { + ok: false, + trashDir: id, + count: 0, + bytes: 0, + restoredPaths: [], + error: resumeErr, + }; + } + } + + const pendingSections: RestorePendingSections = { + state: priorPending ? priorPending.pending.state : true, + logs: priorPending ? priorPending.pending.logs : Boolean(satelliteBackup?.logs), + memories: priorPending ? priorPending.pending.memories : Boolean(satelliteBackup?.memories), + goals: priorPending ? priorPending.pending.goals : Boolean(satelliteBackup?.goals), + }; + + const needAnySatellite = pendingSections.logs || pendingSections.memories || pendingSections.goals; + if (pendingSections.state) { + const needsThreads = entries.some(e => typeof e.threadId === "string") + || Boolean(satelliteBackup?.threadIds?.length) + || Boolean(satelliteBackup?.threads?.length); + if (needsThreads && (!paths.state || !existsSync(paths.state))) { + return { + ok: false, + trashDir: id, + count: 0, + bytes: 0, + restoredPaths: [], + error: "db_reconcile_failed", + }; + } + const probe = probeStateDbWritable(codexHome, busyTimeoutMs); + if (!probe.ok) { + return { + ok: false, + trashDir: id, + count: 0, + bytes: 0, + restoredPaths: [], + error: probe.error === "codex_busy" ? "codex_busy" : "db_reconcile_failed", + }; + } + } + + // Acquire only the satellite locks still needed so a busy DB for an already- + // finished section cannot block resume. Locks happen before moves on a fresh + // attempt so failure stays retryable (nothing has left the stage yet). + let satelliteLocks: SatelliteWriteLocks | undefined; + if (needAnySatellite) { + try { + satelliteLocks = beginSatelliteWriteLocks(paths, busyTimeoutMs, { + logs: pendingSections.logs, + memories: pendingSections.memories, + goals: pendingSections.goals, + }); + } catch (error) { + return { + ok: false, + trashDir: id, + count: 0, + bytes: 0, + restoredPaths: [], + error: mapDbError(error) === "codex_busy" ? "codex_busy" : "db_reconcile_failed", + }; + } + } + + const failBeforeMoves = (error: RestoreErrorCode): RestoreResult => { + if (satelliteLocks) rollbackAllSatelliteLocks(satelliteLocks); + return { ok: false, trashDir: id, count: 0, bytes: 0, restoredPaths: [], error }; + }; + + // Plan renames: staged basename → original archived_sessions path. + // Resume accepts destinations already restored by this incomplete attempt. + const alreadyMoved: StagedFile[] = []; + const toMove: StagedFile[] = []; + for (const entry of entries) { + for (const rel of entry.physicalRelPaths) { + const base = basename(rel); + const from = join(stageDir, base); + let to: string; + try { + to = absFromRel(codexHome, rel); + } catch { + return failBeforeMoves("invalid_trash"); + } + const fromExists = existsSync(from); + const toExists = existsSync(to); + if (toExists && acceptedDest.has(rel) && !fromExists) { + alreadyMoved.push({ from, to, relPath: rel }); + continue; + } + if (toExists) { + return failBeforeMoves("dest_exists"); + } + if (!fromExists) { + return failBeforeMoves("fs_failed"); + } + toMove.push({ from, to, relPath: rel }); + } + } + + const planned = [...alreadyMoved, ...toMove]; + const restoredPaths = [...new Set(entries.map(e => e.relPath))]; + const bytes = entries.reduce((sum, e) => sum + (e.bytes || 0), 0); + const partialCounts = { count: restoredPaths.length, bytes, restoredPaths }; + + let pendingWriteCount = 0; + const persistPending = (): void => { + pendingWriteCount += 1; + const isInitial = pendingWriteCount === 1; + writeRestorePending( + stageDir, + { + version: 1, + filesRestored: true, + acceptedDestRels: planned.map(m => m.relPath), + pending: { ...pendingSections }, + }, + { + failWrite: Boolean(isInitial && hooks?.failInitialPendingWrite), + failBeforeRename: Boolean(!isInitial && hooks?.failPendingWriteBeforeRename), + }, + ); + }; + + // Durable resume marker before any rollout leaves the stage. Crash after a + // later move can still accept destinations from this marker. + try { + persistPending(); + } catch { + return failBeforeMoves("fs_failed"); + } + + const newlyMoved: StagedFile[] = []; + try { + mkdirSync(join(codexHome, ARCHIVED_SESSIONS_DIR), { recursive: true }); + for (const item of toMove) { + // Atomic no-replace (.trash ↔ archived_sessions). Mid-loop failure keeps + // already-placed dests and the durable planned acceptedDestRels marker. + renameNoReplace(item.from, item.to); + newlyMoved.push(item); + if ( + hooks?.failAfterMoveCount !== undefined + && newlyMoved.length >= hooks.failAfterMoveCount + ) { + throw new Error("test_fail_after_move_count"); + } + } + } catch (error) { + // Marker was written before any move. Never reverse successful renames or + // drop/narrow acceptedDestRels — resume must accept placed dests and finish + // the remaining staged files. + if (satelliteLocks) rollbackAllSatelliteLocks(satelliteLocks); + const placed = [...alreadyMoved, ...newlyMoved]; + const placedPhysical = new Set(placed.map(m => m.relPath)); + const partialEntries = entries.filter(e => + e.physicalRelPaths.every(rel => placedPhysical.has(rel)), + ); + const midMoveRestored = [...new Set(partialEntries.map(e => e.relPath))]; + return { + ok: false, + trashDir: id, + count: midMoveRestored.length, + bytes: partialEntries.reduce((sum, e) => sum + (e.bytes || 0), 0), + restoredPaths: midMoveRestored, + error: isExistError(error) ? "dest_exists" : "fs_failed", + }; + } + + const moved = [...alreadyMoved, ...newlyMoved]; + + /** + * Never compensate DBs or restage files after moves. Keep restored files, + * persist which sections remain, and return accurate partial counts. + */ + const abortAfterMoves = (error: RestoreErrorCode): RestoreResult => { + if (satelliteLocks) { + rollbackAllSatelliteLocks(satelliteLocks); + satelliteLocks = undefined; + } + try { + persistPending(); + } catch { + /* best-effort — files already restored; prior atomic marker remains */ + } + return { ok: false, trashDir: id, ...partialCounts, error }; + }; + + if (hooks?.holdAfterFileMovesMs !== undefined) { + const holdMs = Math.max(0, Math.floor(hooks.holdAfterFileMovesMs)); + if (holdMs > 0) { + const deadline = Date.now() + holdMs; + while (Date.now() < deadline) { /* test-only spin wait */ } + } + } + + if (hooks?.failAfterFileMoves) { + return abortAfterMoves("fs_failed"); + } + + if (pendingSections.state) { + const threadsRestored = restoreThreadsFromManifest( + paths.state, + entries, + satelliteBackup, + busyTimeoutMs, + codexHome, + ); + if (!threadsRestored.ok) { + return abortAfterMoves( + threadsRestored.error === "codex_busy" ? "codex_busy" : "db_reconcile_failed", + ); + } + pendingSections.state = false; + try { + persistPending(); + } catch { + return abortAfterMoves("fs_failed"); + } + } + + if (hooks?.failAfterStateCommit) { + return abortAfterMoves("db_reconcile_failed"); + } + + if (satelliteLocks && satelliteBackup) { + const locks = satelliteLocks; + try { + // Commit one satellite DB at a time; uncommitted txs roll back via + // rollbackAllSatelliteLocks. Completed sections are cleared in pending. + if (pendingSections.logs && satelliteBackup.logs) { + if (!locks.logs) throw new Error("missing_logs_lock"); + if (!tableExists(locks.logs.db, "logs")) throw new Error("missing_logs_table"); + insertRowsConflictIgnore(locks.logs.db, "logs", satelliteBackup.logs.rows); + commitSatelliteLock(locks.logs); + locks.logs = undefined; + pendingSections.logs = false; + persistPending(); + if (hooks?.failAfterFirstSatelliteCommit) { + throw new Error("test_fail_after_first_satellite"); + } + } + if (pendingSections.memories && satelliteBackup.memories) { + if (!locks.memories) throw new Error("missing_memories_lock"); + const mem = satelliteBackup.memories; + if (!tableExists(locks.memories.db, "stage1_outputs")) { + throw new Error("missing_stage1_outputs_table"); + } + insertRowsConflictIgnore(locks.memories.db, "stage1_outputs", mem.stage1); + if (tableExists(locks.memories.db, "jobs")) { + insertRowsConflictIgnore(locks.memories.db, "jobs", mem.stage1Jobs); + if (mem.consolidateTouched) { + restoreConsolidateGlobalJob( + locks.memories.db, + mem.consolidateJob, + mem.consolidatePostImage, + ); + } + } + commitSatelliteLock(locks.memories); + locks.memories = undefined; + pendingSections.memories = false; + persistPending(); + if (hooks?.failAfterFirstSatelliteCommit && !satelliteBackup.logs) { + throw new Error("test_fail_after_first_satellite"); + } + } + if (pendingSections.goals && satelliteBackup.goals) { + if (!locks.goals) throw new Error("missing_goals_lock"); + const g = satelliteBackup.goals; + if (!tableExists(locks.goals.db, "thread_goals")) { + throw new Error("missing_thread_goals_table"); + } + insertRowsConflictIgnore(locks.goals.db, "thread_goals", g.goals); + if (tableExists(locks.goals.db, "thread_goal_continuation_deferrals")) { + insertRowsConflictIgnore( + locks.goals.db, + "thread_goal_continuation_deferrals", + g.deferrals, + ); + } + commitSatelliteLock(locks.goals); + locks.goals = undefined; + pendingSections.goals = false; + persistPending(); + if ( + hooks?.failAfterFirstSatelliteCommit + && !satelliteBackup.logs + && !satelliteBackup.memories + ) { + throw new Error("test_fail_after_first_satellite"); + } + } + // Close any locks acquired for DBs that had no pending work / backup rows. + rollbackAllSatelliteLocks(locks); + satelliteLocks = undefined; + } catch (error) { + return abortAfterMoves( + mapDbError(error) === "codex_busy" ? "codex_busy" : "db_reconcile_failed", + ); + } + } + + if ( + pendingSections.state + || pendingSections.logs + || pendingSections.memories + || pendingSections.goals + ) { + return abortAfterMoves("db_reconcile_failed"); + } + + // Completeness gate: every planned file must sit at its restored path, and the stage + // must hold no leftover rollout files, before we destroy the quarantine evidence. + for (const item of moved) { + if (!existsSync(item.to) || existsSync(item.from)) { + return abortAfterMoves("fs_failed"); + } + } + try { + if (hooks?.failAtLeftoverStageGate) { + return abortAfterMoves("fs_failed"); + } + for (const name of readdirSync(stageDir)) { + if ( + name === "manifest.json" + || name === SATELLITE_BACKUP_FILE + || name === RESTORE_PENDING_FILE + ) { + continue; + } + if (!isRolloutFileName(name)) continue; + return abortAfterMoves("fs_failed"); + } + } catch { + return abortAfterMoves("fs_failed"); + } + + if (!finalizeRestoredStage(stageDir, codexHome, hooks)) { + return { + ok: false, + trashDir: id, + ...partialCounts, + error: "fs_failed", + }; + } + removeEmptyTrashRoot(codexHome); + + return { + ok: true, + trashDir: id, + ...partialCounts, + }; +} diff --git a/src/storage/policy-job.ts b/src/storage/policy-job.ts index e647cd66ddf..6000bebe5a8 100644 --- a/src/storage/policy-job.ts +++ b/src/storage/policy-job.ts @@ -6,6 +6,11 @@ * proxy event loop stays responsive. */ import type { CleanupMode, CleanupResult } from "./cleanup"; +import { resolveCodexHomeDir } from "../codex/home"; +import { + endStorageMutation, + tryBeginStorageMutation, +} from "./storage-mutation-coordinator"; import { isPolicyDue, readStorageCleanupPolicyFromConfig, @@ -20,7 +25,7 @@ export interface PolicyJobOutcome { ok: boolean; skipped?: PolicySkipReason; deferred?: "codex_busy"; - error?: CleanupResult["error"] | "evaluation_failed" | "worker_failed"; + error?: CleanupResult["error"] | "evaluation_failed" | "worker_failed" | "storage_mutation_busy"; mode?: CleanupMode; freedBytes?: number; removed?: number; @@ -71,6 +76,14 @@ let livePolicyApply: ((policy: PolicyRunResult["policy"]) => void) | undefined; let cancelActiveRun: (() => void) | null = null; /** Bumped on abort/reset so a late worker completion cannot clobber newer job state. */ let runGeneration = 0; +/** CODEX_HOME whose mutation slot this job holds (parent thread only). */ +let heldMutationHome: string | undefined; + +function releaseHeldMutationSlot(): void { + if (heldMutationHome === undefined) return; + endStorageMutation(heldMutationHome); + heldMutationHome = undefined; +} export function setStorageCleanupPolicyJobLiveApply( apply: ((policy: PolicyRunResult["policy"]) => void) | null, @@ -119,6 +132,7 @@ export function resetStorageCleanupPolicyJobForTests(): void { } inflight = null; testHooks = null; + releaseHeldMutationSlot(); state = { status: "idle" }; } @@ -129,6 +143,7 @@ export function abortStorageCleanupPolicyJob(): void { try { activeWorker.terminate(); } catch { /* */ } activeWorker = null; } + releaseHeldMutationSlot(); if (state.status === "running") { state = { ...state, @@ -187,6 +202,17 @@ function applyFailed(message: string): void { }; } +function applyMutationBusy(): void { + state = { + status: "idle", + reason: state.reason, + startedAt: state.startedAt, + finishedAt: Date.now(), + lastError: "storage_mutation_busy", + lastOutcome: { ok: false, error: "storage_mutation_busy" }, + }; +} + function runInWorker(opts: RequestPolicyRunOptions & { blockMs?: number }): Promise { return new Promise((resolve, reject) => { const requestId = crypto.randomUUID(); @@ -254,6 +280,15 @@ function runInWorker(opts: RequestPolicyRunOptions & { blockMs?: number }): Prom async function executeJob(opts: RequestPolicyRunOptions): Promise { const generation = ++runGeneration; + const codexHome = opts.codexHome ?? resolveCodexHomeDir(); + const gate = tryBeginStorageMutation("policy", codexHome); + if (!gate.acquired) { + if (generation === runGeneration) { + applyMutationBusy(); + } + return; + } + heldMutationHome = codexHome; try { const blockMs = testHooks?.blockMs; let result: PolicyRunResult; @@ -264,13 +299,14 @@ async function executeJob(opts: RequestPolicyRunOptions): Promise { result = runStorageCleanupPolicy({ reason: opts.reason, force: opts.force === true, - ...(opts.codexHome ? { codexHome: opts.codexHome } : {}), + codexHome, ...(opts.busyTimeoutMs !== undefined ? { busyTimeoutMs: opts.busyTimeoutMs } : {}), ...(typeof blockMs === "number" && blockMs > 0 ? { holdAfterLoadMs: blockMs } : {}), }); } else { result = await runInWorker({ ...opts, + codexHome, ...(typeof blockMs === "number" && blockMs > 0 ? { blockMs } : {}), }); } @@ -281,7 +317,7 @@ async function executeJob(opts: RequestPolicyRunOptions): Promise { if (generation !== runGeneration) return; applyFailed(err instanceof Error ? err.message : "worker_failed"); } finally { - if (generation === runGeneration) inflight = null; + releaseHeldMutationSlot(); } } @@ -303,7 +339,11 @@ export function requestStorageCleanupPolicyRun( ...(state.lastOutcome ? { lastOutcome: state.lastOutcome } : {}), }; - inflight = executeJob(opts); + const job = executeJob(opts); + inflight = job; + void job.finally(() => { + if (inflight === job) inflight = null; + }); return { accepted: true, state: getStorageCleanupPolicyJobState() }; } diff --git a/src/storage/policy.ts b/src/storage/policy.ts index 51813fa8cba..f716376c5a4 100644 --- a/src/storage/policy.ts +++ b/src/storage/policy.ts @@ -15,7 +15,8 @@ import { executeArchivedCleanup, listArchivedCandidates, previewExactArchivedCleanup, - selectOldestPercent, + selectOldestPercentSkippingPendingRestore, + selectReduceToBytesSkippingPendingRestore, type CleanupMode, type CleanupResult, type ExecuteCleanupOptions, @@ -350,7 +351,7 @@ export function selectPolicyPreview( if (reduceTo !== undefined) { // Exact candidate set — do not approximate via percent (would over-delete). - const desired = selectReduceToBytes(all, reduceTo); + const desired = selectReduceToBytesSkippingPendingRestore(all, reduceTo, codexHome); const preview = previewExactArchivedCleanup(desired, codexHome); return { archivedBytes, @@ -358,13 +359,13 @@ export function selectPolicyPreview( count: preview.count, bytes: preview.bytes, digest: preview.digest, - candidateRelPaths: desired.map(c => c.relPath), + candidateRelPaths: preview.candidates.map(c => c.relPath), }; } // Reuse the already-listed candidates — avoid a second archive directory walk. const percent = Math.min(100, Math.max(0, Math.floor(removePct ?? 0))); - const selected = selectOldestPercent(all, percent); + const selected = selectOldestPercentSkippingPendingRestore(all, percent, codexHome); return { archivedBytes, percent, diff --git a/src/storage/restore-job.ts b/src/storage/restore-job.ts new file mode 100644 index 00000000000..31181a38899 --- /dev/null +++ b/src/storage/restore-job.ts @@ -0,0 +1,242 @@ +/** + * Single-flight controller for trash restore runs. + * + * Heavy work (file moves, SQLite reconcile) runs in a Bun Worker so the proxy + * event loop stays responsive while the management API awaits the outcome. + * + * Restore shares the CODEX_HOME storage-mutation coordinator with manual cleanup + * and (Phase 3) policy-driven cleanup. Concurrent callers receive + * `storage_mutation_busy` (409) instead of queueing. + */ +import { resolveCodexHomeDir } from "../codex/home"; +import { restoreTrashEntry, type RestoreResult, type RestoreTestHooks } from "./cleanup"; +import { + resetStorageMutationCoordinatorForTests, + setStorageMutationCoordinatorTestHooks, + withStorageMutationSlot, + type StorageMutationCoordinatorTestHooks, +} from "./storage-mutation-coordinator"; + +export interface RestoreJobTestHooks extends StorageMutationCoordinatorTestHooks { + /** + * When true, run on the main thread via dynamic import + optional sleep. + * Responsiveness tests must leave this unset so work stays in a Worker. + */ + runInProcess?: boolean; + /** Expose GET /api/storage/trash/restore/test-stream for responsiveness tests. */ + enableTestStream?: boolean; + /** Forwarded to restoreTrashEntry _test hooks. */ + restoreTest?: RestoreTestHooks; +} + +const WORKER_TIMEOUT_MS = 10 * 60 * 1000; + +const WORKER_REJECTION_CODES: Record = { + restore_worker_timeout: "restore_worker_timeout", + aborted: "restore_worker_aborted", + worker_failed: "restore_worker_failed", +}; + +/** Map a rejected worker promise to a precise restore outcome (exported for tests). */ +export function restoreResultFromWorkerRejection(err: unknown, trashId: string): RestoreResult { + const raw = err instanceof Error ? err.message : String(err); + const error = WORKER_REJECTION_CODES[raw] ?? "restore_worker_failed"; + const detail = + error === "restore_worker_failed" && raw !== "worker_failed" && raw.trim().length > 0 + ? raw + : undefined; + console.error( + `[storage] trash restore worker failed (${error}) trashId=${trashId}${detail ? `: ${detail}` : ""}`, + ); + return { + ok: false, + count: 0, + bytes: 0, + restoredPaths: [], + error, + ...(detail ? { message: detail } : {}), + }; +} + +let activeWorker: Worker | null = null; +let testHooks: RestoreJobTestHooks | null = null; +let cancelActiveRun: (() => void) | null = null; + +export function setRestoreTrashJobTestHooks(hooks: RestoreJobTestHooks | null): void { + testHooks = hooks; + setStorageMutationCoordinatorTestHooks(hooks); +} + +export function resetRestoreTrashJobForTests(): void { + if (activeWorker) { + try { activeWorker.terminate(); } catch { /* */ } + activeWorker = null; + } + cancelActiveRun?.(); + cancelActiveRun = null; + testHooks = null; + resetStorageMutationCoordinatorForTests(); +} + +/** Terminate an in-flight worker during process shutdown. */ +export function abortRestoreTrashJob(): void { + if (activeWorker) { + try { activeWorker.terminate(); } catch { /* */ } + activeWorker = null; + } + cancelActiveRun?.(); + cancelActiveRun = null; +} + +/** Test-only SSE/text stream served from the proxy while a worker is blocked. */ +export function getRestoreTrashTestStreamResponse(): Response | null { + if (!testHooks?.enableTestStream) return null; + const encoder = new TextEncoder(); + return new Response( + new ReadableStream({ + async start(controller) { + for (let i = 0; i < 8; i++) { + controller.enqueue(encoder.encode(`chunk-${i}\n`)); + await Bun.sleep(50); + } + controller.close(); + }, + }), + { headers: { "content-type": "text/plain; charset=utf-8" } }, + ); +} + +function runInWorker(opts: { + trashId: string; + codexHome?: string; + busyTimeoutMs?: number; + blockMs?: number; + restoreTest?: RestoreTestHooks; +}): Promise { + return new Promise((resolve, reject) => { + const requestId = crypto.randomUUID(); + let settled = false; + const worker = new Worker(new URL("./restore-worker.ts", import.meta.url).href); + activeWorker = worker; + + const timer = setTimeout(() => { + if (settled) return; + settled = true; + cancelActiveRun = null; + try { worker.terminate(); } catch { /* */ } + if (activeWorker === worker) activeWorker = null; + reject(new Error("restore_worker_timeout")); + }, WORKER_TIMEOUT_MS); + + const finish = (fn: () => void) => { + if (settled) return; + settled = true; + cancelActiveRun = null; + clearTimeout(timer); + if (activeWorker === worker) activeWorker = null; + try { worker.terminate(); } catch { /* */ } + fn(); + }; + + cancelActiveRun = () => { + finish(() => reject(new Error("aborted"))); + }; + + worker.onmessage = (event: MessageEvent) => { + const data = event.data; + if (!data || typeof data !== "object") return; + const msg = data as Record; + if (msg.requestId !== requestId) return; + if (msg.type === "done" && msg.result && typeof msg.result === "object") { + finish(() => resolve(msg.result as RestoreResult)); + return; + } + if (msg.type === "error") { + const message = typeof msg.message === "string" ? msg.message : "worker_failed"; + finish(() => reject(new Error(message))); + } + }; + + worker.onerror = (err: ErrorEvent) => { + finish(() => reject(err.error instanceof Error ? err.error : new Error(err.message || "worker_failed"))); + }; + + worker.postMessage({ + type: "run", + requestId, + trashId: opts.trashId, + ...(opts.codexHome ? { codexHome: opts.codexHome } : {}), + ...(opts.busyTimeoutMs !== undefined ? { busyTimeoutMs: opts.busyTimeoutMs } : {}), + ...(opts.blockMs !== undefined ? { blockMs: opts.blockMs } : {}), + ...(opts.restoreTest ? { restoreTest: opts.restoreTest } : {}), + env: { + ...(process.env.CODEX_HOME ? { CODEX_HOME: process.env.CODEX_HOME } : {}), + ...(process.env.OPENCODEX_HOME ? { OPENCODEX_HOME: process.env.OPENCODEX_HOME } : {}), + }, + }); + }); +} + +async function executeRestore(opts: { + trashId: string; + codexHome?: string; + busyTimeoutMs?: number; + _test?: RestoreTestHooks; +}): Promise { + const blockMs = testHooks?.blockMs; + const restoreTest = opts._test ?? testHooks?.restoreTest; + + if (testHooks?.runInProcess) { + if (typeof blockMs === "number" && blockMs > 0) await Bun.sleep(blockMs); + return restoreTrashEntry(opts.trashId, { + ...(opts.codexHome ? { codexHome: opts.codexHome } : {}), + ...(opts.busyTimeoutMs !== undefined ? { busyTimeoutMs: opts.busyTimeoutMs } : {}), + ...(restoreTest ? { _test: restoreTest } : {}), + }); + } + + try { + return await runInWorker({ + trashId: opts.trashId, + ...(opts.codexHome ? { codexHome: opts.codexHome } : {}), + ...(opts.busyTimeoutMs !== undefined ? { busyTimeoutMs: opts.busyTimeoutMs } : {}), + ...(typeof blockMs === "number" && blockMs > 0 ? { blockMs } : {}), + ...(restoreTest ? { restoreTest } : {}), + }); + } catch (err) { + return restoreResultFromWorkerRejection(err, opts.trashId); + } +} + +function busyRestoreResult(): RestoreResult { + return { + ok: false, + count: 0, + bytes: 0, + restoredPaths: [], + error: "storage_mutation_busy", + }; +} + +/** + * Run trash restore off the event loop under the shared storage-mutation gate. + * Returns `storage_mutation_busy` when cleanup or another restore is in flight. + */ +export async function runRestoreTrashEntryJob( + trashId: string, + options?: { + codexHome?: string; + busyTimeoutMs?: number; + _test?: RestoreTestHooks; + }, +): Promise { + const codexHome = options?.codexHome ?? resolveCodexHomeDir(); + const result = await withStorageMutationSlot("restore", codexHome, () => + executeRestore({ trashId, ...options }), + ); + if (result && typeof result === "object" && "ok" in result && result.ok === false + && "error" in result && result.error === "storage_mutation_busy") { + return busyRestoreResult(); + } + return result as RestoreResult; +} diff --git a/src/storage/restore-worker.ts b/src/storage/restore-worker.ts new file mode 100644 index 00000000000..6d527977b48 --- /dev/null +++ b/src/storage/restore-worker.ts @@ -0,0 +1,52 @@ +/** + * Worker-thread entry for trash restore runs. + * Keeps file moves and SQLite reconcile off the proxy event loop. + */ +import { restoreTrashEntry, type RestoreResult, type RestoreTestHooks } from "./cleanup"; + +interface RunMessage { + type: "run"; + requestId: string; + trashId: string; + codexHome?: string; + busyTimeoutMs?: number; + /** Test-only: block before restore so responsiveness tests can probe /healthz. */ + blockMs?: number; + restoreTest?: RestoreTestHooks; + /** Env snapshot — Workers may not see parent mutations on all platforms. */ + env?: { CODEX_HOME?: string; OPENCODEX_HOME?: string }; +} + +function isRunMessage(data: unknown): data is RunMessage { + if (!data || typeof data !== "object" || Array.isArray(data)) return false; + const o = data as Record; + return o.type === "run" && typeof o.requestId === "string" && typeof o.trashId === "string"; +} + +declare const self: Worker; + +self.onmessage = async (event: MessageEvent) => { + if (!isRunMessage(event.data)) return; + const { requestId, trashId, codexHome, busyTimeoutMs, blockMs, restoreTest, env } = event.data; + try { + if (env?.CODEX_HOME) process.env.CODEX_HOME = env.CODEX_HOME; + if (env?.OPENCODEX_HOME) process.env.OPENCODEX_HOME = env.OPENCODEX_HOME; + if (typeof blockMs === "number" && Number.isFinite(blockMs) && blockMs > 0) { + await Bun.sleep(Math.floor(blockMs)); + } + const result = restoreTrashEntry(trashId, { + ...(codexHome ? { codexHome } : {}), + ...(busyTimeoutMs !== undefined ? { busyTimeoutMs } : {}), + ...(restoreTest ? { _test: restoreTest } : {}), + }); + self.postMessage({ type: "done", requestId, result }); + } catch (err) { + self.postMessage({ + type: "error", + requestId, + message: err instanceof Error ? err.message : "worker_failed", + }); + } +}; + +export type { RestoreResult }; diff --git a/src/storage/storage-mutation-coordinator.ts b/src/storage/storage-mutation-coordinator.ts new file mode 100644 index 00000000000..05d7965df34 --- /dev/null +++ b/src/storage/storage-mutation-coordinator.ts @@ -0,0 +1,109 @@ +/** + * Single-flight gate for CODEX_HOME storage mutations (cleanup, restore, policy). + * + * Manual cleanup, trash restore, and (Phase 3) policy-driven cleanup share one + * in-flight slot per resolved CODEX_HOME. A second caller receives + * `storage_mutation_busy` immediately — no per-operation queues. + */ +import { resolve } from "node:path"; +import { resolveCodexHomeDir } from "../codex/home"; + +export type StorageMutationKind = "cleanup" | "restore" | "policy"; + +export type StorageMutationBusyError = "storage_mutation_busy"; + +export interface StorageMutationCoordinatorTestHooks { + /** Block after acquiring the slot, before mutation work (race tests). */ + blockMs?: number; +} + +interface ActiveSlot { + kind: StorageMutationKind; + startedAt: number; +} + +const slots = new Map(); +let testHooks: StorageMutationCoordinatorTestHooks | null = null; + +function slotKey(codexHome?: string): string { + return resolve(codexHome ?? resolveCodexHomeDir()); +} + +export function setStorageMutationCoordinatorTestHooks( + hooks: StorageMutationCoordinatorTestHooks | null, +): void { + testHooks = hooks; +} + +export function resetStorageMutationCoordinatorForTests(): void { + testHooks = null; + slots.clear(); +} + +export function getActiveStorageMutation( + codexHome?: string, +): { kind: StorageMutationKind; startedAt: number } | null { + const active = slots.get(slotKey(codexHome)); + return active ? { kind: active.kind, startedAt: active.startedAt } : null; +} + +export function tryBeginStorageMutation( + kind: StorageMutationKind, + codexHome?: string, +): { acquired: true } | { acquired: false; error: StorageMutationBusyError } { + const key = slotKey(codexHome); + if (slots.has(key)) { + return { acquired: false, error: "storage_mutation_busy" }; + } + slots.set(key, { kind, startedAt: Date.now() }); + return { acquired: true }; +} + +export function endStorageMutation(codexHome?: string): void { + slots.delete(slotKey(codexHome)); +} + +async function applyCoordinatorBlock(): Promise { + const blockMs = testHooks?.blockMs; + if (typeof blockMs === "number" && Number.isFinite(blockMs) && blockMs > 0) { + await Bun.sleep(Math.floor(blockMs)); + } +} + +/** + * Phase 3 policy worker integration — wrap policy-driven cleanup FS/DB work. + * Returns `{ ok: false, error: 'storage_mutation_busy' }` when another mutation + * holds the CODEX_HOME slot. + */ +export async function runPolicyStorageMutation( + codexHome: string | undefined, + work: () => T | Promise, +): Promise { + const gate = tryBeginStorageMutation("policy", codexHome); + if (!gate.acquired) { + return { ok: false, error: "storage_mutation_busy" }; + } + try { + await applyCoordinatorBlock(); + return await work(); + } finally { + endStorageMutation(codexHome); + } +} + +export async function withStorageMutationSlot( + kind: StorageMutationKind, + codexHome: string | undefined, + work: () => T | Promise, +): Promise { + const gate = tryBeginStorageMutation(kind, codexHome); + if (!gate.acquired) { + return { ok: false, error: "storage_mutation_busy" }; + } + try { + await applyCoordinatorBlock(); + return await work(); + } finally { + endStorageMutation(codexHome); + } +} diff --git a/tests/api-storage-cleanup.test.ts b/tests/api-storage-cleanup.test.ts index 9b88427f794..b52a30e3a99 100644 --- a/tests/api-storage-cleanup.test.ts +++ b/tests/api-storage-cleanup.test.ts @@ -250,3 +250,146 @@ describe("POST /api/storage/cleanup", () => { } }); }); + +describe("GET /api/storage/trash + POST restore", () => { + test("lists relative trash entries and restores without host paths", async () => { + seedArchived(isolatedCodexHome!.path); + const server = startServer(0); + try { + const previewRes = await fetch(new URL("/api/storage/cleanup/preview", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50 }), + }); + const preview = await previewRes.json(); + const cleanupRes = await fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50, mode: "quarantine", digest: preview.digest }), + }); + expect(cleanupRes.status).toBe(200); + const cleanup = await cleanupRes.json(); + expect(cleanup.trashDir).toMatch(/^\.trash\//); + + const listRes = await fetch(new URL("/api/storage/trash", server.url)); + expect(listRes.status).toBe(200); + const listed = await listRes.json(); + expect(listed.entries).toHaveLength(1); + expect(listed.entries[0].id).toBe(cleanup.trashDir); + expect(listed.entries[0].fileCount).toBe(1); + expect(JSON.stringify(listed)).not.toContain(isolatedCodexHome!.path.replaceAll("\\", "\\\\")); + + const restoreRes = await fetch(new URL("/api/storage/trash/restore", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ id: cleanup.trashDir }), + }); + expect(restoreRes.status).toBe(200); + const restored = await restoreRes.json(); + expect(restored.ok).toBe(true); + expect(restored.count).toBe(1); + expect(restored.restoredPaths).toEqual(["archived_sessions/rollout-old.jsonl"]); + expect(JSON.stringify(restored)).not.toContain(isolatedCodexHome!.path.replaceAll("\\", "\\\\")); + expect(existsSync(join(isolatedCodexHome!.path, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + + const listAfter = await (await fetch(new URL("/api/storage/trash", server.url))).json(); + expect(listAfter.entries).toEqual([]); + } finally { + await server.stop(true); + } + }); + + test("restore returns 409 when Codex DB is busy", async () => { + seedArchived(isolatedCodexHome!.path); + const server = startServer(0); + try { + const previewRes = await fetch(new URL("/api/storage/cleanup/preview", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50 }), + }); + const preview = await previewRes.json(); + const cleanupRes = await fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50, mode: "quarantine", digest: preview.digest }), + }); + const cleanup = await cleanupRes.json(); + + const locker = new Database(join(isolatedCodexHome!.path, "state_5.sqlite")); + locker.exec("BEGIN EXCLUSIVE"); + try { + const restoreRes = await fetch(new URL("/api/storage/trash/restore", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ id: cleanup.trashDir }), + }); + expect(restoreRes.status).toBe(409); + const body = await restoreRes.json(); + expect(body.ok).toBe(false); + expect(body.error).toBe("codex_busy"); + } finally { + locker.exec("ROLLBACK"); + locker.close(); + } + } finally { + await server.stop(true); + } + }); + + test("rejects invalid and missing trash ids", async () => { + const server = startServer(0); + try { + const invalid = await fetch(new URL("/api/storage/trash/restore", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ id: "../etc/passwd" }), + }); + expect(invalid.status).toBe(400); + expect((await invalid.json()).error).toBe("invalid_trash"); + + const missing = await fetch(new URL("/api/storage/trash/restore", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ id: ".trash/999999999" }), + }); + expect(missing.status).toBe(404); + expect((await missing.json()).error).toBe("missing_trash"); + } finally { + await server.stop(true); + } + }); + + test("restore returns 409 when destination archived file already exists", async () => { + seedArchived(isolatedCodexHome!.path); + const server = startServer(0); + try { + const previewRes = await fetch(new URL("/api/storage/cleanup/preview", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50 }), + }); + const preview = await previewRes.json(); + const cleanupRes = await fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50, mode: "quarantine", digest: preview.digest }), + }); + const cleanup = await cleanupRes.json(); + expect(cleanup.ok).toBe(true); + + writeFileSync(join(isolatedCodexHome!.path, "archived_sessions", "rollout-old.jsonl"), "COLLISION"); + const restoreRes = await fetch(new URL("/api/storage/trash/restore", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ id: cleanup.trashDir }), + }); + expect(restoreRes.status).toBe(409); + const body = await restoreRes.json(); + expect(body.ok).toBe(false); + expect(body.error).toBe("dest_exists"); + } finally { + await server.stop(true); + } + }); +}); diff --git a/tests/api-storage-policy.test.ts b/tests/api-storage-policy.test.ts index ef872476606..4dead36800a 100644 --- a/tests/api-storage-policy.test.ts +++ b/tests/api-storage-policy.test.ts @@ -7,6 +7,10 @@ import { saveConfig } from "../src/config"; import { startServer } from "../src/server"; import type { OcxConfig } from "../src/types"; import { installIsolatedCodexHome, type IsolatedCodexHome } from "./helpers/isolated-codex-home"; +import { + resetArchivedCleanupJobForTests, + setArchivedCleanupJobTestHooks, +} from "../src/storage/cleanup-job"; import { resetStorageCleanupPolicyJobForTests, setStorageCleanupPolicyJobTestHooks, @@ -106,12 +110,15 @@ beforeEach(() => { saveConfig(baseConfig()); stopStorageCleanupScheduler(); resetStorageCleanupPolicyJobForTests(); + resetArchivedCleanupJobForTests(); }); afterEach(() => { stopStorageCleanupScheduler(); resetStorageCleanupPolicyJobForTests(); setStorageCleanupPolicyJobTestHooks(null); + resetArchivedCleanupJobForTests(); + setArchivedCleanupJobTestHooks(null); if (previousHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previousHome; isolatedCodexHome?.restore(); @@ -283,7 +290,10 @@ describe("storage cleanup policy API", () => { }, { timeout: 30_000 }); test("blocked worker completion preserves concurrent policy PUT edits", async () => { - setStorageCleanupPolicyJobTestHooks({ blockMs: 1_200 }); + // Long hold so a slow Windows CI worker spawn can load the enabled snapshot + // before the concurrent PUT lands inside holdAfterLoadMs. + const blockMs = 1_500; + setStorageCleanupPolicyJobTestHooks({ blockMs }); seedArchived(isolatedCodexHome!.path); const server = startServer(0); try { @@ -307,17 +317,21 @@ describe("storage cleanup policy API", () => { expect(runStart.started).toBe(true); expect(runStart.job?.status).toBe("running"); - // Wait until the job is visibly running, then edit policy while the worker holds. - const editDeadline = Date.now() + 2_000; + // Status flips to running before the worker loads policy — wait for that marker, + // then allow spawn+load margin before editing during the hold window. + const editDeadline = Date.now() + 5_000; + let sawRunning = false; while (Date.now() < editDeadline) { const peek = await fetch(new URL("/api/storage/cleanup-policy", server.url)); const peekBody = await peek.json() as { job?: { status?: string } }; - if (peekBody.job?.status === "running") break; + if (peekBody.job?.status === "running") { + sawRunning = true; + break; + } await Bun.sleep(20); } - - // Let the worker load the start-of-job snapshot, then edit during the hold window. - await Bun.sleep(450); + expect(sawRunning).toBe(true); + await Bun.sleep(800); const put = await fetch(new URL("/api/storage/cleanup-policy", server.url), { method: "PUT", @@ -337,6 +351,7 @@ describe("storage cleanup policy API", () => { const done = await waitForJobIdle(server.url, runStart.job!.startedAt); expect(done.job.lastOutcome?.ok).toBe(true); + expect(done.job.lastOutcome?.skipped).toBeUndefined(); expect(done.job.lastOutcome?.removed).toBe(1); expect(done.enabled).toBe(false); expect(done.lastRun?.removed).toBe(1); @@ -366,4 +381,62 @@ describe("storage cleanup policy API", () => { resetStorageCleanupPolicyJobForTests(); } }, { timeout: 30_000 }); + + test("storage_mutation_busy clears inflight so a later policy run can start", async () => { + setArchivedCleanupJobTestHooks({ blockMs: 600 }); + seedArchived(isolatedCodexHome!.path); + const server = startServer(0); + try { + await fetch(new URL("/api/storage/cleanup-policy", server.url), { + method: "PUT", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + enabled: true, + trigger: { archivedBytesOver: 50 }, + target: { removeOldestPercent: 50 }, + schedule: "manual", + mode: "quarantine", + }), + }); + + const previewRes = await fetch(new URL("/api/storage/cleanup/preview", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50 }), + }); + const preview = await previewRes.json() as { digest: string }; + const cleanupPromise = fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50, mode: "quarantine", digest: preview.digest }), + }); + await Bun.sleep(50); + + const blockedRun = await fetch(new URL("/api/storage/cleanup-policy/run", server.url), { + method: "POST", + }); + expect(blockedRun.status).toBe(200); + const blockedStart = await blockedRun.json() as { job: { startedAt: number } }; + const blockedDone = await waitForJobIdle(server.url, blockedStart.job.startedAt); + expect(blockedDone.job.lastOutcome?.ok).toBe(false); + expect(blockedDone.job.lastOutcome?.error).toBe("storage_mutation_busy"); + + await cleanupPromise; + + const retryRun = await fetch(new URL("/api/storage/cleanup-policy/run", server.url), { + method: "POST", + }); + expect(retryRun.status).toBe(200); + const retryStart = await retryRun.json() as { started?: boolean; job: { startedAt: number } }; + expect(retryStart.started).toBe(true); + const retryDone = await waitForJobIdle(server.url, retryStart.job.startedAt); + expect(retryDone.job.lastOutcome?.ok).toBe(true); + expect(retryDone.job.lastOutcome?.skipped).toBeUndefined(); + expect(retryDone.job.lastOutcome?.removed).toBe(1); + } finally { + await server.stop(true); + stopStorageCleanupScheduler(); + resetStorageCleanupPolicyJobForTests(); + } + }, { timeout: 30_000 }); }); diff --git a/tests/storage-cleanup.test.ts b/tests/storage-cleanup.test.ts index 85bbc0d6ea3..a5b03048085 100644 --- a/tests/storage-cleanup.test.ts +++ b/tests/storage-cleanup.test.ts @@ -4,8 +4,11 @@ import { existsSync, mkdirSync, mkdtempSync, + readdirSync, readFileSync, + renameSync, rmSync, + unlinkSync, utimesSync, writeFileSync, } from "node:fs"; @@ -15,8 +18,10 @@ import { computePreviewDigest, executeArchivedCleanup, listArchivedCandidates, + listTrashEntries, normalizeArchivedRolloutPath, previewArchivedCleanup, + restoreTrashEntry, selectOldestPercent, type ExecuteCleanupOptions, } from "../src/storage/cleanup"; @@ -369,6 +374,7 @@ describe("executeArchivedCleanup", () => { } }); + // Windows CI: SQLite lock contention across satellite DBs can exceed the default 5s. test("busy final satellite lock rolls back earlier satellite write locks", () => { home = buildHome({ withSatelliteStores: true }); const goalsLocker = new Database(join(home, "goals_1.sqlite")); @@ -412,7 +418,7 @@ describe("executeArchivedCleanup", () => { try { logsRead?.close(); } catch { /* */ } try { memoriesRead?.close(); } catch { /* */ } } - }); + }, { timeout: 20_000 }); test("rolls back staged renames when a later rename fails", () => { home = buildHome(); @@ -588,6 +594,7 @@ describe("executeArchivedCleanup", () => { expect(ids).toEqual(["active"]); }); + // Windows CI: multi-satellite permanent cleanup can exceed the default 5s under lock/IO load. test("permanent cleanup removes logs, goals, and memory rows for deleted threads", () => { home = buildHome({ withSatelliteStores: true }); const result = runWithDigest(100, "permanent", home); @@ -632,7 +639,7 @@ describe("executeArchivedCleanup", () => { const ids = state.query<{ id: string }, []>("SELECT id FROM threads").all().map(r => r.id); state.close(); expect(ids).toEqual(["active"]); - }); + }, { timeout: 20_000 }); test("threads read failure leaves every file and database unchanged", () => { home = buildHome({ withSatelliteStores: true }); @@ -714,6 +721,7 @@ describe("executeArchivedCleanup", () => { expect(stateAfter.query("SELECT id, rollout_path, archived FROM threads ORDER BY id").all()).toEqual(threads); stateAfter.close(); }, + { timeout: 30_000 }, ); test("satellite restore failure keeps recovery trashDir and manifest", () => { @@ -871,6 +879,7 @@ describe("executeArchivedCleanup", () => { state.close(); }, { timeout: 30_000 }); + // Windows CI: same multi-satellite restore profile as above (timed out at 5s on PR #558). test("concurrent consolidate enqueue watermark change is preserved on restore", () => { home = buildHome({ withSatelliteStores: true }); const result = runWithDigest(100, "permanent", home, { @@ -908,3 +917,882 @@ describe("executeArchivedCleanup", () => { state.close(); }, { timeout: 30_000 }); }); + +describe("listTrashEntries + restoreTrashEntry", () => { + test("round-trip quarantine → restore returns files and threads", () => { + home = buildHome(); + const original = readFileSync(join(home, "archived_sessions", "rollout-old.jsonl"), "utf8"); + const quarantined = runWithDigest(50, "quarantine", home, { now: 1_700_000_000_100 }); + expect(quarantined.ok).toBe(true); + expect(quarantined.trashDir).toBe(".trash/1700000000100"); + + const listed = listTrashEntries(home); + expect(listed).toHaveLength(1); + expect(listed[0]!.id).toBe(".trash/1700000000100"); + expect(listed[0]!.fileCount).toBe(1); + expect(listed[0]!.mode).toBe("quarantine"); + expect(listed[0]!.quarantinedAt).toBe(1_700_000_000_100); + expect(JSON.stringify(listed)).not.toContain(home.replaceAll("\\", "\\\\")); + + const restored = restoreTrashEntry(".trash/1700000000100", { codexHome: home }); + expect(restored.ok).toBe(true); + expect(restored.count).toBe(1); + expect(restored.restoredPaths).toEqual(["archived_sessions/rollout-old.jsonl"]); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(readFileSync(join(home, "archived_sessions", "rollout-old.jsonl"), "utf8")).toBe(original); + expect(existsSync(join(home, ".trash", "1700000000100"))).toBe(false); + expect(listTrashEntries(home)).toEqual([]); + + const db = new Database(join(home, "state_5.sqlite"), { readonly: true }); + const row = db.query<{ id: string; rollout_path: string; archived: number | null }, []>( + "SELECT id, rollout_path, archived FROM threads WHERE id='told'", + ).get(); + db.close(); + expect(row).toEqual({ + id: "told", + rollout_path: "archived_sessions/rollout-old.jsonl", + archived: 1, + }); + expect(existsSync(join(home, "sessions", "2026", "05", "27", "rollout-active.jsonl"))).toBe(true); + }); + + test("restore refuses when Codex DB is busy", () => { + home = buildHome(); + const quarantined = runWithDigest(50, "quarantine", home, { now: 1_700_000_000_200 }); + expect(quarantined.ok).toBe(true); + + const locker = new Database(join(home, "state_5.sqlite")); + locker.exec("BEGIN EXCLUSIVE"); + try { + const restored = restoreTrashEntry(".trash/1700000000200", { + codexHome: home, + busyTimeoutMs: 1, + }); + expect(restored.ok).toBe(false); + expect(restored.error).toBe("codex_busy"); + expect(existsSync(join(home, ".trash", "1700000000200", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(false); + } finally { + locker.exec("ROLLBACK"); + locker.close(); + } + }); + + test("rejects missing, invalid, and path-escaping trash ids", () => { + home = buildHome(); + expect(restoreTrashEntry(".trash/999", { codexHome: home }).error).toBe("missing_trash"); + expect(restoreTrashEntry("../etc/passwd", { codexHome: home }).error).toBe("invalid_trash"); + expect(restoreTrashEntry(".trash/../sessions", { codexHome: home }).error).toBe("invalid_trash"); + expect(restoreTrashEntry(".trash/not-an-epoch", { codexHome: home }).error).toBe("invalid_trash"); + expect(restoreTrashEntry("", { codexHome: home }).error).toBe("invalid_trash"); + }); + + test("refuses restore when destination archived file already exists", () => { + home = buildHome(); + const quarantined = runWithDigest(50, "quarantine", home, { now: 1_700_000_000_300 }); + expect(quarantined.ok).toBe(true); + writeFileSync(join(home, "archived_sessions", "rollout-old.jsonl"), "COLLISION"); + const restored = restoreTrashEntry(".trash/1700000000300", { codexHome: home }); + expect(restored.ok).toBe(false); + expect(restored.error).toBe("dest_exists"); + expect(existsSync(join(home, ".trash", "1700000000300", "rollout-old.jsonl"))).toBe(true); + }); + + test("quarantine retains satellite-backup and restores satellite + state dependents", () => { + home = buildHome({ withSatelliteStores: true, withDynamicTools: true, withSpawnEdges: true }); + // 100%: spawn edge told→tmid stays inside the delete set (cross-boundary edges refuse cleanup). + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_000_400 }); + expect(quarantined.ok).toBe(true); + const backupPath = join(home, ".trash", "1700000000400", "satellite-backup.json"); + expect(existsSync(backupPath)).toBe(true); + const backup = JSON.parse(readFileSync(backupPath, "utf8")) as { + threadIds: string[]; + threads?: Array>; + dynamicTools?: Array>; + spawnEdges?: Array>; + logs?: { path: string; rows: unknown[] }; + }; + expect(backup.threadIds).toContain("told"); + expect(backup.threads?.some(r => r.id === "told")).toBe(true); + expect(backup.dynamicTools?.length).toBeGreaterThan(0); + expect(backup.spawnEdges?.length).toBeGreaterThan(0); + expect(backup.logs?.rows.length).toBeGreaterThan(0); + + // Simulate Codex rotating to a newer logs DB — restore must remap to current home. + renameSync(join(home, "logs_2.sqlite"), join(home, "logs_3.sqlite")); + + const restored = restoreTrashEntry(".trash/1700000000400", { codexHome: home }); + expect(restored.ok).toBe(true); + expect(existsSync(backupPath)).toBe(false); + + const state = new Database(join(home, "state_5.sqlite"), { readonly: true }); + expect(state.query("SELECT id FROM threads WHERE id='told'").get()).toEqual({ id: "told" }); + expect(state.query("SELECT COUNT(*) AS n FROM thread_dynamic_tools WHERE thread_id='told'").get()) + .toEqual({ n: 1 }); + expect(state.query("SELECT COUNT(*) AS n FROM thread_spawn_edges WHERE parent_thread_id='told' OR child_thread_id='told'").get()) + .toEqual({ n: 1 }); + state.close(); + + const logs = new Database(join(home, "logs_3.sqlite"), { readonly: true }); + expect(logs.query("SELECT COUNT(*) AS n FROM logs WHERE thread_id='told'").get()).toEqual({ n: 1 }); + logs.close(); + }); + + test("rejects malformed satellite-backup.json without destroying trash", () => { + home = buildHome(); + const quarantined = runWithDigest(50, "quarantine", home, { now: 1_700_000_000_500 }); + expect(quarantined.ok).toBe(true); + writeFileSync(join(home, ".trash", "1700000000500", "satellite-backup.json"), "{truncated"); + const restored = restoreTrashEntry(".trash/1700000000500", { codexHome: home }); + expect(restored.ok).toBe(false); + expect(restored.error).toBe("db_reconcile_failed"); + expect(existsSync(join(home, ".trash", "1700000000500", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(false); + }); + + test("refuses restore when manifest has threads but state DB is absent", () => { + home = buildHome(); + const quarantined = runWithDigest(50, "quarantine", home, { now: 1_700_000_000_600 }); + expect(quarantined.ok).toBe(true); + unlinkSync(join(home, "state_5.sqlite")); + const restored = restoreTrashEntry(".trash/1700000000600", { codexHome: home }); + expect(restored.ok).toBe(false); + expect(restored.error).toBe("db_reconcile_failed"); + expect(existsSync(join(home, ".trash", "1700000000600", "rollout-old.jsonl"))).toBe(true); + }); + + test("partial purge survivors restore only remaining physical files", () => { + home = buildHome(); + writeFileSync(join(home, "archived_sessions", "rollout-old.jsonl.zst"), "ZST"); + utimesSync(join(home, "archived_sessions", "rollout-old.jsonl.zst"), OLD, OLD); + const preview = previewArchivedCleanup(50, home); + const result = executeArchivedCleanup({ + percent: 50, + mode: "permanent", + digest: preview.digest, + codexHome: home, + now: 1_700_000_000_700, + _test: { failPurgeBasenames: ["rollout-old.jsonl"] }, + }); + expect(result.ok).toBe(false); + expect(result.trashDir).toBe(".trash/1700000000700"); + const manifest = JSON.parse( + readFileSync(join(home, ".trash", "1700000000700", "manifest.json"), "utf8"), + ) as { entries: Array<{ physicalRelPaths: string[] }> }; + expect(manifest.entries[0]!.physicalRelPaths).toEqual(["archived_sessions/rollout-old.jsonl"]); + // Twin was purged; stage only has the survivor. + expect(existsSync(join(home, ".trash", "1700000000700", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(home, ".trash", "1700000000700", "rollout-old.jsonl.zst"))).toBe(false); + + const restored = restoreTrashEntry(".trash/1700000000700", { codexHome: home }); + expect(restored.ok).toBe(true); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + }); + + test("rejects mixed valid+malformed manifest entries as invalid_trash without touching staged files", () => { + home = buildHome(); + const stage = join(home, ".trash", "1700000000800"); + mkdirSync(stage, { recursive: true }); + writeFileSync(join(stage, "rollout-old.jsonl"), "OLD-STAGE"); + writeFileSync(join(stage, "rollout-mid.jsonl"), "MID-STAGE"); + writeFileSync(join(stage, "manifest.json"), JSON.stringify({ + quarantinedAt: 1_700_000_000_800, + mode: "quarantine", + entries: [ + { + relPath: "archived_sessions/rollout-old.jsonl", + bytes: 9, + mtimeMs: OLD.getTime(), + physicalRelPaths: ["archived_sessions/rollout-old.jsonl"], + threadId: "told", + rolloutPath: "archived_sessions/rollout-old.jsonl", + archived: 1, + }, + { + relPath: "archived_sessions/rollout-mid.jsonl", + bytes: 9, + mtimeMs: MID.getTime(), + // Malformed: non-string path must reject the entire manifest (no per-entry filter). + physicalRelPaths: ["archived_sessions/rollout-mid.jsonl", null], + threadId: "tmid", + rolloutPath: "archived_sessions/rollout-mid.jsonl", + archived: 1, + }, + ], + })); + + const restored = restoreTrashEntry(".trash/1700000000800", { codexHome: home }); + expect(restored.ok).toBe(false); + expect(restored.error).toBe("invalid_trash"); + expect(readFileSync(join(stage, "rollout-old.jsonl"), "utf8")).toBe("OLD-STAGE"); + expect(readFileSync(join(stage, "rollout-mid.jsonl"), "utf8")).toBe("MID-STAGE"); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(readFileSync(join(home, "archived_sessions", "rollout-old.jsonl"), "utf8")).toBe("OLD".repeat(10)); + expect(existsSync(join(home, "archived_sessions", "rollout-mid.jsonl"))).toBe(true); + expect(readFileSync(join(home, "archived_sessions", "rollout-mid.jsonl"), "utf8")).toBe("MID".repeat(20)); + }); + + test("legacy quarantine without satellite-backup reconstructs production-shaped thread from rollout", () => { + const dir = mkdtempSync(join(tmpdir(), "ocx-cleanup-legacy-")); + home = dir; + mkdirSync(join(dir, "archived_sessions"), { recursive: true }); + + const rolloutBody = [ + JSON.stringify({ + type: "session_meta", + timestamp: "2026-01-01T00:00:00.000Z", + payload: { + id: "told", + model_provider: "openai", + source: "cli", + cwd: "/tmp/project", + }, + }), + JSON.stringify({ + type: "event_msg", + timestamp: "2026-01-01T00:00:01.000Z", + payload: { type: "user_message", message: "restore me please" }, + }), + ].join("\n") + "\n"; + + const stage = join(dir, ".trash", "1700000000900"); + mkdirSync(stage, { recursive: true }); + writeFileSync(join(stage, "rollout-old.jsonl"), rolloutBody); + writeFileSync(join(stage, "manifest.json"), JSON.stringify({ + quarantinedAt: 1_700_000_000_900, + mode: "quarantine", + entries: [ + { + relPath: "archived_sessions/rollout-old.jsonl", + bytes: Buffer.byteLength(rolloutBody), + mtimeMs: OLD.getTime(), + physicalRelPaths: ["archived_sessions/rollout-old.jsonl"], + threadId: "told", + rolloutPath: "archived_sessions/rollout-old.jsonl", + archived: 1, + }, + ], + })); + // Intentionally no satellite-backup.json — Phase-2 legacy quarantine shape. + + const db = new Database(join(dir, "state_5.sqlite")); + db.exec(`CREATE TABLE threads ( + id TEXT PRIMARY KEY, + rollout_path TEXT NOT NULL, + model_provider TEXT NOT NULL, + source TEXT NOT NULL, + first_user_message TEXT NOT NULL, + has_user_event INTEGER NOT NULL DEFAULT 0, + archived INTEGER, + archived_at INTEGER + )`); + db.close(); + + const restored = restoreTrashEntry(".trash/1700000000900", { codexHome: home }); + expect(restored.ok).toBe(true); + expect(existsSync(join(dir, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(stage)).toBe(false); + + const state = new Database(join(dir, "state_5.sqlite"), { readonly: true }); + const row = state.query<{ + id: string; + rollout_path: string; + model_provider: string; + source: string; + first_user_message: string; + has_user_event: number; + archived: number | null; + }, []>( + `SELECT id, rollout_path, model_provider, source, first_user_message, has_user_event, archived + FROM threads WHERE id='told'`, + ).get(); + state.close(); + expect(row).toEqual({ + id: "told", + rollout_path: "archived_sessions/rollout-old.jsonl", + model_provider: "openai", + source: "cli", + first_user_message: "restore me please", + has_user_event: 1, + archived: 1, + }); + }); + + test.each([ + ["failAfterStateCommit", { failAfterStateCommit: true }, "db_reconcile_failed"], + ["failAfterFirstSatelliteCommit", { failAfterFirstSatelliteCommit: true }, "db_reconcile_failed"], + ["failAtLeftoverStageGate", { failAtLeftoverStageGate: true }, "fs_failed"], + ] as const)( + "injected %s leaves partial restore with pending marker and retry succeeds", + (_name, hook, error) => { + home = buildHome({ + withSatelliteStores: true, + withDynamicTools: true, + withSpawnEdges: true, + }); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_001_000 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + const failed = restoreTrashEntry(trashId, { codexHome: home, _test: { ...hook } }); + expect(failed.ok).toBe(false); + expect(failed.error).toBe(error); + // Files stay restored — accurate partial counts, no restage. + expect(failed.count).toBe(3); + expect(failed.restoredPaths).toEqual([ + "archived_sessions/rollout-old.jsonl", + "archived_sessions/rollout-mid.jsonl", + "archived_sessions/rollout-new.jsonl", + ]); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(stage, "rollout-old.jsonl"))).toBe(false); + expect(existsSync(join(stage, "manifest.json"))).toBe(true); + expect(existsSync(join(stage, "restore-pending.json"))).toBe(true); + + const pending = JSON.parse(readFileSync(join(stage, "restore-pending.json"), "utf8")) as { + filesRestored: boolean; + acceptedDestRels: string[]; + pending: { state: boolean; logs: boolean; memories: boolean; goals: boolean }; + }; + expect(pending.filesRestored).toBe(true); + expect(pending.acceptedDestRels).toContain("archived_sessions/rollout-old.jsonl"); + + const retried = restoreTrashEntry(trashId, { codexHome: home }); + expect(retried.ok).toBe(true); + expect(retried.count).toBe(3); + expect(existsSync(stage)).toBe(false); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + + const stateAfter = new Database(join(home, "state_5.sqlite"), { readonly: true }); + expect(stateAfter.query("SELECT id FROM threads WHERE id='told'").get()).toEqual({ id: "told" }); + expect(stateAfter.query("SELECT COUNT(*) AS n FROM thread_dynamic_tools WHERE thread_id='told'").get()) + .toEqual({ n: 1 }); + stateAfter.close(); + const logsAfter = new Database(join(home, "logs_2.sqlite"), { readonly: true }); + expect(logsAfter.query("SELECT COUNT(*) AS n FROM logs WHERE thread_id='told'").get()).toEqual({ n: 1 }); + logsAfter.close(); + }, + { timeout: 20_000 }, + ); + + test("late failure after logs commit keeps metadata, persists pending sections, and resume preserves pre-existing rows", () => { + // Regression for the old non-atomic path: compensate logs then hit busy on + // state — which deleted logs while leaving state+files. Prefer partial+resume. + home = buildHome({ + withSatelliteStores: true, + withDynamicTools: true, + withSpawnEdges: true, + }); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_001_100 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + const stateSeed = new Database(join(home, "state_5.sqlite")); + stateSeed.exec(`INSERT INTO threads VALUES ( + 'told','archived_sessions/rollout-old.jsonl',1,1,'legacy' + )`); + stateSeed.exec(`INSERT INTO thread_dynamic_tools VALUES ('told',0,'pre','d','{}')`); + stateSeed.close(); + const logsSeed = new Database(join(home, "logs_2.sqlite")); + logsSeed.exec( + `INSERT INTO logs (id, ts, level, target, thread_id, estimated_bytes) VALUES (1,1,'INFO','pre','told',10)`, + ); + logsSeed.close(); + const memSeed = new Database(join(home, "memories_1.sqlite")); + memSeed.exec(`INSERT INTO stage1_outputs VALUES ('told',1,'pre-m','pre-s',1,0)`); + memSeed.close(); + const goalsSeed = new Database(join(home, "goals_1.sqlite")); + goalsSeed.exec(`INSERT INTO thread_goals VALUES ('told','g1','pre','complete',0,0,1,1)`); + goalsSeed.close(); + + const failed = restoreTrashEntry(trashId, { + codexHome: home, + _test: { failAfterFirstSatelliteCommit: true }, + }); + expect(failed.ok).toBe(false); + expect(failed.error).toBe("db_reconcile_failed"); + expect(failed.count).toBe(3); + + // Files stay; no restage; no metadata compensation. + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(stage, "rollout-old.jsonl"))).toBe(false); + expect(existsSync(join(stage, "restore-pending.json"))).toBe(true); + + const pending = JSON.parse(readFileSync(join(stage, "restore-pending.json"), "utf8")) as { + pending: { state: boolean; logs: boolean; memories: boolean; goals: boolean }; + }; + expect(pending.pending).toEqual({ + state: false, + logs: false, + memories: true, + goals: true, + }); + + // Pre-existing conflict-ignored rows stay; newly restored mid/new rows stay. + const state = new Database(join(home, "state_5.sqlite"), { readonly: true }); + expect(state.query("SELECT COUNT(*) AS n FROM threads WHERE id IN ('told','tmid','tnew')").get()) + .toEqual({ n: 3 }); + expect( + state.query("SELECT name FROM thread_dynamic_tools WHERE thread_id='told' AND position=0").get(), + ).toEqual({ name: "pre" }); + state.close(); + const logs = new Database(join(home, "logs_2.sqlite"), { readonly: true }); + expect( + logs.query("SELECT ts, target, estimated_bytes FROM logs WHERE id=1").get(), + ).toEqual({ ts: 1, target: "pre", estimated_bytes: 10 }); + expect(logs.query("SELECT COUNT(*) AS n FROM logs WHERE thread_id IN ('tmid','tnew')").get()) + .toEqual({ n: 2 }); + logs.close(); + + // State locked after logs would have been compensated under the old design — + // resume must still finish without dest_exists and without deleting pre rows. + let locker: Database | undefined; + let retried: ReturnType; + try { + locker = new Database(join(home, "state_5.sqlite")); + locker.exec("BEGIN EXCLUSIVE"); + // State already done in pending — busy state must not block satellite resume. + retried = restoreTrashEntry(trashId, { codexHome: home, busyTimeoutMs: 1 }); + } finally { + try { locker?.exec("ROLLBACK"); } catch { /* */ } + try { locker?.close(); } catch { /* */ } + } + expect(retried!.ok).toBe(true); + expect(retried!.count).toBe(3); + expect(existsSync(stage)).toBe(false); + + const mem = new Database(join(home, "memories_1.sqlite"), { readonly: true }); + expect(mem.query("SELECT raw_memory FROM stage1_outputs WHERE thread_id='told'").get()) + .toEqual({ raw_memory: "pre-m" }); + expect( + mem.query("SELECT COUNT(*) AS n FROM stage1_outputs WHERE thread_id IN ('told','tmid','tnew')").get(), + ).toEqual({ n: 2 }); // fixture seeds told+tmid only + mem.close(); + const goals = new Database(join(home, "goals_1.sqlite"), { readonly: true }); + expect(goals.query("SELECT objective FROM thread_goals WHERE thread_id='told'").get()) + .toEqual({ objective: "pre" }); + expect( + goals.query("SELECT COUNT(*) AS n FROM thread_goals WHERE thread_id IN ('told','tmid','tnew')").get(), + ).toEqual({ n: 2 }); // fixture seeds told+tmid only + goals.close(); + }, { timeout: 20_000 }); + + test("leftover-stage failure never restages files and retry accepts destinations", () => { + // Regression for reverse-move failure after metadata compensation: restage + // could leave metadata deleted while files remained at dest. We never restage. + home = buildHome({ withSatelliteStores: true, withDynamicTools: true }); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_001_200 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + const logsSeed = new Database(join(home, "logs_2.sqlite")); + logsSeed.exec( + `INSERT INTO logs (id, ts, level, target, thread_id, estimated_bytes) VALUES (1,42,'INFO','pre','told',99)`, + ); + logsSeed.close(); + + const failed = restoreTrashEntry(trashId, { + codexHome: home, + _test: { failAtLeftoverStageGate: true }, + }); + expect(failed.ok).toBe(false); + expect(failed.error).toBe("fs_failed"); + expect(failed.count).toBe(3); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(stage, "rollout-old.jsonl"))).toBe(false); + expect(existsSync(join(stage, "restore-pending.json"))).toBe(true); + + const pending = JSON.parse(readFileSync(join(stage, "restore-pending.json"), "utf8")) as { + pending: { state: boolean; logs: boolean; memories: boolean; goals: boolean }; + }; + expect(pending.pending).toEqual({ + state: false, + logs: false, + memories: false, + goals: false, + }); + + // Pre-existing log preserved; satellite rows from this restore remain. + const logs = new Database(join(home, "logs_2.sqlite"), { readonly: true }); + expect( + logs.query("SELECT ts, target, estimated_bytes FROM logs WHERE id=1").get(), + ).toEqual({ ts: 42, target: "pre", estimated_bytes: 99 }); + expect(logs.query("SELECT COUNT(*) AS n FROM logs WHERE thread_id IN ('told','tmid','tnew')").get()) + .toEqual({ n: 3 }); + logs.close(); + + const retried = restoreTrashEntry(trashId, { codexHome: home }); + expect(retried.ok).toBe(true); + expect(retried.count).toBe(3); + expect(existsSync(stage)).toBe(false); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + + const logsAfter = new Database(join(home, "logs_2.sqlite"), { readonly: true }); + expect( + logsAfter.query("SELECT ts, target FROM logs WHERE id=1").get(), + ).toEqual({ ts: 42, target: "pre" }); + logsAfter.close(); + }, { timeout: 20_000 }); + + test("initial restore-pending write failure moves no files", () => { + home = buildHome({ withSatelliteStores: true }); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_001_300 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + const failed = restoreTrashEntry(trashId, { + codexHome: home, + _test: { failInitialPendingWrite: true }, + }); + expect(failed.ok).toBe(false); + expect(failed.error).toBe("fs_failed"); + expect(failed.count).toBe(0); + expect(existsSync(join(stage, "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(false); + expect(existsSync(join(stage, "restore-pending.json"))).toBe(false); + + const retried = restoreTrashEntry(trashId, { codexHome: home }); + expect(retried.ok).toBe(true); + expect(retried.count).toBe(3); + expect(existsSync(stage)).toBe(false); + }, { timeout: 20_000 }); + + test("interrupted pending update preserves the previous valid marker", () => { + home = buildHome({ withSatelliteStores: true }); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_001_400 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + const failed = restoreTrashEntry(trashId, { + codexHome: home, + _test: { failPendingWriteBeforeRename: true }, + }); + expect(failed.ok).toBe(false); + expect(failed.error).toBe("fs_failed"); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(stage, "restore-pending.json"))).toBe(true); + + const pending = JSON.parse(readFileSync(join(stage, "restore-pending.json"), "utf8")) as { + filesRestored: boolean; + acceptedDestRels: string[]; + pending: { state: boolean; logs: boolean; memories: boolean; goals: boolean }; + }; + // Atomic rename never landed the post-state update — prior marker remains. + expect(pending.filesRestored).toBe(true); + expect(pending.pending).toEqual({ + state: true, + logs: true, + memories: true, + goals: true, + }); + expect(pending.acceptedDestRels).toContain("archived_sessions/rollout-old.jsonl"); + + const retried = restoreTrashEntry(trashId, { codexHome: home }); + expect(retried.ok).toBe(true); + expect(retried.error).toBeUndefined(); + expect(existsSync(stage)).toBe(false); + }, { timeout: 20_000 }); + + test("crash after file move retries without dest_exists or fs_failed", () => { + home = buildHome({ withSatelliteStores: true }); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_001_500 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + const crashed = restoreTrashEntry(trashId, { + codexHome: home, + _test: { failAfterFileMoves: true }, + }); + expect(crashed.ok).toBe(false); + expect(crashed.error).toBe("fs_failed"); + expect(crashed.count).toBe(3); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(stage, "rollout-old.jsonl"))).toBe(false); + expect(existsSync(join(stage, "restore-pending.json"))).toBe(true); + + const retried = restoreTrashEntry(trashId, { codexHome: home }); + expect(retried.ok).toBe(true); + expect(retried.error).not.toBe("dest_exists"); + expect(retried.error).not.toBe("fs_failed"); + expect(retried.count).toBe(3); + expect(existsSync(stage)).toBe(false); + }, { timeout: 20_000 }); + + test("mid-move failure keeps placed dest, marker, and resumes without dest_exists", () => { + // First rename succeeds, second throws. Do not reverse the first file or drop + // the durable planned acceptedDestRels — retry must finish both states. + home = buildHome({ withSatelliteStores: true }); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_001_550 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + const failed = restoreTrashEntry(trashId, { + codexHome: home, + _test: { failAfterMoveCount: 1 }, + }); + expect(failed.ok).toBe(false); + expect(failed.error).toBe("fs_failed"); + expect(failed.count).toBe(1); + expect(failed.restoredPaths).toEqual(["archived_sessions/rollout-old.jsonl"]); + + // First dest stays; remaining rollouts stay staged; marker keeps full plan. + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(stage, "rollout-old.jsonl"))).toBe(false); + expect(existsSync(join(stage, "rollout-mid.jsonl"))).toBe(true); + expect(existsSync(join(stage, "rollout-new.jsonl"))).toBe(true); + expect(existsSync(join(stage, "restore-pending.json"))).toBe(true); + + const pending = JSON.parse(readFileSync(join(stage, "restore-pending.json"), "utf8")) as { + filesRestored: boolean; + acceptedDestRels: string[]; + pending: { state: boolean; logs: boolean; memories: boolean; goals: boolean }; + }; + expect(pending.filesRestored).toBe(true); + expect(pending.acceptedDestRels).toEqual([ + "archived_sessions/rollout-old.jsonl", + "archived_sessions/rollout-mid.jsonl", + "archived_sessions/rollout-new.jsonl", + ]); + expect(pending.pending).toEqual({ + state: true, + logs: true, + memories: true, + goals: true, + }); + + const retried = restoreTrashEntry(trashId, { codexHome: home }); + expect(retried.ok).toBe(true); + expect(retried.error).not.toBe("dest_exists"); + expect(retried.error).not.toBe("fs_failed"); + expect(retried.count).toBe(3); + expect(existsSync(stage)).toBe(false); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(home, "archived_sessions", "rollout-mid.jsonl"))).toBe(true); + expect(existsSync(join(home, "archived_sessions", "rollout-new.jsonl"))).toBe(true); + }, { timeout: 20_000 }); + + test("malformed restore-pending.json is not treated as a fresh restore", () => { + home = buildHome({ withSatelliteStores: true }); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_001_600 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + writeFileSync(join(stage, "restore-pending.json"), "{not-valid-json", "utf8"); + const failed = restoreTrashEntry(trashId, { codexHome: home }); + expect(failed.ok).toBe(false); + expect(failed.error).toBe("fs_failed"); + expect(failed.count).toBe(0); + // Stage intact — no silent fresh restore that would move files under a corrupt marker. + expect(existsSync(join(stage, "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(false); + expect(readFileSync(join(stage, "restore-pending.json"), "utf8")).toBe("{not-valid-json"); + }, { timeout: 20_000 }); + + test("resume with owed satellite sections and missing backup fails closed", () => { + home = buildHome({ withSatelliteStores: true }); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_001_700 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + const partial = restoreTrashEntry(trashId, { + codexHome: home, + _test: { failAfterFirstSatelliteCommit: true }, + }); + expect(partial.ok).toBe(false); + expect(existsSync(join(stage, "restore-pending.json"))).toBe(true); + expect(existsSync(join(stage, "satellite-backup.json"))).toBe(true); + const pendingAfterPartial = JSON.parse(readFileSync(join(stage, "restore-pending.json"), "utf8")) as { + pending: { logs: boolean; memories: boolean; goals: boolean }; + }; + expect( + pendingAfterPartial.pending.logs + || pendingAfterPartial.pending.memories + || pendingAfterPartial.pending.goals, + ).toBe(true); + + unlinkSync(join(stage, "satellite-backup.json")); + const failed = restoreTrashEntry(trashId, { codexHome: home }); + expect(failed.ok).toBe(false); + expect(failed.error).toBe("db_reconcile_failed"); + expect(existsSync(stage)).toBe(true); + expect(existsSync(join(stage, "manifest.json"))).toBe(true); + }, { timeout: 20_000 }); + + test("resume with owed logs section but missing logs in backup fails closed", () => { + home = buildHome({ withSatelliteStores: true }); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_001_800 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + const partial = restoreTrashEntry(trashId, { + codexHome: home, + _test: { failAfterStateCommit: true }, + }); + expect(partial.ok).toBe(false); + + const backupPath = join(stage, "satellite-backup.json"); + const backup = JSON.parse(readFileSync(backupPath, "utf8")) as Record; + delete backup.logs; + writeFileSync(backupPath, JSON.stringify(backup)); + + const pending = JSON.parse(readFileSync(join(stage, "restore-pending.json"), "utf8")) as { + pending: { logs: boolean; memories: boolean; goals: boolean }; + }; + expect(pending.pending.logs).toBe(true); + + const failed = restoreTrashEntry(trashId, { codexHome: home }); + expect(failed.ok).toBe(false); + expect(failed.error).toBe("db_reconcile_failed"); + expect(existsSync(stage)).toBe(true); + expect(existsSync(join(stage, "manifest.json"))).toBe(true); + expect(existsSync(join(stage, "restore-pending.json"))).toBe(true); + }, { timeout: 20_000 }); + + test("failed tombstone rename keeps stage recoverable and listed", () => { + home = buildHome(); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_001_900 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + const failed = restoreTrashEntry(trashId, { + codexHome: home, + _test: { failStageTombstoneRename: true }, + }); + expect(failed.ok).toBe(false); + expect(failed.error).toBe("fs_failed"); + expect(failed.count).toBe(3); + expect(existsSync(stage)).toBe(true); + expect(existsSync(join(stage, "manifest.json"))).toBe(true); + expect(listTrashEntries(home).some(e => e.id === trashId)).toBe(true); + + const retried = restoreTrashEntry(trashId, { codexHome: home }); + expect(retried.ok).toBe(true); + expect(existsSync(stage)).toBe(false); + expect(listTrashEntries(home)).toEqual([]); + }, { timeout: 20_000 }); + + test("tombstone delete failure reports success without phantom trash entry", () => { + home = buildHome(); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_002_000 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + const restored = restoreTrashEntry(trashId, { + codexHome: home, + _test: { failTombstoneDelete: true }, + }); + expect(restored.ok).toBe(true); + expect(existsSync(stage)).toBe(false); + expect(listTrashEntries(home)).toEqual([]); + + const trashRoot = join(home, ".trash"); + const tombstones = readdirSync(trashRoot).filter(n => n.startsWith(".tombstone-")); + expect(tombstones.length).toBe(1); + }, { timeout: 20_000 }); + + test("cleanup rejects overlap with accepted restore-pending destinations after state-commit failure", () => { + home = buildHome({ withSatelliteStores: true }); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_002_000 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + const restoredRel = "archived_sessions/rollout-old.jsonl"; + + const partial = restoreTrashEntry(trashId, { + codexHome: home, + _test: { failAfterStateCommit: true }, + }); + expect(partial.ok).toBe(false); + expect(existsSync(join(home, restoredRel))).toBe(true); + expect(existsSync(join(stage, "restore-pending.json"))).toBe(true); + + const overlapDigest = computePreviewDigest( + selectOldestPercent(listArchivedCandidates(home), 100), + 100, + ); + const blocked = executeArchivedCleanup({ + percent: 100, + mode: "quarantine", + digest: overlapDigest, + codexHome: home, + }); + expect(blocked.ok).toBe(false); + expect(blocked.error).toBe("restore_pending_overlap"); + expect(existsSync(join(home, restoredRel))).toBe(true); + expect(existsSync(stage)).toBe(true); + + const retry = restoreTrashEntry(trashId, { codexHome: home }); + expect(retry.ok).toBe(true); + expect(existsSync(join(home, restoredRel))).toBe(true); + }, { timeout: 20_000 }); + + test("cleanup rejects overlap with accepted restore-pending destinations after file-move failure", () => { + home = buildHome(); + const quarantined = runWithDigest(100, "quarantine", home, { now: 1_700_000_002_100 }); + expect(quarantined.ok).toBe(true); + const trashId = quarantined.trashDir!; + const stage = join(home, ...trashId.split("/")); + + const partial = restoreTrashEntry(trashId, { + codexHome: home, + _test: { failAfterMoveCount: 1 }, + }); + expect(partial.ok).toBe(false); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(stage, "rollout-mid.jsonl"))).toBe(true); + expect(existsSync(join(stage, "restore-pending.json"))).toBe(true); + + const overlapDigest = computePreviewDigest( + selectOldestPercent(listArchivedCandidates(home), 100), + 100, + ); + const blocked = executeArchivedCleanup({ + percent: 100, + mode: "permanent", + digest: overlapDigest, + codexHome: home, + }); + expect(blocked.ok).toBe(false); + expect(blocked.error).toBe("restore_pending_overlap"); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(stage)).toBe(true); + + const retry = restoreTrashEntry(trashId, { codexHome: home }); + expect(retry.ok).toBe(true); + }, { timeout: 20_000 }); + + test("percent preview backfills past pending oldest archive for manual cleanup", () => { + home = buildHome(); + const stage = join(home, ".trash", "1700000"); + mkdirSync(stage, { recursive: true }); + writeFileSync(join(stage, "restore-pending.json"), JSON.stringify({ + version: 1, + filesRestored: true, + acceptedDestRels: ["archived_sessions/rollout-old.jsonl"], + pending: { state: true, logs: false, memories: false, goals: false }, + })); + + const preview = previewArchivedCleanup(34, home); + expect(preview.count).toBe(1); + expect(preview.candidates.map(c => c.relPath)).toEqual(["archived_sessions/rollout-mid.jsonl"]); + + const result = runWithDigest(34, "quarantine", home, { now: 1_700_000_001_000 }); + expect(result.ok).toBe(true); + expect(result.count).toBe(1); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(home, "archived_sessions", "rollout-mid.jsonl"))).toBe(false); + expect(existsSync(join(home, "archived_sessions", "rollout-new.jsonl"))).toBe(true); + }); +}); diff --git a/tests/storage-mutation-race.test.ts b/tests/storage-mutation-race.test.ts new file mode 100644 index 00000000000..bb841cd5834 --- /dev/null +++ b/tests/storage-mutation-race.test.ts @@ -0,0 +1,413 @@ +/** + * Regression: cleanup and restore must not mutate CODEX_HOME concurrently. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { Database } from "bun:sqlite"; +import { + existsSync, + mkdirSync, + mkdtempSync, + readdirSync, + readFileSync, + rmSync, + utimesSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../src/config"; +import { startServer } from "../src/server"; +import type { OcxConfig } from "../src/types"; +import { + resetArchivedCleanupJobForTests, + setArchivedCleanupJobTestHooks, +} from "../src/storage/cleanup-job"; +import { + resetStorageCleanupPolicyJobForTests, + setStorageCleanupPolicyJobTestHooks, +} from "../src/storage/policy-job"; +import { + resetRestoreTrashJobForTests, + runRestoreTrashEntryJob, + setRestoreTrashJobTestHooks, +} from "../src/storage/restore-job"; +import { + resetStorageMutationCoordinatorForTests, +} from "../src/storage/storage-mutation-coordinator"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "./helpers/isolated-codex-home"; + +let testDir = ""; +let previousHome: string | undefined; +let isolatedCodexHome: IsolatedCodexHome | null = null; + +function baseConfig(): OcxConfig { + return { + port: 0, + hostname: "127.0.0.1", + defaultProvider: "openai", + providers: { + openai: { + adapter: "openai-responses", + baseUrl: "https://api.openai.com/v1", + authMode: "forward", + }, + }, + } as OcxConfig; +} + +function seedArchivedPair(codexHome: string): void { + mkdirSync(join(codexHome, "archived_sessions")); + writeFileSync(join(codexHome, "archived_sessions", "rollout-old.jsonl"), "o".repeat(100)); + writeFileSync(join(codexHome, "archived_sessions", "rollout-new.jsonl"), "n".repeat(200)); + utimesSync(join(codexHome, "archived_sessions", "rollout-old.jsonl"), new Date("2026-01-01"), new Date("2026-01-01")); + utimesSync(join(codexHome, "archived_sessions", "rollout-new.jsonl"), new Date("2026-06-01"), new Date("2026-06-01")); + const db = new Database(join(codexHome, "state_5.sqlite")); + db.exec(`CREATE TABLE threads (id TEXT PRIMARY KEY, rollout_path TEXT NOT NULL, archived INTEGER)`); + db.exec(`INSERT INTO threads VALUES + ('told','archived_sessions/rollout-old.jsonl',1), + ('tnew','archived_sessions/rollout-new.jsonl',1) + `); + db.close(); +} + +function threadCount(codexHome: string): number { + const db = new Database(join(codexHome, "state_5.sqlite")); + const row = db.query("SELECT COUNT(*) AS c FROM threads").get() as { c: number }; + db.close(); + return row.c; +} + +function trashStageCount(codexHome: string): number { + const trashRoot = join(codexHome, ".trash"); + if (!existsSync(trashRoot)) return 0; + return readdirSync(trashRoot).filter(name => !name.startsWith(".")).length; +} + +async function previewDigest(serverUrl: string, percent: number): Promise<{ digest: string; count: number }> { + const res = await fetch(new URL("/api/storage/cleanup/preview", serverUrl), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent }), + }); + expect(res.status).toBe(200); + const body = await res.json(); + return { digest: body.digest, count: body.count }; +} + +async function enablePolicyAndRun(serverUrl: string): Promise<{ startedAt: number }> { + await fetch(new URL("/api/storage/cleanup-policy", serverUrl), { + method: "PUT", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + enabled: true, + trigger: { archivedBytesOver: 50 }, + target: { removeOldestPercent: 50 }, + schedule: "manual", + mode: "quarantine", + }), + }); + const run = await fetch(new URL("/api/storage/cleanup-policy/run", serverUrl), { method: "POST" }); + expect(run.status).toBe(200); + const body = await run.json(); + expect(body.started).toBe(true); + return { startedAt: body.job.startedAt as number }; +} + +async function waitForPolicyJob( + serverUrl: string, + startedAt: number, + timeoutMs = 20_000, +): Promise<{ job: { lastOutcome?: { ok?: boolean; error?: string; removed?: number } } }> { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + const res = await fetch(new URL("/api/storage/cleanup-policy", serverUrl)); + const body = await res.json() as { + job: { + status: string; + startedAt?: number; + lastOutcome?: { ok?: boolean; error?: string; removed?: number }; + }; + }; + if (body.job.status === "idle" && body.job.lastOutcome && body.job.startedAt === startedAt) { + return body; + } + await Bun.sleep(50); + } + throw new Error("policy job did not finish"); +} + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + isolatedCodexHome = installIsolatedCodexHome("ocx-storage-mutation-race-codex-"); + testDir = mkdtempSync(join(tmpdir(), "ocx-storage-mutation-race-")); + process.env.OPENCODEX_HOME = testDir; + saveConfig(baseConfig()); + resetRestoreTrashJobForTests(); + resetArchivedCleanupJobForTests(); + resetStorageCleanupPolicyJobForTests(); + resetStorageMutationCoordinatorForTests(); +}); + +afterEach(() => { + resetRestoreTrashJobForTests(); + resetArchivedCleanupJobForTests(); + resetStorageCleanupPolicyJobForTests(); + resetStorageMutationCoordinatorForTests(); + setRestoreTrashJobTestHooks(null); + setArchivedCleanupJobTestHooks(null); + setStorageCleanupPolicyJobTestHooks(null); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + isolatedCodexHome?.restore(); + isolatedCodexHome = null; + if (testDir) rmSync(testDir, { recursive: true, force: true }); + testDir = ""; +}); + +describe("storage mutation coordinator", () => { + test("policy run is rejected while restore holds the shared mutation slot", async () => { + const home = isolatedCodexHome!.path; + setRestoreTrashJobTestHooks({ blockMs: 400, runInProcess: true }); + seedArchivedPair(home); + + const server = startServer(0); + try { + const preview = await previewDigest(server.url, 50); + const cleanupRes = await fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50, mode: "quarantine", digest: preview.digest }), + }); + const cleanup = await cleanupRes.json(); + const trashId = cleanup.trashDir as string; + + const restorePromise = runRestoreTrashEntryJob(trashId, { codexHome: home }); + await Bun.sleep(50); + + const { startedAt } = await enablePolicyAndRun(server.url); + const done = await waitForPolicyJob(server.url, startedAt); + expect(done.job.lastOutcome?.ok).toBe(false); + expect(done.job.lastOutcome?.error).toBe("storage_mutation_busy"); + + const restoreResult = await restorePromise; + expect(restoreResult.ok).toBe(true); + } finally { + await server.stop(true); + } + }, { timeout: 30_000 }); + + test("policy run is rejected while manual cleanup holds the shared mutation slot", async () => { + const home = isolatedCodexHome!.path; + setArchivedCleanupJobTestHooks({ blockMs: 1200 }); + seedArchivedPair(home); + + const server = startServer(0); + try { + const preview = await previewDigest(server.url, 50); + const cleanupPromise = fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50, mode: "quarantine", digest: preview.digest }), + }); + await Bun.sleep(80); + + const { startedAt } = await enablePolicyAndRun(server.url); + const done = await waitForPolicyJob(server.url, startedAt); + expect(done.job.lastOutcome?.ok).toBe(false); + expect(done.job.lastOutcome?.error).toBe("storage_mutation_busy"); + + const cleanupRes = await cleanupPromise; + expect(cleanupRes.status).toBe(200); + } finally { + await server.stop(true); + } + }, { timeout: 30_000 }); + + test("manual cleanup and restore are rejected while policy job holds the shared mutation slot", async () => { + const home = isolatedCodexHome!.path; + setStorageCleanupPolicyJobTestHooks({ blockMs: 1200 }); + seedArchivedPair(home); + + const server = startServer(0); + try { + await enablePolicyAndRun(server.url); + await Bun.sleep(80); + + const preview = await previewDigest(server.url, 50); + const cleanupRes = await fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50, mode: "quarantine", digest: preview.digest }), + }); + expect(cleanupRes.status).toBe(409); + expect((await cleanupRes.json()).error).toBe("storage_mutation_busy"); + + const restoreRes = await fetch(new URL("/api/storage/trash/restore", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ id: ".trash/missing" }), + }); + expect(restoreRes.status).toBe(409); + expect((await restoreRes.json()).error).toBe("storage_mutation_busy"); + } finally { + await server.stop(true); + } + }, { timeout: 30_000 }); + + test("cleanup quarantine and permanent are rejected while restore holds slot after file moves", async () => { + const home = isolatedCodexHome!.path; + const holdMs = 2500; + setRestoreTrashJobTestHooks({ + restoreTest: { holdAfterFileMovesMs: holdMs }, + }); + seedArchivedPair(home); + + const server = startServer(0); + try { + const preview = await previewDigest(server.url, 50); + const cleanupRes = await fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50, mode: "quarantine", digest: preview.digest }), + }); + expect(cleanupRes.status).toBe(200); + const cleanup = await cleanupRes.json(); + const trashId = cleanup.trashDir as string; + const trashStage = join(home, trashId); + expect(existsSync(trashStage)).toBe(true); + + const remainingPreview = await previewDigest(server.url, 50); + expect(remainingPreview.count).toBe(1); + + const restorePromise = fetch(new URL("/api/storage/trash/restore", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ id: trashId }), + }); + + const restoredPath = join(home, "archived_sessions", "rollout-old.jsonl"); + const movedDeadline = Date.now() + 8000; + while (!existsSync(restoredPath) && Date.now() < movedDeadline) { + await Bun.sleep(20); + } + expect(existsSync(restoredPath)).toBe(true); + expect(existsSync(join(trashStage, "rollout-old.jsonl"))).toBe(false); + expect(existsSync(join(trashStage, "restore-pending.json"))).toBe(true); + + const quarantineDuring = await fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + percent: 50, + mode: "quarantine", + digest: remainingPreview.digest, + }), + }); + expect(quarantineDuring.status).toBe(409); + expect((await quarantineDuring.json()).error).toBe("storage_mutation_busy"); + expect(existsSync(join(home, "archived_sessions", "rollout-new.jsonl"))).toBe(true); + expect(trashStageCount(home)).toBe(1); + + const previewPermanent = await previewDigest(server.url, 100); + const permanentDuring = await fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + percent: 100, + mode: "permanent", + digest: previewPermanent.digest, + }), + }); + expect(permanentDuring.status).toBe(409); + expect((await permanentDuring.json()).error).toBe("storage_mutation_busy"); + + const restoreRes = await restorePromise; + expect(restoreRes.status).toBe(200); + const restored = await restoreRes.json(); + expect(restored.ok).toBe(true); + expect(existsSync(restoredPath)).toBe(true); + expect(threadCount(home)).toBe(2); + expect(readFileSync(restoredPath, "utf8")).toBe("o".repeat(100)); + } finally { + await server.stop(true); + } + }, { timeout: 45_000 }); + + test("restore is rejected while cleanup holds the shared mutation slot", async () => { + const home = isolatedCodexHome!.path; + const blockMs = 1200; + setArchivedCleanupJobTestHooks({ blockMs }); + seedArchivedPair(home); + + const server = startServer(0); + try { + const preview = await previewDigest(server.url, 50); + const cleanupPromise = fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50, mode: "quarantine", digest: preview.digest }), + }); + + await Bun.sleep(80); + + const restoreAttempt = await fetch(new URL("/api/storage/trash/restore", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ id: ".trash/never" }), + }); + expect(restoreAttempt.status).toBe(409); + expect((await restoreAttempt.json()).error).toBe("storage_mutation_busy"); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + expect(existsSync(join(home, "archived_sessions", "rollout-new.jsonl"))).toBe(true); + expect(trashStageCount(home)).toBe(0); + + const cleanupRes = await cleanupPromise; + expect(cleanupRes.status).toBe(200); + const cleanup = await cleanupRes.json(); + expect(cleanup.ok).toBe(true); + expect(cleanup.trashDir).toMatch(/^\.trash\//); + expect(existsSync(join(home, "archived_sessions", "rollout-old.jsonl"))).toBe(false); + expect(existsSync(join(home, "archived_sessions", "rollout-new.jsonl"))).toBe(true); + expect(threadCount(home)).toBe(1); + } finally { + await server.stop(true); + } + }, { timeout: 30_000 }); + + test("second restore while first is in flight returns storage_mutation_busy", async () => { + const home = isolatedCodexHome!.path; + setRestoreTrashJobTestHooks({ blockMs: 800, runInProcess: true }); + seedArchivedPair(home); + + const server = startServer(0); + try { + const preview = await previewDigest(server.url, 50); + const cleanupRes = await fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 50, mode: "quarantine", digest: preview.digest }), + }); + const cleanup = await cleanupRes.json(); + const trashId = cleanup.trashDir as string; + + const first = fetch(new URL("/api/storage/trash/restore", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ id: trashId }), + }); + await Bun.sleep(50); + const second = await fetch(new URL("/api/storage/trash/restore", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ id: trashId }), + }); + expect(second.status).toBe(409); + expect((await second.json()).error).toBe("storage_mutation_busy"); + + const firstRes = await first; + expect(firstRes.status).toBe(200); + } finally { + await server.stop(true); + } + }, { timeout: 30_000 }); +}); diff --git a/tests/storage-policy.test.ts b/tests/storage-policy.test.ts index e0d36222a57..318959b274f 100644 --- a/tests/storage-policy.test.ts +++ b/tests/storage-policy.test.ts @@ -156,6 +156,67 @@ describe("selection helpers", () => { expect(percentForAtLeastCount(4, 2)).toBe(50); }); + test("selectPolicyPreview backfills percent past pending oldest archive", () => { + const dir = seedHome([ + { name: "rollout-old.jsonl", bytes: 100, when: OLD }, + { name: "rollout-mid.jsonl", bytes: 100, when: MID }, + { name: "rollout-new.jsonl", bytes: 100, when: NEW }, + ]); + const stage = join(dir, ".trash", "99000"); + mkdirSync(stage, { recursive: true }); + writeFileSync(join(stage, "restore-pending.json"), JSON.stringify({ + version: 1, + filesRestored: true, + acceptedDestRels: ["archived_sessions/rollout-old.jsonl"], + pending: { state: true, logs: false, memories: false, goals: false }, + })); + + const preview = selectPolicyPreview( + policy({ + enabled: true, + trigger: { archivedBytesOver: 1 }, + target: { removeOldestPercent: 34 }, + schedule: "manual", + mode: "quarantine", + }), + dir, + ); + expect(preview.count).toBe(1); + expect(preview.bytes).toBe(100); + }); + + test("selectPolicyPreview reduceToBytes skips pending oldest and keeps backfilling", () => { + const dir = seedHome([ + { name: "rollout-old.jsonl", bytes: 100, when: OLD }, + { name: "rollout-mid.jsonl", bytes: 100, when: MID }, + { name: "rollout-new.jsonl", bytes: 100, when: NEW }, + ]); + const stage = join(dir, ".trash", "99001"); + mkdirSync(stage, { recursive: true }); + writeFileSync(join(stage, "restore-pending.json"), JSON.stringify({ + version: 1, + filesRestored: true, + acceptedDestRels: ["archived_sessions/rollout-old.jsonl"], + pending: { state: true, logs: false, memories: false, goals: false }, + })); + + const preview = selectPolicyPreview( + policy({ + enabled: true, + trigger: { archivedBytesOver: 1 }, + target: { reduceToBytes: 100 }, + schedule: "manual", + mode: "quarantine", + }), + dir, + ); + expect(preview.count).toBe(2); + expect(preview.candidateRelPaths).toEqual([ + "archived_sessions/rollout-mid.jsonl", + "archived_sessions/rollout-new.jsonl", + ]); + }); + test("selectPolicyPreview honors removeOldestPercent", () => { const dir = seedHome([ { name: "rollout-old.jsonl", bytes: 50, when: OLD }, diff --git a/tests/storage-restore-job-errors.test.ts b/tests/storage-restore-job-errors.test.ts new file mode 100644 index 00000000000..eb45d7b992b --- /dev/null +++ b/tests/storage-restore-job-errors.test.ts @@ -0,0 +1,36 @@ +import { describe, expect, test } from "bun:test"; +import { restoreResultFromWorkerRejection } from "../src/storage/restore-job"; + +describe("restoreResultFromWorkerRejection", () => { + test("maps worker timeout to restore_worker_timeout", () => { + const result = restoreResultFromWorkerRejection(new Error("restore_worker_timeout"), ".trash/1"); + expect(result).toMatchObject({ + ok: false, + error: "restore_worker_timeout", + count: 0, + bytes: 0, + restoredPaths: [], + }); + expect(result.message).toBeUndefined(); + }); + + test("maps cancel to restore_worker_aborted", () => { + const result = restoreResultFromWorkerRejection(new Error("aborted"), ".trash/1"); + expect(result.error).toBe("restore_worker_aborted"); + }); + + test("maps generic worker failure", () => { + const result = restoreResultFromWorkerRejection(new Error("worker_failed"), ".trash/1"); + expect(result.error).toBe("restore_worker_failed"); + expect(result.message).toBeUndefined(); + }); + + test("preserves unexpected worker crash detail", () => { + const result = restoreResultFromWorkerRejection( + new Error("sqlite disk I/O error"), + ".trash/1", + ); + expect(result.error).toBe("restore_worker_failed"); + expect(result.message).toBe("sqlite disk I/O error"); + }); +}); diff --git a/tests/storage-restore-job-responsive.test.ts b/tests/storage-restore-job-responsive.test.ts new file mode 100644 index 00000000000..fb59e7d2f15 --- /dev/null +++ b/tests/storage-restore-job-responsive.test.ts @@ -0,0 +1,151 @@ +/** + * Regression: a blocked restore Worker must not stall /healthz or an active + * streaming response on the proxy event loop. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { Database } from "bun:sqlite"; +import { existsSync, mkdirSync, mkdtempSync, rmSync, utimesSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../src/config"; +import { startServer } from "../src/server"; +import type { OcxConfig } from "../src/types"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "./helpers/isolated-codex-home"; +import { + resetRestoreTrashJobForTests, + setRestoreTrashJobTestHooks, +} from "../src/storage/restore-job"; + +let testDir = ""; +let previousHome: string | undefined; +let previousCleanupTestHooks: string | undefined; +let isolatedCodexHome: IsolatedCodexHome | null = null; + +function baseConfig(): OcxConfig { + return { + port: 0, + hostname: "127.0.0.1", + defaultProvider: "openai", + providers: { + openai: { + adapter: "openai-responses", + baseUrl: "https://api.openai.com/v1", + authMode: "forward", + }, + }, + } as OcxConfig; +} + +function seedArchived(codexHome: string): void { + mkdirSync(join(codexHome, "archived_sessions")); + writeFileSync(join(codexHome, "archived_sessions", "rollout-old.jsonl"), "o".repeat(100)); + utimesSync(join(codexHome, "archived_sessions", "rollout-old.jsonl"), new Date("2026-01-01"), new Date("2026-01-01")); + const db = new Database(join(codexHome, "state_5.sqlite")); + db.exec(`CREATE TABLE threads (id TEXT PRIMARY KEY, rollout_path TEXT NOT NULL, archived INTEGER)`); + db.exec(`INSERT INTO threads VALUES ('told','archived_sessions/rollout-old.jsonl',1)`); + db.close(); +} + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + previousCleanupTestHooks = process.env.OPENCODEX_CLEANUP_TEST_HOOKS; + process.env.OPENCODEX_CLEANUP_TEST_HOOKS = "1"; + isolatedCodexHome = installIsolatedCodexHome("ocx-restore-job-responsive-codex-"); + testDir = mkdtempSync(join(tmpdir(), "ocx-restore-job-responsive-")); + process.env.OPENCODEX_HOME = testDir; + saveConfig(baseConfig()); + resetRestoreTrashJobForTests(); +}); + +afterEach(() => { + resetRestoreTrashJobForTests(); + setRestoreTrashJobTestHooks(null); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + if (previousCleanupTestHooks === undefined) delete process.env.OPENCODEX_CLEANUP_TEST_HOOKS; + else process.env.OPENCODEX_CLEANUP_TEST_HOOKS = previousCleanupTestHooks; + isolatedCodexHome?.restore(); + isolatedCodexHome = null; + if (testDir) rmSync(testDir, { recursive: true, force: true }); + testDir = ""; +}); + +describe("storage trash restore job responsiveness", () => { + test("test-stream route is absent without OPENCODEX_CLEANUP_TEST_HOOKS", async () => { + delete process.env.OPENCODEX_CLEANUP_TEST_HOOKS; + setRestoreTrashJobTestHooks({ enableTestStream: true }); + const server = startServer(0); + try { + const res = await fetch(new URL("/api/storage/trash/restore/test-stream", server.url)); + expect(res.status).toBe(404); + } finally { + await server.stop(true); + } + }); + + test("blocked worker keeps /healthz and streaming response responsive", async () => { + const blockMs = 1200; + setRestoreTrashJobTestHooks({ blockMs, enableTestStream: true }); + seedArchived(isolatedCodexHome!.path); + + const server = startServer(0); + try { + const previewRes = await fetch(new URL("/api/storage/cleanup/preview", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 100 }), + }); + expect(previewRes.status).toBe(200); + const preview = await previewRes.json(); + const cleanupRes = await fetch(new URL("/api/storage/cleanup", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ percent: 100, mode: "quarantine", digest: preview.digest }), + }); + expect(cleanupRes.status).toBe(200); + const cleanup = await cleanupRes.json(); + expect(cleanup.trashDir).toMatch(/^\.trash\//); + + const restoreStarted = Date.now(); + const restorePromise = fetch(new URL("/api/storage/trash/restore", server.url), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ id: cleanup.trashDir }), + }); + + const streamPromise = (async () => { + const res = await fetch(new URL("/api/storage/trash/restore/test-stream", server.url)); + expect(res.ok).toBe(true); + return await res.text(); + })(); + + const healthSamples: number[] = []; + for (let i = 0; i < 6; i++) { + const t0 = Date.now(); + const health = await fetch(new URL("/healthz", server.url)); + expect(health.status).toBe(200); + const elapsed = Date.now() - t0; + if (i > 0) healthSamples.push(elapsed); + await Bun.sleep(40); + } + + const streamText = await streamPromise; + expect(streamText.split("\n").filter(Boolean).length).toBe(8); + + const maxHealthMs = Math.floor(blockMs / 3); + for (const sample of healthSamples) { + expect(sample).toBeLessThan(maxHealthMs); + } + + const restoreRes = await restorePromise; + expect(restoreRes.status).toBe(200); + const restored = await restoreRes.json(); + expect(restored.ok).toBe(true); + expect(Date.now() - restoreStarted).toBeGreaterThanOrEqual(blockMs - 100); + expect(existsSync(join(isolatedCodexHome!.path, "archived_sessions", "rollout-old.jsonl"))).toBe(true); + } finally { + await server.stop(true); + resetRestoreTrashJobForTests(); + } + }, { timeout: 30_000 }); +});