Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Litestreamを用いてDBを永続化 #38

Merged
merged 5 commits into from
Nov 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,93 @@ prismaに関連する設定ファイルやスキーマ定義
### `public/`

静的ファイル(画像やアイコンなど)

## データベースのバックアップと永続化

このプロジェクトでは、SQLiteデータベースのバックアップと永続化にLitestreamを使用しています。

### フローチャート

```mermaid
flowchart LR
subgraph CloudRun[Cloud Run]
direction TB
RemixApp[Remix App]
Litestream[Litestream]
end

subgraph CloudStorage[Cloud Storage]
DB[/prod.db/]
end

RemixApp --> Litestream
Litestream -->|replicate| DB
```

### 動作の仕組み

1. **データベースの復元**

- コンテナ起動時、Google Cloud Storageから最新のバックアップを復元
- 初回起動時やバックアップが存在しない場合は、空のデータベースを使用

2. **リアルタイムレプリケーション**

- アプリケーション実行中、データベースの変更をCloud Storageに自動的にレプリケート
- WALモードの無効化によりPrismaとの互換性を確保

### 制限事項

1. **インスタンス数の制限**

- Cloud Runのインスタンス数は最大1に制限
- 複数インスタンスでの同時実行はデータ整合性の問題を引き起こす可能性あり

2. **Prismaとの互換性**

- PrismaはSQLiteのWALモードに非対応
- 起動時に`PRAGMA journal_mode=DELETE`を実行して対応

### 開発環境との違い

開発環境では通常のSQLiteデータベースとして動作し、Litestreamは使用しません。本番環境(Cloud Run)でのみLitestreamによるバックアップと永続化が有効になります。

### デプロイ時の設定

1. **必要な環境変数**

```
LITESTREAM_BUCKET=<バックアップ先のバケット名>
```

2. **必要な権限**

- Cloud Run サービスアカウントに以下の権限が必要:
- `roles/storage.objectViewer`
- `roles/storage.objectCreator`

### トラブルシューティング

1. **データベースロックエラー**

```
Error: SQLite database error
database is locked
```

対処法:

- アプリケーションの再起動
- WALモードが正しく無効化されているか確認

2. **バックアップの確認**

```sh
# バックアップの存在確認
gcloud storage ls gs://<BUCKET_NAME>/prod.db/
```

### 参考リンク

- [Litestream Documentation](https://litestream.io/)
- [Prisma with SQLite](https://www.prisma.io/docs/orm/overview/databases/sqlite)
49 changes: 34 additions & 15 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,16 @@ SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN groupadd -r appuser && useradd -r -g appuser -s /bin/false appuser
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
curl=8.5.0-2ubuntu10.4 \
unzip=6.0-28ubuntu4.1 \
openssl=3.0.13-0ubuntu3.4 \
ca-certificates=20240203 \
&& rm -rf /var/lib/apt/lists/*
curl \
unzip \
openssl \
ca-certificates \
sqlite3 \
sqlite3-tools \
&& rm -rf /var/lib/apt/lists/* \
&& curl -L https://github.com/benbjohnson/litestream/releases/download/v0.3.13/litestream-v0.3.13-linux-amd64.tar.gz -o /tmp/litestream.tar.gz \
&& tar -C /usr/local/bin -xzf /tmp/litestream.tar.gz \
&& rm /tmp/litestream.tar.gz
USER appuser
WORKDIR /home/appuser
ENV MISE_ROOT="/home/appuser/.local/share/mise"
Expand Down Expand Up @@ -44,28 +49,42 @@ FROM deps AS builder
WORKDIR /app
USER appuser
COPY --chown=appuser:appuser . .
ENV DATABASE_URL=file:/app/prisma/data/deploy.db
RUN mkdir -p prisma/data && \
touch prisma/data/deploy.db && \
ENV DATABASE_URL=file:/app/db/prod.db
RUN mkdir -p db && \
touch db/prod.db && \
bun prisma migrate deploy && \
bun run build
USER root
RUN mkdir -p /tmp/prod/app && \
cp -r build /tmp/prod/app/ && \
cp -r prisma /tmp/prod/app/ && \
cp -r db /tmp/prod/app/ && \
cp package.json /tmp/prod/app/ && \
chown -R 65532:65532 /tmp/prod && \
chmod -R 755 /tmp/prod/app && \
chmod 777 /tmp/prod/app/prisma/data && \
touch /tmp/prod/app/prisma/data/deploy.db && \
chmod 666 /tmp/prod/app/prisma/data/deploy.db
chmod 666 /tmp/prod/app/db/prod.db

FROM gcr.io/distroless/nodejs22-debian12:nonroot AS runner
FROM node:22-slim AS runner
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
sqlite3 \
ca-certificates \
gettext-base \
&& rm -rf /var/lib/apt/lists/*
RUN mkdir -p /app/db && \
chown -R node:node /app
COPY --from=base /usr/local/bin/litestream /usr/local/bin/
COPY litestream.yml.template /etc/litestream.yml.template
COPY run.sh /run.sh
RUN chmod +x /run.sh && \
chown -R node:node /etc/litestream.yml.template
COPY --from=prod-deps /tmp/prod-deps/node_modules ./node_modules
COPY --from=builder /tmp/prod/app ./
ENV NODE_ENV=production \
DATABASE_URL=file:/app/prisma/data/deploy.db
USER 65532:65532
DATABASE_URL=file:/app/db/prod.db
RUN chown -R node:node /app && \
chmod 755 /app/db && \
chmod 666 /app/db/prod.db
USER node
EXPOSE 3000
CMD ["node_modules/@remix-run/serve/dist/cli.js", "build/server/index.js"]
ENTRYPOINT ["/run.sh"]
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,7 @@ bun install --frozen-lockfile
```sh
bun run dev
```

## 関連ドキュメント

- [アーキテクチャの概要](ARCHITECTURE.md)
4 changes: 2 additions & 2 deletions compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ services:
ports:
- "3000:3000"
volumes:
- sqlite_data:/app/prisma/data
- sqlite_data:/app/db
environment:
- NODE_ENV=production
- DATABASE_URL=file:/app/prisma/data/deploy.db
- DATABASE_URL=file:/app/db/prod.db
user: "65532:65532"
restart: unless-stopped

Expand Down
6 changes: 6 additions & 0 deletions litestream.yml.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
dbs:
- path: /app/db/prod.db
replicas:
- type: gcs
bucket: ${LITESTREAM_BUCKET}
path: prod.db
33 changes: 33 additions & 0 deletions run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/bin/sh
set -e

# Check if LITESTREAM_BUCKET is set
if [ -z "${LITESTREAM_BUCKET}" ]; then
echo "Error: LITESTREAM_BUCKET environment variable is not set" >&2
exit 1
fi
sed "s|\${LITESTREAM_BUCKET}|${LITESTREAM_BUCKET}|g" /etc/litestream.yml.template > /app/litestream.yml

# Remove existing db if exists
if [ -f /app/db/prod.db ]; then
mv /app/db/prod.db /app/db/prod.db.bak
fi

# Restore database from GCS if exists
litestream restore -if-replica-exists -config /app/litestream.yml /app/db/prod.db

if [ -f /app/db/prod.db ]; then
echo "Successfully restored database from Cloud Storage"
rm -f /app/db/prod.db.bak
else
echo "No database backup found in Cloud Storage, using original"
mv /app/db/prod.db.bak /app/db/prod.db
fi

# Before starting the app, ensure WAL mode is disabled for Prisma compatibility
sqlite3 /app/db/prod.db "PRAGMA journal_mode=DELETE;"

# Start Litestream replication with the app as the subprocess
exec litestream replicate \
-config /app/litestream.yml \
-exec "node node_modules/@remix-run/serve/dist/cli.js build/server/index.js"