-
Notifications
You must be signed in to change notification settings - Fork 1.5k
docs: salvage database and configuration docs navigation from #2948 #3258
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
d4451df
docs: promote database and configuration pages from drafts to live
eaabb01
docs: fix broken links and add DATABASE_POOL_SIZE to database page
c8c8311
docs: fill gaps in database and onboard pages
0cde3a7
docs: warn about SQLite→Postgres migration incompatibility
be7c231
docs: fix DATABASE_POOL_SIZE default and add database link to quickstart
5d93264
docs: fix 10 wrong defaults and remove 6 phantom env vars
4ee917a
docs: mark superseded draft files
79491c8
docs: clarify configuration reference scope
serrrfirat 074919b
docs: expand configuration variable reference
serrrfirat File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,313 @@ | ||
| --- | ||
| title: Database Backends | ||
| description: PostgreSQL vs libSQL — choosing your database | ||
| --- | ||
|
|
||
| IronClaw supports two database backends: **PostgreSQL** and **libSQL** (embedded SQLite). **PostgreSQL is the default** — if you don't set `DATABASE_BACKEND`, IronClaw will use PostgreSQL and require a `DATABASE_URL`. Choose libSQL for zero-config personal use. | ||
|
|
||
| <Warning> | ||
| If you start IronClaw without configuring a database, you'll see: `DATABASE_URL is required`. Run `ironclaw onboard` or set the environment variables manually. | ||
| </Warning> | ||
|
|
||
| ## Quick Comparison | ||
|
|
||
| | Feature | PostgreSQL (default) | libSQL | | ||
| |---------|----------------------|--------| | ||
| | **Setup** | Requires PostgreSQL server | Zero-dependency, auto-created | | ||
| | **Best For** | Production, multi-user | Personal use, single-user | | ||
| | **Search** | Hybrid (FTS + vector) | FTS only (vector via Turso) | | ||
| | **Scaling** | Horizontal (read replicas) | Single node | | ||
| | **Backup** | pg_dump, replication | File copy, Turso sync | | ||
| | **Size** | 100MB+ installed | ~5MB binary | | ||
|
|
||
| ## PostgreSQL | ||
|
|
||
| Recommended for production deployments, multi-user scenarios, and high-throughput use cases. | ||
|
|
||
| ### Requirements | ||
|
|
||
| - PostgreSQL 15 or later | ||
| - pgvector extension for embeddings | ||
|
|
||
| ### Installation | ||
|
|
||
| <AccordionGroup> | ||
| <Accordion title="Docker Compose (recommended)" icon="container"> | ||
| The fastest way to get started. A `docker-compose.yml` is included in the IronClaw repository: | ||
|
|
||
| ```bash | ||
| cd ironclaw | ||
| docker compose up -d postgres | ||
| ``` | ||
|
|
||
| This starts `pgvector/pgvector:pg16` on `127.0.0.1:5432` with database, user, and password all set to `ironclaw`. | ||
|
|
||
| <Warning> | ||
| These credentials are for local development only. Change `POSTGRES_PASSWORD` for any non-local deployment. | ||
| </Warning> | ||
| </Accordion> | ||
|
|
||
| <Accordion title="Ubuntu/Debian" icon="terminal"> | ||
| ```bash | ||
| sudo apt install postgresql-15 postgresql-15-pgvector | ||
| sudo systemctl enable --now postgresql | ||
| ``` | ||
| </Accordion> | ||
|
|
||
| <Accordion title="macOS (Homebrew)" icon="terminal"> | ||
| ```bash | ||
| brew install postgresql | ||
| brew install pgvector | ||
| brew services start postgresql | ||
| ``` | ||
| </Accordion> | ||
| </AccordionGroup> | ||
|
|
||
| ### Configuration | ||
|
|
||
| ```bash | ||
| # Create database | ||
| sudo -u postgres psql -c "CREATE DATABASE ironclaw;" | ||
| sudo -u postgres psql -c "CREATE USER ironclaw WITH PASSWORD 'your-password';" | ||
| sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE ironclaw TO ironclaw;" | ||
|
|
||
| # Enable pgvector | ||
| sudo -u postgres psql -d ironclaw -c "CREATE EXTENSION IF NOT EXISTS vector;" | ||
| ``` | ||
|
|
||
| ### IronClaw Configuration | ||
|
|
||
| ```bash | ||
| # Select backend (optional — postgres is the default) | ||
| # Accepted values: postgres, postgresql, pg | libsql, turso, sqlite | ||
| export DATABASE_BACKEND=postgres | ||
|
|
||
| # Required for PostgreSQL | ||
| export DATABASE_URL="postgres://ironclaw:***@localhost/ironclaw" | ||
| export DATABASE_POOL_SIZE=30 # default; increase for high-throughput workloads | ||
| ``` | ||
|
|
||
| <Warning> | ||
| `DATABASE_URL` is **required** when using PostgreSQL. If not set, IronClaw will fail to start with the error: `DATABASE_URL is required — Run 'ironclaw onboard' or set DATABASE_URL environment variable`. | ||
| </Warning> | ||
|
|
||
| Or in the wizard: | ||
| 1. Select "PostgreSQL" | ||
| 2. Enter connection string | ||
| 3. Test connection | ||
|
|
||
| ### SSL Modes | ||
|
|
||
| | Mode | Behavior | Use Case | | ||
| |------|----------|----------| | ||
| | `disable` | Never use TLS | Local development | | ||
| | `prefer` | Try TLS, fallback to plaintext | **Default** — works everywhere | | ||
| | `require` | Require TLS | Production with TLS | | ||
|
|
||
| ```bash | ||
| export DATABASE_SSLMODE=require | ||
| ``` | ||
|
|
||
| ## libSQL | ||
|
|
||
| Recommended for personal use, development, and single-user deployments. Zero setup required. | ||
|
|
||
| ### How It Works | ||
|
|
||
| libSQL is an embedded SQLite-compatible database: | ||
| - Database is a single file (`~/.ironclaw/ironclaw.db`) | ||
| - No separate server process | ||
| - Auto-created on first connection | ||
| - Full SQLite feature set | ||
|
|
||
| ### IronClaw Configuration | ||
|
|
||
| ```bash | ||
| export DATABASE_BACKEND=libsql | ||
| export LIBSQL_PATH="/home/user/.ironclaw/ironclaw.db" | ||
| ``` | ||
|
|
||
| Or just use the wizard defaults: | ||
| 1. Select "libSQL" | ||
| 2. Accept default path | ||
| 3. Done! | ||
|
|
||
| ### Turso Cloud Sync | ||
|
|
||
| libSQL supports syncing to Turso for cloud backup: | ||
|
|
||
| ```bash | ||
| export DATABASE_BACKEND=libsql | ||
| export LIBSQL_PATH="/home/user/.ironclaw/ironclaw.db" | ||
| export LIBSQL_URL="libsql://your-db.turso.io" | ||
| export LIBSQL_AUTH_TOKEN="***" | ||
| ``` | ||
|
|
||
| <Warning> | ||
| `LIBSQL_AUTH_TOKEN` is **required** when `LIBSQL_URL` is set. IronClaw will fail to start without it. | ||
| </Warning> | ||
|
|
||
| This keeps a local copy with automatic cloud sync. | ||
|
|
||
| ## Feature Comparison | ||
|
|
||
| ### Hybrid Search | ||
|
|
||
| **PostgreSQL:** Full hybrid search (FTS + vector via RRF) | ||
| ``` | ||
| Keyword matches + semantic similarity | ||
| Reciprocal Rank Fusion ranking | ||
| ``` | ||
|
|
||
| **libSQL:** FTS only (text search) | ||
| ``` | ||
| Keyword matching via FTS5 | ||
| Vector search via Turso cloud only | ||
| ``` | ||
|
|
||
| ### Embeddings | ||
|
|
||
| Both backends support embeddings, but with different implementations: | ||
|
|
||
| | Backend | Embeddings | Notes | | ||
| |---------|------------|-------| | ||
| | PostgreSQL | Yes | pgvector for vector storage | | ||
| | libSQL local | FTS only | No local vector storage | | ||
| | libSQL + Turso | Yes | Via Turso vector indexes | | ||
|
|
||
| <Warning> | ||
| **Encryption at rest:** The local SQLite database stores conversation and workspace data in plaintext. Only secrets (API tokens) are encrypted with AES-256-GCM. If you handle sensitive data, use full-disk encryption (FileVault, LUKS, BitLocker) or choose PostgreSQL with TDE. | ||
| </Warning> | ||
|
|
||
| ## Migration | ||
|
|
||
| ### From libSQL to PostgreSQL | ||
|
|
||
| <Warning> | ||
| SQLite and PostgreSQL have different SQL dialects. A raw `sqlite3 .dump` output may fail on PostgreSQL due to PRAGMA statements, type differences, and quoting. Use a conversion tool like [pgloader](https://pgloader.io/) or [pgloader Docker](https://hub.docker.com/r/dimitri/pgloader) for reliable migration. | ||
| </Warning> | ||
|
|
||
| 1. **Using pgloader (recommended):** | ||
| ```bash | ||
| pgloader ~/.ironclaw/ironclaw.db postgresql://ironclaw:***@localhost/ironclaw | ||
| ``` | ||
|
|
||
| 2. **Manual export (may require editing):** | ||
| ```bash | ||
| sqlite3 ~/.ironclaw/ironclaw.db ".dump" > ironclaw.sql | ||
| # Edit ironclaw.sql to remove PRAGMAs and fix type incompatibilities | ||
| psql -d ironclaw -f ironclaw.sql | ||
| ``` | ||
|
|
||
| 3. **Update IronClaw config:** | ||
| ```bash | ||
| export DATABASE_BACKEND=postgres | ||
| export DATABASE_URL="postgres://user:***@localhost/ironclaw" | ||
| ``` | ||
|
|
||
| 4. **Restart IronClaw** | ||
|
|
||
| ### From PostgreSQL to libSQL | ||
|
|
||
| 1. **Export:** | ||
| ```bash | ||
| pg_dump -h localhost -U ironclaw ironclaw > ironclaw.sql | ||
| ``` | ||
|
|
||
| 2. **Convert and import to SQLite** (requires conversion tools) | ||
|
|
||
| 3. **Update IronClaw config** | ||
|
|
||
| ## When to Choose Which | ||
|
|
||
| ### Choose libSQL if: | ||
|
|
||
| - Running IronClaw on a personal laptop/desktop | ||
| - Single-user deployment | ||
| - Want zero database administration | ||
| - Don't need horizontal scaling | ||
| - FTS-only search is sufficient | ||
|
|
||
| ### Choose PostgreSQL if: | ||
|
|
||
| - Production multi-user deployment | ||
| - Need hybrid (FTS + vector) search locally | ||
| - High-throughput scenario | ||
| - Existing PostgreSQL infrastructure | ||
| - Require advanced backup/recovery | ||
| - Team or shared deployment | ||
|
|
||
| ## Backup | ||
|
|
||
| ### PostgreSQL | ||
|
|
||
| ```bash | ||
| # Backup | ||
| pg_dump -h localhost -U ironclaw ironclaw > backup.sql | ||
|
|
||
| # Restore | ||
| psql -d ironclaw -f backup.sql | ||
| ``` | ||
|
|
||
| ### libSQL | ||
|
|
||
| ```bash | ||
| # Backup (simple file copy) | ||
| cp ~/.ironclaw/ironclaw.db ~/.ironclaw/ironclaw.db.backup | ||
|
|
||
| # Restore | ||
| cp ~/.ironclaw/ironclaw.db.backup ~/.ironclaw/ironclaw.db | ||
|
|
||
| # With Turso: automatic cloud backup | ||
| ``` | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| <AccordionGroup> | ||
| <Accordion title="pgvector not found" icon="x-circle"> | ||
| ```bash | ||
| # Install pgvector | ||
| sudo apt install postgresql-15-pgvector | ||
|
|
||
| # Or compile manually | ||
| git clone https://github.com/pgvector/pgvector.git | ||
| cd pgvector | ||
| make | ||
| sudo make install | ||
| ``` | ||
| </Accordion> | ||
|
|
||
| <Accordion title="Database file is locked" icon="lock"> | ||
| ```bash | ||
| # Find and kill process | ||
| lsof ~/.ironclaw/ironclaw.db | ||
| kill -9 <pid> | ||
|
|
||
| # Or wait for it to release | ||
| ``` | ||
| </Accordion> | ||
|
|
||
| <Accordion title="Connection refused (PostgreSQL)" icon="network"> | ||
| ```bash | ||
| # Check PostgreSQL is running | ||
| sudo systemctl status postgresql | ||
|
|
||
| # Check listen addresses | ||
| sudo -u postgres psql -c "SHOW listen_addresses;" | ||
|
|
||
| # Should be '*' or 'localhost' | ||
| ``` | ||
| </Accordion> | ||
| </AccordionGroup> | ||
|
|
||
| ## Next Steps | ||
|
|
||
| <CardGroup cols={2}> | ||
| <Card title="Configuration" icon="settings" href="/capabilities/configuration"> | ||
| Full environment variable reference | ||
| </Card> | ||
|
|
||
| <Card title="VPS Install" icon="server" href="/infrastructure/droplet"> | ||
| Production deployment guide with PostgreSQL | ||
| </Card> | ||
| </CardGroup> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The username in this example
DATABASE_URLisuser, which is inconsistent with the userironclawthat is created in the manual setup instructions on line 71 and used in other examples throughout this document (e.g., lines 86, 192, and 214). To avoid confusion for users following the guide, it would be best to consistently useironclawas the username.