Add workspaces

.gitignore

@@ -80,7 +80,7 @@ node_modules/
 # AI
 .claude
 .planning
-docs/plans
+docs/plans/
 
 # Build artifacts
 *.egg-info/

.hydra_config/config.yaml

@@ -177,6 +177,7 @@ ray:
   indexer:
     max_task_retries: ${oc.decode:${oc.env:RAY_MAX_TASK_RETRIES, 2}}
     serialize_timeout: ${oc.decode:${oc.env:INDEXER_SERIALIZE_TIMEOUT, 3600}}
+    vectordb_timeout: ${oc.decode:${oc.env:VECTORDB_TIMEOUT, 30}}
     concurrency_groups:
       default: ${oc.decode:${oc.env:INDEXER_DEFAULT_CONCURRENCY, 1000}}
       update: ${oc.decode:${oc.env:INDEXER_UPDATE_CONCURRENCY, 100}}

CLAUDE.md

@@ -118,6 +118,7 @@ The `extra` field in API responses is a JSON string: `{"sources": [filtered_sour
 - `partition.py` - Partition management (multi-tenant document collections)
 - `users.py` - User and membership management
 - `queue.py` - Task queue monitoring
+- `workspaces.py` - Workspace CRUD and file management
 - `tools.py` - Tools like `extractText` at `/v1/tools/execute` (tool param requires JSON: `{"name": "extractText"}`)
 
 ### User Management & Authentication
@@ -129,6 +130,8 @@ The system uses token-based authentication with role-based access control (RBAC)
 - `files` - File records with `file_id`, `partition_name`, `file_metadata`, `created_by` (FK to users), `relationship_id`, `parent_id`
 - `partition_memberships` - Join table linking users to partitions with roles (`owner`, `editor`, `viewer`)
 - `partitions` - Document collections with cascade delete to files and memberships
+- `workspaces` - Named file subsets within a partition for scoped search/chat
+- `workspace_files` - Join table linking workspaces to files
 
 **Authentication Flow** (`openrag/api.py` - `AuthMiddleware`):
 1. Token extracted from `Authorization: Bearer <token>` header (or `?token=` query param for `/static` routes)

docs/content/docs/documentation/API.mdx

@@ -496,6 +496,7 @@ curl -X 'POST' 'http://localhost:8080/v1/chat/completions' \
 }'
 ```
 
+
 ### 🔧 Tools
 
 Tools are useful features that can be called directly by the client.

docs/content/docs/documentation/data_model.md

@@ -17,7 +17,10 @@ Implemented using **SQLAlchemy ORM** with PostgreSQL as the backend.
 erDiagram
     partitions ||--o{ files : contains
     partitions ||--o{ partition_memberships : has
+    partitions ||--o{ workspaces : has
     users ||--o{ partition_memberships : belongs_to
+    workspaces ||--o{ workspace_files : has
+    files ||--o{ workspace_files : referenced_by
 
     partitions {
         int id PK
@@ -50,6 +53,21 @@ erDiagram
         varchar role
         datetime added_at
     }
+
+    workspaces {
+        int id PK
+        varchar workspace_id UK
+        varchar partition_name FK
+        varchar display_name
+        int created_by FK
+        datetime created_at
+    }
+
+    workspace_files {
+        int id PK
+        varchar workspace_id FK
+        varchar file_id FK
+    }
 ```
 
 ### `users`
@@ -133,6 +151,38 @@ Defines the many-to-many relationship between users and partitions with role-bas
 
 ---
 
+### `workspaces`
+
+Groups files within a partition into named subsets for scoped search and chat. See [Workspaces](/openrag/documentation/workspaces/) for full details.
+
+| Column          | Type | Description |
+|------------------|------|-------------|
+| `id`             | Integer (PK) | Internal identifier |
+| `workspace_id`   | String (unique) | Client-facing workspace identifier |
+| `partition_name` | String (FK → `partitions.partition`, CASCADE) | Owning partition |
+| `display_name`   | String (nullable) | Human-readable name |
+| `created_by`     | Integer (FK → `users.id`, SET NULL) | User who created the workspace |
+| `created_at`     | DateTime | Timestamp of creation |
+
+**Relationships:** `files` many-to-many → `File` (via `workspace_files`)
+
+---
+
+### `workspace_files`
+
+Join table linking workspaces to files.
+
+| Column          | Type | Description |
+|------------------|------|-------------|
+| `id`             | Integer (PK) | Internal identifier |
+| `workspace_id`   | String (FK → `workspaces.workspace_id`, CASCADE) | Workspace reference |
+| `file_id`        | String | File reference (no FK constraint — referential integrity enforced at application level) |
+
+**Constraints:**
+- `UniqueConstraint(workspace_id, file_id)` → a file appears at most once per workspace
+
+---
+
 ## Milvus Schema
 
 Milvus stores document chunks with their vector embeddings. The collection uses dynamic fields for flexible metadata.
@@ -193,6 +243,7 @@ flowchart LR
 |------|:----------:|:------:|-----------|
 | Partition metadata | ✓ | - | Referential integrity, access control |
 | File inventory | ✓ | - | Single source of truth for uploaded files |
+| Workspace membership | ✓ | - | File grouping resolved at query time |
 | User accounts & roles | ✓ | - | Authentication, ACID compliance |
 | Document chunks | - | ✓ | Optimized for vector operations |
 | Dense embeddings | - | ✓ | HNSW similarity search |

docs/content/docs/documentation/sql_migration.mdx

@@ -42,17 +42,37 @@ It will create a new migration script in the `openrag/scripts/migrations/alembic
 
 ### Step 2: Apply the Migration
 
-Once you've reviewed the generated migration script and confirmed it looks correct, apply it to your database:
+Once you've reviewed the generated migration script and confirmed it looks correct, apply it to your database.
 
-```bash title="Apply migrations"
+#### Option A: Docker (production)
+
+Rebuilds the image to ensure the migration runs against the exact code being deployed:
+
+```bash title="Apply migrations (Docker)"
 docker compose up -d rdb
 docker compose \
     run --no-deps --build --rm \
     --entrypoint "uv run alembic -c /app/openrag/scripts/migrations/alembic/alembic.ini upgrade head" \
     openrag; docker compose down
 ```
 
-This upgrades your database schema to the latest revision.
+#### Option B: Docker exec (stack already running)
+
+If the stack is already running, skip the rebuild:
+
+```bash title="Apply migrations (exec)"
+docker compose exec openrag \
+    uv run alembic -c /app/openrag/scripts/migrations/alembic/alembic.ini upgrade head
+```
+
+#### Option C: Local (development)
+
+Run alembic directly with your local venv — no Docker needed:
+
+```bash title="Apply migrations (local)"
+DATABASE_URL=postgresql://user:pass@localhost:5432/openrag \
+    uv run alembic -c openrag/scripts/migrations/alembic/alembic.ini upgrade head
+```
 
 :::tip[Automatic Migrations on Code Update]
 When you pull the latest changes to your local code, check if a new migration script has been added. If so, make sure to apply the migration before restarting the service.

docs/content/docs/documentation/workspaces.md

@@ -0,0 +1,213 @@
+---
+title: Workspaces
+description: Group files into workspaces for scoped search and chat
+---
+
+Workspaces let you organize files within a partition into named subsets. When searching or chatting, you can target a specific workspace so that only its files are considered — without affecting the underlying partition structure.
+
+---
+
+## Concepts
+
+- A **workspace** belongs to exactly one partition
+- A file can belong to **multiple workspaces** (or none)
+- Workspaces do not duplicate files — they reference existing partition files
+- Deleting a workspace deletes **orphaned files** (files not in any other workspace) from the partition automatically
+- Deleting a file removes it from all workspaces it belongs to
+
+---
+
+## Data Model
+
+```mermaid
+erDiagram
+    partitions ||--o{ workspaces : contains
+    workspaces ||--o{ workspace_files : has
+    files ||--o{ workspace_files : referenced_by
+
+    workspaces {
+        int id PK
+        varchar workspace_id UK
+        varchar partition_name FK
+        varchar display_name
+        int created_by FK
+        datetime created_at
+    }
+
+    workspace_files {
+        int id PK
+        varchar workspace_id FK
+        varchar file_id FK
+    }
+```
+
+**Constraints:**
+- `UniqueConstraint(workspace_id)` — workspace IDs are globally unique
+- `UniqueConstraint(workspace_id, file_id)` on `workspace_files` — a file appears at most once per workspace
+- Cascade delete: dropping a workspace removes its `workspace_files` rows
+
+---
+
+## API Endpoints
+
+All workspace endpoints live under `/partition/{partition}/workspaces`.
+
+### Workspace CRUD
+
+| Method | Endpoint | Auth | Description |
+|--------|----------|------|-------------|
+| POST | `/partition/{partition}/workspaces` | Editor | Create a workspace |
+| GET | `/partition/{partition}/workspaces` | Viewer | List all workspaces in partition |
+| GET | `/partition/{partition}/workspaces/{workspace_id}` | Viewer | Get workspace details |
+| DELETE | `/partition/{partition}/workspaces/{workspace_id}` | Owner | Delete workspace and orphaned files |
+
+### Workspace File Management
+
+| Method | Endpoint | Auth | Description |
+|--------|----------|------|-------------|
+| POST | `/partition/{partition}/workspaces/{workspace_id}/files` | Editor | Add files to workspace |
+| GET | `/partition/{partition}/workspaces/{workspace_id}/files` | Viewer | List files in workspace |
+| DELETE | `/partition/{partition}/workspaces/{workspace_id}/files/{file_id}` | Editor | Remove file from workspace |
+
+---
+
+### Create a Workspace
+
+```bash
+curl -X POST "$BASE_URL/partition/my-partition/workspaces" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"workspace_id": "project-alpha", "display_name": "Project Alpha"}'
+```
+
+```json
+{"status": "created", "workspace_id": "project-alpha"}
+```
+
+### List Workspaces
+
+```bash
+curl "$BASE_URL/partition/my-partition/workspaces" \
+  -H "Authorization: Bearer $TOKEN"
+```
+
+```json
+{
+  "workspaces": [
+    {
+      "workspace_id": "project-alpha",
+      "partition_name": "my-partition",
+      "display_name": "Project Alpha",
+      "created_by": 1,
+      "created_at": "2026-03-06T10:00:00"
+    }
+  ]
+}
+```
+
+### Add Files to a Workspace
+
+```bash
+curl -X POST "$BASE_URL/partition/my-partition/workspaces/project-alpha/files" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"file_ids": ["report.pdf", "notes.md"]}'
+```
+
+```json
+{"status": "added", "file_ids": ["report.pdf", "notes.md"]}
+```
+
+### Delete a Workspace
+
+```bash
+curl -X DELETE "$BASE_URL/partition/my-partition/workspaces/project-alpha" \
+  -H "Authorization: Bearer $TOKEN"
+```
+
+```json
+{"status": "deleted", "orphaned_files_deleted": 1}
+```
+
+Files that belonged **only** to the deleted workspace are automatically removed from the partition. Files shared with other workspaces are preserved.
+
+---
+
+## Upload with Workspace Assignment
+
+Files can be added to one or more workspaces at upload time using the `workspace_ids` form parameter:
+
+```bash
+curl -X POST "$BASE_URL/indexer/partition/my-partition/file/my-file-id" \
+  -H "Authorization: Bearer $TOKEN" \
+  -F "file=@document.pdf" \
+  -F 'metadata={"mimetype": "application/pdf"}' \
+  -F 'workspace_ids=["project-alpha", "project-beta"]'
+```
+
+The `workspace_ids` field accepts a JSON array of workspace IDs. Each workspace must exist in the target partition, otherwise the request is rejected with a 404.
+
+---
+
+## Workspace-Scoped Search
+
+Pass the `workspace` query parameter to restrict search results to files in that workspace:
+
+```bash
+curl "$BASE_URL/search/partition/my-partition?text=quarterly+results&workspace=project-alpha" \
+  -H "Authorization: Bearer $TOKEN"
+```
+
+Only chunks from files belonging to the `project-alpha` workspace are returned.
+
+### Multi-Partition Search
+
+The `workspace` parameter also works with multi-partition search:
+
+```bash
+curl "$BASE_URL/search?partitions=my-partition&text=quarterly+results&workspace=project-alpha" \
+  -H "Authorization: Bearer $TOKEN"
+```
+
+---
+
+## Workspace-Scoped Chat
+
+To scope a chat completion to a workspace, include the `workspace` field in the request metadata:
+
+```bash
+curl -X POST "$BASE_URL/v1/chat/completions" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "openrag-my-partition",
+    "messages": [{"role": "user", "content": "Summarize the Q1 results"}],
+    "metadata": {"workspace": "project-alpha"}
+  }'
+```
+
+The RAG pipeline resolves the workspace to its file list and filters the vector search accordingly.
+
+---
+
+## Deletion Behavior
+
+### Deleting a Workspace
+
+```mermaid
+flowchart LR
+    A[Delete workspace] --> B[Find workspace files]
+    B --> C{File in other workspaces?}
+    C -->|Yes| D[Keep file]
+    C -->|No| E[Delete orphaned file from partition]
+    D --> F[Done]
+    E --> F
+```
+
+### Deleting a File
+
+When a file is deleted from a partition (via `DELETE /partition/{partition}/file/{file_id}`), it is automatically removed from all workspaces that reference it.
+
+### Deleting a Partition
+
+When a partition is deleted, all its workspaces and workspace-file associations are cascade-deleted along with the files.