feat: Unkey Builds

.gitignore

@@ -24,5 +24,6 @@ dist
 .secrets.json
 certs/
 deployment/data/*
+deployment/config/depot.json
 metald.db
 bin/

QUICKSTART-DEPLOY.md

@@ -9,9 +9,105 @@ This guide will help you get the Unkey deployment platform up and running locall
 - A terminal/command line
 - dnsmasq (for wildcard DNS setup)
 
-## Step 1: Start the Platform
+## Step 1: Configure Build Backend
 
-1. Set up the API key for ctrl service authentication by adding it to `go/apps/ctrl/.env`:
+**You must configure the build backend before starting Docker Compose.** This script generates the `.env` file that Docker Compose requires.
+
+### Why This Matters
+
+The platform builds **unknown user code** in isolation for security. When users deploy their applications, the system:
+
+1. **Isolates each build** - User code runs in separate containers/VMs to prevent interference
+2. **Caches layers** - Speeds up rebuilds by reusing unchanged Docker layers
+3. **Stores artifacts** - Uses S3-compatible storage for built images and artifacts
+
+You have two options:
+
+- **Docker**: Fully local setup, slower builds, good for development
+- **Depot**: Fast remote builds with persistent layer caching
+
+Navigate to the deployment directory:
+
+```bash
+cd deployment
+```
+
+### Option A: Local Docker
+
+Fully local setup using Docker with MinIO for S3 storage:
+
+```bash
+./setup-build-backend.sh docker
+```
+
+This configures:
+
+- Local Docker builds (no external services)
+- MinIO S3 storage running in Docker
+
+### Option B: Depot
+
+Remote builds with persistent caching for faster iteration. Create `deployment/config/depot.json`:
+
+```json
+{
+  "token": "depot_org_YOUR_TOKEN_HERE",
+  "s3_url": "https://your-s3-endpoint.com",
+  "s3_access_key_id": "your_access_key",
+  "s3_access_key_secret": "your_secret_key"
+}
+```
+
+Then run:
+
+```bash
+./setup-build-backend.sh depot
+```
+
+This configures:
+
+- Remote Depot infrastructure for builds
+- Persistent layer caching across builds
+- Your S3-compatible storage for artifacts
+
+**Note:** The token must start with `depot_org_` or the script will fail validation.
+
+### Kubernetes: Additional Setup for Depot and S3
+
+If you're deploying to Kubernetes, you must create secrets for Depot and S3 credentials:
+
+**1. Create Depot credentials secret:**
+
+```bash
+kubectl create secret generic depot-credentials \
+  --from-literal=token=depot_org_YOUR_TOKEN_HERE \
+  --from-literal=s3-url=https://your-s3-endpoint.com \
+  --from-literal=s3-access-key-id=your_access_key \
+  --from-literal=s3-access-key-secret=your_secret_key \
+  --namespace=unkey
+```
+
+**2. Create Depot registry credentials secret:**
+
+```bash
+kubectl create secret docker-registry depot-registry \
+  --docker-server=registry.depot.dev \
+  --docker-username=x-token \
+  --docker-password=depot_org_YOUR_TOKEN_HERE \
+  --namespace=unkey
+```
+
+**Critical:** These secrets must exist before deploying the `ctrl` service to Kubernetes. The `ctrl` deployment references these secrets for:
+
+- Building user code via Depot
+- Storing build contexts in S3
+- Pulling built images from Depot's registry
+
+Without these secrets, deployments will fail with authentication errors.
+
+## Step 2: Configure API Keys
+
+1. Set up the API key for ctrl service authentication in `go/apps/ctrl/.env`:
 
 ```bash
 UNKEY_API_KEY="your-local-dev-key"
@@ -24,76 +120,87 @@ CTRL_URL="http://127.0.0.1:7091"
 CTRL_API_KEY="your-local-dev-key"
 ```
 
-Note: Use the same API key value in both files for authentication to work properly.
+**Critical:** Use the same API key value in both files for authentication to work.
+
+## Step 3: Start the Platform
 
-3. Start all necessary services using Docker Compose:
+From the project root directory, start all services:
 
 ```bash
-docker compose -f ./deployment/docker-compose.yaml up mysql planetscale clickhouse redis s3 dashboard gw metald ctrl -d --build
+docker compose -f ./deployment/docker-compose.yaml up mysql planetscale clickhouse redis s3 dashboard gw krane ctrl -d --build
 ```
 
-This will start:
-- **mysql**: Database for storing workspace, project, and deployment data
+This starts:
+
+- **mysql**: Database for workspace, project, and deployment data
 - **planetscale**: PlanetScale HTTP simulator for database access
 - **clickhouse**: Analytics database for metrics and logs
 - **redis**: Caching layer for session and temporary data
-- **s3**: MinIO S3-compatible storage for assets and vault data
+- **s3**: MinIO S3-compatible storage for assets and vault data (when using docker backend)
 - **dashboard**: Web UI for managing deployments (port 3000)
 - **gw**: Gateway service for routing traffic (ports 80/443)
-- **metald**: VM/container management service (port 8090)
+- **krane**: VM/container management service (port 8090)
 - **ctrl**: Control plane service for managing deployments (port 7091)
 
-4. Set up wildcard DNS for `unkey.local`:
+## Step 4: Set Up DNS and Certificates
+
+1. Set up wildcard DNS for `unkey.local`:
 
 ```bash
 ./deployment/setup-wildcard-dns.sh
 ```
 
-5. **OPTIONAL**: Install self-signed certificate for HTTPS (to avoid SSL errors):
+2. **OPTIONAL**: Install self-signed certificate for HTTPS (to avoid SSL errors):
 
 ```bash
 # For macOS
 sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ./deployment/certs/unkey.local.crt
 ```
 
-Note: Certificates should be mounted to `deployment/certs`. You can skip this if you're fine with SSL errors in your browser.
+Note: Certificates are in `deployment/certs`. You can skip this if you're fine with SSL warnings in your browser.
 
-## Step 2: Set Up Your Workspace
+## Step 5: Set Up Your Workspace
 
 1. Open your browser and navigate to the dashboard:
 
 ```
 http://localhost:3000
 ```
 
-2. Create a workspace and copy its id
+2. Create a workspace and copy its ID
 
-3. Create a new project by filling out the form:
+3. Create a new project by going to:
 
-Go to http://localhost:3000/projects
+```
+http://localhost:3000/projects
+```
+
+Fill out the form:
 
 - **Name**: Choose any name (e.g., "My Test App")
 - **Slug**: This will auto-generate based on the name
 - **Git URL**: Optional, leave blank for testing
 
-4. After creating the project, **copy the Project ID** from the project details. It will look like:
+4. After creating the project, **copy the Project ID** from the project details
 
-## Step 3: Deploy a Version
+## Step 6: Deploy a Version
 
 1. Navigate to the go directory:
 
 ```bash
 cd go
 ```
 
-2. Set up API key authentication for the CLI (choose one option):
+2. Set up API key authentication for the CLI:
 
 **Option A: Environment variable (recommended)**
+
 ```bash
 export API_KEY="your-local-dev-key"
 ```
 
 **Option B: CLI flag**
+
 Use `--api-key="your-local-dev-key"` in the command below.
 
 3. Deploy using the CLI with your copied IDs:
@@ -105,50 +212,52 @@ go run . deploy \
   --project-id="REPLACE_ME" \
   --control-plane-url="http://127.0.0.1:7091" \
   --api-key="your-local-dev-key" \
-  --keyspace-id="REPLACE_ME" # This is optional if you want key verifications
+  --keyspace-id="REPLACE_ME"  # Optional, only needed for key verifications
 ```
 
 Replace the placeholder values:
-- `REPLACE_ME` with your actual workspace ID, project ID, and keyspace ID
-- `your-local-dev-key` with the same API key value you set in steps 1 and 2
+
+- `REPLACE_ME` with your actual workspace ID, project ID, and keyspace ID (if needed)
+- `your-local-dev-key` with the same API key value you set in Step 2
 - Keep `--context=./demo_api` as shown (there's a demo API in that folder)
 
 **Note**: If using Option A (environment variable), you can omit the `--api-key` flag from the command.
 
-3. The CLI will:
-   - Build a Docker image from the demo_api code
+4. The CLI will:
+   - Build a Docker image from the demo_api code (using your configured build backend)
    - Create a deployment on the Unkey platform
    - Show real-time progress through deployment stages
-   - Deploy using metald's VM/container backend
+   - Deploy using Krane's VM/container backend
 
-## Step 4: Test Your Deployment
+## Step 7: Test Your Deployment
 
-1. Once deployment completes, test the API in the unkey root directory:
+1. Once deployment completes, test the API from the project root directory:
 
 ```bash
 curl --cacert ./deployment/certs/unkey.local.crt https://REPLACE_ME/v1/liveness
 ```
 
-Replace:
-- `REPLACE_ME` (URL) with your deployment domain
+Replace `REPLACE_ME` with your deployment domain.
 
 **Note:** The liveness endpoint is public and doesn't require authentication. For protected endpoints, include an Authorization header:
 
 ```bash
-curl --cacert ./deployment/certs/unkey.local.crt -H "Authorization: Bearer YOUR_API_KEY" https://YOUR_DOMAIN/protected/endpoint
+curl --cacert ./deployment/certs/unkey.local.crt \
+  -H "Authorization: Bearer YOUR_API_KEY" \
+  https://YOUR_DOMAIN/protected/endpoint
 ```
 
-2. Return to the dashboard to monitor your deployment:
+2. Monitor your deployment in the dashboard:
 
 ```
 http://localhost:3000/deployments
 ```
 
-### Important: Your Application Must Listen on the PORT Environment Variable
+## Important: Application Port Configuration
 
-**Your deployed application MUST read the `PORT` environment variable and listen on that port.** The platform sets `PORT=8080` in the container, and your code needs to use this value.
+**Your deployed application MUST read the `PORT` environment variable and listen on that port.** The platform sets `PORT=8080` in the container.
 
-**Example for different languages:**
+**Examples for different languages:**
 
 ```javascript
 // Node.js
@@ -178,7 +287,31 @@ The demo_api already follows this pattern and listens on the PORT environment va
 
 ## Troubleshooting
 
-- If you see "port is already allocated" errors, the system will automatically retry with a new random port
-- Check container logs: `docker logs <container-name>`
-- Verify the demo_api is listening on the PORT environment variable (should be 8080)
-- Make sure your Dockerfile exposes the correct port (8080 in the demo_api example)
+### Build Issues
+
+- **"depot login failed"**: Check your depot token in `deployment/config/depot.json` - it must start with `depot_org_`
+- **"S3 connection failed"**: Verify S3 credentials in your depot.json or ensure MinIO is running (for docker backend)
+- **Slow builds**: Switch to depot backend for faster builds with layer caching
+- **Stuck builds**: If you are stuck with a deployment go to `http://localhost:9070/ui/invocations` and kill the ongoing invocation.
+
+### Kubernetes Issues
+
+- **"secret not found"**: Ensure you created both `depot-credentials` and `depot-registry` secrets before deploying ctrl
+- **"imagePullBackOff"**: Verify depot-registry secret credentials are correct and token is valid
+- **"build context upload failed"**: Check depot-credentials secret has valid S3 credentials
+
+### Deployment Issues
+
+- **"port is already allocated"**: The system will automatically retry with a new random port
+- **Application not responding**: Verify your app listens on the PORT environment variable (should be 8080)
+- **Dockerfile issues**: Ensure your Dockerfile exposes the correct port (8080 in the demo_api example)
+
+### Service Logs
+
+Check container logs for any service:
+
+```bash
+docker logs <container-name>
+```
+
+Service names: `mysql`, `planetscale`, `clickhouse`, `redis`, `s3`, `dashboard`, `gw`, `krane`, `ctrl`

deployment/docker-compose.yaml

@@ -167,7 +167,7 @@ services:
     image: bitnamilegacy/minio:2025.7.23-debian-12-r5
     ports:
       - 3902:3902
-      - 2903:2903
+      - 3903:3903
     environment:
       MINIO_ROOT_USER: minio_root_user
       MINIO_ROOT_PASSWORD: minio_root_password
@@ -286,6 +286,10 @@ services:
       UNKEY_DOCKER_SOCKET: "/var/run/docker.sock"
       UNKEY_DEPLOYMENT_EVICTION_TTL: "10m"
 
+      UNKEY_REGISTRY_URL: "${UNKEY_REGISTRY_URL:-}"
+      UNKEY_REGISTRY_USERNAME: "${UNKEY_REGISTRY_USERNAME:-}"
+      UNKEY_REGISTRY_PASSWORD: "${UNKEY_REGISTRY_PASSWORD:-}"
+
   restate:
     networks:
       - default
@@ -355,8 +359,27 @@ services:
       UNKEY_VAULT_S3_ACCESS_KEY_SECRET: "minio_root_password"
       UNKEY_VAULT_MASTER_KEYS: "Ch9rZWtfMmdqMFBJdVhac1NSa0ZhNE5mOWlLSnBHenFPENTt7an5MRogENt9Si6wms4pQ2XIvqNSIgNpaBenJmXgcInhu6Nfv2U="
 
+      # Build configuration
+      UNKEY_BUILD_S3_URL: "${UNKEY_BUILD_S3_URL:-http://s3:3902}"
+      UNKEY_BUILD_S3_EXTERNAL_URL: "${UNKEY_BUILD_S3_EXTERNAL_URL:-}" # For CLI/external access
+      UNKEY_BUILD_S3_BUCKET: "build-contexts"
+      UNKEY_BUILD_S3_ACCESS_KEY_ID: "${UNKEY_BUILD_S3_ACCESS_KEY_ID:-minio_root_user}"
+      UNKEY_BUILD_S3_ACCESS_KEY_SECRET: "${UNKEY_BUILD_S3_ACCESS_KEY_SECRET:-minio_root_password}"
+
       # API key for simple authentication (temporary, will be replaced with JWT)
       UNKEY_API_KEY: "your-local-dev-key"
+      # Build backend configuration
+      UNKEY_BUILD_BACKEND: "${UNKEY_BUILD_BACKEND:-docker}"
+      UNKEY_BUILD_PLATFORM: "linux/amd64"
+      UNKEY_DOCKER_SOCKET: "/var/run/docker.sock"
+      UNKEY_DEPLOYMENT_EVICTION_TTL: "10m"
+      # Registry configuration (used by both Docker and Depot backends)
+      UNKEY_REGISTRY_URL: "${UNKEY_REGISTRY_URL:-registry.depot.dev}"
+      UNKEY_REGISTRY_USERNAME: "${UNKEY_REGISTRY_USERNAME:-x-token}"
+      UNKEY_REGISTRY_PASSWORD: "${UNKEY_REGISTRY_PASSWORD:-${DEPOT_TOKEN:-}}"
+      # Depot-specific configuration (only needed when UNKEY_BUILD_BACKEND=depot)
+      UNKEY_DEPOT_API_URL: "https://api.depot.dev"
+      UNKEY_DEPOT_PROJECT_REGION: "us-east-1"
 
   otel:
     networks: