-
Notifications
You must be signed in to change notification settings - Fork 609
chore: local k8s dev #3943
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: local k8s dev #3943
Changes from all commits
2ba5921
1c33c01
1bdae1e
2d2f54d
8ce300d
0c334e9
3b0de47
4360e42
ef51b2d
7f94eb6
85e3f12
548f067
12d0f9c
76a7675
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| FROM busybox:latest | ||
|
|
||
| # Copy the pre-built binary | ||
| COPY bin/unkey /unkey | ||
|
|
||
| LABEL org.opencontainers.image.source=https://github.com/unkeyed/unkey/go | ||
| LABEL org.opencontainers.image.description="Unkey API" | ||
|
|
||
| ENTRYPOINT ["/unkey"] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| # Kubernetes Development Setup | ||
|
|
||
| Run Unkey services locally using Kubernetes instead of Docker Compose. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Docker Desktop with Kubernetes enabled OR OrbStack with Kubernetes enabled | ||
| - kubectl | ||
|
|
||
| Check requirements: | ||
| ```bash | ||
| make k8s-check | ||
| ``` | ||
|
|
||
| ## Quick Start | ||
|
|
||
| Start everything: | ||
| ```bash | ||
| make k8s-up | ||
| ``` | ||
|
|
||
| Start with hot reloading (requires Tilt): | ||
| ```bash | ||
| make dev | ||
| ``` | ||
|
|
||
| ## Individual Services | ||
|
|
||
| ```bash | ||
| make start-mysql | ||
| make start-clickhouse | ||
| make start-redis | ||
| make start-s3 | ||
| make start-api | ||
| make start-gw | ||
| make start-ctrl | ||
| ``` | ||
|
|
||
| ## Management | ||
|
|
||
| ```bash | ||
| # Stop everything | ||
| make k8s-down | ||
|
|
||
| # Reset environment | ||
| make k8s-reset | ||
|
|
||
| # View services | ||
| kubectl get pods -n unkey | ||
| kubectl get services -n unkey | ||
| ``` | ||
|
|
||
| ## Tilt (Optional) | ||
|
|
||
| Start specific services: | ||
| ```bash | ||
| tilt up -- --services=mysql --services=clickhouse | ||
| tilt up -- --services=api --services=gw --services=ctrl | ||
| tilt up -- --services=all | ||
| ``` | ||
|
|
||
| Stop Tilt: | ||
| ```bash | ||
| tilt down | ||
| ``` | ||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -1,4 +1,4 @@ | ||||||||||||||||||||
| .PHONY: install tools fmt test-unit test-integration test-integration-long test-stress test build generate pull up clean | ||||||||||||||||||||
| .PHONY: install tools fmt test-unit test-integration test-integration-long test-stress test build generate pull up clean k8s-check k8s-up k8s-down k8s-reset k8s-status start-mysql start-ctrl start-all dev | ||||||||||||||||||||
|
|
||||||||||||||||||||
| # Detect OS and set GOMAXPROCS accordingly | ||||||||||||||||||||
| UNAME_S := $(shell uname -s) | ||||||||||||||||||||
|
|
@@ -68,3 +68,106 @@ test-integration: export SIMULATION_TEST=false | |||||||||||||||||||
| test-integration: up | ||||||||||||||||||||
| @echo "Running integration tests w/$(PARALLEL_PROCS) parallel test processes" | ||||||||||||||||||||
| @go test -tags=integration -json -failfast -timeout=15m -parallel=$(PARALLEL_PROCS) ./... | tparse -all -progress -smallscreen | ||||||||||||||||||||
|
|
||||||||||||||||||||
| # ============================================================================ | ||||||||||||||||||||
| # Kubernetes Development Commands | ||||||||||||||||||||
| # ============================================================================ | ||||||||||||||||||||
|
|
||||||||||||||||||||
| k8s-check: ## Check if kubectl is available and cluster is running | ||||||||||||||||||||
| @command -v kubectl >/dev/null 2>&1 || { echo "ERROR: kubectl not found. Install from: https://kubernetes.io/docs/tasks/tools/"; exit 1; } | ||||||||||||||||||||
| @kubectl cluster-info >/dev/null 2>&1 || { echo "ERROR: Kubernetes cluster not available. Enable Kubernetes in Docker Desktop/OrbStack"; exit 1; } | ||||||||||||||||||||
| @echo "Kubernetes cluster is available" | ||||||||||||||||||||
|
|
||||||||||||||||||||
| k8s-up: k8s-check ## Deploy all services to current Kubernetes cluster | ||||||||||||||||||||
| @echo "Building Docker images..." | ||||||||||||||||||||
| @docker build -t unkey/mysql:latest -f ../deployment/Dockerfile.mysql ../ | ||||||||||||||||||||
| @docker build -t unkey/clickhouse:latest -f ../deployment/Dockerfile.clickhouse ../ | ||||||||||||||||||||
| @docker build -t unkey:latest . | ||||||||||||||||||||
| @echo "Creating namespace..." | ||||||||||||||||||||
| @kubectl apply -f k8s/manifests/namespace.yaml | ||||||||||||||||||||
| @echo "Applying Kubernetes manifests..." | ||||||||||||||||||||
| @kubectl apply -f k8s/manifests/ | ||||||||||||||||||||
| @echo "Waiting for services to be ready..." | ||||||||||||||||||||
| @kubectl wait --for=condition=ready pod -l app=mysql -n unkey --timeout=180s | ||||||||||||||||||||
| @kubectl wait --for=condition=ready pod -l app=clickhouse -n unkey --timeout=180s | ||||||||||||||||||||
| @kubectl wait --for=condition=ready pod -l app=s3 -n unkey --timeout=180s | ||||||||||||||||||||
| @kubectl wait --for=condition=ready pod -l app=api -n unkey --timeout=180s | ||||||||||||||||||||
| @kubectl wait --for=condition=ready pod -l app=gw -n unkey --timeout=180s | ||||||||||||||||||||
| @kubectl wait --for=condition=ready pod -l app=ctrl -n unkey --timeout=180s | ||||||||||||||||||||
| @kubectl wait --for=condition=ready pod -l app=dashboard -n unkey --timeout=180s | ||||||||||||||||||||
| @echo "Kubernetes environment is ready!" | ||||||||||||||||||||
| @echo "" | ||||||||||||||||||||
| @echo "Services accessible via NodePort on localhost - check actual ports with:" | ||||||||||||||||||||
| @echo "" | ||||||||||||||||||||
| @echo "Check NodePort assignments: make k8s-ports" | ||||||||||||||||||||
| @make k8s-status | ||||||||||||||||||||
|
|
||||||||||||||||||||
|
Comment on lines
+81
to
+104
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add waits for all dependencies, configurable timeout, and implement referenced status/ports targets.
Apply: +k8s_wait_timeout ?= 300
k8s-up: k8s-check ## Deploy all services to current Kubernetes cluster
@echo "Building Docker images..."
@docker build -t unkey/mysql:latest -f ../deployment/Dockerfile.mysql ../
@docker build -t unkey/clickhouse:latest -f ../deployment/Dockerfile.clickhouse ../
@docker build -t unkey:latest .
@echo "Creating namespace..."
@kubectl apply -f k8s/manifests/namespace.yaml
@echo "Applying Kubernetes manifests..."
@kubectl apply -f k8s/manifests/
@echo "Waiting for services to be ready..."
- @kubectl wait --for=condition=ready pod -l app=mysql -n unkey --timeout=180s
- @kubectl wait --for=condition=ready pod -l app=clickhouse -n unkey --timeout=180s
- @kubectl wait --for=condition=ready pod -l app=s3 -n unkey --timeout=180s
- @kubectl wait --for=condition=ready pod -l app=api -n unkey --timeout=180s
- @kubectl wait --for=condition=ready pod -l app=gw -n unkey --timeout=180s
- @kubectl wait --for=condition=ready pod -l app=ctrl -n unkey --timeout=180s
- @kubectl wait --for=condition=ready pod -l app=dashboard -n unkey --timeout=180s
+ @kubectl wait --for=condition=ready pod -l app=mysql -n unkey --timeout=$(k8s_wait_timeout)s
+ @kubectl wait --for=condition=ready pod -l app=clickhouse -n unkey --timeout=$(k8s_wait_timeout)s
+ @kubectl wait --for=condition=ready pod -l app=redis -n unkey --timeout=$(k8s_wait_timeout)s
+ @kubectl wait --for=condition=ready pod -l app=planetscale -n unkey --timeout=$(k8s_wait_timeout)s || true
+ @kubectl wait --for=condition=ready pod -l app=s3 -n unkey --timeout=$(k8s_wait_timeout)s
+ @kubectl wait --for=condition=ready pod -l app=otel-collector -n unkey --timeout=$(k8s_wait_timeout)s || true
+ @kubectl wait --for=condition=ready pod -l app=api -n unkey --timeout=$(k8s_wait_timeout)s
+ @kubectl wait --for=condition=ready pod -l app=gw -n unkey --timeout=$(k8s_wait_timeout)s
+ @kubectl wait --for=condition=ready pod -l app=ctrl -n unkey --timeout=$(k8s_wait_timeout)s
+ @kubectl wait --for=condition=ready pod -l app=dashboard -n unkey --timeout=$(k8s_wait_timeout)s
@echo "Kubernetes environment is ready!"
@echo ""
@echo "Services accessible via NodePort on localhost - check actual ports with:"
@echo ""
@echo "Check NodePort assignments: make k8s-ports"
- @make k8s-status
+ @$(MAKE) k8s-statusAdd these targets: 🤖 Prompt for AI Agents |
||||||||||||||||||||
| k8s-stop: ## Stop all pods (scale deployments to 0) | ||||||||||||||||||||
| @echo "Stopping all pods..." | ||||||||||||||||||||
| @kubectl scale deployment --all --replicas=0 -n unkey | ||||||||||||||||||||
| @echo "All pods stopped" | ||||||||||||||||||||
|
|
||||||||||||||||||||
|
Comment on lines
+105
to
+109
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Stop also scales StatefulSets (MySQL/ClickHouse likely use them). Current rule only scales Deployments. k8s-stop: ## Stop all pods (scale deployments to 0)
@echo "Stopping all pods..."
@kubectl scale deployment --all --replicas=0 -n unkey
+ @kubectl scale statefulset --all --replicas=0 -n unkey
@echo "All pods stopped"📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||
| k8s-down: ## Delete all services from current Kubernetes cluster | ||||||||||||||||||||
| @echo "Deleting all services..." | ||||||||||||||||||||
| @kubectl delete namespace unkey --ignore-not-found=true | ||||||||||||||||||||
| @echo "Services deleted" | ||||||||||||||||||||
|
|
||||||||||||||||||||
| k8s-reset: k8s-down k8s-up ## Reset the entire Kubernetes environment | ||||||||||||||||||||
|
|
||||||||||||||||||||
| # Helper function to deploy a service | ||||||||||||||||||||
| define deploy-service | ||||||||||||||||||||
| @echo "Starting $(2)..." | ||||||||||||||||||||
| $(if $(3),@docker build -t $(3) $(4)) | ||||||||||||||||||||
| @kubectl apply -f k8s/manifests/namespace.yaml | ||||||||||||||||||||
| @kubectl apply -f k8s/manifests/$(1).yaml | ||||||||||||||||||||
| @kubectl wait --for=condition=ready pod -l app=$(1) -n unkey --timeout=180s | ||||||||||||||||||||
| $(if $(5),@kubectl wait --for=condition=ready pod -l app=$(5) -n unkey --timeout=180s) | ||||||||||||||||||||
| @echo "$(2) is ready!" | ||||||||||||||||||||
| endef | ||||||||||||||||||||
|
|
||||||||||||||||||||
| start-mysql: k8s-check ## Deploy only MySQL | ||||||||||||||||||||
| $(call deploy-service,mysql,MySQL,unkey/mysql:latest,-f ../deployment/Dockerfile.mysql ../) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| start-clickhouse: k8s-check ## Deploy only ClickHouse | ||||||||||||||||||||
| @docker build -t unkey/clickhouse:latest -f ../deployment/Dockerfile.clickhouse ../ | ||||||||||||||||||||
| $(call deploy-service,clickhouse,ClickHouse) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| start-redis: k8s-check ## Deploy only Redis | ||||||||||||||||||||
| $(call deploy-service,redis,Redis) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| start-s3: k8s-check ## Deploy only S3 (MinIO) | ||||||||||||||||||||
| $(call deploy-service,s3,S3 (MinIO)) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| start-planetscale: k8s-check ## Deploy only PlanetScale HTTP driver | ||||||||||||||||||||
| $(call deploy-service,planetscale,PlanetScale HTTP driver) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| start-observability: k8s-check ## Deploy only Observability stack | ||||||||||||||||||||
| $(call deploy-service,observability,Observability stack,,,otel-collector) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| start-api: k8s-check ## Deploy only API service (3 replicas) | ||||||||||||||||||||
| $(call deploy-service,api,API,unkey:latest,.) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| start-gw: k8s-check ## Deploy only Gateway service | ||||||||||||||||||||
| $(call deploy-service,gw,Gateway,unkey:latest,.) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| start-ctrl: k8s-check ## Deploy only ctrl service | ||||||||||||||||||||
| $(call deploy-service,ctrl,Control Plane,unkey:latest,.) | ||||||||||||||||||||
|
|
||||||||||||||||||||
|
Comment on lines
+147
to
+155
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 💡 Verification agent 🧩 Analysis chainEnsure manifests use the locally built image and sane pull policy. You build unkey:latest; make sure api/gw/ctrl manifests reference image: unkey:latest with imagePullPolicy: IfNotPresent (or Never for local). 🏁 Script executed: #!/bin/bash
# Verify images/pullPolicies for api/gw/ctrl
rg -nC2 -g 'k8s/manifests/*.yaml' -e 'name:\s*(api|gw|ctrl)' -e 'image:\s*unkey:latest' -e 'imagePullPolicy:'Length of output: 148 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "PWD: $(pwd)"
echo
echo "Searching for 'deploy-service' usages..."
rg -n --hidden -S "deploy-service" || true
echo
echo "Listing k8s/manifests directories if they exist..."
[ -d k8s ] && ls -la k8s || echo "no k8s dir"
[ -d k8s/manifests ] && ls -la k8s/manifests || echo "no k8s/manifests dir"
echo
echo "Searching repo for image references to 'unkey' and for imagePullPolicy (with context)..."
rg -n --hidden -S -C2 -E 'image:\s*.*unkey(:latest)?|imagePullPolicy:|name:\s*(api|gw|ctrl)' || true
echo
echo "Listing all YAML files found (first 200 lines) for manual inspection:"
rg --hidden --files -g '*.yml' -g '*.yaml' -S | sed -n '1,200p' || trueLength of output: 15270 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "=== Show deploy-service macro from go/Makefile ==="
awk 'BEGIN{p=0} /define deploy-service/{p=1} p{print NR": "$0} /endef/{if(p==1){print NR": "$0; exit}}' go/Makefile || true
echo
echo "=== Show Makefile lines around start-api/start-gw/start-ctrl ==="
sed -n '110,170p' go/Makefile || true
echo
for f in go/k8s/manifests/api.yaml go/k8s/manifests/gw.yaml go/k8s/manifests/ctrl.yaml; do
echo "=== Grep relevant keys in $f ==="
rg -n -C1 -E '^\s*(image:|imagePullPolicy:|name:|containers:)' "$f" || true
echo
echo "=== Full file ($f) head (first 200 lines) ==="
sed -n '1,200p' "$f" || true
echo
done
echo "=== Search repo for any 'image: .*unkey' or 'imagePullPolicy' occurrences ==="
rg -n --hidden -S -E 'image:\s*.*unkey(:latest)?|imagePullPolicy:' || trueLength of output: 14386 Fix image-tag mismatch: make built image tags match k8s manifests (or update manifests to match builds). Makefile start-api/start-gw/start-ctrl build/tag unkey:latest, but go/k8s/manifests/api.yaml, gw.yaml, ctrl.yaml reference unkey-api:latest / unkey-gw:latest / unkey-ctrl:latest (they already set imagePullPolicy: Never). Action: either change go/Makefile to build/tag the per-service images (recommended — e.g., build unkey-api:latest, unkey-gw:latest, unkey-ctrl:latest) or update the manifests to use unkey:latest and set imagePullPolicy to IfNotPresent (or keep Never for local). Files: go/Makefile (start-api/start-gw/start-ctrl), go/k8s/manifests/{api,gw,ctrl}.yaml. 🤖 Prompt for AI Agents |
||||||||||||||||||||
| start-dashboard: k8s-check ## Deploy only dashboard service | ||||||||||||||||||||
| $(call deploy-service,dashboard,Dashboard,unkey/dashboard:latest,-f ../apps/dashboard/Dockerfile ../) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| start-unkey-services: start-api start-gw start-ctrl start-dashboard ## Deploy all Unkey services | ||||||||||||||||||||
|
|
||||||||||||||||||||
| start-dependencies: start-mysql start-clickhouse start-redis start-s3 start-planetscale start-observability ## Deploy all dependency services | ||||||||||||||||||||
|
|
||||||||||||||||||||
| start-all: start-dependencies start-unkey-services ## Deploy all services individually | ||||||||||||||||||||
|
|
||||||||||||||||||||
| dev: ## Start with Tilt (if available) or fallback to k8s-up | ||||||||||||||||||||
| @if command -v tilt >/dev/null 2>&1; then \ | ||||||||||||||||||||
| echo "Starting with Tilt..."; \ | ||||||||||||||||||||
| tilt up; \ | ||||||||||||||||||||
| else \ | ||||||||||||||||||||
| echo "Tilt not found, using k8s-up instead"; \ | ||||||||||||||||||||
| echo "Install Tilt from: https://docs.tilt.dev/install.html"; \ | ||||||||||||||||||||
| make k8s-up; \ | ||||||||||||||||||||
| fi | ||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Document ‘k8s-status’ target and warn about destructive reset.
Expose status helper and add a data-loss warning for reset.
+Warning:
make k8s-resetmay delete persistent volumes and data (mysql/clickhouse/minio). Back up data or use ephemeral test DBs before running.In go/K8S_DEVELOPMENT.md around lines 39 to 51, the Make targets section should
expose a status helper and warn about data loss from reset: add a new "make
k8s-status" entry under a renamed subsection (e.g., "Status and services") and
insert a clear warning that "make k8s-reset" may delete persistent volumes and
data (mysql/clickhouse/minio), advising to back up data or use ephemeral test
DBs before running; ensure the new lines appear before the kubectl commands and
format the warning as a separate paragraph or block for visibility.