minikube helm chart

README.md

@@ -4,6 +4,7 @@ queries against Wikipedia and sister projects databases.
 
 ## Setting up a local dev environment ##
 
+# docker-compose
 Quarry uses [Docker](https://docs.docker.com/engine/install/) to set up a local
 environment. You can set it up by:
 
@@ -18,19 +19,43 @@ will imediatelly be taken into account.
 A worker node is also created to execute your queries in the background (uses the
 same image). Finally, redis and two database instances are also started.
 
-One database is your quarry database the other is a wikireplica-like database
-named `mywiki`. This (or `mywiki_p`) is the correct thing to enter in the
-database field on all local test queries.
-
-In your local environment, you can query Quarry internal db itself. Use then
-"quarry" as database name.
-
 To stop, run `docker-compose stop` or hit CTRL-C on the terminal your docker-compose
 is running in. After that, to start with code changes, you'll want to `docker-compose down`
 to clean up. Also, this creates a docker volume where sqlite versions of query
 results are found. That will not be cleaned up unless you run `docker-compose down -v`
 
 
+
+# minikube
+It is possible to run a quarry system inside [minikube](https://minikube.sigs.k8s.io/docs/)!
+At this time, you need to set it up with a cluster version before 1.22, most likely.
+
+First build the containers:
+```
+eval $(minikube docker-env)
+docker build . -t quarry:01
+cd docker-replica/
+docker build . -t mywiki:01
+```
+
+You will need to install minikube (tested on minikube 1.23) and [helm](https://helm.sh) and kubectl on your system. When you are confident those are working, start minikube with:
+ - `minikube start --kubernetes-version=v1.23.15`
+ - `minikube addons enable ingress`
+ - `kubectl create namespace quarry`
+ - `helm -n quarry install quarry helm-quarry -f helm-quarry/dev-env.yaml`
+
+The rest of the setup instructions will display on screen as long as the install is successful.
+
+# local databases
+Both local setups will create two databases.
+
+One database is your quarry database the other is a wikireplica-like database
+named `mywiki`. This (or `mywiki_p`) is the correct thing to enter in the
+database field on all local test queries.
+
+The other database is the Quarry internal db. In your local environment, you can query Quarry internal db itself. Use then
+"quarry" as database name.
+
 ### Updating existing containers ###
 
 If you had already run a dev environment (that is, ran `docker-compose up`) you might want to update

docker-compose.yml

@@ -30,7 +30,7 @@ services:
       - "redis"
 
   redis:
-    image: redis:alpine
+    image: redis:alpine3.18
 
   # wikireplica simulator
   mywiki:

docker-replica/Dockerfile

@@ -0,0 +1,3 @@
+FROM mariadb:10.4-focal
+
+COPY replica.sql /docker-entrypoint-initdb.d/replica.sql

helm-quarry/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

helm-quarry/Chart.yaml

@@ -0,0 +1,4 @@
+apiVersion: v2
+description: A Helm chart for quarry
+name: helm-quarry
+version: "1.0"

helm-quarry/dev-env.yaml

@@ -0,0 +1,10 @@
+localdev:
+  enabled: true
+web:
+  replicas: 1
+  memory: "300Mi"
+  cpu: "100m"
+worker:
+  replicas: 1
+  memory: "400Mi"
+  cpu: "100m"

helm-quarry/dev_config.yaml

@@ -0,0 +1,31 @@
+# The OAUTH tokens are callbacks to a quarry.local domain
+# it is ok that they are public.
+OAUTH_CONSUMER_TOKEN: "deeb8dbabbf157cb0ed120b89971d800"
+OAUTH_SECRET_TOKEN: "38fdce038bbf848453a856aa329f8a06c456c791"
+SECRET_KEY: 'aaarggghhpleaserescuemeiamstuckinarandomnumbergeneratorfactory'
+DEBUG: True
+DB_HOST: 'db'
+DB_NAME: 'quarry'
+DB_USER: 'quarry'
+DB_PASSWORD: 'quarry'
+broker_url: 'redis://redis'
+result_backend: 'redis://redis'
+worker_concurrency: 24  # Since all tasks are IO bound
+task_acks_late: True  # Tasks are idempotent!
+task_track_started: True
+worker_prefetch_multiplier: 1 # Tasks can run for a long time
+# Just query the quarry database itself.
+REPLICA_DOMAIN: ''
+REPLICA_HOST: 'mywiki'
+REPLICA_DB: 'mywiki_p'
+REPLICA_USER: 'repl'
+REPLICA_PASSWORD: 'repl'
+REPLICA_PORT: 3306
+OUTPUT_PATH_TEMPLATE: '/results/%s/%s/%s.sqlite'
+REDIS_HOST: 'redis'
+REDIS_PORT: 6379
+REDIS_DB: 0
+QUERY_TIME_LIMIT: 60  # 1 minute
+QUERY_RESULTS_PER_PAGE: 50
+KILLER_LOG_PATH: 'killer.log'
+MAINTENANCE_MSG: 'This is your local development environment.'

helm-quarry/prod-env.yaml

@@ -0,0 +1,11 @@
+localdev:
+  enabled: false
+
+web:
+  replicas: 8
+  memory: "300Mi"
+  cpu: "100m"
+worker:
+  replicas: 2
+  memory: "700Mi"
+  cpu: "100m"

helm-quarry/prod_config.yaml

@@ -0,0 +1,33 @@
+## Replace these with prod values and encrypt
+
+# The OAUTH tokens are callbacks to a quarry.local domain
+# it is ok that they are public.
+OAUTH_CONSUMER_TOKEN: "deeb8dbabbf157cb0ed120b89971d800"
+OAUTH_SECRET_TOKEN: "38fdce038bbf848453a856aa329f8a06c456c791"
+SECRET_KEY: 'aaarggghhpleaserescuemeiamstuckinarandomnumbergeneratorfactory'
+DEBUG: True
+DB_HOST: 'db'
+DB_NAME: 'quarry'
+DB_USER: 'quarry'
+DB_PASSWORD: 'quarry'
+broker_url: 'redis://redis'
+result_backend: 'redis://redis'
+worker_concurrency: 24  # Since all tasks are IO bound
+task_acks_late: True  # Tasks are idempotent!
+task_track_started: True
+worker_prefetch_multiplier: 1 # Tasks can run for a long time
+# Just query the quarry database itself.
+REPLICA_DOMAIN: ''
+REPLICA_HOST: 'mywiki'
+REPLICA_DB: 'mywiki_p'
+REPLICA_USER: 'repl'
+REPLICA_PASSWORD: 'repl'
+REPLICA_PORT: 3306
+OUTPUT_PATH_TEMPLATE: '/results/%s/%s/%s.sqlite'
+REDIS_HOST: 'redis'
+REDIS_PORT: 6379
+REDIS_DB: 0
+QUERY_TIME_LIMIT: 60  # 1 minute
+QUERY_RESULTS_PER_PAGE: 50
+KILLER_LOG_PATH: 'killer.log'
+MAINTENANCE_MSG: 'This is your local development environment.'

helm-quarry/schema.sql

@@ -0,0 +1 @@
+../schema.sql

helm-quarry/templates/NOTES.txt

@@ -0,0 +1,14 @@
+Thank you for installing {{ .Chart.Name }}!
+
+An essential part of setting up your dev environment is adding an entry to your /etc/hosts file.
+Get the IP of minikube with:
+minikube ip
+Add this line to your /etc/hosts file:
+<ip address> quarry.local
+
+You can override any values you need in a file called dev-values.yaml.
+
+If you create one, run:
+helm -n {{ .Release.Namespace }} upgrade {{ .Release.Name }} helm-quarry/ -f dev-values.yaml
+Happy hacking.
+

helm-quarry/templates/config_yaml.yaml

@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: config
+data:
+{{ if .Values.localdev.enabled }}
+  config.yaml: {{ .Files.Get "dev_config.yaml" | quote }}
+{{ else }}
+  config.yaml: {{ .Files.Get "prod_config.yaml" | quote }}
+{{ end }}

helm-quarry/templates/db_deployment.yaml

@@ -0,0 +1,36 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: db
+  labels:
+    app: db
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: db
+  template:
+    metadata:
+      labels:
+        app: db
+    spec:
+      containers:
+        - name: db
+          image: mariadb:10.1.48-bionic
+          imagePullPolicy: Always
+          env:
+          - name: MYSQL_USER
+            value: "quarry"
+          - name: MYSQL_PASSWORD
+            value: "quarry"
+          - name: MYSQL_DATABASE
+            value: "quarry"
+          - name: MYSQL_RANDOM_ROOT_PASSWORD
+            value: "1"
+          volumeMounts:
+          - name: schema
+            mountPath: /docker-entrypoint-initdb.d/
+      volumes:
+        - name: schema
+          configMap:
+            name: db-schema

helm-quarry/templates/db_schema_cm.yaml

@@ -0,0 +1,7 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: db-schema
+data:
+  schema.sql: |-
+{{ .Files.Get "schema.sql" | indent 4}}

helm-quarry/templates/db_service.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: db
+spec:
+  selector:
+    app: db
+  ports:
+  - name: db
+    port: 3306
+    targetPort: 3306

helm-quarry/templates/mywiki_deployment.yaml

@@ -0,0 +1,29 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mywiki
+  labels:
+    app: mywiki
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: mywiki
+  template:
+    metadata:
+      labels:
+        app: mywiki
+    spec:
+      containers:
+        - name: mywiki
+          image: mywiki:01 # replace with quay.io
+          imagePullPolicy: Never
+          env:
+          - name: MYSQL_USER
+            value: "repl"
+          - name: MYSQL_PASSWORD
+            value: "repl"
+          - name: MYSQL_DATABASE
+            value: "repl"
+          - name: MYSQL_RANDOM_ROOT_PASSWORD
+            value: "1"

helm-quarry/templates/mywiki_service.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: mywiki
+spec:
+  selector:
+    app: mywiki
+  ports:
+  - name: mywiki
+    port: 3306
+    targetPort: 3306

helm-quarry/templates/redis_deployment.yaml

@@ -0,0 +1,20 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: redis
+  labels:
+    app: redis
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: redis
+  template:
+    metadata:
+      labels:
+        app: redis
+    spec:
+      containers:
+        - name: redis
+          image: redis:alpine3.18
+          imagePullPolicy: Always

helm-quarry/templates/redis_service.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis
+spec:
+  selector:
+    app: redis
+  ports:
+  - name: redis
+    port: 6379
+    targetPort: 6379

helm-quarry/templates/results_pv.yaml

@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: results
+spec:
+  storageClassName: manual
+  capacity:
+    storage: 1Gi
+  accessModes:
+    - ReadWriteMany
+  hostPath:
+    path: "/results"

helm-quarry/templates/results_pvc.yaml

@@ -0,0 +1,9 @@
+kind: PersistentVolumeClaim
+apiVersion: v1
+metadata:
+  name: results
+spec:
+  accessModes: [ReadWriteMany]
+  resources:
+    requests:
+      storage: 1Gi