This repository contains source code for my own homelab's kubernetes instance. It is used to define & configure deployed apps, build charts that are then used by gitops mechanisms to deploy them on my cluster.
Thanks to cdk8s
, the project builds helm charts & apps charts using golang code and stores them in the generated
branch. This branch is then used by Flux CD to reconcile the k8s state with the built charts.
The kubernetes cluster is installed on multiple Mini PCs running Fedora, using k3s.
Most important apps installed on my cluster are:
- longhorn is used to provide distributed physical volumes across the cluster;
- vault & External Secrets Operator are used for most secrets. A few secrets are deployed using sealed-secrets, to unlock encrypted volumes;
- dex-idp is linked with my personal gitea instance to provide oauth based SSO to services supporting it;
- There also is an authentik instance linked with a bunch of services for SSO authentication and authorization;
- traefik-forward-auth protects a few services behind an oauth authn/authz firewall for apps not linked to dex-idp;
- cert-manager generates TLS certificates for ingresses on demand;
- PostgreSQL, Minio, ScyllaDB, MariaDB operators are installed to provide databases instances;
- NATS handles message queues;
- kube-prometheus-stack, along with grafana, provides metrics monitoring;
- blackbox-exporter is installed too;
- Grafana's loki with promtail are setup to aggregate and store logs;
- tempo for traces processing;
- linkerd, Kyverno, Tekton, trivy and temporal are present in the charts for testing;
- A bunch of end-users apps are installed, such as:
- wallabag, a visual to-read list;
- freshrss, a RSS aggregator;
- privatebin, a secure pastebin;
- paperless-ngx provides document management;
- yopass, to share secrets securely;
- bookstack, a platform to organise and share information;
- IT-Tools, some handy tools for engineers;
- vaultwarden, a bitwarden compatible password manager;
- send, simple, private file sharing;
- snippetbox, a code snippet portal;
- excalidraw, virtual collaborative whiteboard;
- wikijs, wiki as stated in its name;
- redmine, because project management;
- microbin another pastebin kind of instance;
- memos for note taking.
When a PR is merged in main
, a Drone CI-based pipeline is started to build cdk8s dependencies, then build charts, and finally push generated charts in the generated
branch. This branch is pulled every 2 minutes by flux
. See .drone.yml
.
Initial installation of Flux CD
comes with the flux
binary. The whole installation procedure is available on the official website.
Please start fluxcd
:
flux bootstrap git \
--url=ssh://[email protected]/mycroft/k8s-home.git \
--private-key-file=/tmp/rsa \
--branch=generated \
--path=generated/
Dump key file:
k get secret -n flux-system -o yaml flux-system | yq .data.identity -r | base64 -d > /tmp/rsa
Then re-run the bootstrap
command as seen in the Installation
section.
To check outdated Helm charts & images, use:
go build && ./k8s-home -check-version
Services must be updated by modifying versions in source code. Note all containers are not checked in with this command.
Secrets can be either stored in source code thanks to sealed-secrets
or in vault
and deployed as a Secret
in Kubernetes with external-secrets
.
Ingresses can be protected thanks to traefik-forward-auth
which perform an Oauth
process using my own private and external gitea
instance.
traefik should/must be reconfigured to redirect all http://
requests to https://
.
On master node, edit /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
and add the following snippet in valuesContent
section:
valuesContent: |-
ports:
web:
redirectTo: websecure
k3s will automatically redeploy the helm chart.
Also, it should allow IngressRoutes
to use middlewares
from another namespaces the ingressroutes is in (disabled by default)
providers:
kubernetesCRD:
allowCrossNamespace: true
Note: Do not edit traefik.yaml
. It is overwritten on restart.
port-forward the 9000 port and reach http://localhost:9000/dashboard/
:
> k port-forward -n kube-system (k get pods -n kube-system | grep ^traefik | cut -d' ' -f1) 9000:9000
Forwarding from 127.0.0.1:9000 -> 9000
Forwarding from [::1]:9000 -> 9000
Generate a login and use it on https://kubernetes-dashboard.services.mkz.me/
kubectl -n kubernetes-dashboard create token admin
You need a recent kubectl binary to be able to do this!
This is fully described on https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md
Vault was manually configured to allow kubernetes based authentication and linking a custom policy to allow External-Secrets queries.
As we're using transient service account token, there is no longer need to have a service account secret and to maintain it. More about this part can be found in the external-secrets docs about hashicorp vault provider.
Documentation about this can be found at https://developer.hashicorp.com/vault/docs/auth/kubernetes
Most likely what to be done is:
$ vault auth enable kubernetes
$ vault write auth/kubernetes/config \
kubernetes_host=https://10.43.0.1:443
$ vault write external-secrets readwrite /path/to/external-secrets.vault.hcl
$ vault write auth/kubernetes/role/external-secrets \
bound_service_account_names=external-secrets \
bound_service_account_namespaces=external-secrets \
policies=external-secrets \
ttl=24h
This should be sufficient to create valid secretstores & externalsecrets objects.
Add/edit the annotation:
kubectl annotate --overwrite -n monitoring helmrelease/prometheus reconcile.fluxcd.io/requestedAt="$(date +%s)"
After fixing the issue, suspend & resume the installation:
> flux suspend hr -n kubernetes-dashboard kubernetes-dashboard
► suspending helmrelease kubernetes-dashboard in kubernetes-dashboard namespace
✔ helmrelease suspended
> flux resume hr -n kubernetes-dashboard kubernetes-dashboard
► resuming helmrelease kubernetes-dashboard in kubernetes-dashboard namespace
✔ helmrelease resumed
◎ waiting for HelmRelease reconciliation
✔ HelmRelease reconciliation completed
✔ applied revision 5.11.0
It is required to not use the default externalTrafficPolicy to be able to get real-ip.
Please use the following configuration:
# cat /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
additionalArguments:
- "--serverstransport.insecureskipverify=true"
service:
spec:
externalTrafficPolicy: Local
# kubectl apply -f /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
# kubectl get svc -o yaml -n kube-system traefik | grep -i policy
externalTrafficPolicy: Local
internalTrafficPolicy: Cluster
See https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml for more values.
See https://github.com/bitnami-labs/sealed-secrets:
# Create a json/yaml-encoded Secret somehow:
# (note use of `--dry-run` - this is just a local file!)
echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o json >mysecret.json
# Warning! Do not forget the namespace & the secret name. By default SealedSecret is very strict about those.
# This is the important bit:
# (note default format is json!)
kubeseal <mysecret.json >mysealedsecret.json
# At this point mysealedsecret.json is safe to upload to Github,
# post on Twitter, etc.
# Eventually:
kubectl create -f mysealedsecret.json
To create and be able to use a PostgreSQL database, a few steps are required:
- In
postgresql.go
, add a record in the list. Commit and push the file; It will spawn the database and create secrets in thepostgres
namespace; - When database is ready, connect to database with psql to allow user to create tables:
$ k exec -ti -n postgres postgres-instance-0 -- psql -U dex-admin dex
dex=# GRANT CREATE ON SCHEMA public TO PUBLIC;
GRANT
- Retrieve username & password from
Secret
in thepostgres
namespace:
$ k get secret -n postgres dex-admin.postgres-instance.credentials.postgresql.acid.zalan.do -o yaml | yq -r .data.username | base64 -d
...
$ k get secret -n postgres dex-admin.postgres-instance.credentials.postgresql.acid.zalan.do -o yaml | yq -r .data.password | base64 -d
...
-
Create a record in
vault
(ie:namespaces/<namespace>/postgresql
) and create anusername
and apassword
record with the values retrieved in theSecret
. -
In the target namespace, create an ExternalSecretStore and an ExternalSecret. There is a lot of examples in the repository.