🪨 Talos K8s Cluster 🔬

Repository for test infrastructure and Kubernetes cluster using GitOps practices.

Held together using Proxmox VE, OpenTofu, Talos, Kubernetes, Argo CD and copious amounts of YAML with some help from Renovate.

---

📖 Overview

This repository hosts the IaC (Infrastructure as Code) configuration for my test lab.

The test lab is backed by Proxmox VE hypervisor nodes with VMs bootstrapped using OpenTofu/Terraform.

Most of the services run on Talos flavoured Kubernetes, though I'm also running a TrueNAS VM for storage and Home Assistant VM for home automation.

To organise all the configuration I've opted for an approach using Kustomized Helm with Argo CD which I've explained in more detail in this article.

I journal my test lab journey over at my self-hosted blog.

🧑‍💻 Getting Started

If you're new to Kubernetes I've written a fairly thorough guide on Bootstrapping k3s with Cilium. In the article I try to guide you from a fresh Debian 12 Bookworm install to a working cluster using the k3s flavour of Kubernetes with Cilium as a CNI and IngressController.

I've also written an article on how to get started with Kubernetes on Proxmox if virtualisation is more your thing.

The current iteration of my test lab runs on Talos Kubernetes and is set up according to this article.

⚙️ Core Components

• Proxmox VE: Server management and KVM hypervisor.
• OpenTofu: Open source infrastructure as code tool.
• Cilium: eBPF-based Networking, Observability, Security.
• Proxmox CSI Plugin: CSI driver for storage
• Argo CD: Declarative, GitOps continuous delivery tool for Kubernetes.
• Cert-manager: Cloud native certificate management.
• Sealed-secrets: Encrypt your Secret into a SealedSecret, which is safe to store - even inside a public repository.
• Keycloak: Open source identity and access management
• Gateway API: Next generation of Kubernetes Ingress
• AdGuardHome: Domain name server backed by Unbound
• Netbird: Completely self hosted VPN solution

🗃️ Folder Structure

.
├── 📂 docs                # Documentation
├── 📂 k8s                 # Kubernetes manifests
│   ├── 📂 apps            # Applications
│   ├── 📂 infra           # Infrastructure components
│   └── 📂 sets            # Bootstrapping ApplicationSets
└── 📂 tofu                # Tofu configuration
    ├── 📂 home-assistant  # Home Assistant VM
    └── 📂 kubernetes      # Kubernetes VM configuration
        ├── 📂 bootstrap   # Kubernetes bootstrap config
        └── 📂 talos       # Talos configuration 

Kubernetes Components Dependency Graph

flowchart TD
        cert-manager --- cloudflare-api-token
        cloudflare-api-token --- sealed-secrets-controller
        gateway-controller --- certificates
        certificates --- cert-manager
        cloudflare-api-token --- git
        gateway-controller --- gateway-api-crds
        cilium-cni --- gateway-api-crds
        sealed-secrets-controller --- openssl-certificates-secret
        openssl-certificates-secret --- openssl-certificates
        openssl-certificates-secret --- talos
        sealed-secrets-controller --- talos

Loading

🏃‍➡️ Setup

NOTE: Proxmox should be deployed and accessible via ssh [email protected]. Local SSH ID file should be copied into /root/.ssh/authorized_keys (~/.ssh/id_rsa.pub)

First, we create Terraform variables file k8s.auto.tfvars in talos/cluster folder with main configuration:

#talos/cluster/k8s.auto.tfvars
api_token_id = "root@pam!terraform"

proxmox = {
  name         = "proxmox"
  cluster_name = "proxmox"
  endpoint     = "https://proxmox.lan:8006"
  insecure     = true
  
  ssh_username            = "root"
  ssh_private_key_file    = "~/.ssh/id_rsa"
}

Start deployment ...

$ cd tf
$ make talos

Bootstrap Kubernetes

Setup necessary components to start ArgoCD (Cilium, Gateway, cert-manager, CSI plugins)

$ make bootstrap-k8s

Create Sealed Secrets

Light LDAP Secrets

NOTE: Light LDAP sealed secrets should be generated only in case main opelssl certificates update link

Deploy ArgoCD

$ make argocd

IMPORTANT: Next steps depend on ArgoCD sync. Verify that applications are synced kubectl get Application -n argocd and gateways are created kubectl get Gateway -n gateway

Install ArgoCD CLI

$ brew install argocd

Retrieve auto generated ArgoCD admin password

$ argocd admin initial-password -n argocd
<generated password>

NOTE: There should be an entry for argocd.lan pointing to 192.168.50.223 in /etc/hosts. That is insecure gateway IP address (k8s/infra/network/gateway/gw-insecure.yaml)

Login into ArgoCD

$ argocd login argocd.lan:80
user: admin
password: <generated password>

Change admin password

$ argocd account update-password

Add ArgoCD account

$ kubectl config get-contexts -o name
admin@talos
$ argocd cluster add admin@talos

ArgoCD WebUI

WebUI should be accessible on https://argocd.emisia.net

Useful Tips:

TIP 1: Create Test Lab

Add test lab branch

NOTE: Make sure the code is in main git branch

$ git branch testlab-branch
$ git checkout testlab-branch

Setup testlab folder

Make testlab folder

$ mkdir k8s/testlab

Add ArgoCD project manifest

#File k8s/testlab/project.xml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: testlab
  namespace: argocd
spec:
  sourceRepos:
    - 'https://github.com/luminosita/k8s-cluster-talos'
  destinations:
    - namespace: 'argocd'
      server: '*'
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'

IMPORTANT: Make sure to all namespaces used by test application deployments into spec.destinations

IMPORTANT: Make sure to all additional source code repositories if used by ArgoCD test application manifest into spec.sourceRepos

Add ArgoCD test application manifest

#File k8s/testlab/test-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-app
  namespace: argocd
spec:
  project: testlab
  source:
    repoURL: https://github.com/luminosita/k8s-cluster-talos
    targetRevision: testlab-branch
    path: k8s/testlab/test-app
  destination:
    namespace: argocd
    name: in-cluster
  syncPolicy:
    automated:
      selfHeal: true
      prune: true

IMPORTANT: Make sure to place proper spec.source.path and spec.source.targetRevision values

Add Kustomization manifest

#File k8s/testlab/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd

resources:
  - project.yaml
  - test-app.yaml