Belacca infrastructure

Public host-level source of truth for the three VPS nodes running Belacca native k3s production on the three maintained production servers.

Current topology

Node	Public IPv4	Current role
belacca-k3s-01	169.58.97.73	native k3s server, embedded-etcd member, and public Traefik edge
belacca-k3s-02	169.58.143.41	native k3s server, embedded-etcd member, and public Traefik edge
belacca-k3s-03	169.58.143.42	native k3s server, embedded-etcd member, and public Traefik edge

All three machines are Debian 13, 6 vCPU, 11 GiB RAM, and approximately 197 GiB usable ext4 storage. They communicate over their public network addresses; no private interface has been observed. Cluster-internal ports are therefore restricted to the three node addresses and the explicitly approved operator access.

The native control plane is complete: all three nodes use k3s with embedded etcd. Native Traefik now runs on all three servers: .73, .41, and .42 each serve the public application edge.

Current platform state

• Flux is installed and reconciling the native cluster from belacca-gitops. The root, Longhorn, Traefik, cert-manager, TLS, routing, Pong, portfolio, analytics, Dex, Headlamp, and Flux Web Kustomizations are Ready.
• Public DNS-only A records for application hosts contain .73, .41, and .42; k3s-api.belacca.com remains .41 and .42 only. This is direct DNS round-robin, not health-aware failover.
• Kubernetes Secrets for the native cluster use SOPS encryption with age. Only encrypted manifests and the age recipient belong in GitOps; the age private key is held by Flux and backed up outside Git.
• Longhorn V1 is installed and reconciled through the native cluster's Flux path at chart 1.12.0. Its host prerequisites, including open-iscsi, iscsid, dm_crypt, and restricted replica/control traffic, are in place. The longhorn StorageClass is deliberately not the cluster default until stateful workload migration is reviewed.
• Native Traefik is Flux-managed on .73, .41, and .42 and is the active public production ingress. The private native Prometheus diagnostic foundation and live-tested Flux-to-Alertmanager-to-Telegram notification path are owned by GitOps; external SLO evidence remains owned by the status repository. The UID 0 low-port-binding exception remains a later hardening review item.

See the parent migration plan for the completed foundation work and the remaining cutover gates. The host hardening contract, provider boundary, recovery test, key rotation, and audit procedure are in docs/HOST-HARDENING-RECOVERY.md, with the machine-readable policy in ansible/inventories/production/hardening-policy.yml. The bounded native one-node failure/reboot procedure is in docs/NATIVE-FAILURE-DRILL.md; it is the only checked-in procedure that may request a planned native-node reboot. The latest sanitized execution outcome is recorded in docs/NATIVE-FAILURE-DRILL-STATUS.md.

Production traffic and current blockers

Native Traefik on .73, .41, and .42 owns public application traffic. Direct DNS is round-robin rather than health-aware failover (application records include all three native addresses).

The native Kubernetes API is accessed directly at https://169.58.143.41:6443, with k3s-api.belacca.com also resolving to .41 and .42 through direct DNS round-robin. This is operational but not a health-aware HA endpoint. A provider VIP or L4 load balancer remains a hardening option; do not invent or commit provider credentials.

The native Traefik instances are not behind a health-aware public load balancer. Cloudflare DNS-only records currently contain all three native addresses for application hosts; cert-manager DNS-01 and namespace-local TLS Secrets are active. The external status publisher checks the public portfolio, Pong, analytics collector, and alias redirects, while native Prometheus remains diagnostic. Monitor all three native edges and remove an unhealthy address manually until a health-aware design is provisioned. Do not broaden API, etcd, kubelet, overlay, or Longhorn firewall rules to compensate.

State migration status: completed local handoff

Pong, GoatCounter, and Dex writers were quiesced. Their SQLite files were copied, integrity-checked, restored into native Longhorn-backed RWO PVCs, and verified by native startup and functional tests. External backup retention and an isolated restore rehearsal remain follow-up work. The following controls remain important for future state operations:

• a valid native kubeconfig and current context using the intended stable API endpoint and verified API certificate identity;
• native Longhorn live health and capacity, including healthy managers, nodes, disks, volumes, engines, and replicas with no unexplained degraded, faulted, or unknown state;
• reviewed target PVC contracts for every stateful workload, including StorageClass, size, access mode, filesystem, mount ownership, reclaim behavior, and single-writer requirements;
• identifiable, integrity-checked backup artifacts with a successful restore result, not merely a backup plan or an untested copy; and
• an approved quiescence procedure that stops or fences writers, verifies that writes have ceased, and captures the final consistent backup.

Native Longhorn PVCs are now active production state. containers and local volumes were removed; do not recreate or rely on them as a rollback target. Encrypted off-cluster backups, isolated restore rehearsal, one-node failure drills, authenticated operator journeys, and a health-aware API/ingress endpoint remain unproven hardening work.

Boundaries

• This repository contains no passwords, private keys, kubeconfigs, join tokens, OAuth credentials, or application data.
• belacca-gitops owns Kubernetes resources, Flux reconciliation, application ingress, and encrypted Secret manifests.
• This repository owns Debian prerequisites, SSH/firewall posture, k3s host configuration, and storage prerequisites. Native public-edge operation does not change that ownership boundary.
• SOPS/age encrypted Kubernetes Secrets are stored in the public GitOps repository as ciphertext and decrypted by Flux. The age private key remains out of Git and out of this repository.
• application, ingress, and state handoff checks. Retain Git history and follow the post-cutover external-backup and failure-drill plan.

Operator setup

The automation host needs Ansible and the dedicated key for audits and any future, explicitly approved host changes:

sudo apt-get update
sudo apt-get install -y ansible-core
chmod 600 ~/.ssh/belacca-infra_ed25519
ansible all -m ping
ansible-playbook ansible/playbooks/audit.yml
./scripts/check-inventory.sh

The native cluster is already bootstrapped. For an existing cluster, use the idempotent rolling maintenance playbook to ensure k3s and iSCSI start after a host reboot; do not rerun the bootstrap playbook:

ansible-playbook ansible/playbooks/ensure-boot.yml

The playbook runs with serial: 1, enables k3s.service, iscsid.service, and open-iscsi.service, and installs a k3s systemd drop-in ordering k3s after network-online.target and iscsid.service. Kubernetes workloads, Flux, and Longhorn then recover through their normal controllers. Any other host change must use a reviewed maintenance window. Never put a join token in the repository; if a future approved operation requires one, supply it only at runtime and unset it afterward.

k3s HA requirements

The three server nodes use embedded etcd. Required node-to-node traffic is:

• TCP 2379-2380: embedded etcd, server nodes only
• TCP 6443: Kubernetes API, cluster peers and approved operator access only
• TCP 10250: kubelet, cluster peers only
• UDP 51820: Flannel WireGuard-native overlay, cluster peers only
• Longhorn frontend and replica/control ports: storage nodes only
• TCP 80/443: public ingress only where an edge is intentionally exposed

The planned stable API name is k3s-api.belacca.com, but it is not currently backed by the recommended single Contabo floating VIP with active/passive failover or by any other validated load-balancing design. The direct .41:6443 endpoint remains the temporary read-only/audit path until that work is completed. Never expose etcd, kubelet, overlay, or Longhorn ports to the public Internet.

Storage

k3s HA does not replicate PersistentVolumes. Longhorn V1 is installed through the native cluster's Flux path at chart 1.12.0, with the required host packages, iscsid, dm_crypt, and restricted inter-node storage ports. Its installation is not proof of live volume health or migration readiness; use current read-only Longhorn status to prove healthy managers, nodes, disks, volumes, engines, replicas, and capacity before restoration. Its replicated volumes improve node recovery but do not make SQLite or other single-writer workloads safe for multiple writers. Review and explicitly select the longhorn StorageClass through the target PVC contracts as stateful workloads are migrated.

Secret lifecycle

Use SOPS with age for Kubernetes Secrets:

1. Generate an age key and back up the private key offline.
2. Store only the public recipient and encrypted Secret manifests in the public GitOps repository.
3. Install the private age key only as the Flux decryption Secret in flux-system.
4. Back up the Flux key and the encrypted repository independently.
5. Rotate application credentials without committing plaintext values.

Do not use this repository, GitHub Actions logs, shell history, or .env files for plaintext credentials. The hardening completion gate is intentionally fail-closed until provider firewall, console recovery, and native recovery runtime evidence are reviewed:

python3 scripts/validate_hardening.py --require-runtime-evidence