Issue with Job/tobs-grafana-db init container

[alexpdp7]

Installing the new version of tobs in my new cluster does not work correctly.

The grafana-db init container fails due to name resolution.

$ kubectl logs -n tobs tobs-grafana-db-c5j2x -c init-db 
Checking if tobs.tobs.svc is up
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'tobs.tobs.svc'
Num times failed: 0
Waiting 5 seconds
Checking if tobs.tobs.svc is up
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'tobs.tobs.svc'
Num times failed: 1
Waiting 5 seconds
Checking if tobs.tobs.svc is up
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'tobs.tobs.svc'
Num times failed: 2
Waiting 5 seconds
Checking if tobs.tobs.svc is up
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'tobs.tobs.svc'
Num times failed: 3
Waiting 5 seconds
Checking if tobs.tobs.svc is up
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'tobs.tobs.svc'
Num times failed: 4
Waiting 5 seconds
Checking if tobs.tobs.svc is up
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'tobs.tobs.svc'
Num times failed: 5
Failed 5 times while waiting for tobs.tobs.svc, exiting

I can reproduce the issue easily:

$ kubectl run -i --tty busybox --image=busybox:1.28 --restart=Never -n tobs
If you don't see a command prompt, try pressing enter.
/ # nslookup tobs.tobs.svc
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'tobs.tobs.svc'
/ # nslookup tobs
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      tobs
Address 1: 10.98.169.108 tobs.tobs.svc.cluster.local
/ # nslookup tobs.tobs.svc.cluster.local
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      tobs.tobs.svc.cluster.local
Address 1: 10.98.169.108 tobs.tobs.svc.cluster.local

There are many issues with DNS resolution with busybox (see https://bugs.busybox.net/show_bug.cgi?id=14671, docker-library/busybox#48 (comment)) and ndots.

If I use the rockylinux:8 container, install bind-utils, the same name resolution works:

$ kubectl run -i --tty busybox --image=rockylinux:8 --restart=Never -n tobs
If you don't see a command prompt, try pressing enter.
[root@busybox /]# dnf install bind-utils
...
[root@busybox /]# nslookup tobs.tobs.svc
Server:		10.96.0.10
Address:	10.96.0.10#53

Name:	tobs.tobs.svc.cluster.local
Address: 10.98.169.108

Environment

• tobs installation method:

• cli

• tobs version:

Tobs CLI Version: 0.10.0, latest tobs helm chart version: 0.10.0

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:38:19Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

Single-node kubeadm-provisioned cluster on top of Fedora CoreOS, using crio. See https://github.com/alexpdp7/proxmox-k8s (passes sonobuoy, except for the tests that require multiple nodes)

[alexpdp7]

I thought I'd managed to reproduce the underlying DNS issue with kind/minikube/k3s, but I cannot. So closing this, because it seems more an issue with my cluster (although busybox:uclibc has a long history of DNS issues).

In any case, making Grafana using SQLite seems to be a good idea :)

[alexpdp7]

In case someone has the same problem, those are the steps I used to install the Helm chart, overriding the busybox image.

Install cert-manager (see https://cert-manager.io/docs/installation/):

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml

$ git clone [email protected]:timescale/tobs.git
$ cd tobs/charts
$ git checkout release-0.10

Apply this change:

$ git diff
diff --git a/chart/templates/grafana-db-user-job.yaml b/chart/templates/grafana-db-user-job.yaml
index 1c97fc7..4cb1ea5 100644
--- a/chart/templates/grafana-db-user-job.yaml
+++ b/chart/templates/grafana-db-user-job.yaml
@@ -50,7 +50,7 @@ spec:
           name: {{ $.Release.Name }}-grafana-db
       initContainers:
       - name: init-db
-        image: busybox:1.28
+        image: busybox:1.28-glibc
         volumeMounts:
         - name: sql-volume
           mountPath: /wait-for-ts.sh

NOTE: if you use an editor to perform the change, make sure it does not leave a backup file behind. Helm will try to create things twice if there are backup files.

$ rm Chart.lock
$ helm dependency build
$ helm install --create-namespace -n tobs --wait tobs . --debug