Apache Kafka event source does not work with minikube local docker

[aslom]

Expected Behavior

Apache Kafka source should work in minikube with local docker

Actual Behavior

export KO_DOCKER_REPO=ko.local and because ImagePullPolicy: "Always" is used in contrib/kafka/pkg/reconciler/resources/receive_adapter.go then source is ImagePullBackOff

$ kubectl get pods
NAME                                              READY   STATUS             RESTARTS   AGE
event-display-49mvc-deployment-68b86668f8-ktz76   3/3     Running            0          32s
kafka-source-vlgfq-675b76cbf5-49pp7               1/2     ImagePullBackOff   0          25s

 $ kubectl logs kafka-source-vlgfq-675b76cbf5-49pp7 -c receive-adapter
Error from server (BadRequest): container "receive-adapter" in pod "kafka-source-vlgfq-675b76cbf5-49pp7" is waiting to start: trying and failing to pull image

$ kubectl describe pod kafka-source-vlgfq-675b76cbf5-49pp7
...
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  81s                default-scheduler  Successfully assigned default/kafka-source-vlgfq-675b76cbf5-49pp7 to minikube
  Normal   Pulled     79s                kubelet, minikube  Container image "docker.io/istio/proxy_init:1.0.2" already present on machine
  Normal   Created    79s                kubelet, minikube  Created container
  Normal   Started    79s                kubelet, minikube  Started container
  Normal   Pulled     77s                kubelet, minikube  Container image "docker.io/istio/proxyv2:1.0.2" already present on machine
  Normal   Created    77s                kubelet, minikube  Created container
  Normal   Started    77s                kubelet, minikube  Started container
  Normal   Pulling    34s (x3 over 77s)  kubelet, minikube  pulling image "ko.local/receive_adapter-a3ffe8766802b0f13e2a5cc64eceba7c:c144fed62d699399c402f7a4110a633eef8a23e9686e3a0c44154c94de6da298"
  Warning  Failed     34s (x3 over 77s)  kubelet, minikube  Failed to pull image "ko.local/receive_adapter-a3ffe8766802b0f13e2a5cc64eceba7c:c144fed62d699399c402f7a4110a633eef8a23e9686e3a0c44154c94de6da298": rpc error: code = Unknown desc = Error response from daemon: Get https://ko.local/v2/: dial tcp: lookup ko.local: Temporary failure in name resolution
  Warning  Failed     34s (x3 over 77s)  kubelet, minikube  Error: ErrImagePull
  Normal   BackOff    7s (x5 over 76s)   kubelet, minikube  Back-off pulling image "ko.local/receive_adapter-a3ffe8766802b0f13e2a5cc64eceba7c:c144fed62d699399c402f7a4110a633eef8a23e9686e3a0c44154c94de6da298"
  Warning  Failed     7s (x5 over 76s)   kubelet, minikube  Error: ImagePullBackOff

Steps to Reproduce the Problem

1. Deploy controller will fail until changed to contrib/kafka/config/500-controller.yaml to IfNotPresent

      - name: manager
        image: github.com/knative/eventing-sources/contrib/kafka/cmd/controller
        imagePullPolicy: IfNotPresent

2. After editing 500-controller.yaml follow https://github.com/knative/eventing-sources/tree/master/contrib/kafka/samples but use export KO_DOCKER_REPO=ko.local
3. See errors above kubectl get pods

Additional Info

The default pull policy is IfNotPresent in k8s:
https://kubernetes.io/docs/concepts/containers/images/

[aslom]

I was looking to compare how other sources are doing and it seems to use IfNotPresent -https://github.com/knative/eventing-sources/blob/1fbf7302f71deb10540acc3b7a938f287ecc1b30/pkg/reconciler/containersource/resources/deployment.go#L76
https://github.com/knative/eventing-sources/search?q=ImagePullPolicy&unscoped_q=ImagePullPolicy

[aslom]

It seems that best option is to remove ImagePullPolicy or make it configurable to override default IfNotPresent? https://kubernetes.io/docs/concepts/containers/images/

[aslom]

@dubee going to change kafka source adapter to use default IfNotPresent but will allow to override it with new optional env variable set in eventing-sources/contrib/kafka/config/500-controller.yaml

    spec:
      serviceAccountName: kafka-controller-manager
      containers:
      - name: manager
        image: github.com/knative/eventing-sources/contrib/kafka/cmd/controller
        env:
        - name: KAFKA_RA_IMAGE
          value: github.com/knative/eventing-sources/contrib/kafka/cmd/receive_adapter
        - name: KAFKA_RA_IMAGE_PULL_POLLICY
          value: Always

and remove imagePullPolicy: Always

[dubee]

The optional pull policy override sounds like the way to go.

[dubee]

/assign @aslom

[lionelvillard]

@aslom this is typically done using Kustomize: https://github.com/kubernetes-sigs/kustomize, as an overlay

[bbrowning]

Why make this an option instead of just changing it from Always to IfNotPresent? Won't this reference an image reference be an image digest once we cut a release, and thus IfNotPresent sufficient?

[bbrowning]

Since my last comment discusses implementation, copying it over to the PR where we can discuss there.

[aslom]

Fixed by #303