Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nifikop 1.6.0 duplicates flow on canvas (deploys two instead of one) #346

Closed
mostafamohajeri opened this issue Jan 5, 2024 · 9 comments
Closed

nifikop 1.6.0 duplicates flow on canvas (deploys two instead of one) #346

mostafamohajeri opened this issue Jan 5, 2024 · 9 comments
Labels
bug Something isn't working community

Comments

@mostafamohajeri
Copy link

What steps will reproduce the bug?

Hi, thanks for the help and the work, here's the method to reproduce

  1. In minikube
  2. follow the starting guide
  3. start minikube minikube start --driver=docker --cpus=2 --memory=8000 --extra-config=kubelet.cgroup-driver=systemd
  4. install cert-manager kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.2/cert-manager.yaml
  5. install nifikop helm install nifikop oci://ghcr.io/konpyutaika/helm-charts/nifikop --namespace=nifi --version 1.6.0 --set image.tag=v1.6.0-release --set resources.requests.memory=256Mi --set resources.requests.cpu=250m --set resources.limits.memory=256Mi --set resources.limits.cpu=250m --set namespaces={"nifi"} --create-namespace
  6. deploy zookeeper helm install zookeeper bitnami/zookeeper --namespace=zookeeper --set resources.requests.memory=256Mi --set resources.requests.cpu=250m --set resources.limits.memory=256Mi --set resources.limits.cpu=250m --set global.storageClass=standard --set networkPolicy.enabled=true --set replicaCount=1 --create-namespace

deploy everything:

  1. kubectl apply -f cluster.yaml -n nifi
  2. kubectl apply -f paramcontext.yaml -n nifi
  3. kubectl apply -f registry.yaml -n nifi
  4. kubectl apply -f flow.yaml -n nifi

cluster

apiVersion: nifi.konpyutaika.com/v1
kind: NifiCluster
metadata:
  name: simplenifi
spec:
  service:
    headlessEnabled: true
    labels:
      cluster-name: simplenifi
  zkAddress: "zookeeper.zookeeper:2181"
  zkPath: /simplenifi
  externalServices:
    - metadata:
        labels:
          cluster-name: driver-simplenifi
      name: driver-ip
      spec:
        portConfigs:
          - internalListenerName: http
            port: 18443
        type: LoadBalancer
  clusterImage: "apache/nifi:1.23.2"
  initContainerImage: "bash:5.2.2"
  oneNifiNodePerNode: true
  readOnlyConfig:
    nifiProperties:
      overrideConfigs: |
        nifi.sensitive.props.key=thisIsABadSensitiveKeyPassword
  pod:
    labels:
      cluster-name: simplenifi
  nodeConfigGroups:
    default_group:
      imagePullPolicy: IfNotPresent
      isNode: true
      serviceAccountName: default
      storageConfigs:
        - mountPath: "/opt/nifi/nifi-current/logs"
          name: logs
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            storageClassName: "standard"
            resources:
              requests:
                storage: 1Gi
      resourcesRequirements:
        limits:
          cpu: "1"
          memory: 1Gi
        requests:
          cpu: "1"
          memory: 1Gi
  nodes:
    - id: 1
      nodeConfigGroup: "default_group"
  propagateLabels: true
  nifiClusterTaskSpec:
    retryDurationMinutes: 10
  listenersConfig:
    internalListeners:
      - containerPort: 8443
        type: http
        name: http
      - containerPort: 6007
        type: cluster
        name: cluster
      - containerPort: 10000
        type: s2s
        name: s2s
      - containerPort: 9090
        type: prometheus
        name: prometheus
      - containerPort: 6342
        type: load-balance
        name: load-balance

param context

# kubectl create secret generic secret-params --from-literal=secret1=yop --from-literal=secret2=yep
apiVersion: nifi.konpyutaika.com/v1
kind: NifiParameterContext
metadata:
  name: dataflow-parameters
spec:
  description: "It is a test"
  clusterRef:
    name: simplenifi
    namespace: nifi
  parameters:
    - name: test
      value: toto
      description: tutu
    - name: test2
      value: toto
      description: toto

registry client

apiVersion: nifi.konpyutaika.com/v1
kind: NifiRegistryClient
metadata:
  name: registryclient
spec:
  clusterRef:
    name: simplenifi
    namespace: nifi
  description: "cool demo"
  uri: "http://reg.local:18080/"

flow

apiVersion: nifi.konpyutaika.com/v1
kind: NifiDataflow
metadata:
  name: testing1234
spec:
#  parentProcessGroupID: "d55f90d4-018c-1000-0000-00004a2c0a5a"
  bucketId: "571b4f9e-7839-412e-be75-3b06aa55dc3e"
  flowId: "7880d07c-4991-4a53-ac44-95ffc9f28850"
  flowVersion: 2
  syncMode: once
  skipInvalidControllerService: true
  skipInvalidComponent: true
  clusterRef:
    name: simplenifi
    namespace: nifi
  registryClientRef:
    name: registryclient
    namespace: nifi
  flowPosition:
    posX: 12
    posY: 200
  parameterContextRef:
    name: dataflow-parameters
    namespace: nifi
  updateStrategy: drain

What is the expected behavior?

one flow being deployed on the canvas

  • this is the behavior in v.1.5.0 *

What do you see instead?

two copies of the flow end up on the canvas

Only one seems to be the correct one:

  • One takes the name of the resource (testing1234) the other takes the name of the flow in the registry
  • One keps sync
  • One enables/starts
  • two events of Creating are generated in minikube dashboard, but one event of Created
  • One object of type NifiDataflow is shown in the dashboard

nifikop log after deploy (not touching the canvas/ui):

{"level":"info","time":"2024-01-05T11:31:27.608Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to patch status for NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T11:31:27.608Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"ad3d2c79-c535-419f-9523-e58636d66ca5","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}
{"level":"info","time":"2024-01-05T11:31:27.704Z","logger":"controllers.NifiDataflow","caller":"controllers/nifidataflow_controller.go:412","msg":"Adding Finalizer for NifiDataflow testing1234"}
{"level":"info","time":"2024-01-05T11:31:27.713Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to update NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T11:31:27.713Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"ffa329e0-4b4d-4fc9-ba25-a5309dd327b3","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}
{"level":"info","time":"2024-01-05T11:31:27.742Z","logger":"controllers.NifiDataflow","caller":"controllers/nifidataflow_controller.go:412","msg":"Adding Finalizer for NifiDataflow testing1234"}
{"level":"info","time":"2024-01-05T11:31:27.831Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to patch status for NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T11:31:27.831Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"a6eee858-299f-45a8-8584-1c2b1c80e399","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}
{"level":"info","time":"2024-01-05T11:31:28.001Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to update NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T11:31:28.001Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"4ddf87ad-4a81-4442-8842-75bb532d8863","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}
{"level":"info","time":"2024-01-05T11:31:32.244Z","logger":"controllers.NifiCluster","caller":"controllers/nificluster_controller.go:135","msg":"NifiCluster starting reconciliation","clusterName":"simplenifi"}
{"level":"info","time":"2024-01-05T11:31:32.357Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:284","msg":"Successfully reconciled cluster","component":"nifi","clusterName":"simplenifi","clusterNamespace":"nifi"}
{"level":"info","time":"2024-01-05T11:31:43.943Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to update status for NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T11:31:43.943Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"b0df95d6-6ba5-4319-b9cd-8b09f4b53292","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}
{"level":"info","time":"2024-01-05T11:31:44.200Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to update NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T11:31:44.200Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"a4421d7f-2aab-49ee-bc37-783386964fdc","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}
{"level":"info","time":"2024-01-05T11:31:52.378Z","logger":"controllers.NifiCluster","caller":"controllers/nificluster_controller.go:135","msg":"NifiCluster starting reconciliation","clusterName":"simplenifi"}
{"level":"info","time":"2024-01-05T11:31:52.470Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:284","msg":"Successfully reconciled cluster","component":"nifi","clusterName":"simplenifi","clusterNamespace":"nifi"}

Possible solution

nifikop v1.5.0 works as expected already

even the same cluster and resources if `helm del' version 1.6.0 and install 1.5.0 next flow deployments work fine. The vice-versa is also true, removing 1.5.0 and installing 1.6.0 brings the duplicating behavior back

have tried deleting minikube profile and other restart/re-installs

NiFiKop version

v1.6.0

Golang version

not installed

Kubernetes version

Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

NiFi version

1.23.2

Additional context

Thanks a lot for the help
@mostafamohajeri mostafamohajeri added bug Something isn't working community labels Jan 5, 2024
@mh013370
Copy link
Member

mh013370 commented Jan 5, 2024

Is it possible to show a screenshot of what the canvas looks like?

I have actually seen this happen before, but I haven't tracked down what causes it.

@mh013370
Copy link
Member

mh013370 commented Jan 5, 2024

I don't see this on any of our v1.6.0 nifikop driven NiFi clusters, but they were all upgraded with flows already in place. Perhaps this happens under a new flow deployment scenario.

@mostafamohajeri
Copy link
Author

hi @mh013370 , thanks for the response
indeed if I delete the duplicate and keep doing kubectl replace flow.yaml the duplicate does not come
but with apply and delete/create it comes back
I will create a screenshot, but the two stack on eachother so normally you won't see them (except for the darkened shadow) unless i move one. I already tested setting the position (hence the 12, 200 xy)

@mostafamohajeri
Copy link
Author

please see the two images, one is the two flows stacked, the flow on top is the "false" one, its name is still the same as the name in the registry (testing) and the other screenshot is after I moved it which shows the "true" flow which has the name of the resource (testing1234), it enables/starts and even if I move it the nifi operator reconciles it back

image
image(1)

@mh013370
Copy link
Member

mh013370 commented Jan 5, 2024 •
edited
Loading

Yes, i have seen exactly this scenario in the past. And like you describe, it doesn't impact anything because the duplicate doesn't get enabled/started by the operator. It should be addressed nonetheless.

Would it be possible to enable debug logging and share the log output? I'm curious what is causing the creation of two process groups in the UI.

https://github.com/konpyutaika/nifikop/blob/master/helm/nifikop/values.yaml#L57

@mostafamohajeri
Copy link
Author

logs a few lines before I deployed the flow until after

{"level":"debug","time":"2024-01-05T14:56:06.851Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:06.875Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Synchronizing parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Synchronizing"}
{"level":"debug","time":"2024-01-05T14:56:06.909Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Synchronized parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Synchronized"}
{"level":"debug","time":"2024-01-05T14:56:06.922Z","logger":"controllers.NifiParameterContext","caller":"controllers/nifiparametercontext_controller.go:339","msg":"Ensured Parameter Context","parameterContext":"dataflow-parameters"}
{"level":"debug","time":"2024-01-05T14:56:06.922Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Reconciled"}
{"level":"debug","time":"2024-01-05T14:56:21.622Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling registry client registryclient","type":"Normal","object":{"kind":"NifiRegistryClient","namespace":"nifi","name":"registryclient","uid":"9dc325b4-02a4-497c-b6f9-2cd48e3114e7","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"31926"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:21.653Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Synchronizing registry client registryclient","type":"Normal","object":{"kind":"NifiRegistryClient","namespace":"nifi","name":"registryclient","uid":"9dc325b4-02a4-497c-b6f9-2cd48e3114e7","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"31926"},"reason":"Synchronizing"}
{"level":"debug","time":"2024-01-05T14:56:21.681Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Synchronized registry client registryclient","type":"Normal","object":{"kind":"NifiRegistryClient","namespace":"nifi","name":"registryclient","uid":"9dc325b4-02a4-497c-b6f9-2cd48e3114e7","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"31926"},"reason":"Synchronized"}
{"level":"debug","time":"2024-01-05T14:56:21.690Z","logger":"controllers.NifiRegistryClient","caller":"controllers/nifiregistryclient_controller.go:271","msg":"Ensured Registry Client","registryClient":"registryclient"}
{"level":"debug","time":"2024-01-05T14:56:21.691Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling registry client registryclient","type":"Normal","object":{"kind":"NifiRegistryClient","namespace":"nifi","name":"registryclient","uid":"9dc325b4-02a4-497c-b6f9-2cd48e3114e7","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"31926"},"reason":"Reconciled"}
{"level":"debug","time":"2024-01-05T14:56:21.924Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:21.958Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Synchronizing parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Synchronizing"}
{"level":"debug","time":"2024-01-05T14:56:21.991Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Synchronized parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Synchronized"}
{"level":"debug","time":"2024-01-05T14:56:22.012Z","logger":"controllers.NifiParameterContext","caller":"controllers/nifiparametercontext_controller.go:339","msg":"Ensured Parameter Context","parameterContext":"dataflow-parameters"}
{"level":"debug","time":"2024-01-05T14:56:22.012Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Reconciled"}
{"level":"info","time":"2024-01-05T14:56:22.245Z","logger":"controllers.NifiCluster","caller":"controllers/nificluster_controller.go:135","msg":"NifiCluster starting reconciliation","clusterName":"simplenifi"}
{"level":"debug","time":"2024-01-05T14:56:22.245Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:85","msg":"reconciling","component":"nifi","clusterName":"simplenifi","clusterNamespace":"nifi"}
{"level":"debug","time":"2024-01-05T14:56:22.245Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:31","msg":"reconciling","kind":"*v1.Service","name":"simplenifi-headless"}
{"level":"debug","time":"2024-01-05T14:56:22.245Z","logger":"events","caller":"recorder/recorder.go:103","msg":"NifiCluster starting reconciliation","type":"Normal","object":{"kind":"NifiCluster","namespace":"nifi","name":"simplenifi","uid":"2bf91600-19f2-4857-aecd-6193874867b8","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"31858"},"reason":"ClusterReconciling"}
{"level":"debug","time":"2024-01-05T14:56:22.249Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:132","msg":"resource is in sync","kind":"*v1.Service"}
{"level":"debug","time":"2024-01-05T14:56:22.249Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:116","msg":"resource updated","name":"simplenifi-headless","namespace":"nifi","kind":""}
{"level":"debug","time":"2024-01-05T14:56:22.249Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:504","msg":"searching for pvc with label because name is empty","nifiCluster":"simplenifi","nodeId":"1","kind":"*v1.PersistentVolumeClaim"}
{"level":"debug","time":"2024-01-05T14:56:22.251Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:132","msg":"resource is in sync","kind":"*v1.PersistentVolumeClaim"}
{"level":"debug","time":"2024-01-05T14:56:22.255Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:31","msg":"reconciling","kind":"*v1.Secret","name":"simplenifi-config-1"}
{"level":"debug","time":"2024-01-05T14:56:22.294Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:132","msg":"resource is in sync","kind":"*v1.Secret"}
{"level":"debug","time":"2024-01-05T14:56:22.294Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:116","msg":"resource updated","name":"simplenifi-config-1","namespace":"nifi","kind":""}
{"level":"debug","time":"2024-01-05T14:56:22.294Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:588","msg":"searching for pod with label because name is empty","clusterName":"simplenifi","nodeId":"1","kind":"*v1.Pod"}
{"level":"debug","time":"2024-01-05T14:56:22.304Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:703","msg":"pod resource is in sync","clusterName":"simplenifi","podName":""}
{"level":"debug","time":"2024-01-05T14:56:22.304Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:31","msg":"reconciling","kind":"*v1.Service","name":"driver-ip"}
{"level":"debug","time":"2024-01-05T14:56:22.305Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:132","msg":"resource is in sync","kind":"*v1.Service"}
{"level":"debug","time":"2024-01-05T14:56:22.306Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:116","msg":"resource updated","name":"driver-ip","namespace":"nifi","kind":""}
{"level":"debug","time":"2024-01-05T14:56:22.369Z","logger":"controllers.NifiCluster","caller":"k8sutil/status.go:298","msg":"Root process group id updated","clusterName":"simplenifi","id":"da0a1f66-018c-1000-3b1a-d0ba0ea7acab"}
{"level":"info","time":"2024-01-05T14:56:22.415Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:284","msg":"Successfully reconciled cluster","component":"nifi","clusterName":"simplenifi","clusterNamespace":"nifi"}
{"level":"debug","time":"2024-01-05T14:56:22.416Z","logger":"controllers.NifiCluster","caller":"controllers/nificluster_controller.go:172","msg":"ensuring finalizers on nificluster","clusterName":"simplenifi"}
{"level":"debug","time":"2024-01-05T14:56:28.428Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34435"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:28.428Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Creating dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34435"},"reason":"Creating"}
{"level":"info","time":"2024-01-05T14:56:28.612Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to patch status for NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T14:56:28.613Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"ba807e74-7ef9-4016-bcfb-7ee44d1d91bb","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}
{"level":"debug","time":"2024-01-05T14:56:28.615Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34435"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:28.615Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Creating dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34435"},"reason":"Creating"}
{"level":"info","time":"2024-01-05T14:56:28.823Z","logger":"controllers.NifiDataflow","caller":"controllers/nifidataflow_controller.go:412","msg":"Adding Finalizer for NifiDataflow testing1234"}
{"level":"debug","time":"2024-01-05T14:56:28.823Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Created dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34440"},"reason":"Created"}
{"level":"info","time":"2024-01-05T14:56:28.834Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to update NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T14:56:28.834Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"3711cf1c-24c6-4971-ac0e-1a290a5fb0e8","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}
{"level":"debug","time":"2024-01-05T14:56:28.836Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34440"},"reason":"Reconciling"}
{"level":"info","time":"2024-01-05T14:56:28.879Z","logger":"controllers.NifiDataflow","caller":"controllers/nifidataflow_controller.go:412","msg":"Adding Finalizer for NifiDataflow testing1234"}
{"level":"info","time":"2024-01-05T14:56:29.015Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to patch status for NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T14:56:29.016Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"5e82b5cb-a31c-49a6-bc76-ad6512123387","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}
{"level":"debug","time":"2024-01-05T14:56:29.017Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34443"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:29.185Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34445"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:29.231Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Syncing dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34445"},"reason":"Synchronizing"}
{"level":"debug","time":"2024-01-05T14:56:29.521Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34445"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:29.614Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Syncing dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34445"},"reason":"Synchronizing"}

@mostafamohajeri
Copy link
Author

I am guessing the Creating dataflow testing1234 based on flow ... should not be happening twice

@juldrixx
Copy link
Contributor

juldrixx commented Jan 9, 2024

It appears that the operator failed to update the resource status to store the dataflow ID. Therefore, during the next reconciliation loop, the operator thinks that the dataflow has not been deployed. So he deploys it again.

But it shoudn't because we changed the update with patch. But maybe it the cause. @mh013370 what do you think?

@juldrixx juldrixx mentioned this issue Jan 9, 2024
Merged
5 tasks
@mostafamohajeri
Copy link
Author

Fixed in 1.7.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working community
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@juldrixx @mostafamohajeri @mh013370