Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Standby operator pod starts reconciling after namespace change #2341

Closed
gyfora opened this issue Apr 11, 2024 · 2 comments · Fixed by #2344
Closed

Standby operator pod starts reconciling after namespace change #2341

gyfora opened this issue Apr 11, 2024 · 2 comments · Fixed by #2344
Assignees

Comments

@gyfora
Copy link

gyfora commented Apr 11, 2024

Bug Report

Based on our production observations it seems to happen that in a HA setup with 2 operator pods, the standby (follower) starts reconciling resources in parallel with the leader after a namespace change event.

This is what we see:

  1. Leader operator reconciles correctly for couple days
  2. At a certain time we get the following log on the standby:
  Changing namespaces on 'flinkdeploymentcontroller' Controller to [...]...

In the Flink operator log this is triggered together with: controller.changeNamespaces(namespaces);

  1. After this log (and changing the namespaces), both the standby and leader starts to reconcile the events, no LeaderElection related logs can be seen otherwise.

This causes all kinds of issues with the managed resources :)

Environment

EKS

@gyfora
Copy link
Author

gyfora commented Apr 11, 2024

cc @csviri

@csviri csviri self-assigned this Apr 11, 2024
@csviri
Copy link
Collaborator

csviri commented Apr 11, 2024

thx @gyfora , we will take a look soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants