Try to get lease namespace if unspecified #1450

honnix · 2022-09-11T17:41:21Z

This PR makes providing lease namespace optional. If unspecified, it will try to read from a file pointed by system property or env var kubenamespace with default value as /var/run/secrets/kubernetes.io/serviceaccount/namespace which is injected by k8s. The code is inspired by https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client-api/src/main/java/io/fabric8/kubernetes/client/Config.java.

The idea is to simplify use code because in most cases users will need to do the same. Please let me know if this makes sense. Thanks.

@csviri @metacosm

honnix · 2022-09-11T17:42:37Z

...k-core/src/main/java/io/javaoperatorsdk/operator/api/config/LeaderElectionConfiguration.java

@@ -59,8 +68,8 @@ public LeaderElectionConfiguration(
    this.identity = identity;
  }

-  public String getLeaseNamespace() {
-    return leaseNamespace;
+  public Optional<String> getLeaseNamespace() {


Note: this is a breaking change. I can mark it as deprecated and provide another method getOptionalLeaseNamespace, if needed.

Ahh normally I would prefer backwards compatible with deprecated. getOptionalLeaseNamespace does not sound too good, maybe just stick with this this time.

Probably would be better, yes. Then again, I don't know if anyone is using Leader Election already…

I think for users, direct usage of this API will be rare even they do use leader election feature. Probably only in unit test.

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/LeaderElectionManager.java

csviri · 2022-09-12T07:35:40Z

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/LeaderElectionManager.java

    if (id == null || id.isBlank()) {
      id = UUID.randomUUID().toString();
    }
    return id;
  }

+  private static Optional<String> tryNamespaceFromPath() {


Not sure if this is needed, could you try instead just use the client's facilities, like:
Config.autoConfigure(null).getNamespace() ?

🤔 Would this lead to the same NS? And actually, we should probably get it from ConfigurationServiceProvider.instance().getClientConfiguration() instead of creating a new Config object that might not match the configured one.

default Config getClientConfiguration() { return Config.autoConfigure(null); }

This seems create a new config as well.

I can give it a try and see whether those tests still work.

I could get tests pass by setting System.setProperty(KUBERNETES_AUTH_TRYKUBECONFIG_SYSTEM_PROPERTY, "false");, but I worry that loading a full config here might not be a correct approach because it confuses users if they do not expect .kube/config to be read at all. The intention of this change is to only make it work in-cluster.

Good point, maybe we can then just stick with the current impl for now

This seems create a new config as well.

That's just the default implementation, though. The Quarkus extension, for example, returns the configuration associated with the client (i.e. the one the client uses to connect to the cluster).

Are there use cases where the config would be configured to use a different namespace than the one associated with the service account? Do we specifically only want the SA namespace regardless of how the Kubernetes config is set up?

@metacosm OK I see what you mean now. Let me try something.

So something like 0c12947 ? If the config was loaded from .kube/config, there is for sure no namespace; if it is in-cluster config, the namespace will be read from /var/run/secrets/kubernetes.io/serviceaccount/namespace if no overridden.

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/LeaderElectionManager.java

csviri · 2022-09-12T07:38:16Z

...k-core/src/main/java/io/javaoperatorsdk/operator/api/config/LeaderElectionConfiguration.java

@@ -59,8 +68,8 @@ public LeaderElectionConfiguration(
    this.identity = identity;
  }

-  public String getLeaseNamespace() {
-    return leaseNamespace;
+  public Optional<String> getLeaseNamespace() {


Ahh normally I would prefer backwards compatible with deprecated. getOptionalLeaseNamespace does not sound too good, maybe just stick with this this time.

metacosm · 2022-09-12T07:45:56Z

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/LeaderElectionManager.java

    if (id == null || id.isBlank()) {
      id = UUID.randomUUID().toString();
    }
    return id;
  }

+  private static Optional<String> tryNamespaceFromPath() {


I think we could reasonably ask for the related method to become public in the Fabric8 client so that we don't have to maintain it here…

Yeah that would be better. Do you think I could give it a shot?

I will give it a try. On the other hand, if we want to get this shipped, we don't need to get blocked because the roundtrip might take some time.

yep, we can adjust it here after, and ship this as it is. Maybe just create an issue so we don't forget it.

metacosm

On the fence regarding the change in API, LGTM otherwise.

csviri

LGTM

csviri · 2022-09-12T11:42:13Z

Shouldn't we consider this rather a bug? and put it to main?

wdyt @metacosm @honnix ?

honnix · 2022-09-12T11:48:32Z

@csviri I guess we can. :) Strictly speaking it is still more like a feature, but given this small breaking change, shipping it sooner seems to be better.

honnix · 2022-09-12T12:00:31Z

I'm making a small tweak.

metacosm · 2022-09-12T12:51:42Z

...ator-framework-core/src/test/java/io/javaoperatorsdk/operator/LeaderElectionManagerTest.java

+    var namespacePath = tempDir.resolve("namespace");
+    Files.writeString(namespacePath, namespace);
+
+    System.setProperty(KUBERNETES_AUTH_TRYKUBECONFIG_SYSTEM_PROPERTY, "false");


Do you know why it's needed to set this property to false?

It is because of https://github.com/fabric8io/kubernetes-client/blob/0742e942eb09f8ed2672f7f82e176b76e595ad50/kubernetes-client-api/src/main/java/io/fabric8/kubernetes/client/Config.java#L600. Otherwise it will try to read .kube/config. By setting this the test case makes sure it tries to read the file.

I'm however still not sure this gets the correct client config.

The order also seems to be wrong:

configurationService.getLeaderElectionConfiguration() .ifPresent(c -> leaderElectionManager.init(c, this.kubernetesClient)); ConfigurationServiceProvider.set(configurationService);

Now the PR as it is, tries to access ConfigurationServiceProvider.instance() before ConfigurationServiceProvider.set(configurationService) is called.

For the use case creating k8s client manually, this seems to be required but maybe it is supposed to be so?

var operator = new Operator( kubernetesClient, configurationServiceOverrider -> configurationServiceOverrider .withCloseClientOnStop(false) .withClientConfiguration(kubernetesClient.getConfiguration()));

Hmm, the instance is already set in ConfigurationServiceProvider::overrideCurrent, so ConfigurationServiceProvider.set(configurationService); doesn't do anything.

What do you mean manually created? In the background it always created, if the namespace is not set it should use always one where the pod is running.

For example:

var kubernetesClient = ...; # build the client; may not be from .kube/config var operator = new Operator( kubernetesClient, configurationServiceOverrider -> configurationServiceOverrider .withCloseClientOnStop(false) .withClientConfiguration(kubernetesClient.getConfiguration()));

If here withClientConfiguration is not called, I think a default configuration will be created, which might be different than the configuration used to build kubernetesClient. Then we have an inconsistency. I'm not sure of the consequence though.

Yes, that is absolutely true. The thing is that on the other hand on quarkus extension it would make issues when the config from config service not used. Note however that if the namespace is configured for a client, it's not our api.
So we have api for namespace config on controller level. And we have api for leader election to config namespace. So I would expect that the leader election is either uses the pod namespace or a setting explicitly for it.

Yeah I understand that. I think as you said this is a separated issue to track, not related to leader election. I can create a new issue.

I modified the existing e2e test a bit. PTAL, thanks.

We probably need to clean things up or, at the very least, document them better. The intention is to create client instance(s) based on the configuration retrieved from the ConfigurationService. I'm not sure that we do follow that ourselves in our code but we should review it. You should only create client instances with a different configuration only for very specific purposes.

metacosm · 2022-09-15T07:00:00Z

...ator-framework-core/src/test/java/io/javaoperatorsdk/operator/LeaderElectionManagerTest.java

+    var namespacePath = tempDir.resolve("namespace");
+    Files.writeString(namespacePath, namespace);
+
+    System.setProperty(KUBERNETES_AUTH_TRYKUBECONFIG_SYSTEM_PROPERTY, "false");


We probably need to clean things up or, at the very least, document them better. The intention is to create client instance(s) based on the configuration retrieved from the ConfigurationService. I'm not sure that we do follow that ourselves in our code but we should review it. You should only create client instances with a different configuration only for very specific purposes.

honnix · 2022-09-21T20:00:59Z

Could you please help merge this so it doesn't stay too long and get conflict? Thank you.

csviri · 2022-09-22T07:38:44Z

yep, fine by me, @metacosm could you take a look pls?

metacosm · 2022-09-22T17:33:45Z

I already approved it… 😁

metacosm · 2022-09-22T17:35:40Z

The only question that remains is whether we should merge it in main or next…

honnix · 2022-09-22T19:03:00Z

If going to main results a bug fix release right way, it might make sense due to the small breaking change. It feels better to ship it sooner than later.

csviri · 2022-09-22T19:04:12Z

I think this is kinda gray area, it's not a bug fix, rather an improvement IMHO. But fine with merging it to the main, and have a release.

honnix · 2022-09-22T20:21:14Z

I think this is kinda gray area, it's not a bug fix, rather an improvement IMHO. But fine with merging it to the main, and have a release.

Yeah agreed. I don't have a strong opinion. Either way is ok for me.

csviri · 2022-09-23T07:13:53Z

merged to main, thank you @honnix !!

honnix · 2022-09-23T07:21:38Z

Thank you all!

metacosm · 2022-09-23T07:42:21Z

Thank you!

Try to get lease namespace if unspecified

b621d9d

honnix commented Sep 11, 2022

View reviewed changes

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/LeaderElectionManager.java Show resolved Hide resolved

honnix commented Sep 11, 2022

View reviewed changes

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/LeaderElectionManager.java Show resolved Hide resolved

honnix changed the base branch from main to next September 11, 2022 17:44

honnix commented Sep 11, 2022

View reviewed changes

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/LeaderElectionManager.java Show resolved Hide resolved

honnix commented Sep 11, 2022

View reviewed changes

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/LeaderElectionManager.java Outdated Show resolved Hide resolved

More precise method name

1317c4f

csviri requested changes Sep 12, 2022

View reviewed changes

metacosm reviewed Sep 12, 2022

View reviewed changes

csviri approved these changes Sep 12, 2022

View reviewed changes

honnix mentioned this pull request Sep 12, 2022

Replace copied code from fabric8 client #1453

Closed

honnix marked this pull request as draft September 12, 2022 12:00

Strip to remove whitespaces

cdd6fea

honnix marked this pull request as ready for review September 12, 2022 12:06

honnix added 2 commits September 12, 2022 14:20

Get namespace from cached cilent config

0c12947

Remove unused method

014e05a

metacosm reviewed Sep 12, 2022

View reviewed changes

honnix added 2 commits September 12, 2022 15:52

Reset

56a69be

Add e2e test to cover optional values

87315bb

honnix force-pushed the try-get-namespace branch from 536f3ed to 87315bb Compare September 13, 2022 11:29

honnix and others added 3 commits September 13, 2022 13:49

POD_NAME is needed for test case

cdbe8d8

Fix scope

1204727

Update namespace-inferred-operator.yaml

a2abd99

metacosm approved these changes Sep 15, 2022

View reviewed changes

csviri self-requested a review September 15, 2022 07:06

csviri approved these changes Sep 23, 2022

View reviewed changes

csviri changed the base branch from next to main September 23, 2022 07:12

csviri merged commit eff9835 into operator-framework:main Sep 23, 2022

honnix deleted the try-get-namespace branch September 23, 2022 07:21

Try to get lease namespace if unspecified #1450

Try to get lease namespace if unspecified #1450

Conversation

honnix commented Sep 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

metacosm Sep 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

metacosm left a comment

Choose a reason for hiding this comment

csviri left a comment

Choose a reason for hiding this comment

csviri commented Sep 12, 2022

honnix commented Sep 12, 2022

honnix commented Sep 12, 2022

Choose a reason for hiding this comment

honnix Sep 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

honnix commented Sep 21, 2022

csviri commented Sep 22, 2022

metacosm commented Sep 22, 2022

metacosm commented Sep 22, 2022 • edited Loading

honnix commented Sep 22, 2022

csviri commented Sep 22, 2022

honnix commented Sep 22, 2022

csviri commented Sep 23, 2022

honnix commented Sep 23, 2022

metacosm commented Sep 23, 2022

honnix commented Sep 11, 2022 •

edited

Loading

metacosm Sep 12, 2022 •

edited

Loading

honnix Sep 12, 2022 •

edited

Loading

metacosm commented Sep 22, 2022 •

edited

Loading