feat(deployments): allows for custom sizing for kubernetes distrbuted deployments in halconfig #724

omawhite · 2017-10-09T17:06:05Z

see spinnaker issue 683

markus-silpala-tgt · 2017-10-09T19:20:40Z

...halyard/deploy/spinnaker/v1/service/distributed/kubernetes/KubernetesDistributedService.java

-import com.netflix.spinnaker.clouddriver.kubernetes.deploy.description.servergroup.KubernetesVolumeMount;
-import com.netflix.spinnaker.clouddriver.kubernetes.deploy.description.servergroup.KubernetesVolumeSource;
-import com.netflix.spinnaker.clouddriver.kubernetes.deploy.description.servergroup.KubernetesVolumeSourceType;
+import com.netflix.spinnaker.clouddriver.kubernetes.deploy.description.servergroup.*;


@omawhite I think IntelliJ did this. We probably want to change it back to list all the packages explicitly.

Yeah there's some setting to keep those expanded by default

markus-silpala-tgt · 2017-10-09T19:28:49Z

...halyard/deploy/spinnaker/v1/service/distributed/kubernetes/KubernetesDistributedService.java

+  // This is super-goofy. What can we do differently for this? Anything in the config parsing logic?
+  default String stringOrNull(Object value) {
+    return value != null ? String.valueOf(value) : null;
+  }


@lwander We'd be interested in your thoughts on this little nugget. The issue arises when the size specification is in form with only numeric characters: "1" as opposed to "1000m" or "1Gi".

We used String.valueOf to ensure that those cases were parsed out of the config as strings. That, in turn, caused problems when a value was unset (null), because String.valueOf(null) evaluates to "null" as a string, which is not what we want to be in place when something isn't set.

Our tactical solution was this stringOrNull method, which just feels weird. Is there something more framework-y in the wider halyard code base that we might leverage for this? Or a natural util-ish place we could stick this little method instead of implementing it twice in the deployment code? We also considered putting it in the CustomSizing class and calling that from both places where it's needed.

Yeah agreed this is a little funky - but this is also the first time where the type isn't explicit in the config. Would it be possible to make it clear that component sizing is mapping string -> string -> string? Or is the fear that this is too restrictive?

We were concerned about being too restrictive, but at this point it probably makes sense to explicitly make it all maps with strings.

lwander

One thing that is missing is restricting the java heap size. @greenkiwi pointed me at this great flag to size your heap to that of the docker container's allocated memory:

ENV JAVA_DOCKER_MEMORY_OPTS '-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm'

lwander · 2017-10-10T13:44:06Z

...halyard/deploy/spinnaker/v1/service/distributed/kubernetes/KubernetesDistributedService.java

-import com.netflix.spinnaker.clouddriver.kubernetes.deploy.description.servergroup.KubernetesVolumeMount;
-import com.netflix.spinnaker.clouddriver.kubernetes.deploy.description.servergroup.KubernetesVolumeSource;
-import com.netflix.spinnaker.clouddriver.kubernetes.deploy.description.servergroup.KubernetesVolumeSourceType;
+import com.netflix.spinnaker.clouddriver.kubernetes.deploy.description.servergroup.*;


Yeah there's some setting to keep those expanded by default

lwander · 2017-10-10T13:45:17Z

...rd-config/src/main/java/com/netflix/spinnaker/halyard/config/model/v1/node/CustomSizing.java

+ */
+@Data
+@EqualsAndHashCode(callSuper = false)
+public class CustomSizing implements Map<String, Map> {


Is the intent here just to provide a type alias?

Heh. Not really.

Initially this class had some structure to it, because it assumed the kubernetes "requests" and "limits" nomenclature. We also extended Node initially, and then removed that after we realized we had nothing to validate because there's no CLI interaction with this data (and we couldn't think of any reasonable validation that wasn't k8s-specific). By the time we removed those things, this had nothing left but the Map implementation; you were the first to realize this. :-)

But now, we've given this class a new reason to exist: it holds our goofy little stringOrNull method, so that it doesn't have to be duplicated in the manual and orca deployment flows.

(We abandoned the idea of explicitly making this a three-layer map with strings at its leaf nodes. No implementation felt clean, and we also realized that when we add replicaSetSize to it later this week, that will be an integer, at the top tier of the map. So we'd be breaking that constraint ourselves.)

Hmm it might still be valuable to validate even if there is no CLI command to generate entries here - but maybe not.

Sounds good

We'd be wide open to it if you can guide us on what to validate in a way that doesn't assume a specific cloud provider, or on a way to detect that we're deploying to k8s and only then do our k8s-specific validations. I would definitely feel better if we validated some basics in the k8s case.

On a later PR, of course. :-)

lwander · 2017-10-10T13:45:30Z

...halyard/deploy/spinnaker/v1/service/distributed/kubernetes/KubernetesDistributedService.java

@@ -373,14 +359,7 @@ default KubernetesContainerDescription buildContainer(String name, ServiceSettin
    readinessProbe.setHandler(handler);
    container.setReadinessProbe(readinessProbe);

-    /* TODO(lwander) this needs work


You guys rock

:-)

To be completely honest, we only moved your TODO into the applyCustomSize method. Because we didn't implement handling of the SMALL/MEDIUM/LARGE specifiers, we kept your placeholder in its new home, below.

Heh I saw that after

lwander · 2017-10-10T13:48:45Z

...halyard/deploy/spinnaker/v1/service/distributed/kubernetes/KubernetesDistributedService.java

+  // This is super-goofy. What can we do differently for this? Anything in the config parsing logic?
+  default String stringOrNull(Object value) {
+    return value != null ? String.valueOf(value) : null;
+  }


Yeah agreed this is a little funky - but this is also the first time where the type isn't explicit in the config. Would it be possible to make it clear that component sizing is mapping string -> string -> string? Or is the fear that this is too restrictive?

edwinavalos · 2017-10-10T14:18:45Z

@lwander should we be append the JAVA_OPTS in this code, or should we be instructing people that they should introduce that in to their service-settings files for individual services?

lwander · 2017-10-10T14:31:48Z

I think it'd be nice to append this in this code for the k8s deployments. They can always be overridden using custom service settings.

omawhite · 2017-10-10T19:13:27Z

Hey @lwander in reference to this environment variable, we noticed that the change wasn't added until java version 8u131, but it appears that the spinnaker components are using 8u111.

ENV JAVA_DOCKER_MEMORY_OPTS '-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm'

omawhite · 2017-10-10T21:05:55Z

@lwander so we found that java version 8u131 isn't available until alpine version 3.6, the containers currently run 3.4. So until that changes people will just have to manually set the JAVA_OPTS in addition to setting custom sizing.

… deployments in halconfig

omawhite · 2017-10-11T16:37:06Z

squashed commits

lwander · 2017-10-11T16:42:47Z

Good catch @omawhite on the java version.

lwander · 2017-10-11T16:44:59Z

I'm ready to merge when you guys feel ready (or you can go ahead @edwinavalos)

markus-silpala-tgt reviewed Oct 9, 2017

View reviewed changes

lwander reviewed Oct 10, 2017

View reviewed changes

feat(deployments): allows for custom sizing for kubernetes distrbuted…

364d482

… deployments in halconfig

omawhite force-pushed the custom-sizing branch from be91f68 to 364d482 Compare October 11, 2017 16:36

lwander approved these changes Oct 11, 2017

View reviewed changes

edwinavalos merged commit 5795c82 into spinnaker:master Oct 11, 2017

edwinavalos mentioned this pull request Oct 17, 2017

Custom sizing spinnaker/spinnaker.github.io#374

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(deployments): allows for custom sizing for kubernetes distrbuted deployments in halconfig #724

feat(deployments): allows for custom sizing for kubernetes distrbuted deployments in halconfig #724

omawhite commented Oct 9, 2017

markus-silpala-tgt Oct 9, 2017

lwander Oct 10, 2017

markus-silpala-tgt Oct 9, 2017

lwander Oct 10, 2017

markus-silpala-tgt Oct 10, 2017

lwander left a comment

lwander Oct 10, 2017

lwander Oct 10, 2017

markus-silpala-tgt Oct 10, 2017

lwander Oct 11, 2017

markus-silpala-tgt Oct 11, 2017

lwander Oct 10, 2017

markus-silpala-tgt Oct 10, 2017

lwander Oct 11, 2017

lwander Oct 10, 2017

edwinavalos commented Oct 10, 2017

lwander commented Oct 10, 2017

omawhite commented Oct 10, 2017

omawhite commented Oct 10, 2017 •

edited

Loading

omawhite commented Oct 11, 2017

lwander commented Oct 11, 2017

lwander commented Oct 11, 2017

feat(deployments): allows for custom sizing for kubernetes distrbuted deployments in halconfig #724

feat(deployments): allows for custom sizing for kubernetes distrbuted deployments in halconfig #724

Conversation

omawhite commented Oct 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lwander left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edwinavalos commented Oct 10, 2017

lwander commented Oct 10, 2017

omawhite commented Oct 10, 2017

omawhite commented Oct 10, 2017 • edited Loading

omawhite commented Oct 11, 2017

lwander commented Oct 11, 2017

lwander commented Oct 11, 2017

omawhite commented Oct 10, 2017 •

edited

Loading