Skip to content

Commit

Permalink
feat(localfiles): Support relative local files to hal config home (#13)
Browse files Browse the repository at this point in the history
* fix(config): parse 'oauthScopes' stanzas that were incorrectly written (spinnaker#1376)

* refactor(config): remove a pointless try/catch block

* refactor(tests): remove an unused import

* fix(config): parse 'oauthScopes' stanzas that were incorrectly written

The old version of Lombok used in Halyard <=1.21 didn't copy
@JsonProperty annotations to the generated methods. This means Jackson
was seeing the field as 'oAuthScopes' and the getter/setter as a
separate 'oauthScopes' property. The same data would be written to both
properties, and during parsing whichever came last in the file would be
persisted into the object.

With new versions of Lombok (>=1.18.8), the @JsonProperty annotation is
copied to the bean methods, so the 'oauthScopes' property disappears and
Jackson can no longer parse those old files.

This commit adds some methods to support parsing the files generated by
previous versions of Halyard, but will no longer write out the
incorrect duplicate data.

* chore(copyright): add a missing copyright header

* chore(dependencies): Autobump korkVersion (spinnaker#1375)

* fix(core): Fix reading of external files as binary instead of text (spinnaker#1380)

* feat(secret): decrypt secrets before sending to deck (spinnaker#1379)

* feat(secret): decrypt secrets before sending to deck

* code review changes

* chore(dependencies): Autobump korkVersion (spinnaker#1383)

* fix(build): remove old usage of korkVersion (spinnaker#1385)

* chore(dependencies): Autobump korkVersion (spinnaker#1384)

* chore(build): Update cloudbuild.yaml file (spinnaker#1388)

Remove the $REPO_NAME variable from the cloudbuild.yaml file; this is being used
to decide the name of the image to push which will not always correspond to the
image name. In particular, if we start publishing both alpine and ubuntu images
we'll want the image name to have a suffix reflecting that.

* chore(dependencies): Autobump korkVersion (spinnaker#1389)

* chore(dependencies): Autobump korkVersion (spinnaker#1390)

* feat(plugins): adding halyard commands for plugins (spinnaker#1386)

* feat(plugins): adding halyard commands for plugins

* chore(refactor): use toMap instead of a concurrentMap collector

* feat(notifications): Add Github Status support (spinnaker#1374)

* chore(dependencies): Autobump korkVersion (spinnaker#1392)

* feat(plugins): enable/disable downloading plugins (spinnaker#1393)

* chore(dependencies): Autobump korkVersion (spinnaker#1394)

* chore(dependencies): Autobump korkVersion (spinnaker#1395)

* feat(kubernetes): Support for tolerations (spinnaker#1396)

* fix(kubernetes): Added support for tolerationn

Signed-off-by: rverma-nikiai <[email protected]>

* feat(kubernetes): Added support for Tolerations

Added support for tolerations

Signed-off-by: rverma-nikiai <[email protected]>

* feat(kubernetes): Support for tolerations

* fix(kubernetes): Added support for Tolerations, fixed access

* chore(dependencies): Autobump korkVersion (spinnaker#1397)

* feat(artifacts): add feature flag to turn on artifactsRewrite (spinnaker#1398)

* feat(deployments): Update component sizing command to allow container component sizing. (spinnaker#1387)

* feat(deploy/kubernetes): Option to select the image variant to deploy (spinnaker#1401)

* refactor(deploy/kubernetes): Removed duplicated code

* feat(deploy/kubernetes): Option to select the image variant to deploy

Supports `slim` and `ubuntu`, the former being the default.
Ubuntu image supported starting with v1.16.0.

* chore(dependencies): Autobump korkVersion (spinnaker#1403)

* fix(docs): s/Kubenretes/Kubernetes (spinnaker#1402)

* fix(ldap): Fix allowing back blank ldap search base when using search filter (spinnaker#1391)

* chore(dependencies): Autobump korkVersion (spinnaker#1404)

* chore(plugins): rename plugin downloading commands to kebab-case (spinnaker#1408)

* chore(dependencies): Autobump korkVersion (spinnaker#1410)

* refactor(deploy): Adapt to changes in kork-config (spinnaker#1407)

* fix(notifications): fix editing notifications in halyard (spinnaker#1413)

Broken in spinnaker#1374

Closes spinnaker/spinnaker#4834

* chore(dependencies): Autobump korkVersion (spinnaker#1415)

* feat(localfiles): Support relative local files to hal config home

Usually when referencing local files in main hal config with a relative
path, halyard throws this error:

Problems in Global:
! ERROR Failed to backup user file: default/files/kubeconfig-main

- Failed to generate config.

With this change, relative files will be automatically resolved relative
from hal config home.

* feat(localfiles): Only prefix files in subdirectories of input prefix

* feat(halyard/localfiles): Use getter for hal config directory

* feat(halyard/localfiles): Error on relative files escaping hal home

* feat(front50): Support AWS S3 SSE (spinnaker#1399)

* feat(operator): Save referenced files to hal config root

* feat(operator): Exception if relative file paths resolve outside root
  • Loading branch information
german-muzquiz authored and ncknt committed Sep 11, 2019
1 parent f708965 commit ac0da7e
Show file tree
Hide file tree
Showing 102 changed files with 2,894 additions and 329 deletions.
1 change: 0 additions & 1 deletion build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@

buildscript {
ext {
korkVersion = "5.3.7"
clouddriverVersion = "5.8.0"
front50Version = "2.1.0"
fiatVersion = "1.0.5"
Expand Down
6 changes: 3 additions & 3 deletions cloudbuild.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ steps:
entrypoint: "bash"
args: [ "-c", "./gradlew halyard-web:installDist -x test"]
- name: 'gcr.io/cloud-builders/docker'
args: ["build", "-t", "gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA", "-t", "gcr.io/$PROJECT_ID/$REPO_NAME:latest", "-f", "Dockerfile.slim", "."]
args: ["build", "-t", "gcr.io/$PROJECT_ID/halyard:$COMMIT_SHA", "-t", "gcr.io/$PROJECT_ID/halyard:latest", "-f", "Dockerfile.slim", "."]
images:
- 'gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA'
- 'gcr.io/$PROJECT_ID/$REPO_NAME:latest'
- 'gcr.io/$PROJECT_ID/halyard:$COMMIT_SHA'
- 'gcr.io/$PROJECT_ID/halyard:latest'
timeout: 25m
340 changes: 339 additions & 1 deletion docs/commands.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions gradle.properties
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#Tue Jul 16 18:16:36 UTC 2019
#Thu Sep 05 14:11:55 UTC 2019
enablePublishing=false
korkVersion=5.8.7
korkVersion=6.6.0
org.gradle.parallel=true
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ public HalCommand() {
registerSubcommand(new TaskCommand());
registerSubcommand(new VersionCommand());
registerSubcommand(new SpinCommand());
registerSubcommand(new PluginCommand());
}

static String getVersion() {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
/*
* Copyright 2019 Armory, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License")
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.netflix.spinnaker.halyard.cli.command.v1;

import com.beust.jcommander.Parameters;
import com.netflix.spinnaker.halyard.cli.command.v1.config.AbstractConfigCommand;
import com.netflix.spinnaker.halyard.cli.command.v1.plugins.*;
import lombok.AccessLevel;
import lombok.Getter;

@Parameters(separators = "=")
public class PluginCommand extends AbstractConfigCommand {
@Getter(AccessLevel.PUBLIC)
private String commandName = "plugins";

@Getter(AccessLevel.PUBLIC)
private String shortDescription = "Show Spinnaker's configured plugins.";

public PluginCommand() {
registerSubcommand(new AddPluginCommand());
registerSubcommand(new EditPluginCommand());
registerSubcommand(new DeletePluginCommand());
registerSubcommand(new ListPluginsCommand());
registerSubcommand(new PluginEnableDisableCommandBuilder().setEnable(true).build());
registerSubcommand(new PluginEnableDisableCommandBuilder().setEnable(false).build());
registerSubcommand(new PluginDownloadingEnableDisableCommandBuilder().setEnable(true).build());
registerSubcommand(new PluginDownloadingEnableDisableCommandBuilder().setEnable(false).build());
}

@Override
protected void executeThis() {
showHelp();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,13 @@
import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.netflix.spinnaker.halyard.cli.command.v1.converter.DeploymentTypeConverter;
import com.netflix.spinnaker.halyard.cli.command.v1.converter.ImageVariantConverter;
import com.netflix.spinnaker.halyard.cli.services.v1.Daemon;
import com.netflix.spinnaker.halyard.cli.services.v1.OperationHandler;
import com.netflix.spinnaker.halyard.cli.ui.v1.AnsiUi;
import com.netflix.spinnaker.halyard.config.model.v1.node.DeploymentEnvironment;
import com.netflix.spinnaker.halyard.config.model.v1.node.DeploymentEnvironment.DeploymentType;
import com.netflix.spinnaker.halyard.config.model.v1.node.DeploymentEnvironment.ImageVariant;
import lombok.AccessLevel;
import lombok.Getter;

Expand Down Expand Up @@ -104,6 +106,17 @@ public class EditDeploymentEnvironmentCommand extends AbstractConfigCommand {
+ "Kubernetes, use this flag to specify the namespace to deploy to (defaults to 'spinnaker')")
private String location;

@Parameter(
names = "--image-variant",
description =
"The container image variant type to use when deploying a distributed installation of Spinnaker.\n"
+ "Warning: variants other than the 'slim' one are only available with Spinnaker v1.16+\n"
+ "\tslim: Based on an Alpine image\n"
+ "\tubuntu: Based on Canonical's ubuntu:bionic image.\n"
+ "Default value: slim",
converter = ImageVariantConverter.class)
private ImageVariant imageVariant;

@Parameter(
names = "--git-upstream-user",
description =
Expand Down Expand Up @@ -194,6 +207,9 @@ protected void executeThis() {
deploymentEnvironment.setLocation(
isSet(location) ? location : deploymentEnvironment.getLocation());

deploymentEnvironment.setImageVariant(
isSet(imageVariant) ? imageVariant : deploymentEnvironment.getImageVariant());

if (originalHash == deploymentEnvironment.hashCode()) {
AnsiUi.failure("No changes supplied.");
return;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,13 @@ public class EditFeaturesCommand extends AbstractConfigCommand {
arity = 1)
private Boolean artifacts = null;

@Parameter(
names = "--artifacts-rewrite",
description =
"Enable new artifact support. Read more at https://www.spinnaker.io/reference/artifacts-with-artifactsrewrite/",
arity = 1)
private Boolean artifactsRewrite = null;

@Parameter(
names = "--mine-canary",
description =
Expand Down Expand Up @@ -119,6 +126,8 @@ protected void executeThis() {
features.setPipelineTemplates(
pipelineTemplates != null ? pipelineTemplates : features.getPipelineTemplates());
features.setArtifacts(artifacts != null ? artifacts : features.getArtifacts());
features.setArtifactsRewrite(
artifactsRewrite != null ? artifactsRewrite : features.getArtifactsRewrite());
features.setMineCanary(mineCanary != null ? mineCanary : features.getMineCanary());
features.setInfrastructureStages(
infrastructureStages != null ? infrastructureStages : features.getInfrastructureStages());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
package com.netflix.spinnaker.halyard.cli.command.v1.config;

import com.beust.jcommander.Parameters;
import com.netflix.spinnaker.halyard.cli.command.v1.config.notifications.github.GithubStatusCommand;
import com.netflix.spinnaker.halyard.cli.command.v1.config.notifications.pubsub.PubsubCommand;
import com.netflix.spinnaker.halyard.cli.command.v1.config.notifications.slack.SlackCommand;
import com.netflix.spinnaker.halyard.cli.command.v1.config.notifications.twilio.TwilioCommand;
Expand All @@ -38,6 +39,7 @@ public class NotificationCommand extends AbstractConfigCommand {
private String shortDescription = "Display the state of Spinnaker's notification settings.";

public NotificationCommand() {
registerSubcommand(new GithubStatusCommand());
registerSubcommand(new PubsubCommand());
registerSubcommand(new SlackCommand());
registerSubcommand(new TwilioCommand());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ protected CustomSizing update(CustomSizing customSizing) {
}

private CustomSizing delete(CustomSizing customSizing) {
customSizing.put(spinnakerService.getCanonicalName(), null);
customSizing.put(spinnakerService.getServiceName(), null);
return customSizing;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,30 @@ public class ComponentSizingEditCommand extends AbstractComponentSizingUpdateCom
+ "sidecar containers (e.g. the monitoring daemon). Example: 1Gi.")
private String limitsMemory;

@Parameter(
names = "--container-requests-cpu",
description =
"Sets the cpu request for the container running the spinnaker service. Example: 250m.")
private String containerRequestsCpu;

@Parameter(
names = "--container-requests-memory",
description =
"Sets the memory request for the container running the spinnaker service. Example: 512Mi.")
private String containerRequestsMemory;

@Parameter(
names = "--container-limits-cpu",
description =
"Sets the cpu limit for the container running the spinnaker service. Example: 1.")
private String containerLimitsCpu;

@Parameter(
names = "--container-limits-memory",
description =
"Sets the memory limit for the container running the spinnaker service. Example: 1Gi.")
private String containerLimitsMemory;

public ComponentSizingEditCommand(SpinnakerService.Type spinnakerService) {
super(spinnakerService, "edit");
}
Expand All @@ -80,11 +104,13 @@ protected CustomSizing update(CustomSizing customSizing) {
private CustomSizing edit(CustomSizing customSizing) {
Map serviceSizing =
customSizing.computeIfAbsent(spinnakerService.getServiceName(), (k) -> new HashMap<>());
edit(serviceSizing);
Map containerSizing =
customSizing.computeIfAbsent(spinnakerService.getCanonicalName(), (k) -> new HashMap<>());
edit(serviceSizing, containerSizing);
return customSizing;
}

private void edit(Map serviceSizing) {
private void edit(Map serviceSizing, Map containerSizing) {
putIfNotNull(serviceSizing, "replicas", replicas);

Map limits = (Map) serviceSizing.computeIfAbsent("limits", (k) -> new HashMap<>());
Expand All @@ -94,6 +120,15 @@ private void edit(Map serviceSizing) {
Map requests = (Map) serviceSizing.computeIfAbsent("requests", (k) -> new HashMap<>());
putIfNotNull(requests, "cpu", requestsCpu);
putIfNotNull(requests, "memory", requestsMemory);

Map containerLimits = (Map) containerSizing.computeIfAbsent("limits", (k) -> new HashMap<>());
putIfNotNull(containerLimits, "cpu", containerLimitsCpu);
putIfNotNull(containerLimits, "memory", containerLimitsMemory);

Map containerRequests =
(Map) containerSizing.computeIfAbsent("requests", (k) -> new HashMap<>());
putIfNotNull(containerRequests, "cpu", containerRequestsCpu);
putIfNotNull(containerRequests, "memory", containerRequestsMemory);
}

private void putIfNotNull(Map map, String key, Object value) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import com.netflix.spinnaker.halyard.cli.services.v1.Daemon;
import com.netflix.spinnaker.halyard.cli.services.v1.OperationHandler;
import com.netflix.spinnaker.halyard.deploy.spinnaker.v1.service.SpinnakerService;
import java.util.HashMap;
import java.util.Map;
import lombok.AccessLevel;
import lombok.Getter;
Expand Down Expand Up @@ -62,11 +63,34 @@ protected void executeThis() {
.setFormat(STRING)
.setUserFormatted(true)
.setOperation(
() ->
Daemon.getDeploymentEnvironment(currentDeployment, !noValidate)
.get()
.getCustomSizing()
.get(spinnakerService.getServiceName()))
() -> {
Map serviceSizing =
Daemon.getDeploymentEnvironment(currentDeployment, !noValidate)
.get()
.getCustomSizing()
.get(spinnakerService.getServiceName());

Map containerSizing =
Daemon.getDeploymentEnvironment(currentDeployment, !noValidate)
.get()
.getCustomSizing()
.get(spinnakerService.getCanonicalName());

if (serviceSizing == null && containerSizing == null) {
return null;
}

Map result = new HashMap();
if (serviceSizing != null) {
result.put(spinnakerService.getServiceName(), serviceSizing);
}

if (containerSizing != null) {
result.put(spinnakerService.getCanonicalName(), containerSizing);
}

return result;
})
.get();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
/*
* Copyright 2019 Netflix, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License")
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.netflix.spinnaker.halyard.cli.command.v1.config.notifications.github;

import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.netflix.spinnaker.halyard.cli.command.v1.config.notifications.AbstractEditNotificationCommand;
import com.netflix.spinnaker.halyard.config.model.v1.node.Notification;
import com.netflix.spinnaker.halyard.config.model.v1.notifications.GithubStatusNotification;

/** Interact with the github notification */
@Parameters(separators = "=")
public class EditGithubStatusCommand
extends AbstractEditNotificationCommand<GithubStatusNotification> {
protected String getNotificationName() {
return "github-status";
}

@Parameter(names = "--token", password = true, description = "Your github account token.")
private String token;

@Override
protected Notification editNotification(GithubStatusNotification notification) {
notification.setToken(isSet(token) ? token : notification.getToken());
return notification;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/*
* Copyright 2019 Netflix, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License")
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.netflix.spinnaker.halyard.cli.command.v1.config.notifications.github;

import com.beust.jcommander.Parameters;
import com.netflix.spinnaker.halyard.cli.command.v1.config.notifications.AbstractNamedNotificationCommand;

/** Interact with the github notification */
@Parameters(separators = "=")
public class GithubStatusCommand extends AbstractNamedNotificationCommand {
protected String getNotificationName() {
return "github-status";
}

public GithubStatusCommand() {
super();
registerSubcommand(new EditGithubStatusCommand());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@

package com.netflix.spinnaker.halyard.cli.command.v1.config.persistentStorage.s3;

import static com.netflix.spinnaker.halyard.config.model.v1.persistentStorage.S3PersistentStore.ServerSideEncryption;

import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.netflix.spinnaker.halyard.cli.command.v1.config.persistentStorage.AbstractPersistentStoreEditCommand;
Expand Down Expand Up @@ -67,6 +69,14 @@ protected String getPersistentStoreType() {
+ " See https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#VirtualHostingExamples.")
private Boolean pathStyleAccess = false;

@Parameter(
names = "--server-side-encryption",
description =
"Use Amazon Server-Side Encryption ('x-amz-server-side-encryption' header). "
+ "Supports 'AES256' (for Amazon S3-managed encryption keys, equivalent to a header value of 'AES256')"
+ " and 'AWSKMS' (for AWS KMS-managed encryption keys, equivalent to a header value of 'aws:kms'.")
private ServerSideEncryption serverSideEncryption;

@Parameter(names = "--assume-role", description = AwsCommandProperties.ASSUME_ROLE_DESCRIPTION)
private String assumeRole;

Expand Down Expand Up @@ -96,6 +106,10 @@ protected S3PersistentStore editPersistentStore(S3PersistentStore persistentStor
isSet(accessKeyId) ? accessKeyId : persistentStore.getAccessKeyId());
persistentStore.setSecretAccessKey(
isSet(secretAccessKey) ? secretAccessKey : persistentStore.getSecretAccessKey());
persistentStore.setServerSideEncryption(
isSet(serverSideEncryption)
? serverSideEncryption
: persistentStore.getServerSideEncryption());

if (persistentStore.getBucket() == null) {
String bucketName = "spin-" + UUID.randomUUID().toString();
Expand Down
Loading

0 comments on commit ac0da7e

Please sign in to comment.