Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(deploy): Scale down orca & rosco during deploys #593

Merged
merged 1 commit into from
Jul 7, 2017
Merged

feat(deploy): Scale down orca & rosco during deploys #593

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 37 additions & 20 deletions ...src/main/java/com/netflix/spinnaker/halyard/deploy/deployment/v1/DistributedDeployer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,16 +276,45 @@ private <T extends Account> void reapRoscoServerGroups(AccountDeploymentDetails<
List<Integer> allRoscos = new ArrayList<>(executionsByServerGroupVersion.keySet());
allRoscos.sort(Integer::compareTo);

int roscoCount = allRoscos.size();
if (roscoCount <= MAX_REMAINING_SERVER_GROUPS) {
cleanupServerGroups(details, roscoService, roscoSettings, executionsByServerGroupVersion, allRoscos);
}

private <S, T extends Account> void cleanupServerGroups(AccountDeploymentDetails<T> details,
DistributedService<S, T> service,
ServiceSettings settings,
Map<Integer, Integer> workloadByVersion,
List<Integer> runningVersions) {
int runningVersionCount = runningVersions.size();

if (runningVersionCount <= 1) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not the usual case, correct? Maybe add logging here so you avoid possibly silently failing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This happens on every fresh deploy - you'll only have one of each service. I'll add a log statement but it's expected to be hit

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, still maybe worth it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed & logged

log.info("There are no extra services to cleanup. Running count = " + runningVersionCount);
return;
}

allRoscos = allRoscos.subList(0, roscoCount - MAX_REMAINING_SERVER_GROUPS);
for (Integer roscoVersion : allRoscos) {
if (executionsByServerGroupVersion.get(roscoVersion) == 0) {
DaemonTaskHandler.message("Reaping old rosco server group sequence " + roscoVersion);
roscoService.deleteVersion(details, roscoSettings, roscoVersion);
/*
* To visualize the array slicing below, say we have 5 running
* server groups, 3 of which we can destroy, one we can scale down,
* and one that's actively serving traffic, as illustrated here:
*
*
* | running - MAX_REMAINING | MAX_REMAINING |
* | No longer in use | backup | active |
* server groups: | v001 | v002 | v003 | v004 | v005 |
*/
List<Integer> killableVersions = runningVersions.subList(0, runningVersionCount - MAX_REMAINING_SERVER_GROUPS);
List<Integer> shrinkableVersions = runningVersions.subList(runningVersionCount - MAX_REMAINING_SERVER_GROUPS, runningVersionCount - 1);

for (Integer version : killableVersions) {
if (workloadByVersion.get(version) == 0) {
DaemonTaskHandler.message("Reaping old server group sequence " + version);
service.deleteVersion(details, settings, version);
}
}

for (Integer version : shrinkableVersions) {
if (workloadByVersion.get(version) == 0) {
DaemonTaskHandler.message("Shrinking old server group sequence " + version);
service.resizeVersion(details, settings, version, 0);
}
}
}
Expand Down Expand Up @@ -318,18 +347,6 @@ private <T extends Account> void reapOrcaServerGroups(AccountDeploymentDetails<T
List<Integer> allOrcas = new ArrayList<>(executionsByServerGroupVersion.keySet());
allOrcas.sort(Integer::compareTo);

int orcaCount = allOrcas.size();
if (orcaCount <= MAX_REMAINING_SERVER_GROUPS) {
return;
}

allOrcas = allOrcas.subList(0, orcaCount - MAX_REMAINING_SERVER_GROUPS);
for (Integer orcaVersion : allOrcas) {
// TODO(lwander) consult clouddriver to ensure this orca isn't enabled
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO was dropped because the orca versions outside of MAX_REMAINING_SERVER_GROUPS cannot be enabled, since the rollback operation requested by halyard ensures the highest sequence number is always the only one serving traffic.

if (executionsByServerGroupVersion.get(orcaVersion) == 0) {
DaemonTaskHandler.message("Reaping old orca instance " + orcaVersion);
orcaService.deleteVersion(details, orcaSettings, orcaVersion);
}
}
cleanupServerGroups(details, orcaService, orcaSettings, executionsByServerGroupVersion, allOrcas);
}
}
1 change: 1 addition & 0 deletions ...netflix/spinnaker/halyard/deploy/spinnaker/v1/service/distributed/DistributedService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ void ensureRunning(AccountDeploymentDetails<A> details,
<S> S connectToService(AccountDeploymentDetails<A> details, SpinnakerRuntimeSettings runtimeSettings, SpinnakerService<S> sidecar);
String connectCommand(AccountDeploymentDetails<A> details, SpinnakerRuntimeSettings runtimeSettings);
void deleteVersion(AccountDeploymentDetails<A> details, ServiceSettings settings, Integer version);
void resizeVersion(AccountDeploymentDetails<A> details, ServiceSettings settings, int version, int targetSize);
boolean isRequiredToBootstrap();
DeployPriority getDeployPriority();
SpinnakerService<T> getService();
Expand Down
7 changes: 7 additions & 0 deletions ...aker/halyard/deploy/spinnaker/v1/service/distributed/google/GoogleDistributedService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,13 @@ default Provider.ProviderType getProviderType() {
return Provider.ProviderType.GOOGLE;
}

@Override
default void resizeVersion(AccountDeploymentDetails<GoogleAccount> details, ServiceSettings serviceSettings, int version, int targetSize) {
String name = getVersionedName(version);
String zone = serviceSettings.getLocation();
GoogleProviderUtils.resize(details, zone, name, targetSize);
}

default String getStartupScript() {
return String.join("\n", "#!/usr/bin/env bash",
"",
Expand Down
9 changes: 9 additions & 0 deletions ...spinnaker/halyard/deploy/spinnaker/v1/service/distributed/google/GoogleProviderUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,15 @@ static String ensureSpinnakerNetworkExists(AccountDeploymentDetails<GoogleAccoun
return String.format("projects/%s/global/networks/%s", project, networkName);
}

static public void resize(AccountDeploymentDetails<GoogleAccount> details, String zone, String managedInstaceGroupName, int targetSize) {
Compute compute = getCompute(details);
try {
compute.instanceGroupManagers().resize(details.getAccount().getProject(), zone, managedInstaceGroupName, targetSize);
} catch (IOException e) {
throw new HalException(FATAL, "Unable to resize instance group manager " + managedInstaceGroupName + ": " + e.getMessage(), e);
}
}

static Compute getCompute(AccountDeploymentDetails<GoogleAccount> details) {
ConfigProblemSetBuilder problemSetBuilder = new ConfigProblemSetBuilder(null);
GoogleNamedAccountCredentials credentials = details.getAccount().getNamedAccountCredentials("", problemSetBuilder);
Expand Down
6 changes: 6 additions & 0 deletions ...yard/deploy/spinnaker/v1/service/distributed/kubernetes/KubernetesDistributedService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,12 @@ default Map<String, List<String>> getAvailabilityZones(ServiceSettings settings)
return availabilityZones;
}

default void resizeVersion(AccountDeploymentDetails<KubernetesAccount> details, ServiceSettings settings, int version, int targetSize) {
String name = getVersionedName(version);
String namespace = getNamespace(settings);
KubernetesProviderUtils.resize(details, namespace, name, targetSize);
}

@Override
default Map<String, Object> buildRollbackPipeline(AccountDeploymentDetails<KubernetesAccount> details, SpinnakerRuntimeSettings runtimeSettings) {
ServiceSettings settings = runtimeSettings.getServiceSettings(getService());
Expand Down
5 changes: 5 additions & 0 deletions ...r/halyard/deploy/spinnaker/v1/service/distributed/kubernetes/KubernetesProviderUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,11 @@ static KubernetesClient getClient(AccountDeploymentDetails<KubernetesAccount> de
return new DefaultKubernetesClient(config);
}

static void resize(AccountDeploymentDetails<KubernetesAccount> details, String namespace, String replicaSetName, int targetSize) {
KubernetesClient client = getClient(details);
client.extensions().replicaSets().inNamespace(namespace).withName(replicaSetName).scale(targetSize);
}

static void upsertSecret(AccountDeploymentDetails<KubernetesAccount> details, Set<Pair<File, String>> files, String secretName, String namespace) {
KubernetesClient client = getClient(details);

Expand Down