Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
dcb871b
AMBARI-22878: Update Service Group API to take list of mpack name ass…
scottduan Jan 30, 2018
e97eb29
AMBARI-22878: Update Service Group API to take list of mpack name ass…
scottduan Jan 30, 2018
da41ca0
Merge branch 'AMBARI-22878-branch-feature-AMBARI-14714' of https://gi…
scottduan Feb 1, 2018
158cd85
Merge branch 'AMBARI-22878-branch-feature-AMBARI-14714' of https://gi…
scottduan Feb 1, 2018
decbf54
Merge branch 'AMBARI-22878-branch-feature-AMBARI-14714' of https://gi…
scottduan Feb 5, 2018
68c1f04
Merge remote-tracking branch 'upstream/branch-feature-AMBARI-14714' i…
scottduan Feb 9, 2018
a2340a6
Merge remote-tracking branch 'upstream/branch-feature-AMBARI-14714' i…
scottduan Feb 9, 2018
04345f8
Merge branch 'AMBARI-22878-branch-feature-AMBARI-14714' of https://gi…
scottduan Feb 9, 2018
90f354f
Merge remote-tracking branch 'upstream/branch-feature-AMBARI-14714' i…
scottduan Feb 9, 2018
a331735
Merge remote-tracking branch 'upstream/branch-feature-AMBARI-14714' i…
scottduan Feb 9, 2018
3de0e93
Merge branch 'AMBARI-22878-branch-feature-AMBARI-14714' of https://gi…
Feb 16, 2018
f74af11
Merge branch 'AMBARI-22878-branch-feature-AMBARI-14714' of https://gi…
Feb 16, 2018
afc6a3b
Merge branch 'AMBARI-22878-branch-feature-AMBARI-14714' of https://gi…
Feb 18, 2018
fb3e005
Merge branch 'AMBARI-22878-branch-feature-AMBARI-14714' of https://gi…
Feb 18, 2018
7bcd6fe
Merge branch 'AMBARI-22878-branch-feature-AMBARI-14714' of https://gi…
Feb 22, 2018
cfe5295
Merge branch 'AMBARI-22878-branch-feature-AMBARI-14714' of https://gi…
Feb 22, 2018
d370602
Merge branch 'AMBARI-22878-branch-feature-AMBARI-14714' of https://gi…
Feb 22, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,20 @@
*/
package org.apache.ambari.server.controller;

import java.util.HashSet;
import java.util.Objects;
import java.util.Set;

public class ServiceGroupRequest {

private String clusterName; // REF
private String serviceGroupName; // GET/CREATE/UPDATE/DELETE
private Set<String> mpackNames; // Associated mpack names

public ServiceGroupRequest(String clusterName, String serviceGroupName) {
this.clusterName = clusterName;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's still a misunderstanding here - I thought it would be cleaner to separate this out into 2 fields so that it's clear to the user what is expected: mpack name and then mpack version. You can combine this and call it a stack ID internally, that's fine - but we should take 2 different properties.

Same goes for API requests where we're currently given name-version as a string. This should be broken out into 2 fields.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am ok for this change.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jayush What do you think?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets keep it as "version" for now for legacy purposes. When we upgrade to ambari 3.0 assuming the existing cluster is on HDP-2.6 stack, calling this as mpack name and mpack version would be misleading.

We should handle this in a separate JIRA if we really want to change this convention for all APIs.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But "version" in legacy terms meant only the numeric value. If we don't want to break it out into 2 references, I think we'd rather use stackId, right?

this.serviceGroupName = serviceGroupName;
mpackNames = new HashSet<>();
}

/**
Expand Down Expand Up @@ -57,9 +61,30 @@ public void setServiceGroupName(String serviceGroupName) {
this.serviceGroupName = serviceGroupName;
}

/**
* @return a list of associated mpack names
*/
public Set<String> getMpackNames() {
return mpackNames;
}

/**
* @param mpackNames a list of associated mpack names
*/
public void setMpackNames(Set<String> mpackNames) {
this.mpackNames.addAll(mpackNames);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method name says set, but it adds to existing ones.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. I copied and wrote,but forgot to update the name.

}

@Override
public String toString() {
return String.format("clusterName=%s, serviceGroupName=%s", clusterName, serviceGroupName);
StringBuilder sb = new StringBuilder();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've been gradually moving to ToStringBuilder here instead

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we should switch this to ToStringBuilder.

sb.append("clusterName="+clusterName+",serviceGroupName="+serviceGroupName);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please avoid + inside append().

if (!mpackNames.isEmpty()) {
sb.append(",mpackNames=");
mpackNames.stream().map(mpackName -> sb.append(mpackName + ","));
sb.deleteCharAt(sb.length()-1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think simply using built-in Set.toString() would be OK. If you prefer explicit formatting, instead of map(append) please use Collectors.joining, which is side-effect free, and also makes deleteCharAt unnecessary.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

}
return sb.toString();
}

@Override
Expand All @@ -73,6 +98,8 @@ public boolean equals(Object obj) {

ServiceGroupRequest other = (ServiceGroupRequest) obj;

// ignore mpackNames, even if they are different, we still consider sgrequests are the same

return Objects.equals(clusterName, other.clusterName) &&
Objects.equals(serviceGroupName, other.serviceGroupName);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@

package org.apache.ambari.server.controller;

import java.util.HashSet;
import java.util.Set;

import io.swagger.annotations.ApiModelProperty;

public class ServiceGroupResponse {
Expand All @@ -26,12 +29,15 @@ public class ServiceGroupResponse {
private Long serviceGroupId;
private String clusterName;
private String serviceGroupName;
private Set<String> mpackNames;

public ServiceGroupResponse(Long clusterId, String clusterName, Long serviceGroupId, String serviceGroupName) {
this.clusterId = clusterId;
this.serviceGroupId = serviceGroupId;
this.clusterName = clusterName;
this.serviceGroupName = serviceGroupName;
this.mpackNames = new HashSet<>();

}

/**
Expand Down Expand Up @@ -90,6 +96,20 @@ public void setServiceGroupName(String serviceGroupName) {
this.serviceGroupName = serviceGroupName;
}

/**
* @return the list of mpack names
*/
public Set<String> getServiceGroupMpackNames() {
return mpackNames;
}

/**
* @param mpackNames the list of mpack names
*/
public void setServiceGroupMpackNames(Set<String> mpackNames) {
this.mpackNames = mpackNames;
}

@Override
public boolean equals(Object o) {
if (this == o) return true;
Expand All @@ -109,6 +129,7 @@ public boolean equals(Object o) {
!serviceGroupName.equals(that.serviceGroupName) : that.serviceGroupName != null) {
return false;
}
// ignore mpackNames

return true;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;

import org.apache.ambari.server.AmbariException;
import org.apache.ambari.server.ClusterNotFoundException;
Expand Down Expand Up @@ -78,6 +79,7 @@ public class ServiceGroupResourceProvider extends AbstractControllerResourceProv
public static final String SERVICE_GROUP_CLUSTER_NAME_PROPERTY_ID = RESPONSE_KEY + PropertyHelper.EXTERNAL_PATH_SEP + "cluster_name";
public static final String SERVICE_GROUP_SERVICE_GROUP_ID_PROPERTY_ID = RESPONSE_KEY + PropertyHelper.EXTERNAL_PATH_SEP + "service_group_id";
public static final String SERVICE_GROUP_SERVICE_GROUP_NAME_PROPERTY_ID = RESPONSE_KEY + PropertyHelper.EXTERNAL_PATH_SEP + "service_group_name";
public static final String SERVICE_GROUP_SERVICE_GROUP_MPACKNAME_PROPERTY_ID = RESPONSE_KEY + PropertyHelper.EXTERNAL_PATH_SEP + "mpacks";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we simply call it SERVICE_GROUP_MPACKNAME_PROPERTY_ID?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I will change it to SERVICE_GROUP_MPACKNAME_PROPERTY_ID



private static Set<String> pkPropertyIds =
Expand Down Expand Up @@ -269,6 +271,11 @@ private ServiceGroupRequest getRequest(Map<String, Object> properties) {
String clusterName = (String) properties.get(SERVICE_GROUP_CLUSTER_NAME_PROPERTY_ID);
String serviceGroupName = (String) properties.get(SERVICE_GROUP_SERVICE_GROUP_NAME_PROPERTY_ID);
ServiceGroupRequest svcRequest = new ServiceGroupRequest(clusterName, serviceGroupName);
Set<Object> mpackNames = (Set<Object>) properties.get(SERVICE_GROUP_SERVICE_GROUP_MPACKNAME_PROPERTY_ID);
if (mpackNames != null) {
Set<String> mpackNamesSet = mpackNames.stream().flatMap(mpack -> ((Map<String, String>)mpack).values().stream()).collect(Collectors.toSet());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you minimize the number of unchecked casts by avoiding the intermediate Set<Object> type, and casting to Set<Map<String,String>> (or better yet, Collection<Map<String,String>>) in the first place?

svcRequest.setMpackNames(mpackNamesSet);
}
return svcRequest;
}

Expand All @@ -292,6 +299,7 @@ public synchronized Set<ServiceGroupResponse> createServiceGroups(Set<ServiceGro

// Already checked that service group does not exist
ServiceGroup sg = cluster.addServiceGroup(request.getServiceGroupName());
sg.setServiceGroupMpackNames(request.getMpackNames());
createdSvcGrps.add(sg.convertToResponse());
}
return createdSvcGrps;
Expand Down Expand Up @@ -422,6 +430,11 @@ private void validateCreateRequests(Set<ServiceGroupRequest> requests, Clusters
}
serviceGroupNames.get(clusterName).add(serviceGroupName);

if (request.getMpackNames().size() != 1) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this change introduces mpacks for service groups, and it also makes exactly one mpack a requirement, it breaks existing code that uses ServiceGroupResourceProvider to create mpacks. For one, it increases the number of failing/erroring unit tests:

[ERROR] Tests run: 5122, Failures: 39, Errors: 231, Skipped: 37
->
[ERROR] Tests run: 5123, Failures: 42, Errors: 296, Skipped: 37

Can you please address those? I think most could be fixed by adding some dummy mpack name in

public static void createServiceGroup(AmbariManagementController controller, String clusterName, String serviceGroupName)
throws AmbariException, AuthorizationException {
ServiceGroupRequest request = new ServiceGroupRequest(clusterName, serviceGroupName);
createServiceGroups(controller, Collections.singleton(request));
}

String errmsg = "Invalid arguments, only one mpack is allowed in the service group " + serviceGroupName;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please distinguish between 0 mpacks and 2+ mpacks. The error message "only one mpack is allowed" is a bit confusing when the request doesn't specify any mpacks for the service group.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not write the description of this bug clearly. At this time, temporarily we only support one mpack per servicegroup, but finally each servicegroup must have one or more mpacks when multiple mpacks support code is implemented. Again, I will add more clearer errmsg here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (request.getMpackNames().size() < 1)
Error should be that "Invalid arguments, atleast one mpack should be specified for a service group"
From backend, we should not restrict the number of mpacks associated to a service group.
@jayush , please correct me if wrong.

throw new IllegalArgumentException(errmsg);
}

Cluster cluster;
try {
cluster = clusters.getCluster(clusterName);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,12 @@

package org.apache.ambari.server.orm.entities;

import java.util.HashSet;
import java.util.List;
import java.util.Set;

import javax.persistence.Column;
import javax.persistence.ElementCollection;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
Expand Down Expand Up @@ -80,6 +83,10 @@ public class ServiceGroupEntity {
@OneToMany(mappedBy="serviceGroupDependency")
private List<ServiceGroupDependencyEntity> dependencies;

@ElementCollection
@Column(name = "name")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we need a corresponding change in the DDL scripts?
(ambari-server/src/main/resources/Ambari-DDL-*-CREATE.sql)

ERROR: relation "servicegroupentity_mpacknames" does not exist

private Set<String> mpackNames = new HashSet<>();

public Long getClusterId() {
return clusterId;
}
Expand Down Expand Up @@ -121,6 +128,14 @@ public void setServiceGroupDependencies(List<ServiceGroupDependencyEntity> servi
this.serviceGroupDependencies = serviceGroupDependencies;
}

public Set<String> getMpackNames() {
return mpackNames;
}

public void setMpackNames(Set<String> mpackNames) {
this.mpackNames = mpackNames;
}

@Override
public boolean equals(Object o) {
if (this == o) return true;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,14 @@ public interface ServiceGroup {

Set<ServiceGroupDependencyResponse> getServiceGroupDependencyResponses();

Set<String> getServiceGroupMpackNames();

void addServiceGroupMpackName(String mpackName);

void addServiceGroupMpackNames(Set<String> mpackNames);

void setServiceGroupMpackNames(Set<String> mpackNames);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please omit ServiceGroup from the method names?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems other methods have ServiceGroup, anyway, I will remove it.


void debugDump(StringBuilder sb);

void refresh();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,34 @@ public long getClusterId() {
return cluster.getClusterId();
}

@Override
public Set<String> getServiceGroupMpackNames() {
ServiceGroupEntity entity = getServiceGroupEntity();
return entity.getMpackNames();
}

@Override
public void addServiceGroupMpackName(String mpackName){
ServiceGroupEntity entity = getServiceGroupEntity();
if (entity.getMpackNames().add(mpackName)) {
serviceGroupDAO.merge(entity);
}
}

@Override
public void addServiceGroupMpackNames(Set<String> mpackNames) {
ServiceGroupEntity entity = getServiceGroupEntity();
entity.getMpackNames().addAll(mpackNames);
serviceGroupDAO.merge(entity);
}

@Override
public void setServiceGroupMpackNames(Set<String> mpackNames) {
ServiceGroupEntity entity = getServiceGroupEntity();
entity.setMpackNames(mpackNames);
serviceGroupDAO.merge(entity);
}

@Override
public Set<ServiceGroupKey> getServiceGroupDependencies() {
return serviceGroupDependencies;
Expand All @@ -149,6 +177,7 @@ public void setServiceGroupDependencies(Set<ServiceGroupKey> serviceGroupDepende
public ServiceGroupResponse convertToResponse() {
ServiceGroupResponse r = new ServiceGroupResponse(cluster.getClusterId(),
cluster.getClusterName(), getServiceGroupId(), getServiceGroupName());
r.setServiceGroupMpackNames(getServiceGroupMpackNames());
return r;
}

Expand Down Expand Up @@ -217,7 +246,14 @@ public Cluster getCluster() {
@Override
public void debugDump(StringBuilder sb) {
sb.append("ServiceGroup={ serviceGroupName=" + getServiceGroupName() + ", clusterName="
+ cluster.getClusterName() + ", clusterId=" + cluster.getClusterId() + "}");
+ cluster.getClusterName() + ", clusterId=" + cluster.getClusterId() );
Set<String> mpackNames = getServiceGroupMpackNames();
if (!mpackNames.isEmpty()) {
sb.append(", mpackNames=");
mpackNames.stream().map(mpackName ->sb.append(mpackName + ","));
sb.deleteCharAt(sb.length()-1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above for similar logic in ServiceGroupRequest.toString().

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

}
sb.append("}");
}

/**
Expand Down
Loading