-
Notifications
You must be signed in to change notification settings - Fork 1.7k
[AMBARI-22878] Update Service Group API to take list of mpack name associated with the service group #234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AMBARI-22878] Update Service Group API to take list of mpack name associated with the service group #234
Changes from 3 commits
dcb871b
e97eb29
da41ca0
158cd85
decbf54
68c1f04
a2340a6
04345f8
90f354f
a331735
3de0e93
f74af11
afc6a3b
fb3e005
7bcd6fe
cfe5295
d370602
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,16 +17,20 @@ | |
| */ | ||
| package org.apache.ambari.server.controller; | ||
|
|
||
| import java.util.HashSet; | ||
| import java.util.Objects; | ||
| import java.util.Set; | ||
|
|
||
| public class ServiceGroupRequest { | ||
|
|
||
| private String clusterName; // REF | ||
| private String serviceGroupName; // GET/CREATE/UPDATE/DELETE | ||
| private Set<String> mpackNames; // Associated mpack names | ||
|
|
||
| public ServiceGroupRequest(String clusterName, String serviceGroupName) { | ||
| this.clusterName = clusterName; | ||
| this.serviceGroupName = serviceGroupName; | ||
| mpackNames = new HashSet<>(); | ||
| } | ||
|
|
||
| /** | ||
|
|
@@ -57,9 +61,30 @@ public void setServiceGroupName(String serviceGroupName) { | |
| this.serviceGroupName = serviceGroupName; | ||
| } | ||
|
|
||
| /** | ||
| * @return a list of associated mpack names | ||
| */ | ||
| public Set<String> getMpackNames() { | ||
| return mpackNames; | ||
| } | ||
|
|
||
| /** | ||
| * @param mpackNames a list of associated mpack names | ||
| */ | ||
| public void addMpackNames(Set<String> mpackNames) { | ||
| if (mpackNames != null) { | ||
| this.mpackNames.addAll(mpackNames); | ||
| } | ||
| } | ||
|
|
||
| @Override | ||
| public String toString() { | ||
| return String.format("clusterName=%s, serviceGroupName=%s", clusterName, serviceGroupName); | ||
| StringBuilder sb = new StringBuilder(); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We've been gradually moving to ToStringBuilder here instead
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, we should switch this to ToStringBuilder. |
||
| sb.append("clusterName=").append(clusterName).append(", serviceGroupName=").append(serviceGroupName); | ||
| if (!mpackNames.isEmpty()) { | ||
| sb.append(",mpackNames=").append(mpackNames.toString()); | ||
| } | ||
| return sb.toString(); | ||
| } | ||
|
|
||
| @Override | ||
|
|
@@ -73,6 +98,8 @@ public boolean equals(Object obj) { | |
|
|
||
| ServiceGroupRequest other = (ServiceGroupRequest) obj; | ||
|
|
||
| // ignore mpackNames, even if they are different, we still consider sgrequests are the same | ||
|
|
||
| return Objects.equals(clusterName, other.clusterName) && | ||
| Objects.equals(serviceGroupName, other.serviceGroupName); | ||
| } | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -19,11 +19,13 @@ | |||||||||||
| package org.apache.ambari.server.controller.internal; | ||||||||||||
|
|
||||||||||||
| import java.util.Arrays; | ||||||||||||
| import java.util.Collection; | ||||||||||||
| import java.util.HashMap; | ||||||||||||
| import java.util.HashSet; | ||||||||||||
| import java.util.Iterator; | ||||||||||||
| import java.util.Map; | ||||||||||||
| import java.util.Set; | ||||||||||||
| import java.util.stream.Collectors; | ||||||||||||
|
|
||||||||||||
| import org.apache.ambari.server.AmbariException; | ||||||||||||
| import org.apache.ambari.server.ClusterNotFoundException; | ||||||||||||
|
|
@@ -78,6 +80,7 @@ public class ServiceGroupResourceProvider extends AbstractControllerResourceProv | |||||||||||
| public static final String SERVICE_GROUP_CLUSTER_NAME_PROPERTY_ID = RESPONSE_KEY + PropertyHelper.EXTERNAL_PATH_SEP + "cluster_name"; | ||||||||||||
| public static final String SERVICE_GROUP_SERVICE_GROUP_ID_PROPERTY_ID = RESPONSE_KEY + PropertyHelper.EXTERNAL_PATH_SEP + "service_group_id"; | ||||||||||||
| public static final String SERVICE_GROUP_SERVICE_GROUP_NAME_PROPERTY_ID = RESPONSE_KEY + PropertyHelper.EXTERNAL_PATH_SEP + "service_group_name"; | ||||||||||||
| public static final String SERVICE_GROUP_MPACKNAME_PROPERTY_ID = RESPONSE_KEY + PropertyHelper.EXTERNAL_PATH_SEP + "mpacks"; | ||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is an awkward construct. Either do the string directly or use PropertyHelper.getPropertyId(...).
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @ncole This is useful for when we document the API using Swagger annotations. The compiler does not allow using strings generated by Anyway, this question does not have much to do with this PR, as @scottduan just followed the existing usage pattern in the lines above.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think mpack:sg is many to many relationship finally. At this time, we just have temporary restriction for 1:1, also I did not validate mpack information. This will be done in the following mpack work.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think it should be M2M - it should be M2O. Since upgrade workflow is based off of SGs, it doesn't make sense to have the ability to mix them at this point. It creates more complexity that will not be used. Also, this relationship shouldn't be based on names, but on specific IDs of the mpack. There shouldn't be ambiguity about which MPack a SG is based off of.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @adoroszlai - it's a constant, spelling should not be an issue. You can't use constants in Swagger? That would be extremely disappointing.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jayush , how do you think? Based on my understand, SG can pick up service from different mpacks, so this should be M2M.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A SG should not be able to pickup services from different mpacks - it will break a lot of upgrade flow and assumptions. Additionally, it needs to be associated with a specific mpack in order to provide a way of querying what SG's are compatible with given mpack.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. mpack provides with service dependence list. So when SG picks up service, theoratically it can choose proper service with the right version (it should be a range). There has a mpack design doc, https://docs.google.com/document/d/13y2T_uN8_nrMsNRuqXLnQAm64Edie0k8_WVNOFvKq0U/edit#heading=h.ykmhjj9wbfa8
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @ncole It's a limitation of annotation "element value" in Java, not specific to Swagger. The value must be a constant expression, not just any runtime constant. |
||||||||||||
|
|
||||||||||||
|
|
||||||||||||
| private static Set<String> pkPropertyIds = | ||||||||||||
|
|
@@ -103,10 +106,12 @@ public class ServiceGroupResourceProvider extends AbstractControllerResourceProv | |||||||||||
| PROPERTY_IDS.add(SERVICE_GROUP_CLUSTER_NAME_PROPERTY_ID); | ||||||||||||
| PROPERTY_IDS.add(SERVICE_GROUP_SERVICE_GROUP_ID_PROPERTY_ID); | ||||||||||||
| PROPERTY_IDS.add(SERVICE_GROUP_SERVICE_GROUP_NAME_PROPERTY_ID); | ||||||||||||
| PROPERTY_IDS.add(SERVICE_GROUP_MPACKNAME_PROPERTY_ID); | ||||||||||||
|
|
||||||||||||
| // keys | ||||||||||||
| KEY_PROPERTY_IDS.put(Resource.Type.Cluster, SERVICE_GROUP_CLUSTER_NAME_PROPERTY_ID); | ||||||||||||
| KEY_PROPERTY_IDS.put(Resource.Type.ServiceGroup, SERVICE_GROUP_SERVICE_GROUP_NAME_PROPERTY_ID); | ||||||||||||
| KEY_PROPERTY_IDS.put(Resource.Type.Mpack, SERVICE_GROUP_MPACKNAME_PROPERTY_ID); | ||||||||||||
| } | ||||||||||||
|
|
||||||||||||
| private Clusters clusters; | ||||||||||||
|
|
@@ -160,6 +165,7 @@ public Set<ServiceGroupResponse> invoke() throws AmbariException, AuthorizationE | |||||||||||
| resource.setProperty(SERVICE_GROUP_CLUSTER_NAME_PROPERTY_ID, response.getClusterName()); | ||||||||||||
| resource.setProperty(SERVICE_GROUP_SERVICE_GROUP_ID_PROPERTY_ID, response.getServiceGroupId()); | ||||||||||||
| resource.setProperty(SERVICE_GROUP_SERVICE_GROUP_NAME_PROPERTY_ID, response.getServiceGroupName()); | ||||||||||||
| resource.setProperty(SERVICE_GROUP_MPACKNAME_PROPERTY_ID, response.getMpackNames()); | ||||||||||||
|
|
||||||||||||
| associatedResources.add(resource); | ||||||||||||
| } | ||||||||||||
|
|
@@ -199,6 +205,8 @@ public Set<ServiceGroupResponse> invoke() throws AmbariException { | |||||||||||
| response.getServiceGroupId(), requestedIds); | ||||||||||||
| setResourceProperty(resource, SERVICE_GROUP_SERVICE_GROUP_NAME_PROPERTY_ID, | ||||||||||||
| response.getServiceGroupName(), requestedIds); | ||||||||||||
| setResourceProperty(resource, SERVICE_GROUP_MPACKNAME_PROPERTY_ID, | ||||||||||||
| response.getMpackNames(), requestedIds); | ||||||||||||
| resources.add(resource); | ||||||||||||
| } | ||||||||||||
| return resources; | ||||||||||||
|
|
@@ -269,6 +277,11 @@ private ServiceGroupRequest getRequest(Map<String, Object> properties) { | |||||||||||
| String clusterName = (String) properties.get(SERVICE_GROUP_CLUSTER_NAME_PROPERTY_ID); | ||||||||||||
| String serviceGroupName = (String) properties.get(SERVICE_GROUP_SERVICE_GROUP_NAME_PROPERTY_ID); | ||||||||||||
| ServiceGroupRequest svcRequest = new ServiceGroupRequest(clusterName, serviceGroupName); | ||||||||||||
| Collection<Map<String,String>> mpackNames = (Collection<Map<String,String>>) properties.get(SERVICE_GROUP_MPACKNAME_PROPERTY_ID); | ||||||||||||
| if (mpackNames != null) { | ||||||||||||
| Set<String> mpackNamesSet = mpackNames.stream().flatMap(mpack -> mpack.values().stream()).collect(Collectors.toSet()); | ||||||||||||
| svcRequest.addMpackNames(mpackNamesSet); | ||||||||||||
| } | ||||||||||||
| return svcRequest; | ||||||||||||
| } | ||||||||||||
|
|
||||||||||||
|
|
@@ -292,6 +305,7 @@ public synchronized Set<ServiceGroupResponse> createServiceGroups(Set<ServiceGro | |||||||||||
|
|
||||||||||||
| // Already checked that service group does not exist | ||||||||||||
| ServiceGroup sg = cluster.addServiceGroup(request.getServiceGroupName()); | ||||||||||||
| sg.setMpackNames(request.getMpackNames()); | ||||||||||||
| createdSvcGrps.add(sg.convertToResponse()); | ||||||||||||
| } | ||||||||||||
| return createdSvcGrps; | ||||||||||||
|
|
@@ -422,6 +436,11 @@ private void validateCreateRequests(Set<ServiceGroupRequest> requests, Clusters | |||||||||||
| } | ||||||||||||
| serviceGroupNames.get(clusterName).add(serviceGroupName); | ||||||||||||
|
|
||||||||||||
| if (request.getMpackNames().size() != 1) { | ||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since this change introduces mpacks for service groups, and it also makes exactly one mpack a requirement, it breaks existing code that uses Can you please address those? I think most could be fixed by adding some dummy mpack name in Lines 39 to 43 in 00f29b9
|
||||||||||||
| String errmsg = "Invalid arguments, " + request.getMpackNames().size() + " mpack(s) found in the service group " + serviceGroupName + ", only one mpack is allowed"; | ||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Use String.format() here.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It seems that concatenation has better performance than string.format().
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For messages like this, we're more interested in readability in code than performance. We've been using String.format() in more and more places to be a bit more consistent with logging syntax (even if they are not identical).
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok, I will update it.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should I use StringBuilder instead? I saw your another comment mentioned it. |
||||||||||||
| throw new IllegalArgumentException(errmsg); | ||||||||||||
| } | ||||||||||||
|
|
||||||||||||
| Cluster cluster; | ||||||||||||
| try { | ||||||||||||
| cluster = clusters.getCluster(clusterName); | ||||||||||||
|
|
||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,9 +18,13 @@ | |
|
|
||
| package org.apache.ambari.server.orm.entities; | ||
|
|
||
| import java.util.HashSet; | ||
| import java.util.List; | ||
| import java.util.Set; | ||
|
|
||
| import javax.persistence.CollectionTable; | ||
| import javax.persistence.Column; | ||
| import javax.persistence.ElementCollection; | ||
| import javax.persistence.Entity; | ||
| import javax.persistence.GeneratedValue; | ||
| import javax.persistence.GenerationType; | ||
|
|
@@ -80,6 +84,14 @@ public class ServiceGroupEntity { | |
| @OneToMany(mappedBy="serviceGroupDependency") | ||
| private List<ServiceGroupDependencyEntity> dependencies; | ||
|
|
||
| @ElementCollection() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Formatting
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you explain a little more there? Format what? Thanks.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I may have meant the next lines down - seems like a lot of whitespace.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I will remove all those whitespace, and put them on the same line. |
||
| @CollectionTable(name = "servicegroup_mpacknames", | ||
| joinColumns = {@JoinColumn(name = "service_group_id", referencedColumnName = "id"), | ||
| @JoinColumn(name = "service_group_cluster_id", referencedColumnName = "cluster_id")}) | ||
| @Column(name = "mpack_name", unique = true, nullable = false) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think the
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually mpack_name should not be unique. It should allow different service group to have the same mpack_name. I will remove this contraint. |
||
|
|
||
| private Set<String> mpackNames = new HashSet<>(); | ||
|
|
||
| public Long getClusterId() { | ||
| return clusterId; | ||
| } | ||
|
|
@@ -121,6 +133,16 @@ public void setServiceGroupDependencies(List<ServiceGroupDependencyEntity> servi | |
| this.serviceGroupDependencies = serviceGroupDependencies; | ||
| } | ||
|
|
||
| public Set<String> getMpackNames() { | ||
| return mpackNames; | ||
| } | ||
|
|
||
| public void setMpackNames(Set<String> mpackNames) { | ||
| if (mpackNames != null) { | ||
| this.mpackNames = mpackNames; | ||
| } | ||
| } | ||
|
|
||
| @Override | ||
| public boolean equals(Object o) { | ||
| if (this == o) return true; | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1156,6 +1156,16 @@ CREATE TABLE alert_notice ( | |
| FOREIGN KEY (history_id) REFERENCES alert_history(alert_id) | ||
| ); | ||
|
|
||
| CREATE TABLE servicegroup_mpackname ( | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Table name is different in this schema (
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Typo, will change it. |
||
| service_group_id BIGINT NOT NULL, | ||
| service_group_cluster_id BIGINT NOT NULL, | ||
| mpack_name VARCHAR(255) NOT NULL, | ||
| CONSTRAINT PK_servicegroup_mpackname PRIMARY KEY (service_group_id, service_group_cluster_id, mpack_name), | ||
| FOREIGN KEY (service_group_id) REFERENCES servicegroups(service_group_id), | ||
| FOREIGN KEY (service_group_cluster_id) REFERENCES servicegroups(service_group_cluster_id) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Constraints/foreign keys in this table are different than in the other DBMS schemas.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. will update. Seems I forgot to update it. |
||
| ); | ||
|
|
||
|
|
||
| CREATE INDEX idx_alert_history_def_id on alert_history(alert_definition_id); | ||
| CREATE INDEX idx_alert_history_service on alert_history(service_name); | ||
| CREATE INDEX idx_alert_history_host on alert_history(host_name); | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think there's still a misunderstanding here - I thought it would be cleaner to separate this out into 2 fields so that it's clear to the user what is expected: mpack name and then mpack version. You can combine this and call it a stack ID internally, that's fine - but we should take 2 different properties.
Same goes for API requests where we're currently given
name-versionas a string. This should be broken out into 2 fields.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am ok for this change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jayush What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets keep it as "version" for now for legacy purposes. When we upgrade to ambari 3.0 assuming the existing cluster is on HDP-2.6 stack, calling this as mpack name and mpack version would be misleading.
We should handle this in a separate JIRA if we really want to change this convention for all APIs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But "version" in legacy terms meant only the numeric value. If we don't want to break it out into 2 references, I think we'd rather use stackId, right?