KAFKA-6145: KIP-441: Add TaskAssignor class config #8541
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package org.apache.kafka.streams.processor.internals.assignment; | ||
|
|
||
| import org.apache.kafka.streams.processor.TaskId; | ||
| import org.apache.kafka.streams.processor.internals.assignment.AssignorConfiguration.AssignmentConfigs; | ||
|
|
||
| import java.util.Map; | ||
| import java.util.Set; | ||
| import java.util.UUID; | ||
|
|
||
| /** | ||
| * A special task assignor implementation to be used as a fallback in case the | ||
| * configured assignor couldn't be invoked. | ||
| * | ||
| * Specifically, this assignor must: | ||
| * 1. ignore the task lags in the ClientState map | ||
| * 2. always return true, indicating that a follow-up rebalance is needed | ||
|
Member
There was a problem hiding this comment. Returning true here will schedule a followup rebalance at the probing interval, but we also schedule a followup rebalance immediately before instantiating this assignor (line 735). Is this intentional? IIUC your proposal was to trigger a followup rebalance right away, which we do by means of the assignment error code. Of course, this is in memory so if the instance crashes and restarts we lose this information. I think we should actually avoid using the
Contributor
Author
There was a problem hiding this comment. Ah, this is a good point. Actually, I overlooked line 735. I'll remove that one. My proposal actually was to just wait for the probing rebalance interval in case the lag computation failed. It seems like this should be ok, since Streams will still make progress in the mean time, and it avoids the pathological case where we could just constantly rebalance if the end-offsets API is down for some reason. |
||
| */ | ||
| public class FallbackPriorTaskAssignor implements TaskAssignor { | ||
| private final StickyTaskAssignor delegate; | ||
|
Comment on lines
+26
to
+35
Contributor
Author
There was a problem hiding this comment. I renamed the PriorTaskAssignor and added a Javadoc to make its role clear. Note that "PriorTaskAssignor" would be an appropriate behavioral name, except that it also always returns "true", and that it must ignore the lags, which is what makes it a "fallback" assignor here. |
||
|
|
||
| public FallbackPriorTaskAssignor() { | ||
| delegate = new StickyTaskAssignor(true); | ||
| } | ||
|
|
||
| @Override | ||
| public boolean assign(final Map<UUID, ClientState> clients, | ||
| final Set<TaskId> allTaskIds, | ||
| final Set<TaskId> standbyTaskIds, | ||
| final AssignmentConfigs configs) { | ||
| delegate.assign(clients, allTaskIds, standbyTaskIds, configs); | ||
| return true; | ||
| } | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Necessary because the test name that JUnit generates for the parameterized StreamsPartitionAssignorTest is slightly too long. I have no way to shorten it because the thing that pushes it over is the fact that there are two package names in the parameterized method name, and there's no control over the format of the test name itself. So, I decided just to truncate the file name instead, which is almost certainly still unique for pretty much any test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only alternative I can think of is to parameterize the "short name" of the TaskAssignor, which seems kind of wacky.
Also, worth noting the impact of truncation is nothing if the file name is still unique. If the name is shared between two tests, then the impact is still nothing if both tests pass. The only observable effect is that if one or both tests fail, their logs would get combined. It seems like we can afford just to defer this problem until it happens, if ever.