Skip to content

HDFS-16671. RBF: RouterRpcFairnessPolicyController supports configurable permit acquire timeout #4597

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ferhui merged 5 commits into from
Jul 28, 2022
Merged

HDFS-16671. RBF: RouterRpcFairnessPolicyController supports configurable permit acquire timeout #4597

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
300d8ee
HDFS-16671. RBF: RouterRpcFairnessPolicyController supports configura…
Jul 20, 2022
bfb9229
HDFS-16671. Modify patch based on comments
Jul 21, 2022
d985449
HDFS-16671. Modify patch based on comment.
Jul 22, 2022
fcbdd2f
HDFS-16671. Modify code based comment
Jul 23, 2022
e5436d5
HDFS-16671. Modify patch based on comments
Jul 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 15 additions & 7 deletions ...che/hadoop/hdfs/server/federation/fairness/AbstractRouterRpcFairnessPolicyController.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import static org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys.DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT_DEFAULT;
import static org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys.DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT_MS;

/**
* Base fairness policy that implements @RouterRpcFairnessPolicyController.
* Internally a map of nameservice to Semaphore is used to control permits.
Expand All @@ -42,15 +45,22 @@ public class AbstractRouterRpcFairnessPolicyController
/** Hash table to hold semaphore for each configured name service. */
private Map<String, Semaphore> permits;

private long acquireTimeoutMs = DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT_DEFAULT;

public void init(Configuration conf) {
this.permits = new HashMap<>();
long timeoutMs = conf.getTimeDuration(DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT_MS,
Copy link
Member

@goiri goiri Jul 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do getTimeDuration() we don't need the prefix in the key DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT_MS.
We should set the default in the XML to "1s"

DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT_DEFAULT, TimeUnit.MILLISECONDS);
if (timeoutMs >= 0) {
acquireTimeoutMs = timeoutMs;
Comment on lines +54 to +55
Copy link
Member

@ayushtkn ayushtkn Jul 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if there is an invalid entry configured and we are moving to using the default value. We should atleast have a warn log. Kind of Invalid value -1 configured for dfs.... should be greater than or equal to 0, Using default value of : 1s instead
something like this

}
}

@Override
public boolean acquirePermit(String nsId) {
try {
LOG.debug("Taking lock for nameservice {}", nsId);
return this.permits.get(nsId).tryAcquire(1, TimeUnit.SECONDS);
return this.permits.get(nsId).tryAcquire(acquireTimeoutMs, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
LOG.debug("Cannot get a permit for nameservice {}", nsId);
}
Expand Down Expand Up @@ -82,15 +92,13 @@ protected int getAvailablePermits(String nsId) {
@Override
public String getAvailableHandlerOnPerNs() {
JSONObject json = new JSONObject();
for (Map.Entry<String, Semaphore> entry : permits.entrySet()) {
permits.forEach((k, v) -> {
try {
String nsId = entry.getKey();
int availableHandler = entry.getValue().availablePermits();
json.put(nsId, availableHandler);
json.put(k, v.availablePermits());
} catch (JSONException e) {
LOG.warn("Cannot put {} into JSONObject", entry.getKey(), e);
LOG.warn("Cannot put {} into JSONObject", k, e);
}
}
});
return json.toString();
}
}
31 changes: 11 additions & 20 deletions ...pache/hadoop/hdfs/server/federation/fairness/StaticRouterRpcFairnessPolicyController.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,13 +50,10 @@ public StaticRouterRpcFairnessPolicyController(Configuration conf) {
init(conf);
}

public void init(Configuration conf)
throws IllegalArgumentException {
public void init(Configuration conf) throws IllegalArgumentException {
super.init(conf);
// Total handlers configured to process all incoming Rpc.
int handlerCount = conf.getInt(
DFS_ROUTER_HANDLER_COUNT_KEY,
DFS_ROUTER_HANDLER_COUNT_DEFAULT);
int handlerCount = conf.getInt(DFS_ROUTER_HANDLER_COUNT_KEY, DFS_ROUTER_HANDLER_COUNT_DEFAULT);

LOG.info("Handlers available for fairness assignment {} ", handlerCount);

Expand All @@ -71,8 +68,7 @@ public void init(Configuration conf)
allConfiguredNS.add(CONCURRENT_NS);
validateHandlersCount(conf, handlerCount, allConfiguredNS);
for (String nsId : allConfiguredNS) {
int dedicatedHandlers =
conf.getInt(DFS_ROUTER_FAIR_HANDLER_COUNT_KEY_PREFIX + nsId, 0);
int dedicatedHandlers = conf.getInt(DFS_ROUTER_FAIR_HANDLER_COUNT_KEY_PREFIX + nsId, 0);
LOG.info("Dedicated handlers {} for ns {} ", dedicatedHandlers, nsId);
if (dedicatedHandlers > 0) {
handlerCount -= dedicatedHandlers;
Expand All @@ -86,7 +82,7 @@ public void init(Configuration conf)
// Assign remaining handlers equally to remaining name services and
// general pool if applicable.
if (!unassignedNS.isEmpty()) {
LOG.info("Unassigned ns {}", unassignedNS.toString());
LOG.info("Unassigned ns {}", unassignedNS);
int handlersPerNS = handlerCount / unassignedNS.size();
LOG.info("Handlers available per ns {}", handlersPerNS);
for (String nsId : unassignedNS) {
Expand All @@ -101,24 +97,20 @@ public void init(Configuration conf)
int existingPermits = getAvailablePermits(CONCURRENT_NS);
if (leftOverHandlers > 0) {
LOG.info("Assigned extra {} handlers to commons pool", leftOverHandlers);
insertNameServiceWithPermits(CONCURRENT_NS,
existingPermits + leftOverHandlers);
insertNameServiceWithPermits(CONCURRENT_NS, existingPermits + leftOverHandlers);
}
LOG.info("Final permit allocation for concurrent ns: {}",
getAvailablePermits(CONCURRENT_NS));
LOG.info("Final permit allocation for concurrent ns: {}", getAvailablePermits(CONCURRENT_NS));
}

private static void logAssignment(String nsId, int count) {
LOG.info("Assigned {} handlers to nsId {} ",
count, nsId);
LOG.info("Assigned {} handlers to nsId {} ", count, nsId);
}

private void validateHandlersCount(Configuration conf, int handlerCount,
Set<String> allConfiguredNS) {
private void validateHandlersCount(Configuration conf,
int handlerCount, Set<String> allConfiguredNS) {
int totalDedicatedHandlers = 0;
for (String nsId : allConfiguredNS) {
int dedicatedHandlers =
conf.getInt(DFS_ROUTER_FAIR_HANDLER_COUNT_KEY_PREFIX + nsId, 0);
int dedicatedHandlers = conf.getInt(DFS_ROUTER_FAIR_HANDLER_COUNT_KEY_PREFIX + nsId, 0);
if (dedicatedHandlers > 0) {
// Total handlers should not be less than sum of dedicated handlers.
totalDedicatedHandlers += dedicatedHandlers;
Expand All @@ -128,8 +120,7 @@ private void validateHandlersCount(Configuration conf, int handlerCount,
}
}
if (totalDedicatedHandlers > handlerCount) {
String msg = String.format(ERROR_MSG, handlerCount,
totalDedicatedHandlers);
String msg = String.format(ERROR_MSG, handlerCount, totalDedicatedHandlers);
LOG.error(msg);
throw new IllegalArgumentException(msg);
}
Expand Down
4 changes: 4 additions & 0 deletions ...hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,10 @@ public class RBFConfigKeys extends CommonConfigurationKeysPublic {
NoRouterRpcFairnessPolicyController.class;
public static final String DFS_ROUTER_FAIR_HANDLER_COUNT_KEY_PREFIX =
FEDERATION_ROUTER_FAIRNESS_PREFIX + "handler.count.";
public static final String DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT_MS =
FEDERATION_ROUTER_FAIRNESS_PREFIX + "acquire.timeout.ms";
public static final long DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT_DEFAULT =
TimeUnit.SECONDS.toMillis(1);

// HDFS Router Federation Rename.
public static final String DFS_ROUTER_FEDERATION_RENAME_PREFIX =
Expand Down
8 changes: 8 additions & 0 deletions hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/resources/hdfs-rbf-default.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -723,6 +723,14 @@
</description>
</property>

<property>
<name>dfs.federation.router.fairness.acquire.timeout.ms</name>
<value>1000</value>
Copy link
Member

@goiri goiri Jul 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1s and remove the ms

<description>
The maximum time, in milliseconds, to wait for a permit.
</description>
</property>

<property>
<name>dfs.federation.router.federation.rename.bandwidth</name>
<value>10</value>
Expand Down
25 changes: 25 additions & 0 deletions .../apache/hadoop/hdfs/server/federation/fairness/TestRouterRpcFairnessPolicyController.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,12 @@
import org.apache.hadoop.hdfs.server.federation.router.FederationUtil;
import org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys;
import org.apache.hadoop.test.GenericTestUtils;
import org.apache.hadoop.util.Time;
import org.junit.Test;
import org.slf4j.LoggerFactory;

import static org.apache.hadoop.hdfs.server.federation.fairness.RouterRpcFairnessConstants.CONCURRENT_NS;
import static org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys.DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT_MS;
import static org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys.DFS_ROUTER_HANDLER_COUNT_KEY;
import static org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys.DFS_ROUTER_MONITOR_NAMENODE;
import static org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys.DFS_ROUTER_FAIR_HANDLER_COUNT_KEY_PREFIX;
Expand Down Expand Up @@ -83,6 +85,29 @@ public void testHandlerAllocationPreconfigured() {
assertFalse(routerRpcFairnessPolicyController.acquirePermit(CONCURRENT_NS));
}

@Test
public void testAcquireTimeout() {
Configuration conf = createConf(40);
conf.setInt(DFS_ROUTER_FAIR_HANDLER_COUNT_KEY_PREFIX + "ns1", 30);
conf.setLong(DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT_MS, 100);
Copy link
Member

@goiri goiri Jul 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setTimeDuration(DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT, 1, TimeUnit.SECONDS)

RouterRpcFairnessPolicyController routerRpcFairnessPolicyController =
FederationUtil.newFairnessPolicyController(conf);

// ns1 should have 30 permits allocated
for (int i = 0; i < 30; i++) {
assertTrue(routerRpcFairnessPolicyController.acquirePermit("ns1"));
}
long acquireBeginTimeMs = Time.monotonicNow();
assertFalse(routerRpcFairnessPolicyController.acquirePermit("ns1"));
long acquireEndTimeMs = Time.monotonicNow();

long acquireTimeMs = acquireEndTimeMs - acquireBeginTimeMs;

// There are some other operations, so acquireTimeMs >= 100ms.
assertTrue(acquireTimeMs >= 100);
assertTrue(acquireTimeMs < 100 + 50);
Copy link
Member

@ayushtkn ayushtkn Jul 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this 50 added as safe margin, kind of that other operations won't take more than 50ms? In that case I doubt this test is gonna go flaky in future, Double check to confirm it isn't gonna cross the safe limit of 50 ever.

Copy link
Contributor Author

@ZanderXu ZanderXu Jul 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ayushtkn Thanks for your review. About this, do you have some good ideas?

Copy link
Member

@ayushtkn ayushtkn Jul 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either we can keep this safe margin way above or remove this. @goiri do you have any suggestions?

}

@Test
public void testAllocationErrorWithZeroHandlers() {
Configuration conf = createConf(0);
Expand Down