Skip to content

HBASE-27784: support quota user overrides #5424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bbeaudreault merged 5 commits into from
Sep 27, 2023
Merged

HBASE-27784: support quota user overrides #5424

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ac13213
Support quota user overrides
Sep 19, 2023
951aec6
initial PR feedback
Sep 25, 2023
3528ca5
removing default quota override key, unifying test
Sep 26, 2023
761b435
add table without override to test
Sep 26, 2023
345e5c8
fetch single request attribute
Sep 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import java.util.EnumSet;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
Expand All @@ -35,8 +36,11 @@
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.RegionStatesCount;
import org.apache.hadoop.hbase.ipc.RpcCall;
import org.apache.hadoop.hbase.ipc.RpcServer;
import org.apache.hadoop.hbase.regionserver.HRegionServer;
import org.apache.hadoop.hbase.regionserver.RegionServerServices;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.yetus.audience.InterfaceAudience;
Expand All @@ -57,6 +61,11 @@ public class QuotaCache implements Stoppable {
private static final Logger LOG = LoggerFactory.getLogger(QuotaCache.class);

public static final String REFRESH_CONF_KEY = "hbase.quota.refresh.period";

// defines the request attribute key which, when provided, will override the request's username
// from the perspective of user quotas. This is also the default request attribute key
public static final String QUOTA_USER_REQUEST_ATTRIBUTE_OVERRIDE_KEY =
"hbase.quota.user.override.key";
Copy link
Contributor Author

@rmdmattingly rmdmattingly Sep 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main advantage of making this configurable is so that users can make use of existing attributes. For example we already submit an "upstream caller" identifier in our request attributes so that they will manifest in the slow query logs (introduced here). It would be nice to make use of that attribute here rather than requiring that we send superfluous bytes over the wire for each request

private static final int REFRESH_DEFAULT_PERIOD = 5 * 60000; // 5min
private static final int EVICT_PERIOD_FACTOR = 5; // N * REFRESH_DEFAULT_PERIOD

Expand All @@ -74,12 +83,15 @@ public class QuotaCache implements Stoppable {
private final ConcurrentHashMap<TableName, Double> tableMachineQuotaFactors =
new ConcurrentHashMap<>();
private final RegionServerServices rsServices;
private final String userOverrideRequestAttributeKey;

private QuotaRefresherChore refreshChore;
private boolean stopped = true;

public QuotaCache(final RegionServerServices rsServices) {
this.rsServices = rsServices;
this.userOverrideRequestAttributeKey = rsServices.getConfiguration()
.get(QUOTA_USER_REQUEST_ATTRIBUTE_OVERRIDE_KEY, QUOTA_USER_REQUEST_ATTRIBUTE_OVERRIDE_KEY);
}

public void start() throws IOException {
Expand Down Expand Up @@ -125,7 +137,7 @@ public QuotaLimiter getUserLimiter(final UserGroupInformation ugi, final TableNa
* @return the quota info associated to specified user
*/
public UserQuotaState getUserQuotaState(final UserGroupInformation ugi) {
return computeIfAbsent(userQuotaCache, ugi.getShortUserName(), UserQuotaState::new,
return computeIfAbsent(userQuotaCache, getQuotaUserName(ugi), UserQuotaState::new,
this::triggerCacheRefresh);
}

Expand Down Expand Up @@ -160,6 +172,23 @@ protected boolean isExceedThrottleQuotaEnabled() {
return exceedThrottleQuotaEnabled;
}

/**
* Applies a request attribute user override if available, otherwise returns the UGI's short
* username
* @param ugi The request's UserGroupInformation
*/
private String getQuotaUserName(final UserGroupInformation ugi) {
Optional<RpcCall> rpcCall = RpcServer.getCurrentCall();
if (
rpcCall.isPresent()
&& rpcCall.get().getRequestAttributes().containsKey(userOverrideRequestAttributeKey)
Copy link
Contributor

@bbeaudreault bbeaudreault Sep 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might recommend just looping the raw rpcCall.get().getHeader().getAttributes() here. For a small N (as i expect attributes to be), it is typically faster to iterate the list than to lookup in a map. This is especially true if we have to actually build the map as well (as we are likely to here). So in a vacuum, this call loops all the attributes and populates a map (which involves hashcode and equals for each, not to mention converting from ByteString to byte[] and string). Then we do a lookup, which involves another hashcode and equals. On the other hand, a loop can exit out early if a match is found and only requires equals.

So if we planned to do many operations on the map, it might start to pay off to do those conversions. But if we are just looking for a single key in an empty or very small list, it'd be cheaper to skip all of that.

Copy link
Contributor Author

@rmdmattingly rmdmattingly Sep 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that makes sense, seems like a reasonable optimization. should we bake that into an RpcCall#getRequestAttribute(String key) method to help this pattern be standard or is that overkill?

Copy link
Contributor

@bbeaudreault bbeaudreault Sep 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, let's do that. We can have it pull from the map if it's non-null, and otherwise loop. Just add javadoc describing the difference between that and getRequestAttributes() and when to use each.

) {
return Bytes
.toString(rpcCall.get().getRequestAttributes().get(userOverrideRequestAttributeKey));
}
return ugi.getShortUserName();
}

/**
* Returns the QuotaState requested. If the quota info is not in cache an empty one will be
* returned and the quota request will be enqueued for the next cache refresh.
Expand Down
111 changes: 111 additions & 0 deletions hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaUserOverride.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.quotas;

import static org.apache.hadoop.hbase.quotas.ThrottleQuotaTestUtil.doPuts;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;

import java.util.concurrent.TimeUnit;
import org.apache.hadoop.hbase.HBaseClassTestRule;
import org.apache.hadoop.hbase.HBaseTestingUtil;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.security.User;
import org.apache.hadoop.hbase.testclassification.MediumTests;
import org.apache.hadoop.hbase.testclassification.RegionServerTests;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.experimental.categories.Category;

@Category({ RegionServerTests.class, MediumTests.class })
public class TestQuotaUserOverride {

@ClassRule
public static final HBaseClassTestRule CLASS_RULE =
HBaseClassTestRule.forClass(TestQuotaUserOverride.class);

private static final HBaseTestingUtil TEST_UTIL = new HBaseTestingUtil();
private static final byte[] FAMILY = Bytes.toBytes("cf");
private static final byte[] QUALIFIER = Bytes.toBytes("q");
private static final int NUM_SERVERS = 1;

private static final TableName TABLE_NAME = TableName.valueOf("TestQuotaUserOverride");

@BeforeClass
public static void setUpBeforeClass() throws Exception {
TEST_UTIL.getConfiguration().setBoolean(QuotaUtil.QUOTA_CONF_KEY, true);
TEST_UTIL.getConfiguration().setInt(QuotaCache.REFRESH_CONF_KEY, 1_000);
TEST_UTIL.getConfiguration().setInt("hbase.client.pause", 1);
TEST_UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 0);
TEST_UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT, 500);
TEST_UTIL.startMiniCluster(NUM_SERVERS);
TEST_UTIL.waitTableAvailable(QuotaTableUtil.QUOTA_TABLE_NAME);
QuotaCache.TEST_FORCE_REFRESH = true;
TEST_UTIL.createTable(TABLE_NAME, FAMILY);
}

@AfterClass
public static void tearDownAfterClass() throws Exception {
EnvironmentEdgeManager.reset();
TEST_UTIL.shutdownMiniCluster();
}

@Test
public void testUserGlobalThrottleWithOverride() throws Exception {
final Admin admin = TEST_UTIL.getAdmin();
final String userOverrideWithQuota = User.getCurrent().getShortName() + "123";
final String userOverrideWithoutQuota = User.getCurrent().getShortName() + "456";

// Add 6req/min limit
admin.setQuota(QuotaSettingsFactory.throttleUser(userOverrideWithQuota,
ThrottleType.REQUEST_NUMBER, 6, TimeUnit.MINUTES));
Thread.sleep(60_000);

Table tableWithThrottle = admin.getConnection().getTableBuilder(TABLE_NAME, null)
.setRequestAttribute(QuotaCache.QUOTA_USER_REQUEST_ATTRIBUTE_OVERRIDE_KEY,
Bytes.toBytes(userOverrideWithQuota))
.build();
Table tableWithoutThrottle = admin.getConnection().getTableBuilder(TABLE_NAME, null)
.setRequestAttribute(QuotaCache.QUOTA_USER_REQUEST_ATTRIBUTE_OVERRIDE_KEY,
Bytes.toBytes(userOverrideWithoutQuota))
.build();

// warm things up
doPuts(10, FAMILY, QUALIFIER, tableWithThrottle);
doPuts(10, FAMILY, QUALIFIER, tableWithoutThrottle);

// should reject some requests
assertTrue(10 > doPuts(10, FAMILY, QUALIFIER, tableWithThrottle));
// should accept all puts
assertEquals(10, doPuts(10, FAMILY, QUALIFIER, tableWithoutThrottle));

// Remove all the limits
admin.setQuota(QuotaSettingsFactory.unthrottleUser(userOverrideWithQuota));
Copy link
Member

@ndimiduk ndimiduk Sep 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testing that removing the limits dynamically is honored -- nice.

Thread.sleep(60_000);
assertEquals(10, doPuts(10, FAMILY, QUALIFIER, tableWithThrottle));
assertEquals(10, doPuts(10, FAMILY, QUALIFIER, tableWithoutThrottle));
}

}
112 changes: 112 additions & 0 deletions ...rver/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaUserOverrideConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.quotas;

import static org.apache.hadoop.hbase.quotas.ThrottleQuotaTestUtil.doPuts;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;

import java.util.concurrent.TimeUnit;
import org.apache.hadoop.hbase.HBaseClassTestRule;
import org.apache.hadoop.hbase.HBaseTestingUtil;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.security.User;
import org.apache.hadoop.hbase.testclassification.MediumTests;
import org.apache.hadoop.hbase.testclassification.RegionServerTests;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.experimental.categories.Category;

@Category({ RegionServerTests.class, MediumTests.class })
public class TestQuotaUserOverrideConfiguration {

@ClassRule
public static final HBaseClassTestRule CLASS_RULE =
HBaseClassTestRule.forClass(TestQuotaUserOverrideConfiguration.class);

private static final HBaseTestingUtil TEST_UTIL = new HBaseTestingUtil();
private static final byte[] FAMILY = Bytes.toBytes("cf");
private static final byte[] QUALIFIER = Bytes.toBytes("q");
private static final int NUM_SERVERS = 1;
private static final String CUSTOM_OVERRIDE_KEY = "foo";

private static final TableName TABLE_NAME =
TableName.valueOf("TestQuotaUserOverrideConfiguration");

@BeforeClass
public static void setUpBeforeClass() throws Exception {
TEST_UTIL.getConfiguration().setBoolean(QuotaUtil.QUOTA_CONF_KEY, true);
TEST_UTIL.getConfiguration().setInt(QuotaCache.REFRESH_CONF_KEY, 1_000);
TEST_UTIL.getConfiguration().setInt("hbase.client.pause", 1);
TEST_UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 0);
TEST_UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT, 500);
TEST_UTIL.getConfiguration().set(QuotaCache.QUOTA_USER_REQUEST_ATTRIBUTE_OVERRIDE_KEY,
CUSTOM_OVERRIDE_KEY);
TEST_UTIL.startMiniCluster(NUM_SERVERS);
TEST_UTIL.waitTableAvailable(QuotaTableUtil.QUOTA_TABLE_NAME);
QuotaCache.TEST_FORCE_REFRESH = true;
TEST_UTIL.createTable(TABLE_NAME, FAMILY);
}

@AfterClass
public static void tearDownAfterClass() throws Exception {
EnvironmentEdgeManager.reset();
TEST_UTIL.shutdownMiniCluster();
}

@Test
public void testUserGlobalThrottleWithCustomOverride() throws Exception {
final Admin admin = TEST_UTIL.getAdmin();
final String userOverrideWithQuota = User.getCurrent().getShortName() + "123";

// Add 6req/min limit
admin.setQuota(QuotaSettingsFactory.throttleUser(userOverrideWithQuota,
ThrottleType.REQUEST_NUMBER, 6, TimeUnit.MINUTES));
Thread.sleep(60_000);

Table tableWithThrottle = admin.getConnection().getTableBuilder(TABLE_NAME, null)
.setRequestAttribute(CUSTOM_OVERRIDE_KEY, Bytes.toBytes(userOverrideWithQuota)).build();
Table tableWithoutThrottle = admin.getConnection().getTableBuilder(TABLE_NAME, null)
.setRequestAttribute(QuotaCache.QUOTA_USER_REQUEST_ATTRIBUTE_OVERRIDE_KEY,
Bytes.toBytes(userOverrideWithQuota))
.build();

// warm things up
doPuts(10, FAMILY, QUALIFIER, tableWithThrottle);
doPuts(10, FAMILY, QUALIFIER, tableWithoutThrottle);

// should reject some requests
assertTrue(10 > doPuts(10, FAMILY, QUALIFIER, tableWithThrottle));
// should accept all puts
assertEquals(10, doPuts(10, FAMILY, QUALIFIER, tableWithoutThrottle));

// Remove all the limits
admin.setQuota(QuotaSettingsFactory.unthrottleUser(userOverrideWithQuota));
Thread.sleep(60_000);
assertEquals(10, doPuts(10, FAMILY, QUALIFIER, tableWithThrottle));
assertEquals(10, doPuts(10, FAMILY, QUALIFIER, tableWithoutThrottle));
}

}