HBASE-23956 Use less resources running tests #1266
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Apache9
requested changes
Mar 10, 2020
| * The default netty event loop config | ||
| */ | ||
| @InterfaceAudience.Private | ||
| class DefaultNettyEventLoopConfig { |
Contributor
There was a problem hiding this comment.
This is not the design...
The design is to create a new EventLoop and make the code use it.
Change a configuration to change the 'default' instance and then make the code use the 'default' instance seems strange...
And also another anti pattern is that, we even do not have a Configuration instance here...
Contributor
There was a problem hiding this comment.
Or let's just remove this class. Move the creation of event loop into the NettyRpcClientConfigHelper directly and honor the hbase.netty.worker.count configuration when creating? Though a bit strange is that the Configuration passed in may change but at least it can avoid reading the configuration from system property...
saintstack
force-pushed
the
2.HBASE-23956
branch
from
2073d24 to
b7ebaae
Compare
Contributor
Author
|
Ok. I think I sorted it. Command-line arg wasn't so bad given we were passing generic for netty anyways but this is a bit better. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests
saintstack
force-pushed
the
2.HBASE-23956
branch
from
b7ebaae to
c12d53f
Compare
Contributor
Author
|
Fix checkstyle and the test failure (related) |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
Ok. Let me merge this. See if it helps w/ the OOME we are seeing around findbugs/spotbugs. Shout if you want amendments to the change you asked me do @Apache9 Thanks. |
asfgit
pushed a commit
that referenced
this pull request
Mar 11, 2020
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests Signed-off-by: Duo Zhang <[email protected]>
|
💔 -1 overall
This message was automatically generated. |
asfgit
pushed a commit
that referenced
this pull request
Mar 11, 2020
Addendum to cleanup pom warning.
asfgit
pushed a commit
that referenced
this pull request
Mar 11, 2020
Addendum to cleanup pom warning.
asfgit
pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: This test was depending on hard-coded number for log cleaners; the parent patch made them less in test context.
asfgit
pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: This test was depending on hard-coded number for log cleaners; the parent patch made them less in test context.
asfgit
pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: Two more tests that had issues because parent cut down the number of default RegionServers launched.
asfgit
pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: Two more tests that had issues because parent cut down the number of default RegionServers launched.
asfgit
pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: TestMultiParallel thread counts were off after parent change.
asfgit
pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: TestNamespaceCommands was using up the limit on nioeventloopgroup.... so create them per run of client.
asfgit
pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: TestNamespaceCommands was using up the limit on nioeventloopgroup.... so create them per run of client.
asfgit
pushed a commit
that referenced
this pull request
Apr 15, 2020
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests Signed-off-by: Duo Zhang <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
Apr 15, 2020
Addendum to cleanup pom warning.
thangTang
pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests Signed-off-by: Duo Zhang <[email protected]>
thangTang
pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum to cleanup pom warning.
thangTang
pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum: This test was depending on hard-coded number for log cleaners; the parent patch made them less in test context.
thangTang
pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum: Two more tests that had issues because parent cut down the number of default RegionServers launched.
thangTang
pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum: TestNamespaceCommands was using up the limit on nioeventloopgroup.... so create them per run of client.
thangTang
pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests Signed-off-by: Duo Zhang <[email protected]>
thangTang
pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum to cleanup pom warning.
thangTang
pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum: This test was depending on hard-coded number for log cleaners; the parent patch made them less in test context.
thangTang
pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum: Two more tests that had issues because parent cut down the number of default RegionServers launched.
thangTang
pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum: TestNamespaceCommands was using up the limit on nioeventloopgroup.... so create them per run of client.
ddupg
pushed a commit
to ddupg/hbase
that referenced
this pull request
Apr 20, 2020
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests Signed-off-by: Duo Zhang <[email protected]>
ddupg
pushed a commit
to ddupg/hbase
that referenced
this pull request
Apr 20, 2020
Addendum to cleanup pom warning.
infraio
pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests Signed-off-by: Duo Zhang <[email protected]>
infraio
pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
…o cleanup pom warning.
infraio
pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
…This test was depending on hard-coded number for log cleaners; the parent patch made them less in test context.
infraio
pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
Addendum: Two more tests that had issues because parent cut down the number of default RegionServers launched.
infraio
pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
…TestMultiParallel thread counts were off after parent change.
infraio
pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
…TestNamespaceCommands was using up the limit on nioeventloopgroup.... so create them per run of client.
sanjeet006py
pushed a commit
to sanjeet006py/hbase
that referenced
this pull request
Sep 26, 2025
* @W-17450732 Add jacoco argline prop * Upgrade jacoco verison
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add being able to configure netty thread counts. Enable socket reuse
(should not have any impact).
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java
Rename the threads we create in here so they are NOT named same was
threads created by Hadoop RPC.
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java
Allow configuring eventloopgroup thread count (so can override for
tests)
hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
Enable socket resuse.
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java
Enable socket resuse and config for how many threads to use.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
Thread name edit; drop the redundant 'Thread' suffix.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
Make closeable and shutdown executor when called.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
Call close on HFileReplicator
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
HDFS creates lots of threads. Use less of it so less threads overall.
hbase-server/src/test/resources/hbase-site.xml
hbase-server/src/test/resources/hdfs-site.xml
Constrain resources when running in test context.
hbase-server/src/test/resources/log4j.properties
Enable debug on netty to see netty configs in our log
pom.xml
Add system properties when we launch JVMs to constrain thread counts in
tests