HBASE-28432 Refactor tools which are under test packaging to a new module hbase-diagnostics #6258

NihalJain · 2024-09-17T18:00:16Z

The purpose of this task is to refactor and move certain tools currently located under the test packaging to a new module, named 'hbase-diagnostics'.

The following tools have been initially identified for relocation(will add more as and when identified):

These tools are valuable beyond the scope of testing and should be accessible in the binary distribution of HBase. However, their current location within the test jars adds unnecessary bloat to the assembly and classpath, and potentially introduces CVE-prone JARs into the binary assemblies. We plan to remove all test jars from assembly with HBASE-28433.

This task involves creating the new 'hbase-diagnostics' module, and moving the identified tools into this module. It also includes ensuring that these tools function correctly in their new location and that their relocation does not negatively impact any existing functionality or dependencies.

Also see draft patch without this change for follow up work HBASE-28433 at #6184

NihalJain · 2024-09-17T18:05:18Z

Supersedes PR #6249, also see #6249 (comment)

Change Summary

Added a new module called hbase-diagnostics (as suggested by @stoty)
- Question: Any better names?
Moved all the targeted tools we intend to move (i.e. PerformanceEvaluation, LoadTestTool, HFilePerformanceEvaluation, ScanPerformanceEvaluation and LoadBalancerPerformanceEvaluation) as part of this task to main of hbase-diagnostics along with all related classes:
- Question: Anything else which we need to handle?
Moved following classses to their corresponding modules' main code:
- RandomDistribution
- KeyProviderForTesting
- LoadTestKVGenerator
- FilterAllFilter
Added a new util DiagnosticToolsCommonUtils and moved method required to loginAndReturnUGI() from HBaseKerberosUtils
Added a new util LoadTestUtil and moved all load test related code from HFileTestUtil and HBaseTestingUtil to here
- Question: Should we merge this to DiagnosticToolsCommonUtils?
- Question: Just hoping it is fine to move code from HBaseTestingUtil as class is marked @InterfaceStability.Evolving, otherwise may have to think some other approach.
Copied generateData() from PerformanceEvaluation in TestHFileOutputFormat2 to break cyclic dependency
Added @InterfaceAudience.Private to all classes moved from test code to main
Replaced UTIL.getConfiguration() with HBaseConfiguration.create() in LoadBalancerPerformanceEvaluation

Also ran basic commands for sanity of above 5 tools in a standalone local cluster.

Newly generated jars' content:

hbase % jar -tf hbase-4.0.0-alpha-1-SNAPSHOT/lib/hbase-diagnostics-4.0.0-alpha-1-SNAPSHOT.jar
META-INF/MANIFEST.MF
META-INF/
org/
org/apache/
org/apache/hadoop/
org/apache/hadoop/hbase/
org/apache/hadoop/hbase/master/
org/apache/hadoop/hbase/master/balancer/
org/apache/hadoop/hbase/util/
org/apache/hadoop/hbase/util/test/
META-INF/maven/
META-INF/maven/org.apache.hbase/
META-INF/maven/org.apache.hbase/hbase-diagnostics/
META-INF/DEPENDENCIES
META-INF/LICENSE
META-INF/NOTICE
org/apache/hadoop/hbase/HFilePerformanceEvaluation$1.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$2.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$3.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$4.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$GaussianRandomReadBenchmark.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$ReadBenchmark.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$RowOrientedBenchmark.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$SequentialReadBenchmark.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$SequentialWriteBenchmark.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$UniformRandomReadBenchmark.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$UniformRandomSmallScan.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation.class
org/apache/hadoop/hbase/PerformanceEvaluation$1$1.class
org/apache/hadoop/hbase/PerformanceEvaluation$1.class
org/apache/hadoop/hbase/PerformanceEvaluation$AppendTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncRandomReadTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncRandomWriteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncScanTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncSequentialReadTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncSequentialWriteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncTableTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$BufferedMutatorTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$CASTableTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$CheckAndDeleteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$CheckAndMutateTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$CheckAndPutTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$CleanMetaTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$CmdDescriptor.class
org/apache/hadoop/hbase/PerformanceEvaluation$Counter.class
org/apache/hadoop/hbase/PerformanceEvaluation$EvaluationMapTask$1.class
org/apache/hadoop/hbase/PerformanceEvaluation$EvaluationMapTask.class
org/apache/hadoop/hbase/PerformanceEvaluation$FilteredScanTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$IncrementTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$MetaRandomReadTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$MetaTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$MetaWriteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomDeleteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomReadTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomScanWithRange10000Test.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomScanWithRange1000Test.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomScanWithRange100Test.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomScanWithRange10Test.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomScanWithRangeTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomSeekScanTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomWriteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$ReverseScanTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$RunResult.class
org/apache/hadoop/hbase/PerformanceEvaluation$ScanTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$SequentialDeleteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$SequentialReadTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$SequentialWriteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$Status.class
org/apache/hadoop/hbase/PerformanceEvaluation$TableTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$Test.class
org/apache/hadoop/hbase/PerformanceEvaluation$TestBase.class
org/apache/hadoop/hbase/PerformanceEvaluation$TestOptions.class
org/apache/hadoop/hbase/PerformanceEvaluation.class
org/apache/hadoop/hbase/PerformanceEvaluationCommons.class
org/apache/hadoop/hbase/ScanPerformanceEvaluation$MyMapper.class
org/apache/hadoop/hbase/ScanPerformanceEvaluation$ScanCounter.class
org/apache/hadoop/hbase/ScanPerformanceEvaluation.class
org/apache/hadoop/hbase/master/balancer/LoadBalancerPerformanceEvaluation.class
org/apache/hadoop/hbase/util/DiagnosticToolsCommonUtils.class
org/apache/hadoop/hbase/util/LoadTestDataGeneratorWithMOB.class
org/apache/hadoop/hbase/util/LoadTestDataGeneratorWithTags.class
org/apache/hadoop/hbase/util/LoadTestTool$1.class
org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.class
org/apache/hadoop/hbase/util/LoadTestTool.class
org/apache/hadoop/hbase/util/LoadTestUtil.class
org/apache/hadoop/hbase/util/MultiThreadedAction$DefaultDataGenerator.class
org/apache/hadoop/hbase/util/MultiThreadedAction$ProgressReporter.class
org/apache/hadoop/hbase/util/MultiThreadedAction.class
org/apache/hadoop/hbase/util/MultiThreadedReader$HBaseReaderThread.class
org/apache/hadoop/hbase/util/MultiThreadedReader.class
org/apache/hadoop/hbase/util/MultiThreadedReaderWithACL$HBaseReaderThreadWithACL$1.class
org/apache/hadoop/hbase/util/MultiThreadedReaderWithACL$HBaseReaderThreadWithACL.class
org/apache/hadoop/hbase/util/MultiThreadedReaderWithACL.class
org/apache/hadoop/hbase/util/MultiThreadedUpdater$1.class
org/apache/hadoop/hbase/util/MultiThreadedUpdater$HBaseUpdaterThread.class
org/apache/hadoop/hbase/util/MultiThreadedUpdater.class
org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL$HBaseUpdaterThreadWithACL$1.class
org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL$HBaseUpdaterThreadWithACL$MutateAccessAction.class
org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL$HBaseUpdaterThreadWithACL.class
org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL.class
org/apache/hadoop/hbase/util/MultiThreadedWriter$HBaseWriterThread.class
org/apache/hadoop/hbase/util/MultiThreadedWriter.class
org/apache/hadoop/hbase/util/MultiThreadedWriterBase$WroteKeysTracker.class
org/apache/hadoop/hbase/util/MultiThreadedWriterBase.class
org/apache/hadoop/hbase/util/MultiThreadedWriterWithACL$HBaseWriterThreadWithACL$WriteAccessAction.class
org/apache/hadoop/hbase/util/MultiThreadedWriterWithACL$HBaseWriterThreadWithACL.class
org/apache/hadoop/hbase/util/MultiThreadedWriterWithACL.class
org/apache/hadoop/hbase/util/test/LoadTestDataGenerator.class
org/apache/hadoop/hbase/util/test/LoadTestDataGeneratorWithACL.class
META-INF/maven/org.apache.hbase/hbase-diagnostics/pom.xml
META-INF/maven/org.apache.hbase/hbase-diagnostics/pom.properties

CC: @stoty

hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtil.java

NihalJain · 2024-09-17T18:24:17Z

This PR is ready for review!

hbase-diagnostics/pom.xml

...src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerPerformanceEvaluation.java

hbase-asyncfs/src/test/java/org/apache/hadoop/hbase/security/HBaseKerberosUtils.java

hbase-diagnostics/src/main/java/org/apache/hadoop/hbase/util/LoadTestUtil.java

hbase-it/pom.xml

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java

ndimiduk

Thanks a lot for the effort here! This is a really great set of improvements.

This reminds me that I need to go back and read "Working Effectively with Legacy Code" by Michael Feathers.

hbase-assembly/src/main/assembly/hadoop-three-compat.xml

hbase-asyncfs/src/test/java/org/apache/hadoop/hbase/security/HBaseKerberosUtils.java

hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/KeyProviderForTesting.java

...src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerPerformanceEvaluation.java

hbase-diagnostics/src/main/java/org/apache/hadoop/hbase/util/DiagnosticToolsCommonUtils.java

hbase-diagnostics/src/main/java/org/apache/hadoop/hbase/util/LoadTestUtil.java

ndimiduk · 2024-09-18T08:05:25Z

hbase-diagnostics/src/main/java/org/apache/hadoop/hbase/util/LoadTestUtil.java

+  public static final String OPT_DATA_BLOCK_ENCODING =
+    ColumnFamilyDescriptorBuilder.DATA_BLOCK_ENCODING.toLowerCase(Locale.ROOT);
+  /** Column family used by the test */
+  public static byte[] DEFAULT_COLUMN_FAMILY = Bytes.toBytes("test_cf");


Details of the schema used by a particular utility should be isolated to that utility, right? Or is there some intention behind sharing schema across these utilities? I'd rather see these kinds of constants repeated in each utility where they're used.

Or maybe this change is big enough as it is, we can save further cleanup for a separate PR.

hbase-it/pom.xml

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java

ndimiduk · 2024-09-18T08:21:00Z

I should also say -- we shouldn't be shy about having more maven modules. Let's have a common module for scope:compile and a common module that is for scope:test. Let's even keep each executable tool to its own module with its own explicit dependency list. It'll make maintaining/pruning them easier in the long term. Plus every module becomes an independent, parallel target for mvn -T.

hbase-diagnostics/
├─ hbase-diagnostics-common/
├─ hbase-diagnostics-testing-common/
├─ hbase-performance-evaluation/
├─ hbase-load-test-tool/
├─ ...

stoty · 2024-09-18T08:24:35Z

I should also say -- we shouldn't be shy about having more maven modules. Let's have a common module for scope:compile and a common module that is for scope:test. Let's even keep each executable tool to its own module with its own explicit dependency list. It'll make maintaining/pruning them easier in the long term. Plus every module becomes an independent, parallel target for mvn -T.
hbase-diagnostics/
├─ hbase-diagnostics-common/
├─ hbase-diagnostics-testing-common/
├─ hbase-performance-evaluation/
├─ hbase-load-test-tool/
├─ ...

As Duo is starting 3.0 preparations, maybe we should leave that for a later ticket.
I'd prefer to make sure that these refactors land in 3.0.

NihalJain · 2024-09-19T15:57:59Z

Thank you @stoty and @ndimiduk for taking time to go over this. Let me go over each review comment one by one and post update.

hbase-diagnostics/src/main/java/org/apache/hadoop/hbase/util/WALPerformanceEvaluationUtil.java

hbase-diagnostics/src/main/java/org/apache/hadoop/hbase/wal/WALPerformanceEvaluation.java

NihalJain · 2024-09-19T19:18:15Z

Commit d6d6246 details:

Removed hbase-diagnostics test jar dependency from hbase-it
Updated ITs to have local util for all shared variables/confs
Cleaned up other classes in similar manner
Fixed test case failure due to log4j
Also added WALPerformanceEvaluation tool which was missed in previous change
Renamed DiagnosticToolsCommonUtils to KerberosUtils
Moved MockRegionServerServices to corresponding main code as needed by WALPerformanceEvaluation
Copied method required by WALPerformanceEvaluation to WALPerformanceEvaluationUtil (draft, need to see can we do any better here)
Added a new class out of TestFSHLogProvider and extracted only test dependent on WALPerformanceEvaluation

This PR may require multiple rounds of review, thank you for your patience. I will try to keep posting new commits in small bites based on feedbacks I receive here.

Should we also create a new module for chaos? I was thinking of doing it as another task of HBASE-28431, if needed? Or else can do i her itself, just that this PR will keep getting larger and larger

NihalJain · 2024-09-19T19:21:10Z

I should also say -- we shouldn't be shy about having more maven modules. Let's have a common module for scope:compile and a common module that is for scope:test. Let's even keep each executable tool to its own module with its own explicit dependency list. It'll make maintaining/pruning them easier in the long term. Plus every module becomes an independent, parallel target for mvn -T.
hbase-diagnostics/
├─ hbase-diagnostics-common/
├─ hbase-diagnostics-testing-common/
├─ hbase-performance-evaluation/
├─ hbase-load-test-tool/
├─ ...
As Duo is starting 3.0 preparations, maybe we should leave that for a later ticket. I'd prefer to make sure that these refactors land in 3.0.

We can decide if we should also target this in current PR. I am okay to update based on this suggested change either here or as another task.

hbase-diagnostics/src/main/java/org/apache/hadoop/hbase/util/KerberosUtils.java

stoty

I did this review a few days ago, but forgot to actually send it, and I was wondering why you haven't responded...

hbase-asyncfs/src/test/java/org/apache/hadoop/hbase/security/HBaseKerberosUtils.java

hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/KeyProviderForTesting.java

hbase-diagnostics/src/main/java/org/apache/hadoop/hbase/util/LoadTestUtil.java

hbase-diagnostics/src/main/java/org/apache/hadoop/hbase/util/WALPerformanceEvaluationUtil.java

...diagnostics/src/test/java/org/apache/hadoop/hbase/wal/TestBoundedRegionGroupingStrategy.java

hbase-it/pom.xml

stoty · 2024-10-12T14:09:07Z

Sure, no problem, I'm just worried that the refactors won't be ready in time for 3.0.

NihalJain · 2024-10-13T19:56:57Z

Sure, no problem, I'm just worried that the refactors won't be ready in time for 3.0.

Hey, I have update the PR as per review comment. Please see if we are good to go.

For this PR, built code, started local hbase instance and ran basic sanity for all of the above 6 tools.

Found issue during testing for WALPerformanceEvaluation, pushed changes as last commit, seems I missed to copy the base implementation. Fixed in latest commit.
Also raised https://issues.apache.org/jira/browse/HBASE-28913 as LoadBalancerPerformanceEvaluation is already broken.
Also please find below newly generated jars' content:

hbase %  jar -tf hbase-4.0.0-alpha-1-SNAPSHOT/lib/hbase-diagnostics-4.0.0-alpha-1-SNAPSHOT.jar
META-INF/MANIFEST.MF
META-INF/
org/
org/apache/
org/apache/hadoop/
org/apache/hadoop/hbase/
org/apache/hadoop/hbase/master/
org/apache/hadoop/hbase/master/balancer/
org/apache/hadoop/hbase/util/
org/apache/hadoop/hbase/util/test/
org/apache/hadoop/hbase/wal/
META-INF/maven/
META-INF/maven/org.apache.hbase/
META-INF/maven/org.apache.hbase/hbase-diagnostics/
META-INF/DEPENDENCIES
META-INF/LICENSE
META-INF/NOTICE
org/apache/hadoop/hbase/HFilePerformanceEvaluation$1.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$2.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$3.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$4.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$GaussianRandomReadBenchmark.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$ReadBenchmark.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$RowOrientedBenchmark.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$SequentialReadBenchmark.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$SequentialWriteBenchmark.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$UniformRandomReadBenchmark.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation$UniformRandomSmallScan.class
org/apache/hadoop/hbase/HFilePerformanceEvaluation.class
org/apache/hadoop/hbase/PerformanceEvaluation$1$1.class
org/apache/hadoop/hbase/PerformanceEvaluation$1.class
org/apache/hadoop/hbase/PerformanceEvaluation$AppendTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncRandomReadTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncRandomWriteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncScanTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncSequentialReadTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncSequentialWriteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncTableTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$AsyncTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$BufferedMutatorTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$CASTableTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$CheckAndDeleteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$CheckAndMutateTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$CheckAndPutTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$CleanMetaTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$CmdDescriptor.class
org/apache/hadoop/hbase/PerformanceEvaluation$Counter.class
org/apache/hadoop/hbase/PerformanceEvaluation$EvaluationMapTask$1.class
org/apache/hadoop/hbase/PerformanceEvaluation$EvaluationMapTask.class
org/apache/hadoop/hbase/PerformanceEvaluation$FilteredScanTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$IncrementTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$MetaRandomReadTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$MetaTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$MetaWriteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomDeleteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomReadTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomScanWithRange10000Test.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomScanWithRange1000Test.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomScanWithRange100Test.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomScanWithRange10Test.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomScanWithRangeTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomSeekScanTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$RandomWriteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$ReverseScanTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$RunResult.class
org/apache/hadoop/hbase/PerformanceEvaluation$ScanTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$SequentialDeleteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$SequentialReadTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$SequentialWriteTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$Status.class
org/apache/hadoop/hbase/PerformanceEvaluation$TableTest.class
org/apache/hadoop/hbase/PerformanceEvaluation$Test.class
org/apache/hadoop/hbase/PerformanceEvaluation$TestBase.class
org/apache/hadoop/hbase/PerformanceEvaluation$TestOptions.class
org/apache/hadoop/hbase/PerformanceEvaluation.class
org/apache/hadoop/hbase/PerformanceEvaluationCommons.class
org/apache/hadoop/hbase/ScanPerformanceEvaluation$MyMapper.class
org/apache/hadoop/hbase/ScanPerformanceEvaluation$ScanCounter.class
org/apache/hadoop/hbase/ScanPerformanceEvaluation.class
org/apache/hadoop/hbase/master/balancer/LoadBalancerPerformanceEvaluation.class
org/apache/hadoop/hbase/util/KerberosUtils.class
org/apache/hadoop/hbase/util/LoadTestDataGeneratorWithMOB.class
org/apache/hadoop/hbase/util/LoadTestDataGeneratorWithTags.class
org/apache/hadoop/hbase/util/LoadTestTool$1.class
org/apache/hadoop/hbase/util/LoadTestTool$WorkerThread.class
org/apache/hadoop/hbase/util/LoadTestTool.class
org/apache/hadoop/hbase/util/LoadTestUtil.class
org/apache/hadoop/hbase/util/MultiThreadedAction$DefaultDataGenerator.class
org/apache/hadoop/hbase/util/MultiThreadedAction$ProgressReporter.class
org/apache/hadoop/hbase/util/MultiThreadedAction.class
org/apache/hadoop/hbase/util/MultiThreadedReader$HBaseReaderThread.class
org/apache/hadoop/hbase/util/MultiThreadedReader.class
org/apache/hadoop/hbase/util/MultiThreadedReaderWithACL$HBaseReaderThreadWithACL$1.class
org/apache/hadoop/hbase/util/MultiThreadedReaderWithACL$HBaseReaderThreadWithACL.class
org/apache/hadoop/hbase/util/MultiThreadedReaderWithACL.class
org/apache/hadoop/hbase/util/MultiThreadedUpdater$1.class
org/apache/hadoop/hbase/util/MultiThreadedUpdater$HBaseUpdaterThread.class
org/apache/hadoop/hbase/util/MultiThreadedUpdater.class
org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL$HBaseUpdaterThreadWithACL$1.class
org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL$HBaseUpdaterThreadWithACL$MutateAccessAction.class
org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL$HBaseUpdaterThreadWithACL.class
org/apache/hadoop/hbase/util/MultiThreadedUpdaterWithACL.class
org/apache/hadoop/hbase/util/MultiThreadedWriter$HBaseWriterThread.class
org/apache/hadoop/hbase/util/MultiThreadedWriter.class
org/apache/hadoop/hbase/util/MultiThreadedWriterBase$WroteKeysTracker.class
org/apache/hadoop/hbase/util/MultiThreadedWriterBase.class
org/apache/hadoop/hbase/util/MultiThreadedWriterWithACL$HBaseWriterThreadWithACL$WriteAccessAction.class
org/apache/hadoop/hbase/util/MultiThreadedWriterWithACL$HBaseWriterThreadWithACL.class
org/apache/hadoop/hbase/util/MultiThreadedWriterWithACL.class
org/apache/hadoop/hbase/util/WALPerformanceEvaluationUtil.class
org/apache/hadoop/hbase/util/test/LoadTestDataGenerator.class
org/apache/hadoop/hbase/util/test/LoadTestDataGeneratorWithACL.class
org/apache/hadoop/hbase/wal/WALPerformanceEvaluation$1.class
org/apache/hadoop/hbase/wal/WALPerformanceEvaluation$WALPutBenchmark.class
org/apache/hadoop/hbase/wal/WALPerformanceEvaluation.class
META-INF/maven/org.apache.hbase/hbase-diagnostics/pom.xml
META-INF/maven/org.apache.hbase/hbase-diagnostics/pom.properties
hbase %

Also, let me resume progress for #6184, which is almost ready, so that we can finish these 2 tasks ASAP.

…dule hbase-tools

- Updated ITs to have local util for all shares data/confs - Cleaned up other classes in similar manner - Fix test case failure due to log4j - Also added WALPerformanceEvaluation tool which was missed in previous change - Renamed DiagnosticToolsCommonUtils to KerberosUtils - Moved MockRegionServerServices to corresponding main code as needed by WALPerformanceEvaluation - Copied method required by WALPerformanceEvaluation to WALPerformanceEvaluationUtil (draft, need to see can we do any better here) - Added a new class out of TestFSHLogProvider and extracted only test dependent on WALPerformanceEvaluation

- Rename KeyProviderForTesting.java to MockAesKeyProvider.java - Added class level comments for copied / moved code. - Break TestBoundedRegionGroupingStrategy to TestBoundedRegionGroupingStrategyWPETool refactoring tests needing WALPerformanceEvaluation into it

- Moved test dependencies after compile dependencies - Removed unused hbase-server dependency - Added class level comments for copied / moved code.

stoty

I think that we should add -diagnostics to the client tarball as well.
(Unless the client tarball is missing some of the dependencies required)

nit: for consistency, IMO we should also add the diagnostics test jar to the assembly, even though we plan to remove all of them in the next step.

NihalJain · 2024-10-14T07:02:14Z

I think that we should add -diagnostics to the client tarball as well.

Sure let me handle this

nit: for consistency, IMO we should also add the diagnostics test jar to the assembly, even though we plan to remove all of them in the next step.

I had checked this for other newer modules: we were not adding test jars for any of them, so I think it is okay to not add. As we anyways plan to remove again in follow up work

…y tarball

NihalJain · 2024-10-14T09:32:41Z

I think that we should add -diagnostics to the client tarball as well.

Sure let me handle this

Fixed.

hbase % ls hbase-4.0.0-alpha-1-SNAPSHOT*/lib/hbase-diag*      
hbase-4.0.0-alpha-1-SNAPSHOT-client/lib/hbase-diagnostics-4.0.0-alpha-1-SNAPSHOT.jar    hbase-4.0.0-alpha-1-SNAPSHOT/lib/hbase-diagnostics-4.0.0-alpha-1-SNAPSHOT.jar
n

stoty

+1 LGTM

Apache-HBase · 2024-10-14T10:54:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 39s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 24s		master passed
+1 💚	compile	8m 15s		master passed
+1 💚	checkstyle	1m 13s		master passed
+1 💚	spotbugs	12m 23s		master passed
+1 💚	spotless	0m 46s		branch has no errors when running spotless:check.
-0 ⚠️	patch	1m 44s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 17s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 7s		the patch passed
+1 💚	compile	8m 12s		the patch passed
-0 ⚠️	javac	8m 12s	/results-compile-javac-root.txt	root generated 63 new + 1214 unchanged - 1 fixed = 1277 total (was 1215)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 11s	/results-checkstyle-root.txt	root: The patch generated 46 new + 66 unchanged - 39 fixed = 112 total (was 105)
+1 💚	xmllint	0m 0s		No new issues.
-1 ❌	spotbugs	1m 48s	/new-spotbugs-hbase-server.html	hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌	spotbugs	7m 52s	/new-spotbugs-root.html	root generated 13 new + 0 unchanged - 0 fixed = 13 total (was 0)
+1 💚	hadoopcheck	11m 53s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 44s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	1m 32s		The patch does not generate ASF License warnings.
		77m 46s

Reason	Tests
SpotBugs	module:hbase-server
	Random object created and used only once in org.apache.hadoop.hbase.util.LoadTestKVGenerator.getValueForRowColumn(int, byte[][]) At LoadTestKVGenerator.java:only once in org.apache.hadoop.hbase.util.LoadTestKVGenerator.getValueForRowColumn(int, byte[][]) At LoadTestKVGenerator.java:[line 111]
SpotBugs	module:root
	Integral division result cast to double or float in org.apache.hadoop.hbase.PerformanceEvaluation.calculateRowsAndSize(PerformanceEvaluation$TestOptions) At PerformanceEvaluation.java:double or float in org.apache.hadoop.hbase.PerformanceEvaluation.calculateRowsAndSize(PerformanceEvaluation$TestOptions) At PerformanceEvaluation.java:[line 3154]
	org.apache.hadoop.hbase.PerformanceEvaluation$RunResult defines compareTo(PerformanceEvaluation$RunResult) and uses Object.equals() At PerformanceEvaluation.java:Object.equals() At PerformanceEvaluation.java:[line 250]
	Random object created and used only once in org.apache.hadoop.hbase.util.LoadTestKVGenerator.getValueForRowColumn(int, byte[][]) At LoadTestKVGenerator.java:only once in org.apache.hadoop.hbase.util.LoadTestKVGenerator.getValueForRowColumn(int, byte[][]) At LoadTestKVGenerator.java:[line 111]
	org.apache.hadoop.hbase.util.LoadTestTool.DEFAULT_NUM_REGIONS_PER_SERVER isn't final but should be At LoadTestTool.java:be At LoadTestTool.java:[line 167]
	org.apache.hadoop.hbase.util.MultiThreadedAction.verifyResultAgainstDataGenerator(Result, boolean, boolean) concatenates strings using + in a loop At MultiThreadedAction.java:using + in a loop At MultiThreadedAction.java:[line 415]
	Integral division result cast to double or float in org.apache.hadoop.hbase.util.MultiThreadedAction$ProgressReporter.run() At MultiThreadedAction.java:double or float in org.apache.hadoop.hbase.util.MultiThreadedAction$ProgressReporter.run() At MultiThreadedAction.java:[line 206]
	org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.createGet(long) concatenates strings using + in a loop At MultiThreadedReader.java:in a loop At MultiThreadedReader.java:[line 318]
	Dead store to rowKey in org.apache.hadoop.hbase.util.MultiThreadedReaderWithACL$HBaseReaderThreadWithACL.queryKey(Get, boolean, long) At MultiThreadedReaderWithACL.java:org.apache.hadoop.hbase.util.MultiThreadedReaderWithACL$HBaseReaderThreadWithACL.queryKey(Get, boolean, long) At MultiThreadedReaderWithACL.java:[line 91]
	Inconsistent synchronization of org.apache.hadoop.hbase.util.MultiThreadedUpdater.writer; locked 75% of time Unsynchronized access at MultiThreadedUpdater.java:75% of time Unsynchronized access at MultiThreadedUpdater.java:[line 80]
	Unwritten field:MultiThreadedUpdaterWithACL.java:[line 94]
	Exception is caught when Exception is not thrown in org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.run(String[]) At WALPerformanceEvaluation.java:is not thrown in org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.run(String[]) At WALPerformanceEvaluation.java:[line 244]
	Format string should use %n rather than n in org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.printUsageAndExit() At WALPerformanceEvaluation.java:rather than n in org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.printUsageAndExit() At WALPerformanceEvaluation.java:[line 414]
	Exception is caught when Exception is not thrown in org.apache.hadoop.hbase.wal.WALPerformanceEvaluation$WALPutBenchmark.run() At WALPerformanceEvaluation.java:is not thrown in org.apache.hadoop.hbase.wal.WALPerformanceEvaluation$WALPutBenchmark.run() At WALPerformanceEvaluation.java:[line 171]

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6258/9/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6258
Optional Tests	dupname asflicense javac codespell detsecrets xmllint hadoopcheck spotless compile spotbugs checkstyle hbaseanti
uname	Linux e8ba27ebb7d1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `01826de`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	190 (vs. ulimit of 30000)
modules	C: hbase-common hbase-client hbase-balancer hbase-asyncfs hbase-server hbase-mapreduce hbase-diagnostics hbase-testing-util hbase-it hbase-assembly . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6258/9/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3 xmllint=20913
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

NihalJain · 2024-10-14T11:01:03Z

Thank you @stoty and @ndimiduk for your patience with reviewing this.

I plan to merge this one to master soon. We already have one approval by @stoty, so should be good to go. Still will keep this PR open for EOD and merge it to code base, if no new reviews / objections.

Apache-HBase · 2024-10-14T15:56:35Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 16s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 30s		master passed
+1 💚	compile	2m 37s		master passed
+1 💚	javadoc	4m 48s		master passed
+1 💚	shadedjars	6m 50s		branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	patch	8m 1s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 20s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 43s		the patch passed
+1 💚	compile	2m 43s		the patch passed
+1 💚	javac	2m 43s		the patch passed
-0 ⚠️	javadoc	0m 20s	/results-javadoc-javadoc-hbase-common.txt	hbase-common generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)
-0 ⚠️	javadoc	2m 30s	/results-javadoc-javadoc-root.txt	root generated 2 new + 93 unchanged - 0 fixed = 95 total (was 93)
+1 💚	shadedjars	6m 39s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	333m 15s		root in the patch passed.
		380m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6258/9/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6258
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 6f2e7fceb301 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `01826de`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6258/9/testReport/
Max. process+thread count	8413 (vs. ulimit of 30000)
modules	C: hbase-common hbase-client hbase-balancer hbase-asyncfs hbase-server hbase-mapreduce hbase-diagnostics hbase-testing-util . hbase-assembly hbase-it U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6258/9/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

…dule hbase-diagnostics (apache#6258) - Move PerformanceEvaluation, LoadTestTool, HFilePerformanceEvaluation, ScanPerformanceEvaluation, LoadBalancerPerformanceEvaluation, and WALPerformanceEvaluation to a new module: hbase-diagnostics Signed-off-by: Istvan Toth <[email protected]> Signed-off-by: Nick Dimiduk <[email protected]> (cherry picked from commit 84f4fb3)

…dule hbase-diagnostics (#6258) (#6367) - Move PerformanceEvaluation, LoadTestTool, HFilePerformanceEvaluation, ScanPerformanceEvaluation, LoadBalancerPerformanceEvaluation, and WALPerformanceEvaluation to a new module: hbase-diagnostics Signed-off-by: Istvan Toth <[email protected]> Signed-off-by: Nick Dimiduk <[email protected]> (cherry picked from commit 84f4fb3)

NihalJain requested review from Apache9, bbeaudreault, ndimiduk and stoty September 17, 2024 18:07

NihalJain commented Sep 17, 2024

View reviewed changes

hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtil.java Show resolved Hide resolved

NihalJain force-pushed the HBASE-28432_final branch from 6972bdb to 96dd5a6 Compare September 17, 2024 18:22

NihalJain commented Sep 17, 2024

View reviewed changes

hbase-diagnostics/pom.xml Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

stoty reviewed Sep 18, 2024

View reviewed changes

ndimiduk reviewed Sep 18, 2024

View reviewed changes

NihalJain commented Sep 19, 2024

View reviewed changes

hbase-diagnostics/src/main/java/org/apache/hadoop/hbase/util/WALPerformanceEvaluationUtil.java Show resolved Hide resolved

NihalJain commented Sep 19, 2024

View reviewed changes

hbase-diagnostics/src/main/java/org/apache/hadoop/hbase/wal/WALPerformanceEvaluation.java Show resolved Hide resolved

NihalJain commented Sep 19, 2024

View reviewed changes

hbase-diagnostics/src/main/java/org/apache/hadoop/hbase/util/KerberosUtils.java Show resolved Hide resolved

This comment has been minimized.

Sign in to view

stoty requested changes Sep 26, 2024

View reviewed changes

NihalJain requested a review from stoty October 13, 2024 19:57

This comment has been minimized.

Sign in to view

NihalJain and others added 7 commits October 14, 2024 01:56

HBASE-28432 Refactor tools which are under test packaging to a new mo…

fe68158

…dule hbase-tools

Remove copies and move depenendent files to coprresponding main modules

090682c

Fix build failure due to RestrictedApi

a55c6b6

Address review comments:

0f789ae

- Rename KeyProviderForTesting.java to MockAesKeyProvider.java - Added class level comments for copied / moved code. - Break TestBoundedRegionGroupingStrategy to TestBoundedRegionGroupingStrategyWPETool refactoring tests needing WALPerformanceEvaluation into it

Address review comments:

9d3341d

- Moved test dependencies after compile dependencies - Removed unused hbase-server dependency - Added class level comments for copied / moved code.

Fix issue in WALPerformanceEvaluation discovered during testing

ad4e429

This comment has been minimized.

Sign in to view

stoty reviewed Oct 14, 2024

View reviewed changes

Fix review comment: Add hbase-diagnostics jar to hbase-client assembl…

01826de

…y tarball

NihalJain force-pushed the HBASE-28432_final branch from ca8bb3d to 01826de Compare October 14, 2024 09:31

NihalJain requested a review from stoty October 14, 2024 09:33

stoty approved these changes Oct 14, 2024

View reviewed changes

ndimiduk approved these changes Oct 14, 2024

View reviewed changes

NihalJain mentioned this pull request Oct 14, 2024

HBASE-28433 Modify the assembly to not include test jars and their tr… #6184

Merged

NihalJain merged commit 84f4fb3 into apache:master Oct 15, 2024

NihalJain mentioned this pull request Oct 15, 2024

HBASE-28432 Refactor tools which are under test packaging to a new module hbase-diagnostics #6367

Merged

HBASE-28432 Refactor tools which are under test packaging to a new module hbase-diagnostics #6258

HBASE-28432 Refactor tools which are under test packaging to a new module hbase-diagnostics #6258

Uh oh!

Conversation

NihalJain commented Sep 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NihalJain commented Sep 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

NihalJain commented Sep 17, 2024

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ndimiduk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ndimiduk Sep 18, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ndimiduk commented Sep 18, 2024

Uh oh!

stoty commented Sep 18, 2024

Uh oh!

NihalJain commented Sep 19, 2024

Uh oh!

Uh oh!

Uh oh!

NihalJain commented Sep 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NihalJain commented Sep 19, 2024

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

stoty left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stoty commented Oct 12, 2024

Uh oh!

NihalJain commented Oct 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

stoty left a comment

Choose a reason for hiding this comment

Uh oh!

NihalJain commented Sep 17, 2024 •

edited

Loading

NihalJain commented Sep 17, 2024 •

edited

Loading

NihalJain commented Sep 19, 2024 •

edited

Loading

stoty left a comment •

edited

Loading

NihalJain commented Oct 13, 2024 •

edited

Loading

NihalJain commented Oct 14, 2024 •

edited

Loading