HDDS-6202. Avoid using jmh-generator-annprocess since it is GPL2.0.

CONTRIBUTING.md

@@ -20,7 +20,8 @@ We welcome contributions of:
       * Unit Tests (JUnit / Java)
       * Acceptance Tests (docker + robot framework)
       * Blockade tests (python + blockade) 
-      * Performance: We have multiple type of load generator / benchmark tools (`ozone freon`, `ozone genesis`), which can be used to test cluster and report problems.
+      * Performance: We have multiple type of load generator / benchmark tools (`ozone freon`),
+        which can be used to test cluster and report problems.
  * **Bug reports** pointing out broken functionality, docs, or suggestions for improvements are always welcome!
 
 ## Who To Contact

hadoop-hdds/docs/content/tools/TestTools.md

@@ -106,131 +106,4 @@ Average Time spent in key write: 00:00:10,894
 Total bytes written: 10240000
 Total Execution time: 00:00:16,898
 ***********************
-```
-
-## Genesis
-
-Genesis is a microbenchmarking tool. It's also included in the distribution (`ozone genesis`) but it doesn't require real cluster. It measures different part of the code in an isolated way (eg. the code which saves the data to the local RocksDB based key value stores)
-
-Example run:
-
-```
- ozone genesis -benchmark=BenchMarkRocksDbStore
-# JMH version: 1.19
-# VM version: JDK 11.0.1, VM 11.0.1+13-LTS
-# VM invoker: /usr/lib/jvm/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/bin/java
-# VM options: -Dproc_genesis -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
-# Warmup: 2 iterations, 1 s each
-# Measurement: 20 iterations, 1 s each
-# Timeout: 10 min per iteration
-# Threads: 4 threads, will synchronize iterations
-# Benchmark mode: Throughput, ops/time
-# Benchmark: org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test
-# Parameters: (backgroundThreads = 4, blockSize = 8, maxBackgroundFlushes = 4, maxBytesForLevelBase = 512, maxOpenFiles = 5000, maxWriteBufferNumber = 16, writeBufferSize = 64)
-
-# Run progress: 0.00% complete, ETA 00:00:22
-# Fork: 1 of 1
-# Warmup Iteration   1: 213775.360 ops/s
-# Warmup Iteration   2: 32041.633 ops/s
-Iteration   1: 196342.348 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   2: 41926.816 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   3: 210433.231 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   4: 46941.951 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   5: 212825.884 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   6: 145914.351 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   7: 141838.469 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   8: 205334.438 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   9: 163709.519 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  10: 162494.608 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  11: 199155.793 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  12: 209679.298 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  13: 193787.574 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  14: 127004.147 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  15: 145511.080 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  16: 223433.864 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  17: 169752.665 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  18: 165217.191 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  19: 191038.476 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  20: 196335.579 ops/s
-                 ?stack: <delayed till summary>
-
-
-
-Result "org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test":
-  167433.864 ?(99.9%) 43530.883 ops/s [Average]
-  (min, avg, max) = (41926.816, 167433.864, 223433.864), stdev = 50130.230
-  CI (99.9%): [123902.981, 210964.748] (assumes normal distribution)
-
-Secondary result "org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test:?stack":
-Stack profiler:
-
-....[Thread state distributions]....................................................................
- 78.9%         RUNNABLE
- 20.0%         TIMED_WAITING
-  1.1%         WAITING
-
-....[Thread state: RUNNABLE]........................................................................
- 59.8%  75.8% org.rocksdb.RocksDB.put
- 16.5%  20.9% org.rocksdb.RocksDB.get
-  0.7%   0.9% java.io.UnixFileSystem.delete0
-  0.7%   0.9% org.rocksdb.RocksDB.disposeInternal
-  0.3%   0.4% java.lang.Long.formatUnsignedLong0
-  0.1%   0.2% org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test
-  0.1%   0.1% java.lang.Long.toUnsignedString0
-  0.1%   0.1% org.apache.hadoop.ozone.genesis.generated.BenchMarkRocksDbStore_test_jmhTest.test_thrpt_jmhStub
-  0.0%   0.1% java.lang.Object.clone
-  0.0%   0.0% java.lang.Thread.currentThread
-  0.4%   0.5% <other>
-
-....[Thread state: TIMED_WAITING]...................................................................
- 20.0% 100.0% java.lang.Object.wait
-
-....[Thread state: WAITING].........................................................................
-  1.1% 100.0% jdk.internal.misc.Unsafe.park
-
-
-
-# Run complete. Total time: 00:00:38
-
-Benchmark                          (backgroundThreads)  (blockSize)  (maxBackgroundFlushes)  (maxBytesForLevelBase)  (maxOpenFiles)  (maxWriteBufferNumber)  (writeBufferSize)   Mode  Cnt       Score       Error  Units
-BenchMarkRocksDbStore.test                           4            8                       4                     512            5000                      16                 64  thrpt   20  167433.864 ? 43530.883  ops/s
-BenchMarkRocksDbStore.test:?stack                    4            8                       4                     512            5000                      16                 64  thrpt              NaN                ---
-```
+```

hadoop-hdds/docs/content/tools/TestTools.zh.md

@@ -107,131 +107,4 @@ Average Time spent in key write: 00:00:10,894
 Total bytes written: 10240000
 Total Execution time: 00:00:16,898
 ***********************
-```
-
-## Genesis
-
-Genesis 是一个微型的基准测试工具，它也包含在发行包中（`ozone genesis`），但是它不需要一个真实的集群，而是采用一种隔离的方法测试不同部分的代码（比如，将数据存储到本地基于 RocksDB 的键值存储中）。
-
-运行示例：
-
-```
- ozone genesis -benchmark=BenchMarkRocksDbStore
-# JMH version: 1.19
-# VM version: JDK 11.0.1, VM 11.0.1+13-LTS
-# VM invoker: /usr/lib/jvm/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/bin/java
-# VM options: -Dproc_genesis -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
-# Warmup: 2 iterations, 1 s each
-# Measurement: 20 iterations, 1 s each
-# Timeout: 10 min per iteration
-# Threads: 4 threads, will synchronize iterations
-# Benchmark mode: Throughput, ops/time
-# Benchmark: org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test
-# Parameters: (backgroundThreads = 4, blockSize = 8, maxBackgroundFlushes = 4, maxBytesForLevelBase = 512, maxOpenFiles = 5000, maxWriteBufferNumber = 16, writeBufferSize = 64)
-
-# Run progress: 0.00% complete, ETA 00:00:22
-# Fork: 1 of 1
-# Warmup Iteration   1: 213775.360 ops/s
-# Warmup Iteration   2: 32041.633 ops/s
-Iteration   1: 196342.348 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   2: 41926.816 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   3: 210433.231 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   4: 46941.951 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   5: 212825.884 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   6: 145914.351 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   7: 141838.469 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   8: 205334.438 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration   9: 163709.519 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  10: 162494.608 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  11: 199155.793 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  12: 209679.298 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  13: 193787.574 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  14: 127004.147 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  15: 145511.080 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  16: 223433.864 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  17: 169752.665 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  18: 165217.191 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  19: 191038.476 ops/s
-                 ?stack: <delayed till summary>
-
-Iteration  20: 196335.579 ops/s
-                 ?stack: <delayed till summary>
-
-
-
-Result "org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test":
-  167433.864 ?(99.9%) 43530.883 ops/s [Average]
-  (min, avg, max) = (41926.816, 167433.864, 223433.864), stdev = 50130.230
-  CI (99.9%): [123902.981, 210964.748] (assumes normal distribution)
-
-Secondary result "org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test:?stack":
-Stack profiler:
-
-....[Thread state distributions]....................................................................
- 78.9%         RUNNABLE
- 20.0%         TIMED_WAITING
-  1.1%         WAITING
-
-....[Thread state: RUNNABLE]........................................................................
- 59.8%  75.8% org.rocksdb.RocksDB.put
- 16.5%  20.9% org.rocksdb.RocksDB.get
-  0.7%   0.9% java.io.UnixFileSystem.delete0
-  0.7%   0.9% org.rocksdb.RocksDB.disposeInternal
-  0.3%   0.4% java.lang.Long.formatUnsignedLong0
-  0.1%   0.2% org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test
-  0.1%   0.1% java.lang.Long.toUnsignedString0
-  0.1%   0.1% org.apache.hadoop.ozone.genesis.generated.BenchMarkRocksDbStore_test_jmhTest.test_thrpt_jmhStub
-  0.0%   0.1% java.lang.Object.clone
-  0.0%   0.0% java.lang.Thread.currentThread
-  0.4%   0.5% <other>
-
-....[Thread state: TIMED_WAITING]...................................................................
- 20.0% 100.0% java.lang.Object.wait
-
-....[Thread state: WAITING].........................................................................
-  1.1% 100.0% jdk.internal.misc.Unsafe.park
-
-
-
-# Run complete. Total time: 00:00:38
-
-Benchmark                          (backgroundThreads)  (blockSize)  (maxBackgroundFlushes)  (maxBytesForLevelBase)  (maxOpenFiles)  (maxWriteBufferNumber)  (writeBufferSize)   Mode  Cnt       Score       Error  Units
-BenchMarkRocksDbStore.test                           4            8                       4                     512            5000                      16                 64  thrpt   20  167433.864 ? 43530.883  ops/s
-BenchMarkRocksDbStore.test:?stack                    4            8                       4                     512            5000                      16                 64  thrpt              NaN                ---
-```
+```

hadoop-hdds/docs/content/tools/_index.md

@@ -62,6 +62,5 @@ Admin commands:
 Test tools:
 
    * **freon** -  Runs the ozone load generator.
-   * **genesis**  - Developer Only, Ozone micro-benchmark application.
 
  For more information see the following subpages:

hadoop-hdds/docs/content/tools/_index.zh.md

@@ -57,6 +57,5 @@ Ozone 有一系列管理 Ozone 的命令行工具。
 测试工具：
 
    * **freon** -  运行 Ozone 负载生成器。
-   * **genesis**  - Ozone 的 benchmark 应用，仅供开发者使用。
 
 更多信息请参见下面的子页面：

hadoop-hdds/server-scm/pom.xml

@@ -106,11 +106,6 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
       <artifactId>assertj-core</artifactId>
       <scope>test</scope>
     </dependency>
-    <dependency>
-      <groupId>org.openjdk.jmh</groupId>
-      <artifactId>jmh-generator-annprocess</artifactId>
-      <scope>test</scope>
-    </dependency>
     <dependency>
       <groupId>org.mockito</groupId>
       <artifactId>mockito-core</artifactId>

hadoop-ozone/dev-support/checks/coverage.sh

@@ -51,7 +51,6 @@ find target/coverage-classes -name proto -type d | xargs rm -rf
 find target/coverage-classes -name generated -type d | xargs rm -rf
 find target/coverage-classes -name v1 -type d | xargs rm -rf
 find target/coverage-classes -name freon -type d | xargs rm -rf
-find target/coverage-classes -name genesis -type d | xargs rm -rf
 
 #generate the reports
 jacoco report "$REPORT_DIR/jacoco-all.exec" --classfiles target/coverage-classes --html "$REPORT_DIR/all" --xml "$REPORT_DIR/all.xml"

hadoop-ozone/dist/src/shell/ozone/ozone

@@ -43,7 +43,6 @@ function ozone_usage
   ozone_add_subcommand "freon" client "runs an ozone data generator"
   ozone_add_subcommand "fs" client "run a filesystem command on Ozone file system. Equivalent to 'hadoop fs'"
   ozone_add_subcommand "genconf" client "generate minimally required ozone configs and output to ozone-site.xml in specified path"
-  ozone_add_subcommand "genesis" client "runs a collection of ozone benchmarks to help with tuning."
   ozone_add_subcommand "getconf" client "get ozone config values from configuration"
   ozone_add_subcommand "jmxget" admin "get JMX exported values from NameNode or DataNode."
   ozone_add_subcommand "om" daemon "Ozone Manager"
@@ -133,22 +132,6 @@ function ozonecmd_case
       OZONE_FREON_OPTS="${OZONE_FREON_OPTS}"
       OZONE_RUN_ARTIFACT_NAME="ozone-tools"
     ;;
-    genesis)
-      ARTIFACT_LIB_DIR="${OZONE_HOME}/share/ozone/lib/ozone-tools"
-      mkdir -p "$ARTIFACT_LIB_DIR"
-      if [[ ! -f "$ARTIFACT_LIB_DIR/jmh-core-1.23.jar" ]]; then
-        echo "jmh-core jar is missing from $ARTIFACT_LIB_DIR, trying to download from maven central (License: GPL + classpath exception)"
-        curl -o "$ARTIFACT_LIB_DIR/jmh-core-1.23.jar"  https://repo1.maven.org/maven2/org/openjdk/jmh/jmh-core/1.23/jmh-core-1.23.jar
-      fi
-
-      if [[ ! -f "$ARTIFACT_LIB_DIR/jopt-simple-4.6.jar" ]]; then
-        echo "jopt jar is missing from $ARTIFACT_LIB_DIR, trying to download from maven central (License: MIT License)"
-        curl -o "$ARTIFACT_LIB_DIR/jopt-simple-4.6.jar" https://repo1.maven.org/maven2/net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6.jar
-      fi
-
-      OZONE_CLASSNAME=org.apache.hadoop.ozone.genesis.Genesis
-      OZONE_RUN_ARTIFACT_NAME="ozone-tools"
-    ;;
     getconf)
       OZONE_CLASSNAME=org.apache.hadoop.ozone.conf.OzoneGetConf;
       OZONE_RUN_ARTIFACT_NAME="ozone-tools"

hadoop-ozone/insight/pom.xml

@@ -42,7 +42,6 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
       <artifactId>hdds-test-utils</artifactId>
       <scope>test</scope>
     </dependency>
-    <!-- Genesis requires server side components -->
     <dependency>
       <groupId>org.apache.ozone</groupId>
       <artifactId>hdds-server-scm</artifactId>

hadoop-ozone/integration-test/pom.xml

@@ -123,11 +123,6 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
       <artifactId>junit-platform-launcher</artifactId>
       <scope>test</scope>
     </dependency>
-    <dependency>
-      <groupId>org.openjdk.jmh</groupId>
-      <artifactId>jmh-generator-annprocess</artifactId>
-      <scope>test</scope>
-    </dependency>
     <dependency>
       <groupId>org.mockito</groupId>
       <artifactId>mockito-core</artifactId>

hadoop-ozone/tools/dev-support/findbugsExcludeFile.xml

@@ -13,10 +13,6 @@
   limitations under the License. See accompanying LICENSE file.
 -->
 <FindBugsFilter>
-    <Match>
-      <Package name="org.apache.hadoop.ozone.genesis.generated" />
-    </Match>
-
     <!-- Test -->
     <Match>
       <Class name="org.apache.hadoop.ozone.recon.TestReconUtils"/>

hadoop-ozone/tools/pom.xml

@@ -42,11 +42,6 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
       <artifactId>hdds-test-utils</artifactId>
       <scope>test</scope>
     </dependency>
-    <!-- Genesis requires server side components -->
-    <dependency>
-      <groupId>org.apache.ozone</groupId>
-      <artifactId>hdds-server-scm</artifactId>
-    </dependency>
     <dependency>
       <groupId>org.apache.ozone</groupId>
       <artifactId>hdds-tools</artifactId>
@@ -93,11 +88,6 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
       <groupId>javax.activation</groupId>
       <artifactId>activation</artifactId>
     </dependency>
-    <dependency>
-      <groupId>org.openjdk.jmh</groupId>
-      <artifactId>jmh-generator-annprocess</artifactId>
-      <scope>provided</scope>
-    </dependency>
     <dependency>
       <groupId>io.dropwizard.metrics</groupId>
       <artifactId>metrics-core</artifactId>