HDDS-6202. Avoid using jmh-generator-annprocess since it is GPL2.0.

[szetszwo]

What changes were proposed in this pull request?

jmh-generator-annprocess is GPL2.0. It is incompatible with Apache License 2.0. It is currently only used by genesis (a micro-benchmark tool).

We may have to rewrite the tool or move it out from the main Ozone code base.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6202

How was this patch tested?

NA

[adoroszlai]

Thanks @szetszwo for the patch.

Further references which I think should be removed:

ozone/hadoop-ozone/dist/src/shell/ozone/ozone

Lines 136 to 151 in a59c64c

	genesis)
	ARTIFACT_LIB_DIR="${OZONE_HOME}/share/ozone/lib/ozone-tools"
	mkdir -p "$ARTIFACT_LIB_DIR"
	if [[ ! -f "$ARTIFACT_LIB_DIR/jmh-core-1.23.jar" ]]; then
	echo "jmh-core jar is missing from $ARTIFACT_LIB_DIR, trying to download from maven central (License: GPL + classpath exception)"
	curl -o "$ARTIFACT_LIB_DIR/jmh-core-1.23.jar" https://repo1.maven.org/maven2/org/openjdk/jmh/jmh-core/1.23/jmh-core-1.23.jar
	fi
	if [[ ! -f "$ARTIFACT_LIB_DIR/jopt-simple-4.6.jar" ]]; then
	echo "jopt jar is missing from $ARTIFACT_LIB_DIR, trying to download from maven central (License: MIT License)"
	curl -o "$ARTIFACT_LIB_DIR/jopt-simple-4.6.jar" https://repo1.maven.org/maven2/net/sf/jopt-simple/jopt-simple/4.6/jopt-simple-4.6.jar
	fi
	OZONE_CLASSNAME=org.apache.hadoop.ozone.genesis.Genesis
	OZONE_RUN_ARTIFACT_NAME="ozone-tools"
	;;

ozone/hadoop-ozone/dist/src/shell/ozone/ozone

Line 46 in a59c64c

ozone_add_subcommand "genesis" client "runs a collection of ozone benchmarks to help with tuning."

ozone/hadoop-ozone/tools/dev-support/findbugsExcludeFile.xml

Lines 16 to 18 in a59c64c

	<Match>
	<Package name="org.apache.hadoop.ozone.genesis.generated" />
	</Match>

ozone/hadoop-ozone/dev-support/checks/coverage.sh

Line 54 in a59c64c

find target/coverage-classes -name genesis -type d | xargs rm -rf

• https://github.com/apache/ozone/blob/a59c64cccbfec49f7178073e477fce6102596aec/hadoop-hdds/docs/content/tools/TestTools.md#genesis
• https://github.com/apache/ozone/blob/a59c64cccbfec49f7178073e477fce6102596aec/hadoop-hdds/docs/content/tools/TestTools.zh.md#genesis

CONTRIBUTING.md
23:      * Performance: We have multiple type of load generator / benchmark tools (`ozone freon`, `ozone genesis`), which can be used to test cluster and report problems.

hadoop-hdds/docs/content/tools/_index.md
65:   * **genesis**  - Developer Only, Ozone micro-benchmark application.

hadoop-hdds/docs/content/tools/_index.zh.md
60:   * **genesis**  - Ozone 的 benchmark 应用，仅供开发者使用。

[szetszwo]

@adoroszlai , thanks a lot for reviewing this.

Suppose genesis is a useful tool. We probably need to move the genesis code to a different maven module or even a different repo. By default, genesis won't be included with Ozone release. It will be available somewhere. Then, we may download it on demand. This is similar to the jmh-core jar in the ozone script.

ozone/hadoop-ozone/dist/src/shell/ozone/ozone

Lines 139 to 142 in a59c64c

	if [[ ! -f "$ARTIFACT_LIB_DIR/jmh-core-1.23.jar" ]]; then
	echo "jmh-core jar is missing from $ARTIFACT_LIB_DIR, trying to download from maven central (License: GPL + classpath exception)"
	curl -o "$ARTIFACT_LIB_DIR/jmh-core-1.23.jar" https://repo1.maven.org/maven2/org/openjdk/jmh/jmh-core/1.23/jmh-core-1.23.jar
	fi

The current genesis doc and scripts may be still useful. So, how about we fix them separately?

[adoroszlai]

Suppose genesis is a useful tool. We probably need to move the genesis code to a different maven module or even a different repo. By default, genesis won't be included with Ozone release. It will be available somewhere. Then, we may download it on demand. This is similar to the jmh-core jar in the ozone script.

I think we should move Genesis to a separate repo with different license. (I'm not sure if Apache allows such license, nor about the mechanics of a license change.)

Separate project is also recommended by JMH, too: "The recommended way to run a JMH benchmark is to use Maven to setup a standalone project that depends on the jar files of your application."

The current genesis doc and scripts may be still useful. So, how about we fix them separately?

ozone genesis won't work because Genesis.java is being removed. I think leaving the script around would be confusing. I'm OK with fixing it separately, though. Please let me know if I should work on that part.

[szetszwo]

ozone genesis won't work because Genesis.java is being removed.

It could possibly work:

1. Suppose genesis.jar, which contains Genesis.class and other classes, is available in maven or somewhere else.
2. Use curl to download the jar in the ozone script.

curl -o "$ARTIFACT_LIB_DIR/genesis.jar"  https://repo1.maven.org/maven2/org/apache/ozone/.../genesis.jar

3. Then, Genesis.class is available locally. It can be executed as usual.

Of course, I am open to use other ways running it.

[szetszwo]

... Please let me know if I should work on that part.

@adoroszlai , It would be great if you are interested in working on that!

[adoroszlai]

ozone genesis won't work because Genesis.java is being removed.

It could possibly work:

1. Suppose genesis.jar, which contains Genesis.class and other classes, is available in maven or somewhere else.

Got it, but my point is it will be broken until we create this separate repo and make it available.

[szetszwo]

@adoroszlai , Okay, just have removed the genesis scripts and docs.

[adoroszlai]

Thanks @szetszwo for updating the patch.

[sodonnel]

For the record, I strongly disagree with how this was handled. Useful code was just removed here because you could not be bothered to setup something else to house the code. I don't recall seeing anything on a mailing list about this.

I get that is should not have been committed in the first place, but it was, and as one of the people who contributed to these benchmarks I feel quit insulted it was just thrown away.

I have now went looking for code to benchmark a change to an area one of these covered and have seen quite a bit of inconvenience trying to track it down.

[sodonnel]

I am also inclined to say this change should be reverted and handled correctly, moving the code elsewhere and notifying the community of its location and a strategy for adding such benchmarks in the future.

[szetszwo]

... Useful code was just removed here ...

@sodonnel , The code may be useful but it violates Apache policy; see https://www.apache.org/legal/resolved.html#category-x . We cannot include it in any Apache projects. Sorry!

I get that is should not have been committed in the first place, ...

Usually, we will reopen the JIRAs and revert the commits. However, there were multiple commits done long time ago. Reverting them becomes impossible or very hard. In such case, we usually remove the code in a separated JIRA. Please don't take it personal.

[szetszwo]

I am also inclined to say this change should be reverted and handled correctly, moving the code elsewhere and notifying the community of its location and a strategy for adding such benchmarks in the future.

I am totally fine if we want to revert this pull request. No hard feelings at all.

[adoroszlai]

@sodonnel I apologize for removing the benchmarks from this repo. I have filtered the commits related to Genesis to a separate repo. It's a personal one for now, but we can work on moving it to apache.