HDDS-4452. findbugs.sh couldn't be executed after a full build

[elek]

What changes were proposed in this pull request?

./hadoop-ozone/dev-support/checks/findbugs.sh – which is a short-cut to execute the CI findbugs check locally – couldn't be executed locally after a full build:

./hadoop-ozone/dev-support/checks/findbugs.sh
....
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.451 s
[INFO] Finished at: 2020-11-11T11:42:40+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.spotbugs:spotbugs-maven-plugin:3.1.12:spotbugs (spotbugs) on project hadoop-hdds: Execution spotbugs of goal com.github.spotbugs:spotbugs-maven-plugin:3.1.12:spotbugs failed: Java returned: 1 -> [Help 1]
[ERROR] 

The problem:

target/classes directory should be either empty/missing or it should contain java classes to make spotbugs work.

On github it works well as an empty checkout is tested. But locally it's possible that a dummy classpath file is created under hadoop-hdds/target/classes which breaks spotbug local execution.

The solution is easy: execute the classpath descriptor generation only if src/main/java dir exists.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-4452

How was this patch tested?

mvn clean install -DskipTests
./hadoop-ozone/dev-support/checks/findbugs.sh

Without the patch it fails as there is a classpath file under hadoop-hdds/target/classes

[elek]

The plugin execution is moved to a profile which is acivated automatically by this fragment:

      <activation>
        <file>
          <exists>src/main/java</exists>
        </file>
      </activation>

Which means the profile will be activated only for the projects where java code exists.

[adoroszlai]

Looks like hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/share/ozone/lib is missing after the build. These two steps are executed in the wrong order:

2020-11-12T09:09:33.6324111Z [INFO] --- maven-dependency-plugin:3.1.1:copy-dependencies (copy-jars) @ hadoop-ozone-dist ---
2020-11-12T09:09:33.6368628Z [INFO] Copying kerb-simplekdc-1.0.1.jar to /mnt/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/share/ozone/lib/kerb-simplekdc-1.0.1.jar
2020-11-12T09:09:33.6419123Z [INFO] Copying hk2-utils-2.5.0.jar to /mnt/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/share/ozone/lib/hk2-utils-2.5.0.jar
...
2020-11-12T09:09:37.3442300Z [INFO] --- exec-maven-plugin:1.3.1:exec (dist) @ hadoop-ozone-dist ---
2020-11-12T09:09:37.6671877Z cp: cannot stat '/mnt/ozone/hadoop-hdds/docs/target/classes/docs': No such file or directory
2020-11-12T09:09:37.6677468Z 
2020-11-12T09:09:37.6691337Z Current directory /mnt/ozone/hadoop-ozone/dist/target
2020-11-12T09:09:37.6692296Z 
2020-11-12T09:09:37.6693293Z $ rm -rf ozone-1.1.0-SNAPSHOT

[elek]

Looks like hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/share/ozone/lib is missing after the build. These two steps are executed in the wrong order:

Seems to be a bug which was not visible until now because the classpath file generation.

[adoroszlai]

Now order is OK, but classpath files are generated into the wrong dir, so they are missing from the jars, and then from the share/ozone/classpath dir, too.

[elek]

Now order is OK, but classpath files are generated into the wrong dir

Oh, that was my first idea to fix it (use different output dir), and it's not supposed to be included in the patch as it would make the modifications more complex. (Added your commits, thanks the suggestions.)

[adoroszlai]

Thanks @elek for updating the patch. Acceptance tests are still red, datanodes fail to start:

datanode_1  | Error: Could not find or load main class org.apache.hadoop.ozone.HddsDatanodeService
datanode_1  | Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.ozone.HddsDatanodeService

because datanode has no classpath file:

bash-4.2$ ozone classpath hadoop-ozone-datanode
/opt/hadoop/bin/../libexec/hadoop-functions.sh: line 2776: /opt/hadoop/share/ozone/classpath/hadoop-ozone-datanode.classpath: No such file or directory
Error: Could not find or load main class org.apache.hadoop.util.Classpath
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.Classpath

as there is only a pom.xml file in hadoop-ozone/datanode.

[adoroszlai]

Latest patch LGTM, and we finally have a green build, too.