HDDS-12777. Use module-specific name for generated config files #8475

adoroszlai · 2025-05-17T11:07:02Z

What changes were proposed in this pull request?

Currently OzoneConfiguration loads all ozone-default-generated.xml files found on the classpath. It needs to do so, because such files are present in multiple modules (hdds-common, ozone-manager, etc.), and each file only has a subset of config properties. Both ozone-default.xml and ozone-default-generated.xml are added via addResource, because addDefaultResource does not allow multiple resources with the same name in different locations (see apache/hadoop#773 (comment)). However, loading all resources produces unexpected results if the classpath contains multiple versions of the same jar.

This patch changes Ozone to use module-specific names for the generated config files, so we can avoid iterating resources.

Add a build step to rename ozone-default-generated.xml after compilation of each module.
Remove config file generation from modules that do not have such configs currently (ozone-s3gateway, ozone-client, ozone-interface-storage).
Move ReconSqlDbConfig to ozone-recon, where it's used. This allows skipping config generation in ozone-reconcodegen.
Remove test-specific config class from ozone-insight, so we can skip config generation in that module, too.
Add resources in OzoneConfiguration via addDefaultResource instead of addResource. This is how Configuration and HdfsConfiguration add their own configs, and is what OzoneConfiguration used to do before HDDS-1469. addDefaultResource also handles duplicate names, so duplicate jars should not result in overridden config. As a side-effect:
- Remove logic from OzoneConfiguration related to isolated classloader (HDDS-922), no longer needed after HDDS-3458.
- Remove the hack introduced by HDDS-3871 for looking for Ozone config properties in core-site.xml. I think this is consistent with HDFS, whose properties should be set in hdfs-site.xml, not core-site.xml:
```
public class HdfsConfiguration extends Configuration {
  static {
    addDeprecatedKeys();


    // adds the default resources
    Configuration.addDefaultResource("hdfs-default.xml");
    Configuration.addDefaultResource("hdfs-rbf-default.xml");
    Configuration.addDefaultResource("hdfs-site.xml");
    Configuration.addDefaultResource("hdfs-rbf-site.xml");
  }
```
Minor HttpFS fixes (included because HttpFS tests were timing out with the initial version of the patch):
- Use Ozone's startup message for HttpFS to show Ozone version and configuration.
- Set timeout for HttpFS acceptance tests.

https://issues.apache.org/jira/browse/HDDS-12777

How was this patch tested?

Checked OM and SCM configuration in Docker Compose cluster:

cd hadoop-ozone/dist/target/ozone-2.1.0-SNAPSHOT/compose/ozone
OZONE_DATANODES=3 ./run.sh -d
open http://localhost:9874/conf
open http://localhost:9876/conf

Verified that items from various config files are present, e.g.:

<property>
  <name>ozone.scm.dead.node.interval</name>
  <value>45s</value>
  <final>false</final>
  <source>ozone-site.xml</source>
</property>
<property>
  <name>hdds.container.ipc.random.port</name>
  <value>false</value>
  <final>false</final>
  <source>ozone-default.xml</source>
</property>
<property>
  <name>hdds.scm.block.deletion.txn.dn.commit.map.limit</name>
  <value>5000000</value>
  <final>false</final>
  <source>hdds-client-default.xml</source>
</property>
<property>
  <name>ipc.server.tcpnodelay</name>
  <value>true</value>
  <final>false</final>
  <source>core-default.xml</source>
</property>
<property>
  <name>hdds.container.balancer.move.timeout</name>
  <value>65m</value>
  <final>false</final>
  <source>hdds-common-default.xml</source>
</property>
<property>
  <name>hadoop.hdds.db.rocksdb.writeoption.sync</name>
  <value>false</value>
  <final>false</final>
  <source>hdds-server-framework-default.xml</source>
</property>

Only OM has items from ozone-manager-default.xml:

<property>
  <name>ozone.om.user.max.volume</name>
  <value>1024</value>
  <final>false</final>
  <source>ozone-manager-default.xml</source>
</property>

And only SCM has items from hdds-server-scm-default.xml:

<property>
  <name>hdds.scm.replication.datanode.delete.container.limit</name>
  <value>40</value>
  <final>false</final>
  <source>hdds-server-scm-default.xml</source>
</property>

CI:
https://github.com/adoroszlai/ozone/actions/runs/15083080061

ChenSammi · 2025-05-19T11:35:34Z

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java

+    addDefaultResource("hdfs-site.xml");
+
+    // One generated file per module
+    addDefaultResource("hdds-common-default.xml");


Can we extra the module name into an array or list, and append the "default.xml" automatically for each module, can let a comment telling about if there is any new module added in future, and the new module uses this annotation based property definition mechanism, module name should be added into this array?

Will do, waiting a bit for any other feedback.

So, now we generate these default xml files per module and load each file once right?

That works, thanks

ChenSammi · 2025-05-19T11:43:01Z

hadoop-hdds/framework/pom.xml

+          </execution>
+          <execution>
+            <id>rename-generated-test-config</id>
+            <phase>process-test-classes</phase>


For the two modules, famework and common, who has this extra "process-test-classes", will both the two generated xml files be loaded during OzoneConfiguration activation?
If only first generated xml file will be loaded, how TestGeneratedConfigurationOverwrite.java can pass?

The config file for test is created in target/test-classes (used in their own unit tests) and is added to ...-tests.jar (used by some other modules' tests).

ChenSammi · 2025-05-19T11:44:16Z

@adoroszlai , thanks for the quick fix. Overall it looks good to me. Just a few minor comments.

umamaheswararao · 2025-05-22T06:35:23Z

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java

-    // distributed to executors by spark driver. Ozone properties which are
-    // added to core-site, will be overridden by properties from adding Resource
-    // ozone-default.xml. So, adding core-site again will help to resolve
-    // this override issue.


@adoroszlai have checked the above comment why we added again core-site.xml here? Just wanted to make sure we are not skipping anything!

@umamaheswararao Please see the "Remove the hack introduced by HDDS-3871" point in PR description.

Thanks @adoroszlai for pointing it.

dombizita

Thanks for working on this @adoroszlai, the HttpFS fixes are looking good to me.

adoroszlai · 2025-05-22T18:03:32Z

Thanks @ChenSammi, @dombizita, @umamaheswararao for the review.

peterxcli · 2025-05-25T07:25:49Z

@adoroszlai sorry to disturb you in weekend, I keep getting error when running mvn verify and the pmd check script.

❯ mvn verify install -DskipShade -DskipTests -Pgo-offline
...
...
[INFO] 314 goals, 314 executed
[ERROR] Failed to execute goal com.coderplus.maven.plugins:copy-rename-maven-plugin:1.0.1:rename (rename-generated-config) on project hdds-common: sourceFile /Users/peter.li/Documents/oss/apache/ozone/hadoop-hdds/common/target/classes/ozone-default-generated.xml does not exist -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hdds-common

my maven version:

❯ mvn --version
Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
Maven home: /opt/homebrew/Cellar/maven/3.9.9/libexec
Java version: 21.0.6, vendor: Azul Systems, Inc., runtime: /Users/peter.li/.sdkman/candidates/java/21.0.6-zulu/zulu-21.jdk/Contents/Home
Default locale: en_TW, platform encoding: UTF-8
OS name: "mac os x", version: "15.5", arch: "aarch64", family: "mac"

Have I missed a setup step, or could something in my environment be causing this? Any pointers would be greatly appreciated.

get same error in my another machine, its maven version is:
(branch is up-to-date)

❯ mvn --version                                          
Apache Maven 3.8.7
Maven home: /usr/share/maven
Java version: 21.0.7, vendor: Ubuntu, runtime: /usr/lib/jvm/java-21-openjdk-amd64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.8.0-59-generic", arch: "amd64", family: "unix"

adoroszlai · 2025-05-25T07:50:55Z

@peterxcli Please use mvn install instead of mvn verify install.

verify and install are both phases of the same lifecycle. Running mvn with some phase executes steps for all prior phases, so verify install compiles and packages stuff twice. This is unnecessary.

The compiler plugin applies some optimization on the second run, apparently skipping annotation processing, which would create ozone-default-generated.xml.

[INFO] --- compiler:3.14.0:compile (default-compile) @ hdds-common ---
[INFO] Compiling 277 source files with javac [debug target 8] to target/classes
...
[INFO] --- copy-rename:1.0.1:rename (rename-generated-config) @ hdds-common ---
[INFO] Renamed hadoop-hdds/common/target/classes/ozone-default-generated.xml to hadoop-hdds/common/target/classes/hdds-common-default.xml
...
[INFO] --- compiler:3.14.0:compile (default-compile) @ hdds-common ---
[INFO] Compiling 53 source files with javac [debug target 8] to target/classes
[WARNING] Overwriting artifact's file from hadoop-hdds/common/target/hdds-common-2.1.0-SNAPSHOT.jar to hadoop-hdds/common/target/classes
[INFO]
[INFO] --- copy-rename:1.0.1:rename (rename-generated-config) @ hdds-common ---
...
[ERROR] Failed to execute goal com.coderplus.maven.plugins:copy-rename-maven-plugin:1.0.1:rename (rename-generated-config) on project hdds-common: sourceFile hadoop-hdds/common/target/classes/ozone-default-generated.xml does not exist

peterxcli · 2025-06-03T09:30:33Z

@adoroszlai Thanks for the clarification!

BTW, I just noticed flaky-test-check CI has same problem: https://github.com/peterxcli/ozone/actions/runs/15413245625/job/43370319701#step:7:843
I guess we need to add clean step in junit.sh, too.

I'm tryna fix it. https://github.com/apache/ozone/compare/master...peterxcli:hdds13169-Intermittent-failure-in-testSnapshotOperationsNotBlockedDuringCompaction?expand=1

…239-container-reconciliation Commits: 80 commits 5e273a4 HDDS-12977. Fail build on dependency problems (apache#8574) 5081ba2 HDDS-13034. Refactor DirectoryDeletingService to use ReclaimableDirFilter and ReclaimableKeyFilter (apache#8546) e936e4d HDDS-12134. Implement Snapshot Cache lock for OM Bootstrap (apache#8474) 31d13de HDDS-13165. [Docs] Python client developer guide. (apache#8556) 9e6955e HDDS-13205. Bump common-custom-user-data-maven-extension to 2.0.3 (apache#8581) 750b629 HDDS-13203. Bump Bouncy Castle to 1.81 (apache#8580) ba5177e HDDS-13202. Bump build-helper-maven-plugin to 3.6.1 (apache#8579) 07ee5dd HDDS-13204. Bump awssdk to 2.31.59 (apache#8582) e1964f2 HDDS-13201. Bump jersey2 to 2.47 (apache#8578) 81295a5 HDDS-13013. [Snapshot] Add metrics and tests for snapshot operations. (apache#8436) b3d75ab HDDS-12976. Clean up unused dependencies (apache#8521) e0f08b2 HDDS-13179. rename-generated-config fails on re-compile without clean (apache#8569) f388317 HDDS-12554. Support callback on completed reconfiguration (apache#8391) c13a3fe HDDS-13154 Link more Grafana dashboard json files to the Observability user doc (apache#8533) 2a761f7 HDDS-11967. [Docs]DistCP Integration in Kerberized environment. (apache#8531) 81fc4c4 HDDS-12550. Use DatanodeID instead of UUID in NodeManager CommandQueue. (apache#8560) 2360af4 HDDS-13169. Intermittent failure in testSnapshotOperationsNotBlockedDuringCompaction (apache#8553) f19789d HDDS-13170. Reclaimable filter should always reclaim entries when buckets and volumes have already been deleted (apache#8551) 315ef20 HDDS-13175. Leftover reference to OM-specific trash implementation (apache#8563) 902e715 HDDS-13159. Refactor KeyManagerImpl for getting deleted subdirectories and deleted subFiles (apache#8538) 46a93d0 HDDS-12817. Addendum rename ecIndex to replicaIndex in chunkinfo output (apache#8552) 19b9b9c HDDS-13166. Set pipeline ID in BlockExistenceVerifier to avoid cached pipeline with different node (apache#8549) b3ff67c HDDS-13068. Validate Container Balancer move timeout and replication timeout configs (apache#8490) 7a7b9a8 HDDS-13139. Introduce bucket layout flag in freon rk command (apache#8539) 3c25e7d HDDS-12595. Add verifier for container replica states (apache#8422) 6d59220 HDDS-13104. Move auditparser acceptance test under debug (apache#8527) 8e8c432 HDDS-13071. Documentation for Container Replica Debugger Tool (apache#8485) 0e8c8d4 HDDS-13158. Bump junit to 5.13.0 (apache#8537) 8e552b4 HDDS-13157. Bump exec-maven-plugin to 3.5.1 (apache#8534) 168f690 HDDS-13155. Bump jline to 3.30.4 (apache#8535) cc1e4d1 HDDS-13156. Bump awssdk to 2.31.54 (apache#8536) 3bfb7af HDDS-13136. KeyDeleting Service should not run for already deep cleaned snapshots (apache#8525) 006e691 HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache (apache#8141) 568b228 HDDS-13067. Container Balancer delete commands should not be sent with an expiration time in the past (apache#8491) 53673c5 HDDS-11244. OmPurgeDirectoriesRequest should clean up File and Directory tables of AOS for deleted snapshot directories (apache#8509) 07f4868 HDDS-13099. ozone admin datanode list ignores --json flag when --id filter is used (apache#8500) 08c0ab8 HDDS-13075. Fix default value in description of container placement policy configs (apache#8511) 58c87a8 HDDS-12177. Set runtime scope where missing (apache#8513) 10c470d HDDS-12817. Add EC block index in the ozone debug replicas chunk-info (apache#8515) 7027ab7 HDDS-13124. Respect config hdds.datanode.use.datanode.hostname when reading from datanode (apache#8518) b8b226c HDDS-12928. datanode min free space configuration (apache#8388) fd3d70c HDDS-13026. KeyDeletingService should also delete RenameEntries (apache#8447) 4c1c6cf HDDS-12714. Create acceptance test framework for debug and repair tools (apache#8510) fff80fc HDDS-13118. Remove duplicate mockito-core dependency from hdds-test-utils (apache#8508) 10d5555 HDDS-13115. Bump awssdk to 2.31.50 (apache#8505) 360d139 HDDS-13017. Fix warnings due to non-test scoped test dependencies (apache#8479) 1db1cca HDDS-13116. Bump jline to 3.30.3 (apache#8504) 322ca93 HDDS-13025. Refactor KeyDeletingService to use ReclaimableKeyFilter (apache#8450) 988b447 HDDS-5287. Document S3 ACL classes (apache#8501) 64bb29d HDDS-12777. Use module-specific name for generated config files (apache#8475) 54ed115 HDDS-9210. Update snapshot chain restore test to incorporate snapshot delete. (apache#8484) 87dfa5a HDDS-13014. Improve PrometheusMetricsSink#normalizeName performance (apache#8438) 7cdc865 HDDS-13100. ozone admin datanode list --json should output a newline at the end (apache#8499) 9cc4194 HDDS-13089. [snapshot] Add an integration test to verify snapshotted data can be read by S3 SDK client (apache#8495) cb9867b HDDS-13065. Refactor SnapshotCache to return AutoCloseSupplier instead of ReferenceCounted (apache#8473) a88ff71 HDDS-10979. Support STANDARD_IA S3 storage class to accept EC replication config (apache#8399) 6ec8f85 HDDS-13080. Improve delete metrics to show number of timeout DN command from SCM (apache#8497) 3bb8858 HDDS-12378. Change default hdds.scm.safemode.min.datanode to 3 (apache#8331) 0171bef HDDS-13073. Set pipeline ID in checksums verifier to avoid cached pipeline with different node (apache#8480) 5c7726a HDDS-11539. OzoneClientCache `@PreDestroy` is never called (apache#8493) a8ed19b HDDS-13031. Implement a Flat Lock resource in OzoneManagerLock (apache#8446) e9e8b30 HDDS-12935. Support unsigned chunked upload with STREAMING-UNSIGNED-PAYLOAD-TRAILER (apache#8366) 7590268 HDDS-13079. Improve logging in DN for delete operation. (apache#8489) 435fe7e HDDS-12870. Fix listObjects corner cases (apache#8307) eb5dabd HDDS-12926. Remove *.tmp.* exclusion in DU (apache#8486) eeb98c7 HDDS-13030. Snapshot Purge should unset deep cleaning flag for next 2 snapshots in the chain (apache#8451) 6bf121e HDDS-13032. Support proper S3OwnerId representation (apache#8478) 5d1b43d HDDS-13076. Refactor OzoneManagerLock class to rename Resource class to LeveledResource (apache#8482) bafe6d9 HDDS-13064. [snapshot] Add test coverage for SnapshotUtils.isBlockLocationInfoSame() (apache#8476) 7035846 HDDS-13040. Add user doc highlighting the difference between Ozone ACL and S3 ACL. (apache#8457) 1825cdf HDDS-13049. Deprecate VolumeName & BucketName in OmKeyPurgeRequest and prevent Key version purge on Block Deletion Failure (apache#8463) 211c76c HDDS-13060. Change NodeManager.addDatanodeCommand(..) to use DatanodeID (apache#8471) f410238 HDDS-13061. Add test for key ACL operations without permission (apache#8472) d1a2f48 HDDS-13057. Increment block delete processed transaction counts regardless of log level (apache#8466) 0cc6fcc HDDS-13043. Replace != with assertNotEquals in TestSCMContainerPlacementRackAware (apache#8470) e1c779a HDDS-13051. Use DatanodeID in server-scm. (apache#8465) 35e1126 HDDS-13042. [snapshot] Add future proofing test cases for unsupported file system API (apache#8458) 619c05d HDDS-13008. Exclude same SST files when calculating full snapdiff (apache#8423) 21b49d3 HDDS-12965. Fix warnings about "used undeclared" dependencies (apache#8468) 8136119 HDDS-13048. Create new module for Recon integration tests (apache#8464) Conflicts: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java

adoroszlai added 4 commits May 17, 2025 09:38

HDDS-12777. ozone-default is used as actual configuration

304775d

simplify resource loading

9471bba

set timeout for httpfs test

9d210c7

use Ozone startup msg in httpfs

41eab9e

adoroszlai self-assigned this May 17, 2025

adoroszlai requested review from ChenSammi, devmadhuu, dombizita, fapifta, jojochuang, nandakumar131, smengcl and xichen01 May 17, 2025 12:12

adoroszlai added the build Pull request that modifies the build process label May 17, 2025

ChenSammi reviewed May 19, 2025

View reviewed changes

umamaheswararao approved these changes May 22, 2025

View reviewed changes

umamaheswararao reviewed May 22, 2025

View reviewed changes

dombizita approved these changes May 22, 2025

View reviewed changes

Merge remote-tracking branch 'origin/master' into HDDS-12777

1e8145a

adoroszlai marked this pull request as draft May 22, 2025 13:56

use list, add comments, add ozone-csi

d9ab1a9

adoroszlai marked this pull request as ready for review May 22, 2025 15:42

adoroszlai merged commit 64bb29d into apache:master May 22, 2025
53 checks passed

adoroszlai deleted the HDDS-12777 branch May 22, 2025 18:03

peterxcli mentioned this pull request Jun 3, 2025

HDDS-13169. Intermittent failure in testSnapshotOperationsNotBlockedDuringCompaction #8553

Merged

HDDS-12777. Use module-specific name for generated config files #8475

HDDS-12777. Use module-specific name for generated config files #8475

Uh oh!

Conversation

adoroszlai commented May 17, 2025

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChenSammi commented May 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dombizita left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adoroszlai commented May 22, 2025

Uh oh!

peterxcli commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adoroszlai commented May 25, 2025

Uh oh!

peterxcli commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

adoroszlai May 19, 2025 •

edited

Loading

peterxcli commented May 25, 2025 •

edited

Loading

peterxcli commented Jun 3, 2025 •

edited

Loading