HDDS-12294. Create config object for OM

[adoroszlai]

What changes were proposed in this pull request?

Create annotated config object OmConfig for Ozone Manager configs. As a first step, move server.list.max.size into it.

This will help reduce boilerplate code required to make properties reconfigurable, which will be useful for speeding up integration tests.

https://issues.apache.org/jira/browse/HDDS-12294

How was this patch tested?

CI:
https://github.com/adoroszlai/ozone/actions/runs/13242815738

[ivandika3]

@adoroszlai Thanks for the patch. Left some minor comments.

which will be useful for speeding up integration tests.

Could you help to explain a bit more about how the config object can speed up the integration tests?

[adoroszlai]

Thanks @ivandika3 for the review.

Could you help to explain a bit more about how the config object can speed up the integration tests?

The config object itself does not speed up tests.

Currently, if we want to change the value of some configs, we need to restart OM, or create a completely new cluster. Some tests (e.g. TestListKeys) set ozone.om.server.list.max.size = 2, to be able to test key list iterator / paging without creating thousands of keys. Most other tests use the default value (1000). This is a simple setting, does not change any state in OM. We could simply apply the latest setting for each new listKeys call.

Reading the value from OzoneConfiguration each time may be a bit expensive. So we should store it in a member variable. Currently it is stored as a final variable in OzoneManagerRequestHandler. It could be simply made mutable in that class, but this approach has some problems:

• Have to expose a method to allow changing it.
• The test needs to know that it is used in OzoneManagerRequestHandler, not in e.g. OzoneManager. It needs to get a reference to the instance.
• If any other code uses the same config, we need to make these changes there, too. While ozone.om.server.list.max.size is used only in OzoneManagerRequestHandler, other settings are used in multiple places.

Config objects also store references, so we avoid lookup from OzoneConfiguration. Accessors and mutators are natural on these, and an object like OzoneManagerRequestHandler does not need to expose its config. Config objects can also be shared among code that needs to use OM config.

So making the config reference mutable allows setting it without cluster restart, and the test can simply set the desired value in its setup. Of course tests need to be tweaked to do that, speedup does not come automatically.

This can be taken a step further, by making the config property reconfigurable (HDDS-3835), which allows changing the property in real clusters. A callback needs to be registered for each reconfigurable property, which tends to get repetitive (this is the boilerplate I mentioned in the PR description):

ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

Lines 530 to 537 in f2d9cc3

	reconfigurationHandler =
	new ReconfigurationHandler("OM", conf, this::checkAdminUserPrivilege)
	.register(OZONE_ADMINISTRATORS, this::reconfOzoneAdmins)
	.register(OZONE_READONLY_ADMINISTRATORS,
	this::reconfOzoneReadOnlyAdmins)
	.register(OZONE_OM_VOLUME_LISTALL_ALLOWED, this::reconfigureAllowListAllVolumes)
	.register(OZONE_KEY_DELETING_LIMIT_PER_TASK,
	this::reconfOzoneKeyDeletingLimitPerTask);

ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

Lines 4973 to 4987 in f2d9cc3

	private String reconfOzoneKeyDeletingLimitPerTask(String newVal) {
	Preconditions.checkArgument(Integer.parseInt(newVal) >= 0,
	OZONE_KEY_DELETING_LIMIT_PER_TASK + " cannot be negative.");
	getConfiguration().set(OZONE_KEY_DELETING_LIMIT_PER_TASK, newVal);
	getKeyManager().getDeletingService()
	.setKeyLimitPerTask(Integer.parseInt(newVal));
	return newVal;
	}
	private String reconfigureAllowListAllVolumes(String newVal) {
	getConfiguration().set(OZONE_OM_VOLUME_LISTALL_ALLOWED, newVal);
	setAllowListAllVolumesFromConfig();
	return String.valueOf(allowListAllVolumes);
	}

Alternatively, a config object can be registered as a whole, and it can handle all its reconfigurable properties (HDDS-8760). See ReplicationManagerConfiguration as an example with many reconfigurable properties.

ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java

Line 842 in f2d9cc3

reconfigurationHandler.register(replicationManager.getConfig());

[ivandika3]

Thank you for the detailed explanation. This is a good direction to take. LGTM +1.

[adoroszlai]

Thanks @ivandika3, @smengcl for the review.