HDDS-10586. Avoid loading network topology layer schema file for every read

[tanvipenumudy]

What changes were proposed in this pull request?

• After the change implemented in: #5391, the network topology layer schema file is now loaded for every key-level read operation, flooding the OM logs (please find the below logs for reference).
• This is because a new NetworkTopologyImpl object is being instantiated for each read. As part of this instantiation, NodeSchemaManager is initialized each time resulting in the loading of the schema file each time.
• To optimize this process, a new NetworkTopologyImpl object is now generated only when the fetched network topology cluster tree differs from the current cluster tree.
• This object generation is now bound by the rate specified by the ozone.om.network.topology.refresh.duration configuration which defaults to 1 hour.
• This ensures that the schema file is also loaded at the same rate, thereby reducing the overhead during read operations.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10586

How was this patch tested?

• Green Git CI: https://github.com/tanvipenumudy/ozone/actions/runs/8416254723
• Tested with a synthetic read workload on a docker-compose cluster (using freon ockrw tool):

ozone freon ockrw -n 10 --percentage-read 100 --percentage-list 0 -v vol1 -b buck1 -p sample

ozone-om logs:

...
2024-03-25 11:25:57 2024-03-25 05:55:57,738 [IPC Server handler 97 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 95 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 40 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 40 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 95 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 71 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
...

Before changes:

docker-compose logs | grep 'Loading network topology layer schema file' | grep 'ozone-om-1' | wc -l
20

After changes:

docker-compose logs | grep 'Loading network topology layer schema file' | grep 'ozone-om-1' | wc -l
1

[tanvipenumudy]

Thank you @ChenSammi for helping identifying this behaviour, could you please take a look at the changes? Thanks!

[tanvipenumudy]

@kerneltime could you please take a look at the change as well, thanks!

[ChenSammi]

The last patch LGTM. Thanks @tanvipenumudy .

[adoroszlai]

Thanks @tanvipenumudy for the patch, @ChenSammi for the review.