[HUDI-5832] add relocated prefix for hbase classes in hbase-site.xml

[stayrascal]

Change Logs

Add relocated prefix(org.apache.hudi) for hbase classes in hbase-site.xml, because we have relocated hbase classes in the bundle jar.

Note:
Maybe it's better to put hbase-site.xml in bundle module instead of hudi-common module. But currently it seems that every bundle jar will relocated hbase classes, so it's fine to modify the hbase-site.xml in hudi-common module directly

Impact

No

Risk level (write none, low medium or high below)

low

Documentation Update

No

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[danny0405]

Thanks for contribution, can we elaborate a little the details?

[stayrascal]

@danny0405 Thanks for review this. I give more context here.

I meet a problem about ClassNotFound exception org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher during loading flink-hbase-connector and hudi-flink-bundle. Because we package hbase-site.xml in the bundle jar, and the value of hbase.status.publisher.class is org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher, but it hudi-flink-bundle, the class location is org/apache/hudi/org/apache/hadoop/hbase/master/ClusterStatusPublisher$MulticastPublisher.

During open Hbase connection and write data to Hbase, the application will load the hbase-site.xml of hudi-flink-bundle jar, and latter try to initialise the class of hbase.status.publisher.class property, and then meet ClassNotFound problem.

So I'm thinking all configured classes properties should start with relocated prefix org.apapche.hudi.

@yihua could you please help to double check if it's reasonable?

[danny0405]

Reasonable, looks good from my side.

[nsivabalan]

they @stayrascal : thanks for the contribution.
can you give us an idea of the impact. What is failing as of now w/o this fix? and do we have any work around for existing users (even if we land this now, older official releases may not have this fix). So, when we plan for any minor version release (like 0.12.3), will help us.

[danny0405]

+1, should good to go

[nsivabalan]

@yihua : can you review this.

[danny0405]

Should we just exclude the hbase-site.xml from our bundle jars? This hbase-site.xml can cause conflict if other jars in the classpath load the hbase-site.xml file.

[yihua]

Unfortunately we cannot exclude the hbase-site.xml, which is added explicitly for the metadata table to work properly with HFile (see the description of #5004).

[danny0405]

Read the description in #5004, but still confused why the wrong class name(unshaded) still works here, does that mean the classpath must contain these classes to make the hudi jar run correctly?

Another question is do we need all the config items?

[yihua]

Reading and writing HFile do not require some of the HBase configs, including the ones fixed by this PR, so solely for reading and writing the metadata table, there should not be any issue. But if other HBase functionality is used elsewhere and the Hudi jar is loaded first by the class loader, the unshaded class names can cause problems.

We need all the configs as they are required for HBase 2.4.9.

[yihua]

@danny0405 Thanks for review this. I give more context here.

I meet a problem about ClassNotFound exception org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher during loading flink-hbase-connector and hudi-flink-bundle. Because we package hbase-site.xml in the bundle jar, and the value of hbase.status.publisher.class is org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher, but it hudi-flink-bundle, the class location is org/apache/hudi/org/apache/hadoop/hbase/master/ClusterStatusPublisher$MulticastPublisher.

During open Hbase connection and write data to Hbase, the application will load the hbase-site.xml of hudi-flink-bundle jar, and latter try to initialise the class of hbase.status.publisher.class property, and then meet ClassNotFound problem.

So I'm thinking all configured classes properties should start with relocated prefix org.apapche.hudi.

@yihua could you please help to double check if it's reasonable?

Makes sense. LGTM.

[yihua]

@stayrascal could you check the CI failure?

[xicm]

@danny0405 Thanks for review this. I give more context here.

I meet a problem about ClassNotFound exception org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher during loading flink-hbase-connector and hudi-flink-bundle. Because we package hbase-site.xml in the bundle jar, and the value of hbase.status.publisher.class is org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher, but it hudi-flink-bundle, the class location is org/apache/hudi/org/apache/hadoop/hbase/master/ClusterStatusPublisher$MulticastPublisher.

During open Hbase connection and write data to Hbase, the application will load the hbase-site.xml of hudi-flink-bundle jar, and latter try to initialise the class of hbase.status.publisher.class property, and then meet ClassNotFound problem.

So I'm thinking all configured classes properties should start with relocated prefix org.apapche.hudi.

@yihua could you please help to double check if it's reasonable?

The shaded pattern can solve this problem, while for hbase.status.publisher.class I am sure we can remove this conf. Hbase can find this class from classpath.
Another question, what if some one just use hudi-common-*.jar?

[danny0405]

Another question, what if some one just use hudi-common-*.jar?

This is a good question, unfortunately we have no good way to solve it unless we publish our own shaded Hbase jars, i.e all the hudi modules depends on the separately published shaded Hbase jars.

[yihua]

Another question, what if some one just use hudi-common-*.jar?

This is a good question, unfortunately we have no good way to solve it unless we publish our own shaded Hbase jars, i.e all the hudi modules depends on the separately published shaded Hbase jars.

Ultimately we'll have our own HFile reader and writer implementation and remove HBase dependency for metadata table.

[stayrascal]

@stayrascal could you check the CI failure?

@yihua Thanks a lot for review this PR. may I check which CI failed? it seems that all CI passed?

[danny0405]

There is a failue for timeout: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=15364&view=logs&j=7601efb9-4019-552e-11ba-eb31b66593b2&t=9688f101-287d-53f4-2a80-87202516f5d0&l=6717, should not be cause by the patch.

[bvaradar]

@yihua @danny0405 : Is this good to be merged ?

[danny0405]

@yihua @danny0405 : Is this good to be merged ?

Yes, should be good to land.

[bvaradar]

@stayrascal : Since this is a old PR, can you please rebase this PR so that we can land it.

[stayrascal]

@stayrascal : Since this is a old PR, can you please rebase this PR so that we can land it.

Thanks @nsivabalan a lot for reminder this, just rebase the latest changes from master branch, and update this PR.

[hudi-bot]

CI report:

• 77bc172 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[danny0405]

The test `` is failing continously after merging the PR:

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hudi.org.apache.hadoop.hbase.http.lib.StaticUserWebFilter not found
	at org.apache.hadoop.conf.Configuration.getClasses(Configuration.java:2404)
	at org.apache.hadoop.hbase.http.HttpServer.getFilterInitializers(HttpServer.java:705)
	at org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:639)
	at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:588)
	at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:107)
	at org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:411)
	at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:96)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:2201)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:688)
	at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:417)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:132)
	... 65 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hudi.org.apache.hadoop.hbase.http.lib.StaticUserWebFilter not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2329)
	at org.apache.hadoop.conf.Configuration.getClasses(Configuration.java:2400)
	... 79 more

The shaded class can not be found in the classpath.

[bvaradar]

@stayrascal : We are reverting this change. Can you please relook at failing tests and reopen the PR after fixing.

[stayrascal]

@stayrascal : We are reverting this change. Can you please relook at failing tests and reopen the PR after fixing.

sorry for reply late, will check it.

[aib628]

Another question, what if some one just use hudi-common-*.jar?

This is a good question, unfortunately we have no good way to solve it unless we publish our own shaded Hbase jars, i.e all the hudi modules depends on the separately published shaded Hbase jars.

This file (hbase-site.xml) specified 'zookeeper.znode.parent' to '/hbase', some hive query from external hbase table will not available when use another 'zookeeper.znode.parent' config

if we not open the config 'metadata.enabled', can we delete this xml file from hudi-hadoop-mr-bundle.jar ?

[yihua]

Another question, what if some one just use hudi-common-*.jar?

This is a good question, unfortunately we have no good way to solve it unless we publish our own shaded Hbase jars, i.e all the hudi modules depends on the separately published shaded Hbase jars.

This file (hbase-site.xml) specified 'zookeeper.znode.parent' to '/hbase', some hive query from external hbase table will not available when use another 'zookeeper.znode.parent' config

if we not open the config 'metadata.enabled', can we delete this xml file from hudi-hadoop-mr-bundle.jar ?

Yes, hbase-site.xml is only used for the bootstrap data table and metadata table. You may delete this xml when explicitly disabling the metadata table.