Skip to content

[HUDI-5832] add relocated prefix for hbase classes in hbase-site.xml #8029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bvaradar merged 1 commit into from
Apr 21, 2023
Merged

[HUDI-5832] add relocated prefix for hbase classes in hbase-site.xml #8029

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 19 additions & 19 deletions hudi-common/src/main/resources/hbase-site.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ possible configurations would overwhelm and obscure the important.
<property>
<name>hbase.master.logcleaner.plugins</name>
<value>
org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner
org.apache.hudi.org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner,org.apache.hudi.org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner,org.apache.hudi.org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner
Copy link
Contributor

@danny0405 danny0405 Mar 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just exclude the hbase-site.xml from our bundle jars? This hbase-site.xml can cause conflict if other jars in the classpath load the hbase-site.xml file.

Copy link
Contributor

@yihua yihua Mar 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately we cannot exclude the hbase-site.xml, which is added explicitly for the metadata table to work properly with HFile (see the description of #5004).

Copy link
Contributor

@danny0405 danny0405 Mar 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the description in #5004, but still confused why the wrong class name(unshaded) still works here, does that mean the classpath must contain these classes to make the hudi jar run correctly?

Another question is do we need all the config items?

Copy link
Contributor

@yihua yihua Mar 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading and writing HFile do not require some of the HBase configs, including the ones fixed by this PR, so solely for reading and writing the metadata table, there should not be any issue. But if other HBase functionality is used elsewhere and the Hudi jar is loaded first by the class loader, the unshaded class names can cause problems.

We need all the configs as they are required for HBase 2.4.9.

</value>
<description>A comma-separated list of BaseLogCleanerDelegate invoked by
the LogsCleaner service. These WAL cleaners are called in order,
Expand All @@ -178,7 +178,7 @@ possible configurations would overwhelm and obscure the important.
<property>
<name>hbase.master.hfilecleaner.plugins</name>
<value>
org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner
org.apache.hudi.org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner,org.apache.hudi.org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner
</value>
<description>A comma-separated list of BaseHFileCleanerDelegate invoked by
the HFileCleaner service. These HFiles cleaners are called in order,
Expand Down Expand Up @@ -328,12 +328,12 @@ possible configurations would overwhelm and obscure the important.
</property>
<property>
<name>hbase.regionserver.hlog.reader.impl</name>
<value>org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader</value>
<value>org.apache.hudi.org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader</value>
<description>The WAL file reader implementation.</description>
</property>
<property>
<name>hbase.regionserver.hlog.writer.impl</name>
<value>org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter</value>
<value>org.apache.hudi.org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter</value>
<description>The WAL file writer implementation.</description>
</property>
<property>
Expand Down Expand Up @@ -393,7 +393,7 @@ possible configurations would overwhelm and obscure the important.
</property>
<property>
<name>hbase.regionserver.region.split.policy</name>
<value>org.apache.hadoop.hbase.regionserver.SteppingSplitPolicy</value>
<value>org.apache.hudi.org.apache.hadoop.hbase.regionserver.SteppingSplitPolicy</value>
<description>
A split policy determines when a region should be split. The various
other split policies that are available currently are BusyRegionSplitPolicy,
Expand Down Expand Up @@ -1369,7 +1369,7 @@ possible configurations would overwhelm and obscure the important.
<name>hbase.coprocessor.master.classes</name>
<value></value>
<description>A comma-separated list of
org.apache.hadoop.hbase.coprocessor.MasterObserver coprocessors that are
org.apache.hudi.org.apache.hadoop.hbase.coprocessor.MasterObserver coprocessors that are
loaded by default on the active HMaster process. For any implemented
coprocessor methods, the listed classes will be called in order. After
implementing your own MasterObserver, just put it in HBase's classpath
Expand Down Expand Up @@ -1694,7 +1694,7 @@ possible configurations would overwhelm and obscure the important.
</property>
<property>
<name>hbase.status.publisher.class</name>
<value>org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher</value>
<value>org.apache.hudi.org.apache.hadoop.hbase.master.ClusterStatusPublisher$MulticastPublisher</value>
<description>
Implementation of the status publication with a multicast message.
</description>
Expand Down Expand Up @@ -1735,14 +1735,14 @@ possible configurations would overwhelm and obscure the important.
</property>
<property>
<name>hbase.rest.filter.classes</name>
<value>org.apache.hadoop.hbase.rest.filter.GzipFilter</value>
<value>org.apache.hudi.org.apache.hadoop.hbase.rest.filter.GzipFilter</value>
<description>
Servlet filters for REST service.
</description>
</property>
<property>
<name>hbase.master.loadbalancer.class</name>
<value>org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer</value>
<value>org.apache.hudi.org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer</value>
<description>
Class used to execute the regions balancing when the period occurs.
See the class comment for more on how it works
Expand All @@ -1760,7 +1760,7 @@ possible configurations would overwhelm and obscure the important.
</property>
<property>
<name>hbase.master.normalizer.class</name>
<value>org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer</value>
<value>org.apache.hudi.org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer</value>
<description>
Class used to execute the region normalization when the period occurs.
See the class comment for more on how it works
Expand Down Expand Up @@ -1812,7 +1812,7 @@ possible configurations would overwhelm and obscure the important.
<name>hbase.procedure.regionserver.classes</name>
<value></value>
<description>A comma-separated list of
org.apache.hadoop.hbase.procedure.RegionServerProcedureManager procedure managers that are
org.apache.hudi.org.apache.hadoop.hbase.procedure.RegionServerProcedureManager procedure managers that are
loaded by default on the active HRegionServer process. The lifecycle methods (init/start/stop)
will be called by the active HRegionServer process to perform the specific globally barriered
procedure. After implementing your own RegionServerProcedureManager, just put it in
Expand All @@ -1823,7 +1823,7 @@ possible configurations would overwhelm and obscure the important.
<name>hbase.procedure.master.classes</name>
<value></value>
<description>A comma-separated list of
org.apache.hadoop.hbase.procedure.MasterProcedureManager procedure managers that are
org.apache.hudi.org.apache.hadoop.hbase.procedure.MasterProcedureManager procedure managers that are
loaded by default on the active HMaster process. A procedure is identified by its signature and
users can use the signature and an instant name to trigger an execution of a globally barriered
procedure. After implementing your own MasterProcedureManager, just put it in HBase's classpath
Expand All @@ -1832,7 +1832,7 @@ possible configurations would overwhelm and obscure the important.
</property>
<property>
<name>hbase.coordinated.state.manager.class</name>
<value>org.apache.hadoop.hbase.coordination.ZkCoordinatedStateManager</value>
<value>org.apache.hudi.org.apache.hadoop.hbase.coordination.ZkCoordinatedStateManager</value>
<description>Fully qualified name of class implementing coordinated state manager.</description>
</property>
<property>
Expand Down Expand Up @@ -1862,10 +1862,10 @@ possible configurations would overwhelm and obscure the important.
</property>
<property>
<name>hbase.http.filter.initializers</name>
<value>org.apache.hadoop.hbase.http.lib.StaticUserWebFilter</value>
<value>org.apache.hudi.org.apache.hadoop.hbase.http.lib.StaticUserWebFilter</value>
<description>
A comma separated list of class names. Each class in the list must extend
org.apache.hadoop.hbase.http.FilterInitializer. The corresponding Filter will
org.apache.hudi.org.apache.hadoop.hbase.http.FilterInitializer. The corresponding Filter will
be initialized. Then, the Filter will be applied to all user facing jsp
and servlet web pages.
The ordering of the list defines the ordering of the filters.
Expand All @@ -1891,7 +1891,7 @@ possible configurations would overwhelm and obscure the important.
</property>
<property>
<name>hbase.replication.rpc.codec</name>
<value>org.apache.hadoop.hbase.codec.KeyValueCodecWithTags</value>
<value>org.apache.hudi.org.apache.hadoop.hbase.codec.KeyValueCodecWithTags</value>
<description>
The codec that is to be used when replication is enabled so that
the tags are also replicated. This is used along with HFileV3 which
Expand Down Expand Up @@ -2009,7 +2009,7 @@ possible configurations would overwhelm and obscure the important.
</property>
<property>
<name>hbase.mob.compactor.class</name>
<value>org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactor</value>
<value>org.apache.hudi.org.apache.hadoop.hbase.mob.compactions.PartitionedMobCompactor</value>
<description>
Implementation of mob compactor, the default one is PartitionedMobCompactor.
</description>
Expand Down Expand Up @@ -2153,15 +2153,15 @@ possible configurations would overwhelm and obscure the important.
<property>
<name>hbase.namedqueue.provider.classes</name>
<value>
org.apache.hadoop.hbase.namequeues.impl.SlowLogQueueService,org.apache.hadoop.hbase.namequeues.impl.BalancerDecisionQueueService,org.apache.hadoop.hbase.namequeues.impl.BalancerRejectionQueueService
org.apache.hudi.org.apache.hadoop.hbase.namequeues.impl.SlowLogQueueService,org.apache.hudi.org.apache.hadoop.hbase.namequeues.impl.BalancerDecisionQueueService,org.apache.hudi.org.apache.hadoop.hbase.namequeues.impl.BalancerRejectionQueueService
</value>
<description>
Default values for NamedQueueService implementors. This comma separated full class names
represent all implementors of NamedQueueService that we would like to be invoked by
LogEvent handler service. One example of NamedQueue service is SlowLogQueueService which
is used to store slow/large RPC logs in ringbuffer at each RegionServer.
All implementors of NamedQueueService should be found under package:
"org.apache.hadoop.hbase.namequeues.impl"
"org.apache.hudi.org.apache.hadoop.hbase.namequeues.impl"
</description>
</property>
<property>
Expand Down