[SUPPORT]

[sstimmel]

See the Following exception in the logs after running deltastreamer. I believe this is shown when we have metrics enabled.
java.lang.NoSuchMethodError: 'int org.apache.hudi.avro.model.HoodieRollbackMetadata.getTotalFilesDeleted()'

A clear and concise description of what you expected to happen.

Environment Description

• Hudi version : 0.12.1

• Spark version : 3.3.1

• Hive version :

• Hadoop version : 3

• Storage (HDFS/S3/GCS..) : S3

• Running on Docker? (yes/no) : yes, kubernetes

Additional context

Add any other context about the problem here.

The following hudi jars are in the classpath
-hudi-aws-bundle.jar

• hudi-cli.jar
  -hudi-hive-sync-bundle.jar
  -hudi-spark-bundle.jar
  -hudi-utilities-slim-bundle.jar

hudi-utilities-slim-bundle does show that class.
10851 Mon Oct 17 22:29:30 UTC 2022 org/apache/hudi/avro/model/HoodieRollbackMetadata.class

Stacktrace

22/12/07 16:27:35 DEBUG org.apache.spark.internal.Logging: Executor 1 on 10.22.164.85 lost, but reason not yet known.
22/12/07 16:27:35 DEBUG org.apache.spark.internal.Logging: Cleaning up TaskScheduler state for tasks [] on failed executor 1
22/12/07 16:27:35 ERROR org.apache.spark.internal.Logging: Uncaught exception in thread dispatcher-CoarseGrainedScheduler
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
	at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:178)
	at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:150)
	at org.apache.spark.rpc.netty.NettyRpcEnv.send(NettyRpcEnv.scala:193)
	at org.apache.spark.rpc.netty.NettyRpcEndpointRef.send(NettyRpcEnv.scala:563)
	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.$anonfun$reviveOffers$1(CoarseGrainedSchedulerBackend.scala:630)
	at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1484)
	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.reviveOffers(CoarseGrainedSchedulerBackend.scala:630)
	at org.apache.spark.scheduler.TaskSchedulerImpl.executorLost(TaskSchedulerImpl.scala:1004)
	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.disableExecutor(CoarseGrainedSchedulerBackend.scala:482)
	at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint.$anonfun$onDisconnected$1(KubernetesClusterSchedulerBackend.scala:328)
	at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint.$anonfun$onDisconnected$1$adapted(KubernetesClusterSchedulerBackend.scala:328)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint.onDisconnected(KubernetesClusterSchedulerBackend.scala:328)
	at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:141)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
	at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
	at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
22/12/07 16:27:35 WARN org.apache.spark.internal.Logging: Kubernetes client has been closed.
22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: MapOutputTrackerMasterEndpoint stopped!
22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: MemoryStore cleared
22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: BlockManager stopped
22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: BlockManagerMaster stopped
22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: OutputCommitCoordinator stopped!
22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoSuchMethodError: 'int org.apache.hudi.avro.model.HoodieRollbackMetadata.getTotalFilesDeleted()'
	at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:774)
	at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1194)
	at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1177)
	at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1165)
	at org.apache.hudi.client.BaseHoodieWriteClient.lambda$startCommitWithTime$97cdbdca$1(BaseHoodieWriteClient.java:966)
	at org.apache.hudi.common.util.CleanerUtils.rollbackFailedWrites(CleanerUtils.java:153)
	at org.apache.hudi.client.BaseHoodieWriteClient.startCommitWithTime(BaseHoodieWriteClient.java:965)
	at org.apache.hudi.client.BaseHoodieWriteClient.startCommitWithTime(BaseHoodieWriteClient.java:958)
	at org.apache.hudi.utilities.deltastreamer.DeltaSync.startCommit(DeltaSync.java:677)
	at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:581)
	at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:336)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$2(HoodieDeltaStreamer.java:204)
	at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:202)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:571)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: Shutdown hook called
22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: Deleting directory /tmp/spark-7a432e99-1f82-46a4-a555-10bd6090a2ba
22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: Deleting directory /tmp/spark-42983bb3-9835-4bf4-9ea5-c68004077e7c
22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: Deleting directory /tmp/spark-6098afd8-d431-4b91-a81e-3ebea9419630
22/12/07 16:27:36 INFO org.apache.spark.internal.Logging: Deleting directory /tmp/spark-f394dce8-567a-461d-b559-c953966adb93
22/12/07 16:27:36 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
22/12/07 16:27:36 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
22/12/07 16:27:36 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.```

[xushiyan]

@sstimmel this is some dependency conflicts, likely caused by hudi-cli or hudi-hive-sync-bundle. can you try removing these 2 and only leaving spark-bundle and utilities-slim bundle? also can you confirm you're using spark3.3 bundle ?

For @lokeshj1703 to follow up.

[sstimmel]

@sstimmel this is some dependency conflicts, likely caused by hudi-cli or hudi-hive-sync-bundle. can you try removing these 2 and only leaving spark-bundle and utilities-slim bundle? also can you confirm you're using spark3.3 bundle ?

For @lokeshj1703 to follow up.

i tried removing hudi-cli jar from classpath. However still see the issue occassionally. I'm not sure how I can remove hudi-hive-sync-bundle, i need that for hive-syncs?, removing that i get different errors. I am using spark3.3 bundle.

[sstimmel]

i removed hudi-sync-bundle and switched over to use hudi-utilities-bundle instead of hudi-utilities-slim-bundle, since that has hive-sync included, but still see the error with that as well.

[sstimmel]

this i believe is resolved now. Having to build a custom hudi-aws-bundle jar that removes hudi-common from it. Also needed to add aws-sts to the bundle to run (Deltastreamer) from Kubernetes (EKS). Would be nice to not have to do that for all future releases though

[yihua]

Hi @sstimmel Thanks for the feedback. I created a ticket to track the improvement: HUDI-5466. Closing this issue now.