java.lang.NoSuchMethodError: 'org.apache.iceberg.shaded.com.fasterxml.jackson.databind.ObjectMapper org.apache.iceberg.util.JsonUtil.mapper()'

[hoaiff]

Hi, I built a docker image cdc from dockerfile on the repo.
I ran it on local without any error but when I ran in server, I received an error as title. In addition, I tried to image on k8s env, and I faced with similar problem
This is a detail error:

cdc_mysql101  | 2024-02-28 04:15:25,071 ERROR [io.deb.ser.ConnectorLifecycle] (pool-7-thread-1) Connector completed: success = 'false', message = 'Stopping connector after error in the application's handler method: 'org.apache.iceberg.shaded.com.fasterxml.jackson.databind.ObjectMapper org.apache.iceberg.util.JsonUtil.mapper()'', error = 'java.lang.NoSuchMethodError: 'org.apache.iceberg.shaded.com.fasterxml.jackson.databind.ObjectMapper org.apache.iceberg.util.JsonUtil.mapper()'': java.lang.NoSuchMethodError: 'org.apache.iceberg.shaded.com.fasterxml.jackson.databind.ObjectMapper org.apache.iceberg.util.JsonUtil.mapper()'
cdc_mysql101  |         at org.apache.iceberg.hive.HiveTableOperations.setSnapshotSummary(HiveTableOperations.java:470)
cdc_mysql101  |         at org.apache.iceberg.hive.HiveTableOperations.setSnapshotStats(HiveTableOperations.java:461)
cdc_mysql101  |         at org.apache.iceberg.hive.HiveTableOperations.setHmsTableParameters(HiveTableOperations.java:440)
cdc_mysql101  |         at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:237)
cdc_mysql101  |         at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
cdc_mysql101  |         at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:400)
cdc_mysql101  |         at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
cdc_mysql101  |         at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
cdc_mysql101  |         at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
cdc_mysql101  |         at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
cdc_mysql101  |         at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:374)
cdc_mysql101  |         at io.debezium.server.iceberg.tableoperator.IcebergTableOperator.addToTablePerSchema(IcebergTableOperator.java:182)
cdc_mysql101  |         at io.debezium.server.iceberg.tableoperator.IcebergTableOperator.addToTable(IcebergTableOperator.java:157)
cdc_mysql101  |         at io.debezium.server.iceberg.IcebergChangeConsumer.handleBatch(IcebergChangeConsumer.java:167)
cdc_mysql101  |         at io.debezium.embedded.ConvertingEngineBuilder.lambda$notifying$2(ConvertingEngineBuilder.java:101)
cdc_mysql101  |         at io.debezium.embedded.EmbeddedEngine.pollRecords(EmbeddedEngine.java:728)
cdc_mysql101  |         at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:475)
cdc_mysql101  |         at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:229)
cdc_mysql101  |         at io.debezium.server.DebeziumServer.lambda$start$1(DebeziumServer.java:170)
cdc_mysql101  |         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
cdc_mysql101  |         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
cdc_mysql101  |         at java.base/java.lang.Thread.run(Unknown Source)
cdc_mysql101  | 
cdc_mysql101  | 2024-02-28 04:15:25,094 INFO  [io.deb.ser.DebeziumServer] (main) Received request to stop the engine
cdc_mysql101  | 2024-02-28 04:15:25,094 INFO  [io.deb.emb.EmbeddedEngine] (main) Stopping the embedded engine
cdc_mysql101  | 2024-02-28 04:15:25,119 INFO  [io.quarkus] (main) debezium-server-iceberg-dist stopped in 0.045s
cdc_mysql101 exited with code 1

[ismailsimsek]

@hoaiff upgraded iceberg version to 1.5.0 could you give it another try?

do you have the iceberg-hive-metastore artifact with the deployment? https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-hive-metastore

[hoaiff]

Thanks @ismailsimsek for reply

upgraded iceberg version to 1.5.0 could you give it another try?

I built the latest pom file but face still the same problem. Besides, I checked the jar files in the lib folder, I found that the version of spark and jackson are not similar in pom file.

This morning, I edited only one version of Iceberg in the pom file and built it into docker image with versions: 1.4.2, and 1.4.3. I tried to run images locally, on my server and pc of my partner:

• ver 1.4.3, in local, it worked well. In server and pc of the partner faced with same error
• ver 1.4.2, in server and pc of partner, it worked well. In local faced with the same error
  I don't understand why this could happen with the same image

[ismailsimsek]

@hoaiff iceberg 1.4.* versions using jackson 2.14.2. somehow this project was pulling 2.15.3

merged a fix. now latest release is having same jackson version as iceberg (2.14.2).

#274

[hoaiff]

@ismailsimsek, I built again your latest fixing, jackson 2.14.2 was pulled but the same error still
This is full log file:
full_log.log
This is application.properties file
application.txt

[quydx]

@ismailsimsek I face the same problem, the error randomly appears in different environments. I pulled the latest code but the error still exists.

[ismailsimsek]

@hoaiff @quydx aligned the iceberg runtime library and other libs. Any chance to test it again?

[hoaiff]

@ismailsimsek I tried your latest fixing but it doesn't work. I still receive the same error.

[hoaiff]

Hi @ismailsimsek, I built the image again and tested it. It seems to work well, but randomly env still error about log (maybe conflict version between SLF4J and Logback)

08:22:34.869 [pool-7-thread-1] INFO io.debezium.server.iceberg.offset.IcebergOffsetBackingStore - Stopped IcebergOffsetBackingStore table:test1.debezium_offset_storage_custom_table
Exception in thread "pool-7-thread-1" java.lang.NoSuchMethodError: 'boolean ch.qos.logback.core.util.OptionHelper.isNotEmtpy(java.lang.Object[])'
	at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:100)
	at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:63)
	at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:119)
	at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
	at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
	at ch.qos.logback.classic.Logger.error(Logger.java:538)
	at io.debezium.server.ConnectorLifecycle.handle(ConnectorLifecycle.java:87)
	at io.debezium.server.ConnectorLifecycle_ClientProxy.handle(Unknown Source)
	at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:500)
	at io.debezium.embedded.ConvertingEngineBuilder$1.run(ConvertingEngineBuilder.java:248)
	at io.debezium.server.DebeziumServer.lambda$start$1(DebeziumServer.java:170)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
08:22:34.911 [main] INFO io.debezium.server.DebeziumServer - Received request to stop the engine
08:22:34.911 [main] INFO io.debezium.embedded.EmbeddedEngine - Stopping the embedded engine
2024-03-04 08:22:34,955 INFO  [io.quarkus] (main) debezium-server-iceberg-dist stopped in 0.085s

[ismailsimsek]

@hoaiff thank you for the feedback i will check

[hoaiff]

@ismailsimsek, I tried to run on my K8s env and appeared error about the log as I commented above
Full log
cdc-865fc5d76c-sdfqg_cdc.log

[ismailsimsek]

@hoaiff this should be fixed now #289

[hoaiff]

Great, @ismailsimsek, thank you so much. I tested it on different environments, and it works well

[ismailsimsek]

Solved by #288 #289 #114

[ismailsimsek]

Thank you @hoaiff for testing and the feedback