Error: java.sql.SQLException: No suitable driver

[Rap70r]

Hello,

When using latest version of metorikku (0.0.148) with jdbc connector (mssql-jdbc-8.4.1.jre8.jar) as an external jar file, it fails with the following error:

User class threw exception: java.sql.SQLException: No suitable driver

When switching back to version 0.0.143, it works fine.

Spark Submit Command:

spark-submit --deploy-mode cluster --jars s3://some_bucket/libs/mssql-jdbc-8.4.1.jre8.jar --class com.yotpo.metorikku.Metorikku s3://some_bucket/libs/metorikku.jar -c s3://some_bucket/some_file_config.yaml

Were there any changes made that require an additional library to the classpath?

Thank you

[golankiviti]

Hi, the changes in the new version shouldn't effect anything. However, this exception usually happens when the executor doesn't have the jar in it's classpath. Try supplying it explicitly to the executor's classpath using spark configuration.
If you can supply us a way to reproduce the issues we can look into it.

[Rap70r]

Hi @golankiviti,

I tried supplying the jar to the classpath using multiple configs with no luck. This only happens for the latest versions. I'm able to run the job with older versions.

The following works fine with version: 0.0.143

spark-submit --deploy-mode cluster --master yarn --conf spark.yarn.maxAppAttempts=1 --jars s3://some_bucket/jars/mssql-jdbc-8.4.1.jre8.jar --class com.yotpo.metorikku.Metorikku s3://some_bucket/jars/metorikku.jar -c s3://some_bucket/args/some_config.yaml

But it gives me that error when running with version: 0.0.148.

I do see significant changes between the 2 versions:

https://github.com/YotpoLtd/metorikku/blob/v0.0.143/src/main/scala/com/yotpo/metorikku/input/readers/jdbc/JDBCInput.scala

VS

https://github.com/YotpoLtd/metorikku/blob/v0.0.148/src/main/scala/com/yotpo/metorikku/input/readers/jdbc/JDBCInput.scala

This is part of my config file:

metrics:
  - s3://some_bucket/args/some_metric.yaml
inputs:
  input_name1:
    jdbc:
      connectionUrl: jdbc:sqlserver://hostname:1433
      table: db_name.dbo.some_table
      options:
        driver: com.microsoft.sqlserver.jdbc.SQLServerDriver
        partitionColumn: some_column
        lowerBound: 2001-12-21 00:00:00.000
        upperBound: 2022-11-09 12:00:00.000
        numPartitions: 5000