Skip to content

[SPARK-15648][SQL] Add teradataDialect for JDBC connection to Teradata #16746

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
klinvill wants to merge 4 commits into from
Closed

[SPARK-15648][SQL] Add teradataDialect for JDBC connection to Teradata #16746

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4b8d9a6
SPARK-15648: Added teradataDialect for JDBC connection
Jan 26, 2017
8258270
SPARK-15648: Added teradataDialect for JDBC connection to Teradata
Jan 26, 2017
0be3e0c
Merge branch 'master' into master
klinvill Feb 10, 2017
91e12e1
Merge branch 'master' into master
klinvill Mar 29, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@ object JdbcDialects {
registerDialect(MsSqlServerDialect)
registerDialect(DerbyDialect)
registerDialect(OracleDialect)
registerDialect(TeradataDialect)

/**
* Fetch the JdbcDialect class corresponding to a given database url.
Expand Down
34 changes: 34 additions & 0 deletions sql/core/src/main/scala/org/apache/spark/sql/jdbc/TeradataDialect.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.spark.sql.jdbc

import java.sql.Types
Copy link
Member

@dongjoon-hyun dongjoon-hyun Jan 31, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A blank line is needed here. You can run the following command line to check that and to confirm after fixing.

$ dev/lint-scala

Copy link
Author

@klinvill klinvill Jan 31, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Fixed in latest commit.


import org.apache.spark.sql.types._


private case object TeradataDialect extends JdbcDialect {

override def canHandle(url: String): Boolean = { url.startsWith("jdbc:teradata") }

override def getJDBCType(dt: DataType): Option[JdbcType] = dt match {
case StringType => Some(JdbcType("VARCHAR(255)", java.sql.Types.VARCHAR))
case BooleanType => Option(JdbcType("CHAR(1)", java.sql.Types.CHAR))
case _ => None
}
Copy link
Member

@dongjoon-hyun dongjoon-hyun Jan 31, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @klinvill .
According to the description and initial PR in SPARK-15648, Teradata didn't support LIMIT query at that time.
Now, does it supports LIMIT?

Copy link
Author

@klinvill klinvill Jan 31, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @dongjoon-hyun,
Teradata still doesn't support LIMIT (it uses TOP instead) but the spark code that was originally using limit has been changed to use "where 1=0 instead".

/**
   * Get the SQL query that should be used to find if the given table exists. Dialects can
   * override this method to return a query that works best in a particular database.
   * @param table  The name of the table.
   * @return The SQL query to use for checking the table.
   */
  def getTableExistsQuery(table: String): String = {
    s"SELECT * FROM $table WHERE 1=0"
  }

  /**
   * The SQL query that should be used to discover the schema of a table. It only needs to
   * ensure that the result set has the same schema as the table, such as by calling
   * "SELECT * ...". Dialects can override this method to return a query that works best in a
   * particular database.
   * @param table The name of the table.
   * @return The SQL query to use for discovering the schema.
   */
  @Since("2.1.0")
  def getSchemaQuery(table: String): String = {
    s"SELECT * FROM $table WHERE 1=0"
  }

Copy link
Member

@dongjoon-hyun dongjoon-hyun Jan 31, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Member

@dongjoon-hyun dongjoon-hyun Jan 31, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you verify if we need to override the followings together?

  override def quoteIdentifier(colName: String): String = ...
  override def getTableExistsQuery(table: String): String = ...
  override def isCascadingTruncateTable(): Option[Boolean] = ...

Copy link
Member

@dongjoon-hyun dongjoon-hyun Jan 31, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about isCascadingTruncateTable? Could you check if Teradata does truncate cascadingly by default for TRUNCATE TABLE statement?

Copy link
Author

@klinvill klinvill Feb 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quoteIdentifier and getTableExistsQuery will both work for Teradata. Teradata does not cascade by default but it also doesn't have a TRUNCATE TABLE command (DELETE is used instead) so any commands that use TRUNCATE TABLE will fail.

}
12 changes: 12 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -917,6 +917,18 @@ class JDBCSuite extends SparkFunSuite
assert(e2.contains("User specified schema not supported with `jdbc`"))
}

test("SPARK-15648: teradataDialect StringType data mapping") {
val teradataDialect = JdbcDialects.get("jdbc:teradata://127.0.0.1/db")
assert(teradataDialect.getJDBCType(StringType).
map(_.databaseTypeDefinition).get == "VARCHAR(255)")
}

test("SPARK-15648: teradataDialect BooleanType data mapping") {
val teradataDialect = JdbcDialects.get("jdbc:teradata://127.0.0.1/db")
assert(teradataDialect.getJDBCType(BooleanType).
map(_.databaseTypeDefinition).get == "CHAR(1)")
}

test("Checking metrics correctness with JDBC") {
val foobarCnt = spark.table("foobar").count()
val res = InputOutputMetricsHelper.run(sql("SELECT * FROM foobar").toDF())
Expand Down