Skip to content

[SPARK-19329][SQL]Reading from or writing to a datasource table with a non pre-existing location should succeed #16672

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
windpiger wants to merge 13 commits into from
Closed

[SPARK-19329][SQL]Reading from or writing to a datasource table with a non pre-existing location should succeed #16672

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
5ec7dd6
[SPARK-19329][SQL]insert data to a not exist location datasource tabl…
windpiger Jan 22, 2017
2196648
modify a test name desc
windpiger Jan 22, 2017
abc57dd
add a param name
windpiger Jan 24, 2017
c3439ff
add read from an non exist path
windpiger Feb 12, 2017
409f7a4
add more test case
windpiger Feb 12, 2017
542f86b
modify a comment
windpiger Feb 12, 2017
2cbb9d6
remove some test code
windpiger Feb 12, 2017
c3e87d3
modify a comment
windpiger Feb 12, 2017
5ebd596
remove hive serde related and add partition test cases
windpiger Feb 13, 2017
334e89f
remove some import
windpiger Feb 13, 2017
b238e8d
move test case to DDLSuit
windpiger Feb 13, 2017
dee844c
remove an redundant import
windpiger Feb 13, 2017
0d947a5
fix some code style
windpiger Feb 14, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,8 @@ class FindDataSourceTable(sparkSession: SparkSession) extends Rule[LogicalPlan]
// TODO: improve `InMemoryCatalog` and remove this limitation.
catalogTable = if (withHiveSupport) Some(table) else None)

LogicalRelation(dataSource.resolveRelation(), catalogTable = Some(table))
LogicalRelation(dataSource.resolveRelation(checkFilesExist = false),
catalogTable = Some(table))
}
})
}
Expand Down
26 changes: 26 additions & 0 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1431,4 +1431,30 @@ class HiveDDLSuite
}
}
}

test("insert data to a table which has altered the table location " +
"to an not exist location should success") {
Copy link
Member

@gatorsmile gatorsmile Feb 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test case names are not accurate after you add new test cases. Actually, could you split the test cases?

withTable("t") {
withTempDir { dir =>
spark.sql(
s"""create table t(a string, b int)
Copy link
Member

@gatorsmile gatorsmile Feb 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General style suggestions. Please use upper case for SQL keywords. For example, in this SQL statement can be improved to

CREATE TABLE t(a STRING, b INT)
USING parquet
OPTIONS(path "xyz")

|using parquet
|options(path "${dir.getAbsolutePath}")
""".stripMargin)
Copy link
Member

@gatorsmile gatorsmile Feb 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A general comment about the style. We prefer to the following indentation styles.

        sql(
          """
            |SELECT '1' AS part, key, value FROM VALUES
            |(1, "one"), (2, "two"), (3, null) AS data(key, value)
          """.stripMargin)

var table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t"))
Copy link
Member

@gatorsmile gatorsmile Feb 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another general comment. Please avoid using var, if possible.

assert(table.location == dir.getAbsolutePath)

var newDir = dir.getAbsolutePath.stripSuffix("/") + "/x"
spark.sql(s"alter table t set location '$newDir'")

table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("t"))
assert(table.location == newDir)
assert(!new File(newDir).exists())
checkAnswer(spark.table("t"), Nil)

spark.sql("insert into table t select 'c', 1")
checkAnswer(spark.table("t"), Row("c", 1) :: Nil)
}
}
}
}