[SPARK-34498][SQL][TESTS] fix the remaining problems in #31560 #31621
Conversation
1. we don't need to implement SessionConfigSupport in simple writable table data source tests. remove it 2. change the schema of `SimpleWritableDataSource`, to match `TestingV2Source`
|
Jenkins test this please |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Test build #135430 has finished for PR 31621 at commit
|
| * Each job moves files from `target/_temporary/uniqueId/` to `target`. | ||
| */ | ||
| class SimpleWritableDataSource extends SimpleTableProvider with SessionConfigSupport { | ||
| class SimpleWritableDataSource extends SimpleTableProvider { |
There was a problem hiding this comment.
can we extend TestingV2Source instead?
There was a problem hiding this comment.
yes, i‘ll fix it
|
|
||
| override def write(record: InternalRow): Unit = { | ||
| out.writeBytes(s"${record.getLong(0)},${record.getLong(1)}\n") | ||
| out.writeBytes(s"${record.getInt(0)},${record.getInt(1)}\n") |
There was a problem hiding this comment.
This change just matches the TestingV2Source.schema?
| } | ||
|
|
||
| override def readSchema(): StructType = tableSchema | ||
| override def readSchema(): StructType = TestingV2Source.schema |
There was a problem hiding this comment.
SimpleScanBuilder.readSchema is already TestingV2Source.schema, we don't need to override it here.
There was a problem hiding this comment.
yes, it‘s unnecessary to override the function.
| private val conf = SparkContext.getActive.get.hadoopConfiguration | ||
|
|
||
| override def schema(): StructType = tableSchema | ||
| override def schema(): StructType = TestingV2Source.schema |
There was a problem hiding this comment.
ditto, SimpleBatchTable.schema is already TestingV2Source.schema
There was a problem hiding this comment.
yes, it‘s unnecessary.
…n MyTable, so delete these.
|
Jenkins test this please |
|
GA passed, merging to master, thanks! |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
4 participants
What changes were proposed in this pull request?
This is a followup of #31560,
In #31560, we added
JavaSimpleWritableDataSourceand left some little problems like unused interfaceSessionConfigSupport、 inconsistent schema betweenJavaSimpleWritableDataSourceandSimpleWritableDataSource.This PR fixes the remaining problems in #31560.
Why are the changes needed?
1.
SessionConfigSupportinJavaSimpleWritableDataSourceandSimpleWritableDataSourceis never used, so we don't need to implement it.2. change the schema of
SimpleWritableDataSource, to matchTestingV2SourceDoes this PR introduce any user-facing change?
NO
How was this patch tested?
existing testsuites