[SPARK-19833][SQL]remove SQLConf.HIVE_VERIFY_PARTITION_PATH, always return empty when the location does not exists #17176
Conversation
…eturn empty when the location does not exists
|
Test build #73991 has finished for PR 17176 at commit
|
windpiger
force-pushed
the
removeHiveVerfiyPath
branch
from
4bb0e28 to
8128567
Compare
| } | ||
| // convert /demo/data/year/month/day to /demo/data/*/*/*/ | ||
| def getPathPatternByPath(parNum: Int, tempPath: Path, partitionName: String): String = { | ||
| // if the partition path does not end with partition name, we should not |
There was a problem hiding this comment.
if the partition location has been altered to another location, we should not do this pattern, or we will list pattern files which does not belong to the partition
|
Test build #73998 has finished for PR 17176 at commit
|
|
Test build #73992 has finished for PR 17176 at commit
|
|
retest this please |
|
Test build #74016 has finished for PR 17176 at commit
|
|
why jenkins failed... |
|
Test build #74072 has finished for PR 17176 at commit
|
| def verifyPartitionPath( | ||
| partitionToDeserializer: Map[HivePartition, Class[_ <: Deserializer]]): | ||
| Map[HivePartition, Class[_ <: Deserializer]] = { | ||
| if (!sparkSession.sessionState.conf.verifyPartitionPath) { |
There was a problem hiding this comment.
after this pr https://github.com/apache/spark/pull/17187， read hive table which does not use stored by will not use HiveTableScanExec.
this function has a bug ,that if the partition path is custom path
- it will still do filter for all partition path in the parameter
partitionToDeserializer, - it will scan the path which does not belong to the table ,e.g. custom path is
/root/a
and the partitionSpec isb=1/c=2, this will lead to scan/because of thegetPathPatternByPath
|
Test build #74106 has finished for PR 17176 at commit
|
|
Test build #74107 has finished for PR 17176 at commit
|
|
@windpiger If you do not have a bandwidth to continue it, how about closing it now? |
| case (partition, partDeserializer) => | ||
| val partPath = partition.getDataLocation | ||
| val fs = partPath.getFileSystem(hadoopConf) | ||
| fs.exists(partPath) |
There was a problem hiding this comment.
Each partition sending an RPC request to the NameNode can result in poor performance
Closes apache#21766 Closes apache#21679 Closes apache#21161 Closes apache#20846 Closes apache#19434 Closes apache#18080 Closes apache#17648 Closes apache#17169 Add: Closes apache#22813 Closes apache#21994 Closes apache#22005 Closes apache#22463 Add: Closes apache#15899 Add: Closes apache#22539 Closes apache#21868 Closes apache#21514 Closes apache#21402 Closes apache#21322 Closes apache#21257 Closes apache#20163 Closes apache#19691 Closes apache#18697 Closes apache#18636 Closes apache#17176 Closes apache#23001 from wangyum/CloseStalePRs. Authored-by: Yuming Wang <[email protected]> Signed-off-by: hyukjinkwon <[email protected]>
4 participants
What changes were proposed in this pull request?
In SPARK-5068, we introduce a SQLConf spark.sql.hive.verifyPartitionPath,
if it is set to true, it will avoid the task failed when the patition location does not exists in the filesystem.
this situation should always return emtpy and don't lead to the task failed, here we remove this conf.
And the function
verifyPartitionPathhas a bug ,that if the partition path is custom pathit will still do filter for all partition path in the parameter
partitionToDeserializer,it will scan the path which does not belong to the table ,e.g. custom path is /root/a
and the partitionSpec is b=1/c=2, this will lead to scan / because of the getPathPatternByPath
How was this patch tested?
modify a test case