Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Spark: Positional deletes creates unpartitioned path
I was doing some work on the Python side: #6775 But ran into an issue when creating some integration tests for testing the positional deletes. I ended up with double slashes: s3://warehouse/default/test_positional_mor_deletes/data//00000-32-70be11f7-3c4b-40e0-b35a-334e97ef6554-00001-deletes.parquet It looks like the Struct is not-null, but the partition not partitioned, therefore it creates a partitioned path, but with the empty struct we'll end up with a double slash `//` that Minio doesn't like. Outputfactory.java ```java public EncryptedOutputFile newOutputFile(PartitionSpec spec, StructLike partition) { // partition is a StructCopy String newDataLocation = locations.newDataLocation(spec, partition, generateFilename()); OutputFile rawOutputFile = io.newOutputFile(newDataLocation); return encryptionManager.encrypt(rawOutputFile); } ``` ClusteredWriter.java ```java // copy the partition key as the key object may be reused this.currentPartition = StructCopy.copy(partition); // partition is a StructProjection this.currentWriter = newWriter(currentSpec, currentPartition); ``` I still have to dig into why there is a StructProjection. Resolves #7678
- Loading branch information