Skip to content

Include key metadata in manifest tables (Spark 4.1) #15041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
huaxingao merged 2 commits into from
Jan 13, 2026
+13 −2
Merged

Include key metadata in manifest tables (Spark 4.1) #15041

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6f080ac
Include key metadata in manifest tables (Spark 4.1)
iamthomaspowell Jan 13, 2026
dd6c5c1
spotless
iamthomaspowell Jan 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,8 @@ protected Dataset<FileInfo> contentFileDS(Table table, Set<Long> snapshotIds) {
"length",
"0 as sequenceNumber",
"partition_spec_id as partitionSpecId",
"added_snapshot_id as addedSnapshotId")
"added_snapshot_id as addedSnapshotId",
"key_metadata as keyMetadata")
.dropDuplicates("path")
.repartition(numShufflePartitions) // avoid adaptive execution combining tasks
.as(ManifestFileBean.ENCODER);
Expand Down
12 changes: 11 additions & 1 deletion spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/ManifestFileBean.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ public class ManifestFileBean implements ManifestFile, Serializable {
private Integer content = null;
private Long sequenceNumber = null;
private Long firstRowId = null;
private byte[] keyMetadata = null;

public static ManifestFileBean fromManifest(ManifestFile manifest) {
ManifestFileBean bean = new ManifestFileBean();
Expand All @@ -48,6 +49,7 @@ public static ManifestFileBean fromManifest(ManifestFile manifest) {
bean.setContent(manifest.content().id());
bean.setSequenceNumber(manifest.sequenceNumber());
bean.setFirstRowId(manifest.firstRowId());
bean.setKeyMetadata(manifest.keyMetadata() == null ? null : manifest.keyMetadata().array());

return bean;
}
Expand Down Expand Up @@ -104,6 +106,14 @@ public void setFirstRowId(Long firstRowId) {
this.firstRowId = firstRowId;
}

public byte[] getKeyMetadata() {
return keyMetadata;
}

public void setKeyMetadata(byte[] keyMetadata) {
this.keyMetadata = keyMetadata;
}

@Override
public String path() {
return path;
Expand Down Expand Up @@ -176,7 +186,7 @@ public List<PartitionFieldSummary> partitions() {

@Override
public ByteBuffer keyMetadata() {
return null;
return keyMetadata == null ? null : ByteBuffer.wrap(keyMetadata);
}

@Override
Expand Down