Docs: Update spec about statistics file snapshot id #6267
Conversation
ajantha-bhat
force-pushed
the
stats
branch
4 times, most recently
from
8537ce6 to
1e0906e
Compare
| private List<Snapshot> snapshots; | ||
| private final Map<String, SnapshotRef> refs; | ||
| private final Map<Long, List<StatisticsFile>> statisticsFiles; | ||
| private final Map<Long, List<StatisticsFile>> statisticsFilesById; |
There was a problem hiding this comment.
I thought better to rename this similar to snapshotsById
| "snapshotId does not match: %s vs %s", | ||
| snapshotId, | ||
| statisticsFile.snapshotId()); | ||
| statisticsFiles.put(statisticsFile.snapshotId(), ImmutableList.of(statisticsFile)); |
There was a problem hiding this comment.
The interface supports returning multiple stats file for one snapshot. But was always overwriting instead of appending. So, fixed it.
There was a problem hiding this comment.
This is not correct. There should be only one stats file for a snapshot that contains all information. Allowing multiple stats files for a snapshot is going to lead to lazy implementations that don't merge the files and slower job planning.
There was a problem hiding this comment.
@rdblue, @findepi:
It is just as per the previous existing interface in the builder.
we had Map<Long, List<StatisticsFile>> statisticsFiles in builder.
If we wanted one stats file per interface then no need to have list for map value?
Also is it possible that there will be one partition level stats (avro/parquet file) and there is one table level NDV file per snapshot. In this case, no need to merge them but can be used based on the need. So, still ok to have multiple stats per snapshot id?
| Assert.assertEquals("Statistics file path", "/some/path/to/stats/file2", statisticsFile.path()); | ||
| Assertions.assertThat(withStatisticsAppended.statisticsFiles()) | ||
| .as("There should be two statistics files registered") | ||
| .hasSize(2) |
There was a problem hiding this comment.
As I fixed the overwrite to append. There will be two stats file for this snapshot id. Hence, updated the testcases.
|
@ajantha-bhat, I'm fine with the update to the spec wording, but I don't think the rest of these changes are needed. There should be only one stats file per snapshot in metadata. There is no value in having multiple stats files and implementations should merge them. |
I think we still need an interface to get the current stats file for snapshot id
Maybe true. But It was just as per the previous existing interface in the builder. Also is it possible that there will be one partition level stats (avro/parquet file) and there is one table level NDV file per snapshot. In this case, no need to merge them but can be used based on the need. So, still ok to have multiple stats per snapshot id? |
No, I don't think this is valuable. This depends too much on some definition of "current", which is about as useful as asking for any stats file. This should be more specific. I agree that we want some way to get a stats file, but I don't think there's much value in the "current" idea.
I wouldn't worry too much about intermediate representations. What we have in the current API is fine for now. I don't think that we should make the changes in this PR. |
ajantha-bhat
changed the title
|
@rdblue: Thanks for the review and suggestions. I have kept it as just the document (spec) update now. |
|
Thanks, @ajantha-bhat! Merging this. |
3 participants
Background:
Spec has a snapshot id in two places, one in
StatisticsFileand another in itsblob metadata.To support the reuse of statistics files, we should have the referenced snapshot id in
StatisticsFile, not the computed-from snapshot id. Hence, updated the spec.Note that PR #6090 is stuck because of confusion around stats file reuse.