Skip to content
This repository was archived by the owner on Feb 6, 2024. It is now read-only.

Add statistics information in table snapshot #77

Closed
findepi wants to merge 2 commits into from
Closed

Add statistics information in table snapshot #77

findepi wants to merge 2 commits into from

Conversation

@findepi
Copy link
Member

@findepi findepi commented May 5, 2022

Add table snapshot's anchor for Iceberg table-level stats as discussed in https://docs.google.com/document/d/1we0BuQbbdqiJS2eUFC_-6TPSuO57GXivzKmcTzApivY

@findepi
Copy link
Member Author

findepi commented May 5, 2022

Currently, based on #69, the format specification itself.

@findepi findepi force-pushed the findepi/stats-in-table branch from 2b7bdb9 to 3820f00 Compare May 5, 2022 13:04
@findepi
Copy link
Member Author

findepi commented May 6, 2022

cc @RussellSpitzer @rdblue

@findepi findepi force-pushed the findepi/stats-in-table branch from 3820f00 to 66819b7 Compare May 6, 2022 12:16
@findepi findepi changed the title Add statistics link in table snapshot Add statistics information in table snapshot May 6, 2022
@findepi findepi mentioned this pull request May 10, 2022
Merged
@findepi
Copy link
Member Author

findepi commented May 10, 2022

Proposed implementation: apache/iceberg#4741

@findepi findepi mentioned this pull request May 12, 2022
Merged
@findepi findepi force-pushed the findepi/stats-in-table branch from 66819b7 to 1551f17 Compare May 12, 2022 14:13
@findepi findepi mentioned this pull request May 12, 2022
Merged
@findepi findepi force-pushed the findepi/stats-in-table branch from 1551f17 to 139f8e2 Compare May 13, 2022 09:51
@findepi
Copy link
Member Author

findepi commented May 13, 2022

@rdblue @danielcweeks @RussellSpitzer please take a look

@findepi findepi force-pushed the findepi/stats-in-table branch from 139f8e2 to 6341e8f Compare May 18, 2022 12:27
@findepi
Copy link
Member Author

findepi commented May 18, 2022

Rebased on current state of #69

@findepi
Copy link
Member Author

findepi commented May 18, 2022

@rdblue did you happen to have time to see this?

@findepi findepi force-pushed the findepi/stats-in-table branch 2 times, most recently from 6a2d0e3 to 5f38d3b Compare May 20, 2022 07:22
findepi added 2 commits May 26, 2022 14:42
@findepi
Add Puffin, indices and stats file format specification
0c603c4 
Add a specification for Puffin format, a container file format to store
indices and stats for Iceberg tables.
@findepi
Add statistics information in table snapshot
934ec5e
@findepi findepi force-pushed the findepi/stats-in-table branch from 5f38d3b to 934ec5e Compare May 26, 2022 12:43
@findepi
Copy link
Member Author

findepi commented May 27, 2022 •
edited
Loading

As @pvary pointed out, there are similar (but not equal) spec.md files in iceberg and iceberg-docs repo.

Both seem to be under development, see below.
@rdblue where should this PR be targeted at?

iceberg master$ git lg ./format/spec.md
abe6ba4ba - Spec: Add note about reserved field id 141 in manifests (#4750) (9 days ago) <Szehon Ho>
387e41468 - Spec: Clarify manifest length is size in bytes, fix a typo (#4793) (10 days ago) <emkornfield>
248350684 - Docs: Default value support feature specification (#4301) (4 weeks ago) <Limian (Raymond) Zhang>
f4c45e8a0 - Spec: Clarify manifest entry file uniqueness (#4272) (9 weeks ago) <emkornfield>
4d2ff3f87 - Spec: Clarify that floating point bounds cannot contain NaNs (#4404) (9 weeks ago) <emkornfield>
dc2862776 - Docs: Clarify "read-optimized" (#4352) (10 weeks ago) <wuwenchi>
dc7e4dca7 - Docs: Restructure docs files for new contributing guideline (#4208) (3 months ago) <Samuel Redai>
02433f22c - Docs: Fix numeration in Metastore Tables section (#4195) (3 months ago) <Christopher Lambert>
90637d664 - Doc: Delete duplicate contents in spec (#4156) (3 months ago) <liliwei>
c47d8cf0a - Docs: add new site markdown files (#4062) (4 months ago) <Jack Ye>

iceberg-docs main$ git lg ./landing-page/content/common/spec.md
4f8dd64 - Fix spec links (#68) (8 weeks ago) <Samuel Redai> (origin/main, origin/HEAD)
b69dc3f - Add anchor tags and table of contents (#63) (2 months ago) <Samuel Redai>
ad06e67 - Update contributing guidelines to direct common page contributions here (#59) (3 months ago) <Samuel Redai>
cb037b9 - Fix spec.md (#47) (4 months ago) <Prashant Singh>
9997c35 - Add missing snapshot image in spec page (#43) (4 months ago) <Samuel Redai>

@findepi
Copy link
Member Author

findepi commented Jun 2, 2022

per #69 (comment)

this is replaced with apache/iceberg#4945
thanks for review to everyone involved

@findepi findepi closed this Jun 2, 2022
@findepi findepi deleted the findepi/stats-in-table branch June 2, 2022 12:13
@findepi findepi mentioned this pull request Jun 2, 2022
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@findepi