diff --git a/docs/_index.md b/docs/_index.md new file mode 100644 index 000000000000..bc8f35c44b4f --- /dev/null +++ b/docs/_index.md @@ -0,0 +1,55 @@ +--- +title: "Introduction" +menu: main +weight: 0 +--- + + +![Iceberg](./img/Iceberg-logo.png) + +**Apache Iceberg is an open table format for huge analytic datasets.** Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink and Hive using a high-performance table format that works just like a SQL table. + +### User experience + +Iceberg avoids unpleasant surprises. Schema evolution works and won't inadvertently un-delete data. Users don't need to know about partitioning to get fast queries. + +* [Schema evolution](evolution#schema-evolution) supports add, drop, update, or rename, and has [no side-effects](evolution#correctness) +* [Hidden partitioning](partitioning) prevents user mistakes that cause silently incorrect results or extremely slow queries +* [Partition layout evolution](evolution#partition-evolution) can update the layout of a table as data volume or query patterns change +* [Time travel](spark-queries#time-travel) enables reproducible queries that use exactly the same table snapshot, or lets users easily examine changes +* Version rollback allows users to quickly correct problems by resetting tables to a good state + +### Reliability and performance + +Iceberg was built for huge tables. Iceberg is used in production where a single table can contain tens of petabytes of data and even these huge tables can be read without a distributed SQL engine. + +* [Scan planning is fast](performance#scan-planning) -- a distributed SQL engine isn't needed to read a table or find files +* [Advanced filtering](performance#data-filtering) -- data files are pruned with partition and column-level stats, using table metadata + +Iceberg was designed to solve correctness problems in eventually-consistent cloud object stores. + +* [Works with any cloud store](reliability) and reduces NN congestion when in HDFS, by avoiding listing and renames +* [Serializable isolation](reliability) -- table changes are atomic and readers never see partial or uncommitted changes +* [Multiple concurrent writers](reliability#concurrent-write-operations) use optimistic concurrency and will retry to ensure that compatible updates succeed, even when writes conflict + +### Open standard + +Iceberg has been designed and developed to be an open community standard with a [specification](../../spec) to ensure compatibility across languages and implementations. + +[Apache Iceberg is open source](../../community), and is developed at the [Apache Software Foundation](https://www.apache.org/). + diff --git a/docs/api/_index.md b/docs/api/_index.md deleted file mode 100644 index 23ff2e83c810..000000000000 --- a/docs/api/_index.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -bookIconFa: fa-connectdevelop -weight: 700 -bookCollapseSection: true ---- \ No newline at end of file diff --git a/docs/api/javadoc/_index.md b/docs/api/javadoc/_index.md deleted file mode 100644 index 11430fc59e2a..000000000000 --- a/docs/api/javadoc/_index.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -title: "Javadocs" -weight: 310 -javadocLink: true ---- \ No newline at end of file diff --git a/docs/asf/_index.md b/docs/asf/_index.md deleted file mode 100644 index 114d1da22398..000000000000 --- a/docs/asf/_index.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -title: "ASF" -bookIconImage: ../img/asf.png -bookCollapseSection: true -weight: 1200 ---- \ No newline at end of file diff --git a/docs/asf/donate/_index.md b/docs/asf/donate/_index.md deleted file mode 100644 index 7a8edb6db17f..000000000000 --- a/docs/asf/donate/_index.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: "Donate" -weight: 400 -bookExternalUrlNewWindow: https://www.apache.org/foundation/sponsorship.html -bookIconFa: fa-external-link ---- - \ No newline at end of file diff --git a/docs/asf/events/_index.md b/docs/asf/events/_index.md deleted file mode 100644 index 0cc7a3b074d2..000000000000 --- a/docs/asf/events/_index.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: "Events" -weight: 500 -bookExternalUrlNewWindow: https://www.apache.org/events/current-event.html -bookIconFa: fa-external-link ---- - \ No newline at end of file diff --git a/docs/asf/license/_index.md b/docs/asf/license/_index.md deleted file mode 100644 index 6e111e7a90d8..000000000000 --- a/docs/asf/license/_index.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: "License" -weight: 100 -bookExternalUrlNewWindow: https://www.apache.org/licenses/ -bookIconFa: fa-external-link ---- - \ No newline at end of file diff --git a/docs/asf/security/_index.md b/docs/asf/security/_index.md deleted file mode 100644 index 198c81f3f4bc..000000000000 --- a/docs/asf/security/_index.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: "Security" -weight: 200 -bookExternalUrlNewWindow: https://www.apache.org/security/ -bookIconFa: fa-external-link ---- - \ No newline at end of file diff --git a/docs/asf/sponsors/_index.md b/docs/asf/sponsors/_index.md deleted file mode 100644 index 0b76124d5041..000000000000 --- a/docs/asf/sponsors/_index.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: "Sponsors" -weight: 300 -bookExternalUrlNewWindow: https://www.apache.org/foundation/thanks.html -bookIconFa: fa-external-link ---- - \ No newline at end of file diff --git a/docs/athena/_index.md b/docs/athena/_index.md deleted file mode 100644 index fbdb137b8457..000000000000 --- a/docs/athena/_index.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: "Amazon Athena" -bookIconImage: ../img/athena-logo.png -bookFlatSection: true -weight: 440 -bookExternalUrlNewWindow: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html ---- - \ No newline at end of file diff --git a/docs/integrations/aws.md b/docs/aws.md similarity index 99% rename from docs/integrations/aws.md rename to docs/aws.md index a159f51abf8b..b34488f22574 100644 --- a/docs/integrations/aws.md +++ b/docs/aws.md @@ -1,6 +1,10 @@ --- title: "AWS" url: aws +menu: + main: + parent: Integrations + weight: 0 --- \ No newline at end of file diff --git a/docs/emr/_index.md b/docs/emr/_index.md deleted file mode 100644 index 0d1fd7f7bd75..000000000000 --- a/docs/emr/_index.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: "Amazon EMR" -bookIconImage: ../img/emr-logo.png -bookFlatSection: true -weight: 450 -bookExternalUrlNewWindow: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-iceberg-use-cluster.html ---- - \ No newline at end of file diff --git a/docs/tables/evolution.md b/docs/evolution.md similarity index 98% rename from docs/tables/evolution.md rename to docs/evolution.md index ae094355f722..3f3f18eefb50 100644 --- a/docs/tables/evolution.md +++ b/docs/evolution.md @@ -1,7 +1,12 @@ --- +title: Evolution url: evolution aliases: - "tables/evolution" +menu: + main: + parent: Tables + weight: 0 --- \ No newline at end of file diff --git a/docs/format/terms.md b/docs/format/terms.md deleted file mode 100644 index 952b80e2ea12..000000000000 --- a/docs/format/terms.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: "Terms" -bookUrlFromBaseURL: /../../terms ---- - \ No newline at end of file diff --git a/docs/format/view-spec.md b/docs/format/view-spec.md deleted file mode 100644 index 982f67cfc9bf..000000000000 --- a/docs/format/view-spec.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: "View Spec" -bookUrlFromBaseURL: /../../view-spec ---- - \ No newline at end of file diff --git a/docs/hive/_index.md b/docs/hive.md similarity index 60% rename from docs/hive/_index.md rename to docs/hive.md index cf47121109c2..d9cc3b041b44 100644 --- a/docs/hive/_index.md +++ b/docs/hive.md @@ -1,8 +1,8 @@ --- -bookIconImage: ../img/hive-logo.png -bookFlatSection: true +title: "Hive" url: hive weight: 400 +menu: main --- \ No newline at end of file diff --git a/docs/project/_index.md b/docs/project/_index.md deleted file mode 100644 index 1dd2bc810e21..000000000000 --- a/docs/project/_index.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -bookIconFa: fa-wrench -bookCollapseSection: true -weight: 900 ---- \ No newline at end of file diff --git a/docs/project/benchmarks.md b/docs/project/benchmarks.md deleted file mode 100644 index 2d45cf267d0c..000000000000 --- a/docs/project/benchmarks.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: "Benchmarks" -bookUrlFromBaseURL: /../../benchmarks -bookHidden: true ---- - \ No newline at end of file diff --git a/docs/project/how-to-release.md b/docs/project/how-to-release.md deleted file mode 100644 index 81364f8c640c..000000000000 --- a/docs/project/how-to-release.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: "How to Release" -bookUrlFromBaseURL: /../../how-to-release ---- - \ No newline at end of file diff --git a/docs/project/multi-engine-support.md b/docs/project/multi-engine-support.md deleted file mode 100644 index 60cd5d0ccd8b..000000000000 --- a/docs/project/multi-engine-support.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: "Multi-Engine Support" -bookUrlFromBaseURL: /../../multi-engine-support ---- - \ No newline at end of file diff --git a/docs/project/roadmap.md b/docs/project/roadmap.md deleted file mode 100644 index c4718d23fd9d..000000000000 --- a/docs/project/roadmap.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: "Roadmap" -bookUrlFromBaseURL: /../../roadmap ---- - \ No newline at end of file diff --git a/docs/project/security.md b/docs/project/security.md deleted file mode 100644 index bc25db12ecca..000000000000 --- a/docs/project/security.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: "Security" -bookUrlFromBaseURL: /../../security ---- - \ No newline at end of file diff --git a/docs/project/trademarks.md b/docs/project/trademarks.md deleted file mode 100644 index 3a02c4b40df2..000000000000 --- a/docs/project/trademarks.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: "Trademarks" -bookUrlFromBaseURL: /../../trademarks ---- - \ No newline at end of file diff --git a/docs/api/python-api-intro.md b/docs/python-api-intro.md similarity index 98% rename from docs/api/python-api-intro.md rename to docs/python-api-intro.md index 4fd9afbb3f25..80c1039d0cef 100644 --- a/docs/api/python-api-intro.md +++ b/docs/python-api-intro.md @@ -1,9 +1,12 @@ --- -weight: 500 title: "Python API" url: python-api-intro aliases: - "python/api-intro" +menu: + main: + parent: "API" + weight: 500 --- \ No newline at end of file diff --git a/docs/tables/reliability.md b/docs/reliability.md similarity index 98% rename from docs/tables/reliability.md rename to docs/reliability.md index 999383fb570b..b06ab2f5e74e 100644 --- a/docs/tables/reliability.md +++ b/docs/reliability.md @@ -1,7 +1,12 @@ --- +title: Reliability" url: reliability aliases: - "tables/reliability" +menu: + main: + parent: Tables + weight: 0 --- diff --git a/docs/tables/_index.md b/docs/tables/_index.md deleted file mode 100644 index 033a32a6f1c4..000000000000 --- a/docs/tables/_index.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -bookIconFa: fa-table -bookFlatSection: true -weight: 100 ---- \ No newline at end of file diff --git a/docs/community/talks.md b/docs/talks.md similarity index 90% rename from docs/community/talks.md rename to docs/talks.md index dcab56a4fd67..0695cf743282 100644 --- a/docs/community/talks.md +++ b/docs/talks.md @@ -1,7 +1,10 @@ --- title: "Talks" -bookUrlFromBaseURL: /../../talks -weight: 300 +relurl: "../../talks" +menu: + main: + parent: Community + weight: 0 --- \ No newline at end of file