Indexing and storage: Cross linking

Author: amotl

docs/explain/index.md

@@ -13,6 +13,11 @@ about applications and use cases of CrateDB, trying to put things into a
 bigger picture and joining things together to help answer the question _why_?
 
 
+:::{rubric} 2021
+:::
+
+- {ref}`indexing-and-storage`
+
 :::{rubric} 2018
 :::
 

docs/feature/index/index.md

@@ -84,23 +84,16 @@ with solutions from other vendors.
 
 
 
-::::{info-card}
-:::{grid-item}
-:columns: auto 9 9 9
-**Blog: Indexing and Storage in CrateDB**
-
-{{ '{}[Indexing and Storage in CrateDB]'.format(blog) }}
-
+::::{card} Blog: Indexing and Storage in CrateDB
+:link: indexing-and-storage
+:link-type: ref
 Learn about the fundamentals of the CrateDB storage layer,
 looking at the three main Lucene structures that are used within CrateDB:
-Inverted Indexes for text values, BKD-trees for numeric values, and Doc Values.
-:::
-:::{grid-item}
-:columns: auto 3 3 3
-{tags-primary}`Fundamentals` \
+Inverted indexes for text values, BKD trees for numeric values, and doc values.
++++
+{tags-primary}`Fundamentals`
 {tags-secondary}`Converged Indexing`
 {tags-secondary}`Deep Dive`
-:::
 ::::
 
 
@@ -159,7 +152,6 @@ bit thin.
 [Elasticsearch for Dummies]: https://dzone.com/articles/elasticsearch-for-dummies
 [Elasticsearch: Documents and Indices]: https://www.elastic.co/guide/en/elasticsearch/reference/current/documents-indices.html
 [Independent comparison of CrateDB and MongoDB using Time Series Benchmark Suite]: https://blog.nyrkio.com/wp-content/uploads/2024/07/Nyrkio-comparison-of-CrateDB-and-MongoDB-with-TSBS-v2.pdf
-[Indexing and Storage in CrateDB]: https://cratedb.com/blog/indexing-and-storage-in-cratedb
 [Searching and Indexing With Apache Lucene]: https://dzone.com/articles/apache-lucene-a-high-performance-and-full-featured
 [Time Series Benchmark on CrateDB and MongoDB]: https://blog.nyrkio.com/2024/07/11/timeseries-benchmark-on-cratedb-and-mongodb/
 [TimescaleDB Time Series Benchmark Suite (TSBS)]: https://github.com/timescale/tsbs

docs/feature/search/fts/index.md

@@ -61,6 +61,7 @@ of a [search engine].
 - {ref}`vector-search`
 - {ref}`hybrid-search`
 - {ref}`query`
+- {ref}`storage-layer`
 
 {tags-primary}`SQL`
 {tags-primary}`Full-Text Search`
@@ -301,41 +302,23 @@ by exploring how to manage a dataset of Netflix titles.
 :::{rubric} Explanations
 :::
 
-::::{info-card}
-:::{grid-item}
-:columns: auto 9 9 9
-**Indexing and Storage in CrateDB**
+:::{card} Indexing and Storage in CrateDB
+:link: indexing-and-storage
+:link-type: ref
 
 This article explores the internal workings of the storage layer in CrateDB,
 with a focus on Lucene's indexing strategies.
 
-{hyper-navigate}`Indexing and Storage in CrateDB <[Indexing and Storage in CrateDB]>`
-
 The CrateDB storage layer is based on Lucene indexes.
-Lucene offers scalable and high-performance indexing which enables efficient search
+Lucene offers scalable and high-performance indexing, which enables efficient search
 and aggregations over documents and rapid updates to the existing documents.
-We will look at the three main Lucene structures that are used within CrateDB:
-Inverted Indexes for text values, BKD-Trees for numeric values, and Doc Values.
-
-:Inverted Index:
-    You will learn how inverted indexes are implemented in Lucene and CrateDB.
-
-:BKD Tree:
-    Better understand the BKD tree, starting from KD trees, and how this data
-    structure supports range queries in CrateDB.
-
-:Doc Values:
-    This data structure supports more efficient querying document fields by id,
-    performs column-oriented retrieval of data, and improves the performance of
-    aggregation and sorting operations.
++++
+CrateDB uses three important Lucene data structures:
+Inverted indexes for text values, BKD trees for numeric values, and doc values.
 
-:::
-:::{grid-item}
-:columns: auto 3 3 3
-{tags-primary}`Introduction` \
+{tags-primary}`Introduction`
 {tags-secondary}`Lucene Indexing`
 :::
-::::
 
 
 :::{card} Indexing Text for Both Effective Search and Accurate Analysis
@@ -374,7 +357,6 @@ effective-search
 [BM25: The Next Generation of Lucene Relevance]: https://opensourceconnections.com/blog/2015/10/16/bm25-the-next-generation-of-lucene-relevation/
 [BM25 vs. Lucene Default Similarity]: https://www.elastic.co/blog/found-bm-vs-lucene-default-similarity
 [full-text search]: https://en.wikipedia.org/wiki/Full_text_search
-[Indexing and Storage in CrateDB]: https://cratedb.com/blog/indexing-and-storage-in-cratedb
 [MATCH predicate]: inv:crate-reference#predicates_match
 [Okapi BM25]: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/okapi_trec3.pdf
 [search engine]: https://en.wikipedia.org/wiki/Search_engine

docs/feature/storage/index.md

@@ -1,3 +1,4 @@
+(storage-internals)=
 (storage-layer)=
 # Storage Layer
 
@@ -11,8 +12,8 @@ The CrateDB storage layer is based on Lucene.
 By default, all fields are indexed,
 nested or not, but the indexing can be turned off selectively.
 
-This page enumerates some concepts of Lucene. The article [Indexing and Storage in
-CrateDB] goes into more details by exploring its internal workings.
+This page enumerates some concepts of Lucene. The article {ref}`indexing-and-storage`
+goes into more details by exploring its internal workings.
 
 ## Lucene
 
@@ -49,7 +50,7 @@ Elasticsearch are building upon the same technologies.
 ## Data structures
 
 CrateDB uses three main data structures of Lucene:
-Inverted indexes for text values, BKD trees for numeric values, and DocValues.
+Inverted indexes for text values, BKD trees for numeric values, and doc values.
 
 - **Inverted index**
 
@@ -69,7 +70,7 @@ Inverted indexes for text values, BKD trees for numeric values, and DocValues.
 
   To optimize numeric range queries, Lucene uses an implementation of the Block KD (BKD)
   tree data structure. The BKD tree index structure is suitable for indexing large
-  multi-dimensional point data sets. It is an I/O-efficient dynamic data structure based
+  multidimensional point data sets. It is an I/O-efficient dynamic data structure based
   on the KD tree. Contrary to its predecessors, the BKD tree maintains its high space
   utilization and excellent query and update performance regardless of the number of
   updates performed on it.
@@ -78,17 +79,25 @@ Inverted indexes for text values, BKD trees for numeric values, and DocValues.
   including fields defined as `TIMESTAMP` types, supporting performant date range
   queries.
 
-- **DocValues**
+- **Doc values**
 
   Because Lucene's inverted index data structure implementation is not optimal for
   finding field values by given document identifier, and for performing column-oriented
-  retrieval of data, the DocValues data structure is used for those purposes instead.
+  retrieval of data, the doc values data structure is used for those purposes instead.
 
-  DocValues is a column-based data storage built at document index time. They store
+  Doc values is a column-based data storage built at document index time. They store
   all field values that are not analyzed as strings in a compact column, making it more
   effective for sorting and aggregations.
 
+## See also
+
+- {ref}`indexing-and-storage`
+
+
+:::{toctree}
+:hidden:
+indexing-and-storage
+:::
 
 
 [column-based store]: https://cratedb.com/docs/crate/reference/en/latest/general/ddl/storage.html
-[Indexing and Storage in CrateDB]: https://cratedb.com/blog/indexing-and-storage-in-cratedb

docs/feature/storage/indexing-and-storage.md

@@ -1,5 +1,4 @@
 (indexing-and-storage)=
-(storage-internals)=
 
 # Indexing and storage in CrateDB
 
@@ -155,6 +154,9 @@ Lucene 6.0 adds an implementation of Block KD (BKD) tree data structure.
 
 ### BKD tree
 
+> Lucene's k-d tree geospatial data structure offers fast single- and
+multidimensional numeric range and geospatial point-in-shape filtering.
+
 To better understand the BKD tree data structure, let's begin with an
 introduction to KD trees. A KD tree is a binary tree for multidimensional
 queries. KD tree shares the same properties as binary search trees (BST), but
@@ -266,10 +268,6 @@ These values are quite fast to access at search time, since they are
 stored column-stride such that only the value for that one field needs
 to be decoded per row searched.
 
-:::{seealso}
--- [Document values with Apache Lucene]
-:::
-
 ### Column store
 
 CrateDB implements a {ref}`column store <crate-reference:ddl-storage-columnstore>`
@@ -305,9 +303,19 @@ the following:
 The use of a column store results in a small disk footprint, thanks to specialized
 compression algorithms such as delta encoding, bit packing, and GCD.
 
-Besides inverted indexes, the Lucene indexing strategy also relies on BKD trees
-and Doc Values that are successfully adopted by CrateDB as well as many popular
-search engines. With a better understanding of the storage layer, we move to
-another interesting topic: [Handling Dynamic Objects in CrateDB].
+## See also
+
+[Introducing Lucene Index Doc Values] is a technical deep dive into
+IndexDocValues introduced with Lucene 4.0.
+
+[Storing multidimensional points using BKD trees] is a comprehensive
+technical explanation about the benefits and design decisions
+behind the BKD tree geospatial data structure coming with Lucene 6.0.
+
+[Document values with Apache Lucene] highlights significant improvements to
+Apache Lucene 7.0 around how doc values are indexed and searched.
+
 
 [Document values with Apache Lucene]: https://www.elastic.co/blog/sparse-versus-dense-document-values-with-apache-lucene
+[Introducing Lucene Index Doc Values]: https://trifork.nl/blog/introducing-lucene-index-doc-values/
+[Storing multidimensional points using BKD trees]: https://www.elastic.co/blog/lucene-points-6-0