Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 0 additions & 9 deletions .github/workflows/docs-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,27 +59,18 @@ jobs:
- Namespace Spec: namespace
EOF
cp docs/src/format/namespace/rest.yaml docs/src/rest.yaml
cp lance-namespace/CONTRIBUTING.md docs/src/community/contributing/namespace.md
# Update contributing .pages to include namespace.md in navbar
echo " - Namespace: namespace.md" >> docs/src/community/contributing/.pages
- name: Copy lance-spark docs
run: |
cp -r lance-spark/docs/src docs/src/integrations/spark
cat >> docs/src/integrations/.pages << 'EOF'
- Apache Spark: spark
EOF
cp lance-spark/CONTRIBUTING.md docs/src/community/contributing/spark.md
# Update contributing .pages to include spark.md in navbar
echo " - Apache Spark: spark.md" >> docs/src/community/contributing/.pages
- name: Copy lance-ray docs
run: |
cp -r lance-ray/docs/src docs/src/integrations/ray
cat >> docs/src/integrations/.pages << 'EOF'
- Ray: ray
EOF
cp lance-ray/CONTRIBUTING.md docs/src/community/contributing/ray.md
# Update contributing .pages to include ray.md in navbar
echo " - Ray: ray.md" >> docs/src/community/contributing/.pages
- name: Deploy
working-directory: docs
run: uv run mkdocs gh-deploy --force
6 changes: 3 additions & 3 deletions docs/src/community/contributing/index.md → CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Currently Lance is implemented in Rust and comes with a Python wrapper. So you'l

## Example Notebooks

Example notebooks are under `examples`.
Example notebooks are under `examples`.
These are standalone notebooks you should be able to download and run.

## Benchmarks
Expand All @@ -50,5 +50,5 @@ These live under `benchmarks`.

## Code of Conduct

We follow the Code of Conduct of [Python Foundation](https://www.python.org/psf/conduct/) and
[Rust Foundation](https://www.rust-lang.org/policies/code-of-conduct).
We follow the Code of Conduct of [Python Foundation](https://www.python.org/psf/conduct/) and
[Rust Foundation](https://www.rust-lang.org/policies/code-of-conduct).
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,4 @@ as a part of the release process.
### Java Generated Doc

Java code documentation is built and published to Maven Central.
You can find the doc page for the specific project at [javadoc.io](https://javadoc.io).
You can find the doc page for the specific project at [javadoc.io](https://javadoc.io).
11 changes: 5 additions & 6 deletions docs/mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,14 @@ theme:
favicon: logo/logo.png
palette:
- scheme: default
primary: indigo
accent: indigo
primary: custom
accent: custom
toggle:
icon: material/brightness-7
name: Switch to dark mode
- scheme: slate
primary: indigo
accent: indigo
primary: custom
accent: custom
toggle:
icon: material/brightness-4
name: Switch to light mode
Expand Down Expand Up @@ -67,13 +67,12 @@ plugins:
proto_dir: ../protos

extra:
generator: false
social:
- icon: fontawesome/brands/github
link: https://github.com/lancedb/lance
- icon: fontawesome/brands/discord
link: https://discord.gg/zMM32dvNtd
- icon: fontawesome/brands/twitter
link: https://twitter.com/lancedb

copyright: © 2025 Lance Format. All rights reserved.

Expand Down
165 changes: 96 additions & 69 deletions docs/overrides/home.html
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,84 @@
.md-main__inner {
margin: 0;
}

/* Style Learn More and View Integrations links */
.lance-feature-section .md-button,
.lance-intro-section .md-button:not(.md-button--primary) {
background-color: transparent;
color: #625EFF;
text-decoration: underline;
border: none;
box-shadow: none;
}

.lance-feature-section .md-button:hover,
.lance-intro-section .md-button:not(.md-button--primary):hover {
background-color: transparent;
color: #757575;
text-decoration: underline;
}
</style>

<script>
// Fix palette toggle jumping to bottom of page
(function() {
// Store scroll position
let savedScrollY = 0;
let isTogglingPalette = false;

// Prevent hash-based scrolling for palette changes
if ('scrollRestoration' in history) {
history.scrollRestoration = 'manual';
}

// Intercept scroll attempts during palette toggle
const preventScroll = function() {
if (isTogglingPalette) {
window.scrollTo(0, savedScrollY);
}
};

// Save scroll position and intercept clicks on palette toggle
document.addEventListener('click', function(e) {
const paletteLabel = e.target.closest('[for^="__palette"]');
if (paletteLabel) {
savedScrollY = window.scrollY;
isTogglingPalette = true;

// Prevent any scrolling for a brief moment
window.addEventListener('scroll', preventScroll, { passive: false });

setTimeout(function() {
isTogglingPalette = false;
window.removeEventListener('scroll', preventScroll);
}, 100);
}
}, true);

// Listen for hash changes from palette toggle
window.addEventListener('hashchange', function(e) {
if (window.location.hash.startsWith('#__palette')) {
e.preventDefault();
// Immediately restore scroll position
window.scrollTo(0, savedScrollY);
// Clean up hash without triggering another hashchange
history.replaceState(null, null, window.location.pathname);
}
}, false);

// Also handle palette input changes directly
document.addEventListener('DOMContentLoaded', function() {
const paletteInputs = document.querySelectorAll('input[name="__palette"]');
paletteInputs.forEach(function(input) {
input.addEventListener('change', function() {
window.scrollTo(0, savedScrollY);
});
});
});
})();
</script>

<!-- Hero Section -->
<section class="mdx-container">
<div class="container">
Expand Down Expand Up @@ -76,7 +152,7 @@ <h2>What is Lance<sup>™</sup>?</h2>
allowing you to build a complete open lakehouse on top of object storage to power your AI workflows.
Lance brings high-performance vector search, full-text search, random access, and feature
engineering capabilities to the lakehouse, while you can still get all the existing lakehouse benefits
like SQL analytics, ACID transactions, time travel, and integrations with open engines (Apache Spark, Ray, Trino, DuckDB, etc.)
like SQL analytics, ACID transactions, time travel, and integrations with open engines (Apache Spark, Ray, PyTorch, Trino, DuckDB, etc.)
and open catalogs (Apache Polaris, Unity Catalog, Apache Gravitino, Hive Metastore, etc.)
</p>
<a href="quickstart" class="md-button md-button--primary">Learn More</a>
Expand All @@ -93,7 +169,7 @@ <h2>Expressive Hybrid Search</h2>
<p>
Lance enables powerful hybrid search combining vector similarity, full-text search,
and SQL analytics on the same dataset. All query types are accelerated by corresponding
secondary indices as part of the Lance specification.
secondary indexes as part of the Lance specification.
</p>
<p>
Run semantic search on embeddings, BM25 search on keywords, and apply complex SQL predicates -
Expand All @@ -102,19 +178,7 @@ <h2>Expressive Hybrid Search</h2>
<a href="quickstart/vector-search" class="md-button">Learn More</a>
</div>
<div class="lance-feature-demo">
<div id="termynal-hybrid-search"
data-termynal=""
data-ty-startdelay="500"
data-ty-typedelay="40"
data-ty-linedelay="700"
style="width: 500px;">
<span data-ty="input" data-ty-prompt=">>>">import lance</span>
<span data-ty="input" data-ty-prompt=">>>">ds = lance.dataset("s3://my-bucket/docs")</span>
<span data-ty="input" data-ty-prompt=">>>">ds.to_table(full_text_query="machine learning")</span>
<span data-ty="input" data-ty-prompt=">>>">ds.to_table(</span>
<span data-ty="input" data-ty-prompt="..."> nearest={"column": "embedding", "q": query_vec, "k": 10},</span>
<span data-ty="input" data-ty-prompt="..."> filter="year > 2020")</span>
</div>
<img src="assets/images/hybrid-search.png" alt="Hybrid Search Example" style="max-width: 500px; width: 100%; height: auto; border-radius: 8px;">
</div>
</div>
</div>
Expand All @@ -127,28 +191,19 @@ <h2>Expressive Hybrid Search</h2>
<div class="lance-feature-text">
<h2>Lightning-fast Random Access</h2>
<p>
Lance delivers 100x faster random access compared to Parquet or Iceberg. With efficient
row-addressing, you can access individual records across multiple files instantly,
making it perfect for real-time ML serving, random sampling, and interactive applications.
Lance delivers 100x faster random access compared to Parquet or Iceberg.
Unlike traditional formats, Lance maintains high performance even when
randomly accessing scattered rows across your entire dataset.
</p>
<p>
Unlike traditional columnar formats, Lance maintains high performance even when
randomly accessing scattered rows across your entire dataset.
With a highly optimized file format plus efficient row-addressing and secondary indexes at table level,
you can access individual records across multiple files instantly,
making it perfect for real-time ML serving, random sampling, and interactive applications.
</p>
<a href="guide/read_and_write#random-access" class="md-button">Learn More</a>
</div>
<div class="lance-feature-demo">
<div id="termynal-random-access"
data-termynal=""
data-ty-startdelay="1500"
data-ty-typedelay="40"
data-ty-linedelay="700"
style="width: 500px;">
<span data-ty="input" data-ty-prompt=">>>">import lance</span>
<span data-ty="input" data-ty-prompt=">>>">dataset = lance.dataset("s3://my-bucket/embeddings")</span>
<span data-ty="input" data-ty-prompt=">>>">table = dataset.take([100, 5000, 1000000])</span>
<span data-ty="input" data-ty-prompt=">>>">dataset.take([0, 1], columns=["id", "vector"])</span>
</div>
<img src="assets/images/random-access.png" alt="Random Access Example" style="max-width: 500px; width: 100%; height: auto; border-radius: 8px;">
</div>
</div>
</div>
Expand All @@ -161,31 +216,18 @@ <h2>Lightning-fast Random Access</h2>
<div class="lance-feature-text">
<h2>Native Multimodal Data Support</h2>
<p>
Store images, videos, audio, text, and embeddings in a single unified format.
Store images, videos, audio, text, and embeddings alongside your traditional tabular data in a single unified format.
Lance's blob encoding efficiently handles large binary objects with lazy loading,
while optimized vector storage accelerates similarity search.
</p>
<p>
Perfect for AI/ML workloads where you need to store raw data alongside embeddings
for multimodal retrieval and generation workflows.
Perfect for AI/ML workloads where you need to store raw data, ML features, generated captions and embeddings
all together for multimodal retrieval and genAI workflows.
</p>
<a href="guide/blob" class="md-button">Learn More</a>
</div>
<div class="lance-feature-demo">
<div id="termynal-multimodal"
data-termynal=""
data-ty-startdelay="2500"
data-ty-typedelay="40"
data-ty-linedelay="700"
style="width: 500px;">
<span data-ty="input" data-ty-prompt=">>>">import lance</span>
<span data-ty="input" data-ty-prompt=">>>">import pyarrow as pa</span>
<span data-ty="input" data-ty-prompt=">>>">schema = pa.schema([</span>
<span data-ty="input" data-ty-prompt="..."> pa.field("video", pa.large_binary(),</span>
<span data-ty="input" data-ty-prompt="..."> metadata={"lance-encoding:blob": "true"}),</span>
<span data-ty="input" data-ty-prompt="..."> pa.field("embedding", pa.list_(pa.float32(), 128))])</span>
<span data-ty="input" data-ty-prompt=">>>">lance.write_dataset(table, "multimodal.lance", schema=schema)</span>
</div>
<img src="assets/images/multimodal-data.png" alt="Multimodal Data Example" style="max-width: 500px; width: 100%; height: auto; border-radius: 8px;">
</div>
</div>
</div>
Expand All @@ -200,7 +242,7 @@ <h2>Data Evolution > Schema Evolution</h2>
<p>
Schema evolution in most open table formats are metadata only and fast.
But when trying to backfill column values in existing rows, a full table rewrite is typically required.
Lance supports efficient schema evolution with backfill, making it perfect for ML
Lance supports data evolution (efficient schema evolution with backfill), making it perfect for ML
feature engineering, embedding and media content management.
</p>
<p>
Expand All @@ -210,20 +252,7 @@ <h2>Data Evolution > Schema Evolution</h2>
<a href="guide/data_evolution" class="md-button">Learn More</a>
</div>
<div class="lance-feature-demo">
<div id="termynal-evolution"
data-termynal=""
data-ty-startdelay="3500"
data-ty-typedelay="40"
data-ty-linedelay="700"
style="width: 500px;">
<span data-ty="input" data-ty-prompt=">>>">import lance</span>
<span data-ty="input" data-ty-prompt=">>>">dataset = lance.dataset("my_data.lance")</span>
<span data-ty="input" data-ty-prompt=">>>">@lance.batch_udf()</span>
<span data-ty="input" data-ty-prompt="...">def add_embeddings(batch):</span>
<span data-ty="input" data-ty-prompt="..."> vectors = model.encode(batch["text"])</span>
<span data-ty="input" data-ty-prompt="..."> return {"embedding": vectors}</span>
<span data-ty="input" data-ty-prompt=">>>">dataset.add_columns(add_embeddings)</span>
</div>
<img src="assets/images/data-evolution.png" alt="Data Evolution Example" style="max-width: 500px; width: 100%; height: auto; border-radius: 8px;">
</div>
</div>
</div>
Expand All @@ -237,11 +266,13 @@ <h2>Data Evolution > Schema Evolution</h2>
<h2>Rich Ecosystem Integrations</h2>
<p>
As an open format, Lance integrates seamlessly with the Python data ecosystem and modern data platforms.
Work with your favorite tools including Pandas, Polars, and PyTorch for data processing and machine learning.
Work with your favorite tools including Pandas, Polars, Ray and PyTorch for data processing and machine learning.
</p>
<p>
Connect with leading query engines like Apache DataFusion, DuckDB, Apache Spark, Trino, and Apache Flink/Fluss
to run SQL analytics and distributed processing on your Lance datasets.
</p>
<a href="integrations/duckdb" class="md-button">View Integrations</a>
<a href="integrations/datafusion" class="md-button">View Integrations</a>
</div>
<div class="lance-feature-demo">
<img src="assets/images/ecosystem-integrations.png" alt="Lance Ecosystem Integrations" style="max-width: 500px; width: 100%; height: auto; border-radius: 8px;">
Expand All @@ -250,10 +281,6 @@ <h2>Rich Ecosystem Integrations</h2>
</div>
</section>

<script
src="assets/javascript/termynal.js"
data-termynal-container="#termynal-hybrid-search|#termynal-random-access|#termynal-multimodal|#termynal-evolution">
</script>

{% endblock %}

Expand Down
Binary file added docs/src/assets/images/data-evolution.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/src/assets/images/hybrid-search.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/src/assets/images/multimodal-data.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/src/assets/images/random-access.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
11 changes: 10 additions & 1 deletion docs/src/assets/stylesheets/home.css
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
/* Lance Homepage Styles */

/* Override with custom color #625EFF site-wide */
:root > * {
--md-primary-fg-color: #625EFF;
--md-primary-fg-color--light: #8481FF;
--md-primary-fg-color--dark: #4A46CC;
--md-accent-fg-color: #625EFF;
--md-accent-fg-color--transparent: rgba(98, 94, 255, 0.1);
}

* {
box-sizing: border-box;
}
Expand Down Expand Up @@ -108,7 +117,7 @@
}

.intro-message .md-button--primary:hover {
box-shadow: 0 8px 20px rgba(102, 126, 234, 0.5);
box-shadow: 0 8px 20px rgba(98, 94, 255, 0.5);
}

.intro-message .md-button:not(.md-button--primary):hover {
Expand Down
1 change: 0 additions & 1 deletion docs/src/community/.pages
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
nav:
- Contributing: contributing
- Lancelot Program: lancelot.md
5 changes: 0 additions & 5 deletions docs/src/community/contributing/.pages

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,4 @@ make integtest
```

To run the tests on OS X, you may need to increase the default limit on the number of open files:
`ulimit -n 2048`
`ulimit -n 2048`
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ If you're working on a performance related feature, benchmarks can be run via:
cargo bench
```

If you want detailed logging and full backtraces, set the following environment variables.
More details can be found [here](../../guide/performance.md#logging).
If you want detailed logging and full backtraces, set the following environment variables.
More details can be found [here](../docs/src/guide/performance.md#logging).

```bash
LANCE_LOG=info RUST_BACKTRACE=FULL <cargo-commands>
Expand Down
Loading