v0.20.0-beta.2

What's Changed

New Features 🎉

• feat: support 4bit PQ on new IVF_PQ by @BubbleCal in #3144

Bug Fixes 🐛

• fix: handle filter on empty partition by @eddyxu in #3151

Performance Improvements 🚀

• perf: optimize reading transactions in commit loop by @wjones127 in #3117

Full Changelog: v0.20.0-beta.1...v0.20.0-beta.2

v0.20.0-beta.1

What's Changed

Breaking Changes 🛠

• feat!: allow passing down existing dataset for write by @wjones127 in #3119
• fix!: low recall with cosine/dot on v3 index types by @BubbleCal in #3141

New Features 🎉

• feat: make dataset version serializable by @albertlockett in #3143

Full Changelog: v0.19.3-beta.1...v0.20.0-beta.1

v0.19.3-beta.1

What's Changed

New Features 🎉

• feat: start recording index details in the mainifest, cache index type lookup by @westonpace in #3131

Bug Fixes 🐛

• fix: fix error about schema is not writable pd to pa by @Jay-ju in #3109

Other Changes

• refactor: remove the queue in LanceArrowWriter to reduce memory usage for spark sink by @SaintBacchus in #3110

New Contributors

• @Jay-ju made their first contribution in #3109

Full Changelog: v0.19.2...v0.19.3-beta.1

v0.19.2

What's Changed

New Features 🎉

• feat: support FTS on multiple fields by @BubbleCal in #3025
• feat: adds support for detached commits by @westonpace in #3028
• feat: add 2.1 read path by @westonpace in #2968
• feat: expose compression-level configuration for general compression by @niyue in #3034
• feat: make it possible to use rowid and rowaddr in filters by @westonpace in #2973
• feat: more flexible and sophisticated handling of non-null constraints by @westonpace in #2467
• feat: support build lance java with rust release mode by @SaintBacchus in #3051
• feat: do brute force search on unindexed data by @BubbleCal in #3036
• feat: bitpack with miniblock by @broccoliSpicy in #3067
• feat: introduce concept of "storage class" wtih separate dataset for "blob" storage data by @westonpace in #3064
• feat: add max length statistics to VariableWidth Datablock by @broccoliSpicy in #3082
• feat: add support for "take" operation to balanced storage by @westonpace in #3079
• feat: enable write variable width data in 2.1 by @broccoliSpicy in #3090
• feat: add compaction support to balanced datasets by @westonpace in #3088
• feat: align buffers in 2.1 files by @westonpace in #3101
• feat: allow inserting subschemas by @wjones127 in #3041
• feat: support miniblock with binary data by @broccoliSpicy in #3099
• feat: add full zip encoding for wide data types by @westonpace in #3114
• feat: fsst compression with mini-block by @broccoliSpicy in #3121

Bug Fixes 🐛

• fix: fix the incorrect assertion for fixed size binary decoder by @niyue in #3050
• fix: verify num_sub_vectors is valid before creating index by @westonpace in #3056
• fix: always return correct batch size by @westonpace in #3066
• fix: fp16 GPU training by @chebbyChefNEQ in #3105

Performance Improvements 🚀

• perf: use total cmp for ordered float by @chebbyChefNEQ in #3059
• perf: parallel search partitions in subindex by @chebbyChefNEQ in #3060
• perf: implement fast path for union many row id tree map by @chebbyChefNEQ in #3087
• perf: transpose the PQ codes to improve search performance by @BubbleCal in #3120

Other Changes

• refactor: fix confusing impl of indexed_fragments by @wjones127 in #3047

Full Changelog: v0.19.1...v0.19.2

v0.19.2-beta.5

What's Changed

New Features 🎉

• feat: align buffers in 2.1 files by @westonpace in #3101
• feat: allow inserting subschemas by @wjones127 in #3041

Bug Fixes 🐛

• fix: fp16 GPU training by @chebbyChefNEQ in #3105

Full Changelog: v0.19.2-beta.4...v0.19.2-beta.5

v0.19.2-beta.4

What's Changed

New Features 🎉

• feat: add max length statistics to VariableWidth Datablock by @broccoliSpicy in #3082
• feat: add support for "take" operation to balanced storage by @westonpace in #3079
• feat: enable write variable width data in 2.1 by @broccoliSpicy in #3090
• feat: add compaction support to balanced datasets by @westonpace in #3088

Performance Improvements 🚀

• perf: implement fast path for union many row id tree map by @chebbyChefNEQ in #3087

Full Changelog: v0.19.2-beta.3...v0.19.2-beta.4

v0.19.2-beta.3

What's Changed

New Features 🎉

• feat: support build lance java with rust release mode by @SaintBacchus in #3051
• feat: do brute force search on unindexed data by @BubbleCal in #3036
• feat: bitpack with miniblock by @broccoliSpicy in #3067
• feat: introduce concept of "storage class" wtih separate dataset for "blob" storage data by @westonpace in #3064

Bug Fixes 🐛

• fix: always return correct batch size by @westonpace in #3066

Full Changelog: v0.19.2-beta.2...v0.19.2-beta.3

v0.19.2-beta.2

What's Changed

New Features 🎉

• feat: adds support for detached commits by @westonpace in #3028
• feat: add 2.1 read path by @westonpace in #2968
• feat: expose compression-level configuration for general compression by @niyue in #3034
• feat: make it possible to use rowid and rowaddr in filters by @westonpace in #2973
• feat: more flexible and sophisticated handling of non-null constraints by @westonpace in #2467

Bug Fixes 🐛

• fix: fix the incorrect assertion for fixed size binary decoder by @niyue in #3050
• fix: verify num_sub_vectors is valid before creating index by @westonpace in #3056

Performance Improvements 🚀

• perf: use total cmp for ordered float by @chebbyChefNEQ in #3059
• perf: parallel search partitions in subindex by @chebbyChefNEQ in #3060

Other Changes

• refactor: fix confusing impl of indexed_fragments by @wjones127 in #3047

Full Changelog: v0.19.2-beta.1...v0.19.2-beta.2

v0.19.2-beta.1

What's Changed

New Features 🎉

• feat: support FTS on multiple fields by @BubbleCal in #3025

Full Changelog: v0.19.1...v0.19.2-beta.1

v0.19.1

What's Changed

Breaking Changes 🛠

• feat!: combine LanceFragment.merge_columns and LanceDataset.add_columns in one implementation by @westonpace in #3021
• feat!: support to customize tokenizer by @BubbleCal in #2992

New Features 🎉

• feat: add a blob encoding for large binary values by @westonpace in #2868
• feat: cuda acceleration for PQ builds/assignments by @jacketsj in #2946
• feat(rust): add checkout_latest to Dataset by @wjones127 in #2991
• feat: support storage options for spark read and write by @SaintBacchus in #2990
• feat: add some basic primitive for working with repdef data by @westonpace in #2998
• feat: one-pass IVF_PQ accelerated builds by @jacketsj in #3001
• feat: add bit-width, cardinality and data-size to datablock statistics by @broccoliSpicy in #2986
• feat: add table config by @dsgibbons in #2820
• feat: expose number of rows covered per delta by @chebbyChefNEQ in #2979
• feat: add the basic encode path for 2.1 by @westonpace in #3002
• feat: add a new BlobFile API that can be used to read blob data by @westonpace in #2983
• feat: make it possible to add columns from existing data by @westonpace in #3010

Bug Fixes 🐛

• fix: filter the vec index in function Index::scalar_index_info by @SaintBacchus in #3000
• fix: update cursor correctly in blob file API by @westonpace in #3018
• fix: create FTS on empty table/strings by @BubbleCal in #3024

Performance Improvements 🚀

• perf: reduce the required memory for indexing FTS by @BubbleCal in #2926

Other Changes

• refactor: convert scan args into struct by @westonpace in #2976

Full Changelog: v0.18.2...v0.19.1