DataType: Blob

[KKould]

tonbo needs to support blob storage, so we need to refer to WiscKey: for implementation

What are the benefits and problems that tonbo can bring by implementing WischKey now?

😀

• reduce write amplification caused by compaction

😅

• high maintenance costs, if another version is maintained like BlobDB, but if supported in the existing version, it will inevitably reduce performance
• when scanning a large amount of data, a large number of random reads of Value will be involved, which may result in reduced sequential read performance.
• implementing WiscKey requires modifying a large amount of code and requires the vLog component. Is the benefit brought by this feature valuable?
• how does vLog manage files on S3? The randomness of data location may cause files on S3 to be modified frequently but only a small number of values ​​are modified for each file.

ref

• Key-Value Separation in LSM Storage Engines
• http://mysql.taobao.org/monthly/2022/09/01/

[ethe]

Are there any references to other implementations that support shared storage, how they deisgns and make trade-offs? I think discussions lacks narrativity and references at the moment so it makes me confusd, it should be better to complete the storytelling with more causation and context at the top of disscussion. I simply list some of my questions here:

reduce write amplification caused by compaction

also blob storage I think, which is Tonbo's user need, I do not mind to talk about any potential solutions here.

it will inevitably reduce performance

how to estimate the reduction? are all of the implementations of kv separation the same in this topic?

implementing WiscKey requires modifying a large amount of code

are there any design introduction about how large amount?

[KKould]

how to estimate the reduction? are all of the implementations of kv separation the same in this topic?

Key and Value are not on the same file, resulting in increased IO costs. At the same time, during data scanning, the key value will cause a large number of random reads and further reduce performance.

implementing WiscKey requires modifying a large amount of code

vLog will introduce considerable complexity, and will also further affect the read and write performance (because after tonbo reads the sstable content, it needs to be forwarded to vLog for reading and writing again)
tonbo itself is lightweight Parquet based, if WiscKey is implemented, will we have a Key.Parquet and Value.Parquet? This defeats our vision of tonbo being able to produce the ability to read Parquet even without using tonbo

[ethe]

and further reduce performance

any new features will potentially reduce performance, if user require the feature, measure it and make tradeoffs. I suggest to write a complete article about these questions.

will we have a Key.Parquet and Value.Parquet?

it might be, and I don't think it breaks our visions, or maybe you should point your thoughts about vision out in disscussions.

[ethe]

IMO the primary vision of tonbo lists here: https://tonbo.io/blog/introducing-tonbo

What if we could have a write path implementation as agile and performant as DataFusion?

the complexity of Tonbo is still much smaller than datafusion in the current, and also datafusion does not mind to improve behaviors by higher complexity.

[crwen]

Here is an article about Key-Value Separation in LSM Storage Engines.

[KKould]

Here is an article about Key-Value Separation in LSM Storage Engines.

It seems that there are many related design references

[ethe]

Yes, please @KKould arrange and evaluate them and update to above.

[KKould]

directly using Byte (like LargeBinaryArray) to store Blob types is a simpler way. Even if it may cause write amplification during Compaction, the compression algorithm of Parquet itself can theoretically reduce part of the space occupied.

actively search for parquet's support for blob types. Welcome to add relevant information.

• https://stackoverflow.com/questions/77609268/loading-parquet-with-large-string
• https://medium.com/vortechsa/when-parquet-columns-get-too-big-206ad7f6ca9c

[loloxwg]

LargeBinaryArray https://docs.pola.rs/api/rust/dev/polars/prelude/type.LargeBinaryArray.html

[ethe]

Repost Slack comment from @KKould

为什么tonbo不适合去实现kv分离

tonbo我理解本身的定位是一个低侵入性且强依赖于parquet的存储引擎，他能够带来的好处是能够让用户简单地使用Rust直接生成对应的parquet文件，并且其本身并没有对parquet有较强的限制，用户几乎可以仅仅把tonbo作为业务代码中的数据写入源并使用datafusion以使用或不使用tonbo皆可的形式对本地或s3的parquet进行读取分析

而kv分离是一个非常局限性的feature，以现在最主流的wisckey的实现方案来看，wisckey对迭代读取的性能影响极其大，因为当迭代读取key时，value也需要对应进行随机点读，即使原文利用ssd随机性能强的特点并且结合预读取块来尽可能使随机读取合并，但这仍然会显著影响其性能。

目前业界内近乎所有应用WiscKey的存储引擎都是仅仅面向于极端大kv场景下所设计的

• Titan
• Badger
• BlobDB

而TerarkDB则比较特殊，有对应的vSSTable对大kv的情况进行特殊的优化(TerarkDB本身支持多种不同的SSTable，这会带来相当的复杂性，比如大kv和小kv混合时，怎样选择对应的SSTable)，但这是此类大KV中唯一适合Tonbo的低侵入方案。

如果非可变SSTable的方案，那么Tonbo必定迎来第二次大重构并且读写性能已经不再适合之前所描述的场景，甚至也不适合使用parquet进行实现，因为parquet本身在这里并没有比较好的方案，对其实现较好的KV分离设计非常困难(除非有一列只记录url或者offset这种极其hack，用户读取到block不知所以，并且强行耦合tonbo才能进行parquet读取的方案)

即使是BlobDB的方案选择再构建一个单独的kv分离版本，即使使用了刚才提到使用url列这样丑陋的方案，那么feature同步造成更多的维护成本尤其实现都不一致的情况下

tonbo实现Blob的方案

• 直接使用Bytes类型，目前使用Bytes来作为Blob使用实质上是符合直接使用parquet的使用方式，如果是先以使用Blob再去选择parquet的角度去看是不合理的，因为被定义为汽车的东西其目的不是为了飞行，如果一开始就想要飞行就不应该是以汽车作为目的。其带来的问题是compaction时，会带来较大的写放大和写流量，因为kv在同一位置(但如果不在同一位置，scanning性能必定差，这是一个取舍)，同时这也是一个业界内普遍存在的问题。

  • 低Level内存化SSTable: https://dl.acm.org/doi/10.1145/3422575.3422802 ，无法缓解写放大但有效降低写流量

• 使用外置文件对对应的value进行单独存储类似KV分离，这会带来相当的复杂性并影响现有各方面的负载(目前最主要的api: scan)，其带来的代价也就是如开头所述

• tonbo实现路由，当用户传入Blob类型使，使用对应的列去记录对应的url并将该列每一个数据都使用单独的文件进行存储，这里带来的问题在于如何保证文件的一致性，引入相当复杂度并且破坏了tonbo产生的parquet的低侵入性用户需要使用url自行获取对应文件，并且如果用户一开始就选择了这样的方案，实质上自己去使用s3的存储再将url作为列的的方案要远胜于我们自己为用户做一个路由，因为用户比我们更清楚他要怎么存以及存哪里

Loro的Blob类型需求解决方案

需求：Loro需要Blob类型对大对象进行存储，所以Tonbo需要Blob来为其服务

• 解构(用户无法接受)：对有规律的数据结构进行映射分为多个KV，使其能够面向自己的业务场景来进行应用
• Tonbo重构：Loro的需求是Tonbo开始的根本，那么需要详细了解Loro的需求以及结构设计重新为其打造
• 定制化：为其实现一个支持Blob类型的Tonbo，回到 tonbo实现Blob的方案