feat: add `CacheReader` for SStable #193

KKould · 2024-10-23T03:34:13Z

No description provided.

github-actions · 2024-10-23T04:26:44Z

Read:

+--------------------------------------------+----------+---------+
|                                            | tonbo    | rocksdb |
+=================================================================+
| random range reads                         | 419415ms | 38294ms |
|--------------------------------------------+----------+---------|
| random range reads                         | 425391ms | 38307ms |
|--------------------------------------------+----------+---------|
| random range projection reads              | 300326ms | 38322ms |
|--------------------------------------------+----------+---------|
| random range projection reads              | 305746ms | 38262ms |
|--------------------------------------------+----------+---------|
| random range projection reads (4 threads)  | 234554ms | 38842ms |
|--------------------------------------------+----------+---------|
| random range projection reads (8 threads)  | 218874ms | 38901ms |
|--------------------------------------------+----------+---------|
| random range projection reads (16 threads) | 209794ms | 37119ms |
|--------------------------------------------+----------+---------|
| random range projection reads (32 threads) | 204734ms | 37198ms |
+--------------------------------------------+----------+---------+

github-actions · 2024-10-23T04:26:47Z

Write:

+-------------------------------+--------+---------+
|                               | tonbo  | rocksdb |
+==================================================+
| bulk load                     | 6398ms | 9453ms  |
|-------------------------------+--------+---------|
| individual writes             | 8366ms | 10541ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8252ms | 10521ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7870ms | 10440ms |
|-------------------------------+--------+---------|
| batch writes                  | 8153ms | 7119ms  |
|-------------------------------+--------+---------|
| batch writes (4 threads)      | 7535ms | 7111ms  |
|-------------------------------+--------+---------|
| batch writes (8 threads)      | 7628ms | 7146ms  |
|-------------------------------+--------+---------|
| removals                      | 3122ms | 3813ms  |
+-------------------------------+--------+---------+

github-actions · 2024-10-23T09:24:46Z

Read:

+--------------------------------------------+----------+---------+
|                                            | tonbo    | rocksdb |
+=================================================================+
| random range reads                         | 210883ms | 38092ms |
|--------------------------------------------+----------+---------|
| random range reads                         | 212152ms | 37892ms |
|--------------------------------------------+----------+---------|
| random range projection reads              | 108384ms | 38052ms |
|--------------------------------------------+----------+---------|
| random range projection reads              | 117106ms | 37964ms |
|--------------------------------------------+----------+---------|
| random range projection reads (4 threads)  | 71706ms  | 38746ms |
|--------------------------------------------+----------+---------|
| random range projection reads (8 threads)  | 56557ms  | 38760ms |
|--------------------------------------------+----------+---------|
| random range projection reads (16 threads) | 53053ms  | 37282ms |
|--------------------------------------------+----------+---------|
| random range projection reads (32 threads) | 55659ms  | 37281ms |
+--------------------------------------------+----------+---------+

github-actions · 2024-10-23T09:24:48Z

Write:

+-------------------------------+--------+---------+
|                               | tonbo  | rocksdb |
+==================================================+
| bulk load                     | 6461ms | 9705ms  |
|-------------------------------+--------+---------|
| individual writes             | 8335ms | 10660ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8258ms | 10528ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7855ms | 10554ms |
|-------------------------------+--------+---------|
| batch writes                  | 7904ms | 7117ms  |
|-------------------------------+--------+---------|
| batch writes (4 threads)      | 7540ms | 7129ms  |
|-------------------------------+--------+---------|
| batch writes (8 threads)      | 7733ms | 7099ms  |
|-------------------------------+--------+---------|
| removals                      | 3102ms | 3835ms  |
+-------------------------------+--------+---------+

github-actions · 2024-10-23T10:13:26Z

Read:

+--------------------------------------------+----------+---------+
|                                            | tonbo    | rocksdb |
+=================================================================+
| random range reads                         | 214187ms | 38355ms |
|--------------------------------------------+----------+---------|
| random range reads                         | 205518ms | 38264ms |
|--------------------------------------------+----------+---------|
| random range projection reads              | 112558ms | 39109ms |
|--------------------------------------------+----------+---------|
| random range projection reads              | 106715ms | 38389ms |
|--------------------------------------------+----------+---------|
| random range projection reads (4 threads)  | 81601ms  | 37308ms |
|--------------------------------------------+----------+---------|
| random range projection reads (8 threads)  | 63640ms  | 37277ms |
|--------------------------------------------+----------+---------|
| random range projection reads (16 threads) | 66594ms  | 36125ms |
|--------------------------------------------+----------+---------|
| random range projection reads (32 threads) | 69212ms  | 36860ms |
+--------------------------------------------+----------+---------+

github-actions · 2024-10-23T10:13:27Z

Write:

+-------------------------------+--------+---------+
|                               | tonbo  | rocksdb |
+==================================================+
| bulk load                     | 6351ms | 9398ms  |
|-------------------------------+--------+---------|
| individual writes             | 8233ms | 10497ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8226ms | 10601ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7786ms | 10548ms |
|-------------------------------+--------+---------|
| batch writes                  | 7901ms | 7131ms  |
|-------------------------------+--------+---------|
| batch writes (4 threads)      | 7402ms | 7212ms  |
|-------------------------------+--------+---------|
| batch writes (8 threads)      | 7618ms | 7178ms  |
|-------------------------------+--------+---------|
| removals                      | 3104ms | 3831ms  |
+-------------------------------+--------+---------+

github-actions · 2024-10-28T10:58:18Z

Read:

+--------------------------------------------+---------+---------+
|                                            | tonbo   | rocksdb |
+================================================================+
| random range reads                         | 38812ms | 36993ms |
|--------------------------------------------+---------+---------|
| random range reads                         | 37890ms | 37047ms |
|--------------------------------------------+---------+---------|
| random range projection reads              | 27648ms | 37716ms |
|--------------------------------------------+---------+---------|
| random range projection reads              | 27502ms | 37632ms |
|--------------------------------------------+---------+---------|
| random range projection reads (4 threads)  | 17209ms | 37810ms |
|--------------------------------------------+---------+---------|
| random range projection reads (8 threads)  | 16577ms | 37857ms |
|--------------------------------------------+---------+---------|
| random range projection reads (16 threads) | 15851ms | 36279ms |
|--------------------------------------------+---------+---------|
| random range projection reads (32 threads) | 15939ms | 36380ms |
+--------------------------------------------+---------+---------+

github-actions · 2024-10-28T10:58:20Z

Write:

+-------------------------------+--------+---------+
|                               | tonbo  | rocksdb |
+==================================================+
| bulk load                     | 6332ms | 9472ms  |
|-------------------------------+--------+---------|
| individual writes             | 8237ms | 10634ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8181ms | 10751ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7736ms | 10606ms |
|-------------------------------+--------+---------|
| batch writes                  | 8005ms | 7097ms  |
|-------------------------------+--------+---------|
| batch writes (4 threads)      | 7829ms | 7114ms  |
|-------------------------------+--------+---------|
| batch writes (8 threads)      | 7666ms | 7162ms  |
|-------------------------------+--------+---------|
| removals                      | 3103ms | 3833ms  |
+-------------------------------+--------+---------+

github-actions · 2024-10-28T13:47:19Z

Read:

+--------------------------------------------+---------+---------+
|                                            | tonbo   | rocksdb |
+================================================================+
| random range reads                         | 38340ms | 37258ms |
|--------------------------------------------+---------+---------|
| random range reads                         | 37380ms | 37050ms |
|--------------------------------------------+---------+---------|
| random range projection reads              | 28851ms | 37093ms |
|--------------------------------------------+---------+---------|
| random range projection reads              | 28759ms | 37230ms |
|--------------------------------------------+---------+---------|
| random range projection reads (4 threads)  | 17397ms | 37081ms |
|--------------------------------------------+---------+---------|
| random range projection reads (8 threads)  | 16864ms | 37081ms |
|--------------------------------------------+---------+---------|
| random range projection reads (16 threads) | 16106ms | 35656ms |
|--------------------------------------------+---------+---------|
| random range projection reads (32 threads) | 16195ms | 35608ms |
+--------------------------------------------+---------+---------+

github-actions · 2024-10-28T13:47:21Z

Write:

+-------------------------------+--------+---------+
|                               | tonbo  | rocksdb |
+==================================================+
| bulk load                     | 6379ms | 9484ms  |
|-------------------------------+--------+---------|
| individual writes             | 8343ms | 10707ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8264ms | 10573ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7833ms | 10687ms |
|-------------------------------+--------+---------|
| batch writes                  | 7971ms | 7128ms  |
|-------------------------------+--------+---------|
| batch writes (4 threads)      | 7854ms | 7049ms  |
|-------------------------------+--------+---------|
| batch writes (8 threads)      | 7777ms | 7094ms  |
|-------------------------------+--------+---------|
| removals                      | 3122ms | 3880ms  |
+-------------------------------+--------+---------+

github-actions · 2024-10-29T08:56:07Z

Read:

+--------------------------------------------+---------+---------+
|                                            | tonbo   | rocksdb |
+================================================================+
| random range reads                         | 38697ms | 37509ms |
|--------------------------------------------+---------+---------|
| random range reads                         | 37618ms | 37520ms |
|--------------------------------------------+---------+---------|
| random range projection reads              | 28282ms | 37897ms |
|--------------------------------------------+---------+---------|
| random range projection reads              | 28054ms | 38006ms |
|--------------------------------------------+---------+---------|
| random range projection reads (4 threads)  | 17511ms | 38125ms |
|--------------------------------------------+---------+---------|
| random range projection reads (8 threads)  | 16451ms | 38094ms |
|--------------------------------------------+---------+---------|
| random range projection reads (16 threads) | 15733ms | 36439ms |
|--------------------------------------------+---------+---------|
| random range projection reads (32 threads) | 15709ms | 36462ms |
+--------------------------------------------+---------+---------+

github-actions · 2024-10-29T08:56:09Z

Write:

+-------------------------------+--------+---------+
|                               | tonbo  | rocksdb |
+==================================================+
| bulk load                     | 6348ms | 9496ms  |
|-------------------------------+--------+---------|
| individual writes             | 8281ms | 10523ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8348ms | 10480ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7836ms | 10461ms |
|-------------------------------+--------+---------|
| batch writes                  | 8015ms | 7132ms  |
|-------------------------------+--------+---------|
| batch writes (4 threads)      | 7560ms | 7065ms  |
|-------------------------------+--------+---------|
| batch writes (8 threads)      | 7739ms | 7070ms  |
|-------------------------------+--------+---------|
| removals                      | 3085ms | 3841ms  |
+-------------------------------+--------+---------+

github-actions · 2024-10-31T10:05:08Z

Read:

+--------------------------------------------+---------+---------+
|                                            | tonbo   | rocksdb |
+================================================================+
| random range reads                         | 38605ms | 37112ms |
|--------------------------------------------+---------+---------|
| random range reads                         | 37172ms | 37372ms |
|--------------------------------------------+---------+---------|
| random range projection reads              | 27770ms | 37692ms |
|--------------------------------------------+---------+---------|
| random range projection reads              | 27845ms | 37594ms |
|--------------------------------------------+---------+---------|
| random range projection reads (4 threads)  | 16679ms | 37588ms |
|--------------------------------------------+---------+---------|
| random range projection reads (8 threads)  | 16381ms | 37661ms |
|--------------------------------------------+---------+---------|
| random range projection reads (16 threads) | 15712ms | 36047ms |
|--------------------------------------------+---------+---------|
| random range projection reads (32 threads) | 15774ms | 36206ms |
+--------------------------------------------+---------+---------+

github-actions · 2024-10-31T10:05:10Z

Write:

+-------------------------------+--------+---------+
|                               | tonbo  | rocksdb |
+==================================================+
| bulk load                     | 6465ms | 9588ms  |
|-------------------------------+--------+---------|
| individual writes             | 8401ms | 10515ms |
|-------------------------------+--------+---------|
| individual writes (4 threads) | 8254ms | 10823ms |
|-------------------------------+--------+---------|
| individual writes (8 threads) | 7834ms | 10595ms |
|-------------------------------+--------+---------|
| batch writes                  | 7950ms | 7195ms  |
|-------------------------------+--------+---------|
| batch writes (4 threads)      | 7609ms | 7150ms  |
|-------------------------------+--------+---------|
| batch writes (8 threads)      | 7658ms | 7155ms  |
|-------------------------------+--------+---------|
| removals                      | 3128ms | 3811ms  |
+-------------------------------+--------+---------+

…creation

github-actions · 2024-11-07T03:25:45Z

Read:

+--------------------------------------------+---------+-------------+---------+----------+
|                                            | tonbo   | tonbo_foyer | rocksdb | tonbo_s3 |
+=========================================================================================+
| random range reads                         | 39305ms | 39288ms     | 36527ms | 48580ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range reads                         | 38747ms | 39197ms     | 36470ms | 43870ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads              | 29407ms | 29771ms     | 36815ms | 33218ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads              | 29413ms | 29778ms     | 36786ms | 33191ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (4 threads)  | 17041ms | 17914ms     | 36778ms | 17739ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (8 threads)  | 16877ms | 16757ms     | 36866ms | 17844ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (16 threads) | 16120ms | 16028ms     | 35448ms | 17094ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (32 threads) | 16253ms | 16066ms     | 35493ms | 17037ms  |
+--------------------------------------------+---------+-------------+---------+----------+

github-actions · 2024-11-07T03:25:47Z

Write:

+-------------------------------+--------+---------+----------+
|                               | tonbo  | rocksdb | tonbo_s3 |
+=============================================================+
| bulk load                     | 6371ms | 9359ms  | 6055ms   |
|-------------------------------+--------+---------+----------|
| individual writes             | 8243ms | 10585ms | 8338ms   |
|-------------------------------+--------+---------+----------|
| individual writes (4 threads) | 8043ms | 10569ms | 8161ms   |
|-------------------------------+--------+---------+----------|
| individual writes (8 threads) | 7796ms | 10651ms | 7792ms   |
|-------------------------------+--------+---------+----------|
| batch writes                  | 7949ms | 7095ms  | 8139ms   |
|-------------------------------+--------+---------+----------|
| batch writes (4 threads)      | 7813ms | 7065ms  | 7614ms   |
|-------------------------------+--------+---------+----------|
| batch writes (8 threads)      | 7735ms | 7093ms  | 7668ms   |
|-------------------------------+--------+---------+----------|
| removals                      | 3113ms | 3791ms  | 3145ms   |
+-------------------------------+--------+---------+----------+

Cargo.toml

tonbo_ext_reader/src/lib.rs

tonbo_ext_reader/Cargo.toml

ethe · 2024-11-11T08:29:38Z

tonbo_ext_reader/Cargo.toml

+edition = "2021"
+
+[dependencies]
+anyhow = "1"


It should better to use Box<dyn std::error::Error + Send + Sync>, I can not see the necessary for anyhow integration.

I tried it, but when using foyer's error in box, the compiler will prompt that anyhow is required.

convert anyhow::Error to Box<dyn std::error::Error + Send + Sync> is definitely possible

bindings/python/src/db.rs

rust-toolchain.toml

ethe · 2024-11-11T08:52:31Z

src/version/set.rs

-        let set = VersionSet::<R> {
+
+        let (meta_cache, range_cache) = C::build_caches(
+            path_to_local(&option.cache_path).unwrap(),


Are there any validations to ensure cache_path is valid to unwrap?

the cache_path is obtained by joining base_path. Base_path should be local, but after I checked just now, base_path does not seem to guarantee that it is local.

ethe · 2024-11-11T09:33:35Z

Basically I can not agree that the cache component awares ParquetMetaData, and split caches to MetaCache and RangeCache, why we could not just keep it simple as Cache<(Ulid, Range<usize>), Block> like https://github.com/risingwavelabs/risingwave/blob/71146a6bf5931e68d5f51917ac72b7070ad0df3f/src/storage/benches/bench_block_cache.rs#L47. In the current, cache reader takes strong intrusive, leaks C: CacheReader to lots of places and lacks operability.

In addition, RangeCache and MetaCache should be a recursive type, they are wrappers to another fusio::Read and AsyncFileReader, and also implement fusio::Read / AsyncFileReader for themselves. Therefore, it is able to be treated as a normal read-only file or a parquet async file reader. The benefit of this design is that except for the construction logic, rest of codes could keep the same as before. And also, cache reader does not need to provide the insert method, if the cache is missed, then just get from inner file or reader and update itself.

github-actions · 2024-11-11T11:17:21Z

Read:

+--------------------------------------------+---------+-------------+---------+----------+
|                                            | tonbo   | tonbo_foyer | rocksdb | tonbo_s3 |
+=========================================================================================+
| random range reads                         | 38582ms | 38954ms     | 36363ms | 47096ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range reads                         | 37145ms | 38109ms     | 36166ms | 30045ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads              | 27864ms | 28133ms     | 36935ms | 29665ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads              | 27873ms | 27916ms     | 36934ms | 29512ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (4 threads)  | 16805ms | 18036ms     | 37439ms | 18019ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (8 threads)  | 16765ms | 17146ms     | 37497ms | 18271ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (16 threads) | 16082ms | 16530ms     | 36096ms | 17475ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (32 threads) | 16139ms | 16481ms     | 36203ms | 17530ms  |
+--------------------------------------------+---------+-------------+---------+----------+

github-actions · 2024-11-11T11:17:23Z

Write:

+-------------------------------+--------+---------+----------+
|                               | tonbo  | rocksdb | tonbo_s3 |
+=============================================================+
| bulk load                     | 6349ms | 9430ms  | 6075ms   |
|-------------------------------+--------+---------+----------|
| individual writes             | 8327ms | 10618ms | 8168ms   |
|-------------------------------+--------+---------+----------|
| individual writes (4 threads) | 8190ms | 10585ms | 8058ms   |
|-------------------------------+--------+---------+----------|
| individual writes (8 threads) | 7788ms | 10601ms | 7795ms   |
|-------------------------------+--------+---------+----------|
| batch writes                  | 8006ms | 7110ms  | 8137ms   |
|-------------------------------+--------+---------+----------|
| batch writes (4 threads)      | 7772ms | 7069ms  | 7644ms   |
|-------------------------------+--------+---------+----------|
| batch writes (8 threads)      | 7683ms | 7114ms  | 7599ms   |
|-------------------------------+--------+---------+----------|
| removals                      | 3086ms | 3794ms  | 3085ms   |
+-------------------------------+--------+---------+----------+

KKould · 2024-11-11T11:30:29Z

.github/workflows/ci.yml

For testing, need to be restored before merging

github-actions · 2024-11-11T11:42:43Z

Read:

+--------------------------------------------+---------+-------------+---------+----------+
|                                            | tonbo   | tonbo_foyer | rocksdb | tonbo_s3 |
+=========================================================================================+
| random range reads                         | 38660ms | 39818ms     | 36776ms | 49374ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range reads                         | 37680ms | 38914ms     | 36800ms | 44368ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads              | 28187ms | 29350ms     | 37221ms | 32876ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads              | 28381ms | 29235ms     | 37151ms | 32767ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (4 threads)  | 17549ms | 16596ms     | 37413ms | 17677ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (8 threads)  | 17228ms | 16657ms     | 37318ms | 17642ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (16 threads) | 16544ms | 16047ms     | 35746ms | 16904ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (32 threads) | 16642ms | 16025ms     | 35717ms | 16896ms  |
+--------------------------------------------+---------+-------------+---------+----------+

github-actions · 2024-11-11T11:42:44Z

Write:

+-------------------------------+--------+---------+----------+
|                               | tonbo  | rocksdb | tonbo_s3 |
+=============================================================+
| bulk load                     | 6351ms | 9443ms  | 6114ms   |
|-------------------------------+--------+---------+----------|
| individual writes             | 8301ms | 10579ms | 8069ms   |
|-------------------------------+--------+---------+----------|
| individual writes (4 threads) | 8131ms | 10777ms | 7936ms   |
|-------------------------------+--------+---------+----------|
| individual writes (8 threads) | 7798ms | 10660ms | 7759ms   |
|-------------------------------+--------+---------+----------|
| batch writes                  | 7960ms | 7157ms  | 7878ms   |
|-------------------------------+--------+---------+----------|
| batch writes (4 threads)      | 7481ms | 7105ms  | 7543ms   |
|-------------------------------+--------+---------+----------|
| batch writes (8 threads)      | 7410ms | 7082ms  | 7541ms   |
|-------------------------------+--------+---------+----------|
| removals                      | 3085ms | 3799ms  | 3039ms   |
+-------------------------------+--------+---------+----------+

github-actions · 2024-11-11T12:07:34Z

Read:

+--------------------------------------------+---------+-------------+---------+----------+
|                                            | tonbo   | tonbo_foyer | rocksdb | tonbo_s3 |
+=========================================================================================+
| random range reads                         | 38244ms | 34537ms     | 36019ms | 48601ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range reads                         | 36950ms | 27226ms     | 36000ms | 43700ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads              | 27859ms | 27921ms     | 36692ms | 31871ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads              | 27570ms | 28102ms     | 36664ms | 31511ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (4 threads)  | 17515ms | 17903ms     | 36960ms | 17755ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (8 threads)  | 16734ms | 17162ms     | 36904ms | 17618ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (16 threads) | 16100ms | 16362ms     | 35453ms | 17049ms  |
|--------------------------------------------+---------+-------------+---------+----------|
| random range projection reads (32 threads) | 16079ms | 16385ms     | 35460ms | 17190ms  |
+--------------------------------------------+---------+-------------+---------+----------+

github-actions · 2024-11-11T12:07:35Z

Write:

+-------------------------------+--------+---------+----------+
|                               | tonbo  | rocksdb | tonbo_s3 |
+=============================================================+
| bulk load                     | 6491ms | 9487ms  | 6050ms   |
|-------------------------------+--------+---------+----------|
| individual writes             | 8142ms | 10667ms | 8238ms   |
|-------------------------------+--------+---------+----------|
| individual writes (4 threads) | 8221ms | 10623ms | 8135ms   |
|-------------------------------+--------+---------+----------|
| individual writes (8 threads) | 7791ms | 10675ms | 7725ms   |
|-------------------------------+--------+---------+----------|
| batch writes                  | 7977ms | 7108ms  | 7980ms   |
|-------------------------------+--------+---------+----------|
| batch writes (4 threads)      | 7794ms | 7080ms  | 7624ms   |
|-------------------------------+--------+---------+----------|
| batch writes (8 threads)      | 7723ms | 7119ms  | 7604ms   |
|-------------------------------+--------+---------+----------|
| removals                      | 3100ms | 3800ms  | 3052ms   |
+-------------------------------+--------+---------+----------+

KKould added the enhancement New feature or request label Oct 23, 2024

KKould requested a review from ethe October 23, 2024 03:34

KKould self-assigned this Oct 23, 2024

KKould force-pushed the feat/cache_read branch from 4ab3a80 to cd33a8f Compare October 23, 2024 08:56

KKould force-pushed the feat/cache_read branch from 4e0e04a to cf742b9 Compare October 28, 2024 10:41

KKould linked an issue Oct 29, 2024 that may be closed by this pull request

Feat: Cache for S3 #192

Closed

1 task

KKould force-pushed the feat/cache_read branch 2 times, most recently from aa4ed14 to 054296b Compare October 31, 2024 09:48

KKould force-pushed the feat/cache_read branch from 4a2dfff to 65c0d0f Compare November 1, 2024 10:49

KKould added 7 commits November 4, 2024 11:03

feat: add CacheReader for SStable

6837a83

chore: use CacheReader on remotes

aa1d66a

refactor: CacheReader globally unify RangeCache to avoid duplicate …

1220569

…creation

chore: codefmt

bdcb2da

chore: local also uses CacheReader

cc62dc0

chore: move cache_reader.rs to other crate

8f34cbe

chore: remove foyer and tonbo coupling

c2d6893

KKould and others added 3 commits November 4, 2024 11:04

chore: fix benchmark

06af52b

bench: add tonbo_s3

a2d10ea

chore: resolve conflicts

e6521b4

KKould force-pushed the feat/cache_read branch from 65c0d0f to e6521b4 Compare November 4, 2024 03:19

KKould added 3 commits November 4, 2024 11:21

ci: reduce benchmark trigger limit

6cce98b

feat: add LruReader for WASM

f24b6f1

chore: sync version

5a27922

KKould force-pushed the feat/cache_read branch from c603571 to 5a27922 Compare November 5, 2024 08:10

chore: add lru_reader on read_bench

3b5e11a

KKould force-pushed the feat/cache_read branch from 60988b9 to 3b5e11a Compare November 5, 2024 08:35

KKould mentioned this pull request Nov 8, 2024

Objective: S3 storage improvement #208

Open

4 tasks

KKould linked an issue Nov 8, 2024 that may be closed by this pull request

Objective: S3 storage improvement #208

Open

4 tasks

ethe requested changes Nov 11, 2024

View reviewed changes

chore: optimization with review comments

89f5d59

KKould force-pushed the feat/cache_read branch from e00775b to 89f5d59 Compare November 11, 2024 11:08

chore: fix example

7ebba62

KKould commented Nov 11, 2024

View reviewed changes

.github/workflows/ci.yml

Copy link

Contributor Author

KKould Nov 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For testing, need to be restored before merging

This was referenced Nov 15, 2024

2.2x lower than redb and sled？ #112

Closed

refactor: breakdown S3 benchmark action from cache reader #225

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add `CacheReader` for SStable #193

feat: add `CacheReader` for SStable #193

KKould commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 29, 2024

github-actions bot commented Oct 29, 2024

github-actions bot commented Oct 31, 2024

github-actions bot commented Oct 31, 2024

github-actions bot commented Nov 7, 2024

github-actions bot commented Nov 7, 2024

ethe Nov 11, 2024

KKould Nov 11, 2024

ethe Nov 12, 2024

ethe Nov 11, 2024

KKould Nov 11, 2024

ethe commented Nov 11, 2024 •

edited

Loading

github-actions bot commented Nov 11, 2024

github-actions bot commented Nov 11, 2024

KKould Nov 11, 2024

github-actions bot commented Nov 11, 2024

github-actions bot commented Nov 11, 2024

github-actions bot commented Nov 11, 2024

github-actions bot commented Nov 11, 2024

feat: add CacheReader for SStable #193

Are you sure you want to change the base?

feat: add CacheReader for SStable #193

Conversation

KKould commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 28, 2024

github-actions bot commented Oct 29, 2024

github-actions bot commented Oct 29, 2024

github-actions bot commented Oct 31, 2024

github-actions bot commented Oct 31, 2024

github-actions bot commented Nov 7, 2024

github-actions bot commented Nov 7, 2024

ethe Nov 11, 2024

Choose a reason for hiding this comment

KKould Nov 11, 2024

Choose a reason for hiding this comment

ethe Nov 12, 2024

Choose a reason for hiding this comment

ethe Nov 11, 2024

Choose a reason for hiding this comment

KKould Nov 11, 2024

Choose a reason for hiding this comment

ethe commented Nov 11, 2024 • edited Loading

github-actions bot commented Nov 11, 2024

github-actions bot commented Nov 11, 2024

KKould Nov 11, 2024

Choose a reason for hiding this comment

github-actions bot commented Nov 11, 2024

github-actions bot commented Nov 11, 2024

github-actions bot commented Nov 11, 2024

github-actions bot commented Nov 11, 2024

feat: add `CacheReader` for SStable #193

feat: add `CacheReader` for SStable #193

ethe commented Nov 11, 2024 •

edited

Loading