-
-
Notifications
You must be signed in to change notification settings - Fork 3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: pebbleds profile and plugin (#10530)
* include pebble as built-in plugin Pebble provides a high-performance alternative to leveldb as the datastore, and will serve as a replacement for badger1. There are a number of tuning parameters available for tuning pebble's performance to your specific needs. Default values are used for any that are not configured or are set to the parameter's zero-value. Requires ipfs/go-ds-pebble#39 Closes #10347 * docs: remove mention of ipfs-ds-convert. Rationale: ipfs-inactive/ipfs-ds-convert#50 * docs: pebbleds profile * test: meaningful t0025-datastores.sh * Update config/init.go * Update docs/config.md * Do not hard-code zero values into pebble config
- Loading branch information
Showing
16 changed files
with
476 additions
and
28 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,6 +3,13 @@ | |
This document describes the different possible values for the `Datastore.Spec` | ||
field in the ipfs configuration file. | ||
|
||
- [flatfs](#flatfs) | ||
- [levelds](#levelds) | ||
- [pebbleds](#pebbleds) | ||
- [badgerds](#badgerds) | ||
- [mount](#mount) | ||
- [measure](#measure) | ||
|
||
## flatfs | ||
|
||
Stores each key value pair as a file on the filesystem. | ||
|
@@ -35,6 +42,35 @@ Uses a leveldb database to store key value pairs. | |
} | ||
``` | ||
|
||
## pebbleds | ||
|
||
Uses [pebble](https://github.com/cockroachdb/pebble) as a key value store. | ||
|
||
```json | ||
{ | ||
"type": "pebbleds", | ||
"path": "<location of pebble inside repo>", | ||
} | ||
``` | ||
|
||
The following options are availble for tuning pebble. | ||
If they are not configured (or assigned their zero-valued), then default values are used. | ||
|
||
* `bytesPerSync`: int, Sync sstables periodically in order to smooth out writes to disk. (default: 512KB) | ||
* `bisableWAL`: true|false, Disable the write-ahead log (WAL) at expense of prohibiting crash recovery. (default: false) | ||
* `cacheSize`: Size of pebble's shared block cache. (default: 8MB) | ||
* `l0CompactionThreshold`: int, Count of L0 files necessary to trigger an L0 compaction. | ||
* `l0StopWritesThreshold`: int, Limit on L0 read-amplification, computed as the number of L0 sublevels. | ||
* `lBaseMaxBytes`: int, Maximum number of bytes for LBase. The base level is the level which L0 is compacted into. | ||
* `maxConcurrentCompactions`: int, Maximum number of concurrent compactions. (default: 1) | ||
* `memTableSize`: int, Size of a MemTable in steady state. The actual MemTable size starts at min(256KB, MemTableSize) and doubles for each subsequent MemTable up to MemTableSize (default: 4MB) | ||
* `memTableStopWritesThreshold`: int, Limit on the number of queued of MemTables. (default: 2) | ||
* `walBytesPerSync`: int: Sets the number of bytes to write to a WAL before calling Sync on it in the background. (default: 0, no background syncing) | ||
* `walMinSyncSeconds`: int: Sets the minimum duration between syncs of the WAL. (default: 0) | ||
|
||
> [!TIP] | ||
> Start using pebble with only default values and configure tuning items are needed for your needs. For a more complete description of these values, see: `https://pkg.go.dev/github.com/cockroachdb/[email protected]#Options` (where `A.B.C` is pebble version from Kubo's `go.mod`). | ||
## badgerds | ||
|
||
Uses [badger](https://github.com/dgraph-io/badger) as a key value store. | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.