Added sharding to experimental features #4569
Conversation
ghost
assigned 
| ipfs config --json Experimental.ShardingEnabled true | ||
| ``` | ||
|
|
||
| ### Road to being a real feature |
There was a problem hiding this comment.
IIRC this was a problem:
- Make sure that objects that don't have to be sharded aren't
There was a problem hiding this comment.
Also, we kind of wanted to generalize this and define a new layer between IPLD and IPFS with sharding.
docs/experimental-features.md
Outdated
| ### State | ||
| Experimental | ||
|
|
||
| Currently, when too many items are added into a unixfs directory, the object gets too large and you may experience issues. To pervent this problem, and generally make working really large directories more efficient, we have implemented a HAMT structure for unixfs. To enable this feature, run: |
There was a problem hiding this comment.
I'd keep the description short, something like: Allows to create directories with unlimited number of entries - currently size of unixfs directories is limited by the maximum block size.
Also, try to keep the lines in docs under 80 characters.
docs/experimental-features.md
Outdated
| @@ -343,3 +343,23 @@ See [Plugin docs](./plugins.md) | |||
|
|
|||
| - [ ] Needs more testing | |||
| - [ ] Make sure there are no unknown major problems | |||
|
|
|||
| ## Sharding / HAMT | |||
There was a problem hiding this comment.
I'd explicitly say directory sharding.
| ipfs config --json Experimental.ShardingEnabled true | ||
| ``` | ||
|
|
||
| ### Road to being a real feature |
There was a problem hiding this comment.
Also, we kind of wanted to generalize this and define a new layer between IPLD and IPFS with sharding.
Not sure about "Road to being a real feature" and also would like to have clarified when this is being used. My understanding is that it'll shard automatically both when using the Files API and also when doing `ipfs add -r` but I'm not 100% about this. License: MIT Signed-off-by: Victor Bjelkholm <[email protected]>
victorb
force-pushed
the
add-sharding-experimental-flag
branch
from
ec6f6b0
to
db3abf3
Compare
|
Thanks for the feedback @Stebalien and @magik6k, I've updated the document now. |
|
Could someone clarify if the sharding always happens or only when using |
|
@victorbjelkholm when sharding is enabled, everything dealing with unixfs directories will use it. |
|
@whyrusleeping thanks, then this is ready to be merged. |
Not sure about "Road to being a real feature" and also would like to have clarified when this is being used. My understanding is that it'll shard automatically both when using the Files API and also when doing
ipfs add -rbut I'm not 100% about this.