-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pinning and Garbage Collection #28
Comments
Merged
achingbrain
added a commit
that referenced
this issue
Feb 24, 2023
Adds the pinning API with the reference counting implementation as specified in #28 - see that issue for discussion Benchmarks prove reference counting is faster than DAG traversal during gc, see [this comment for results & discussion](#36 (comment)). Closes: #28
github-actions bot
pushed a commit
that referenced
this issue
Mar 23, 2023
## @helia/interface-v1.0.0 (2023-03-23) ### Features * add bitswap progress events ([#50](#50)) ([7460719](7460719)), closes [#27](#27) * add pinning API ([#36](#36)) ([270bb98](270bb98)), closes [#28](#28) [/github.com//pull/36#issuecomment-1441403221](https://github.com/ipfs//github.com/ipfs/helia/pull/36/issues/issuecomment-1441403221) [#28](#28) * initial implementation ([#17](#17)) ([343d360](343d360)) ### Bug Fixes * extend blockstore interface ([#55](#55)) ([42308c0](42308c0)) * make all helia args optional ([#37](#37)) ([d15d76c](d15d76c)) * survive a cid causing an error during gc ([#38](#38)) ([5330188](5330188)) * update block events ([#58](#58)) ([d33be53](d33be53)) * update blocks interface to align with interface-blockstore ([#54](#54)) ([202b966](202b966)) ### Dependencies * update interface-store to 5.x.x ([#63](#63)) ([5bf11d6](5bf11d6)) ### Trivial Changes * add release config ([a1c7ed0](a1c7ed0)) * fix ci badge ([50929c0](50929c0)) * release main ([#62](#62)) ([2bce77c](2bce77c)) * update logo ([654a70c](654a70c)) * update publish config ([913ab6a](913ab6a)) * update release please config ([b52d5e3](b52d5e3))
🎉 This issue has been resolved in version @helia/interface-v1.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
github-actions bot
pushed a commit
that referenced
this issue
Mar 23, 2023
## helia-v1.0.0 (2023-03-23) ### Features * add bitswap progress events ([#50](#50)) ([7460719](7460719)), closes [#27](#27) * add pinning API ([#36](#36)) ([270bb98](270bb98)), closes [#28](#28) [/github.com//pull/36#issuecomment-1441403221](https://github.com/ipfs//github.com/ipfs/helia/pull/36/issues/issuecomment-1441403221) [#28](#28) * initial implementation ([#17](#17)) ([343d360](343d360)) ### Bug Fixes * make all helia args optional ([#37](#37)) ([d15d76c](d15d76c)) * survive a cid causing an error during gc ([#38](#38)) ([5330188](5330188)) * update blocks interface to align with interface-blockstore ([#54](#54)) ([202b966](202b966)) * use release version of libp2p ([#59](#59)) ([a3a7c9c](a3a7c9c)) ### Trivial Changes * add release config ([a1c7ed0](a1c7ed0)) * fix ci badge ([50929c0](50929c0)) * release main ([#62](#62)) ([2bce77c](2bce77c)) * update logo ([654a70c](654a70c)) * update publish config ([913ab6a](913ab6a)) * update release please config ([b52d5e3](b52d5e3)) * use wildcards for interop test deps ([29b4fb0](29b4fb0)) ### Dependencies * update interface-store to 5.x.x ([#63](#63)) ([5bf11d6](5bf11d6)) * update sibling dependencies ([ac28d38](ac28d38))
🎉 This issue has been resolved in version helia-v1.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This was referenced Jan 8, 2024
Closed
achingbrain
added a commit
that referenced
this issue
Jan 8, 2024
* deps(dev): bump aegir from 39.0.13 to 40.0.11 Bumps [aegir](https://github.com/ipfs/aegir) from 39.0.13 to 40.0.11. - [Release notes](https://github.com/ipfs/aegir/releases) - [Changelog](https://github.com/ipfs/aegir/blob/master/CHANGELOG.md) - [Commits](ipfs/aegir@v39.0.13...v40.0.11) --- updated-dependencies: - dependency-name: aegir dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: achingbrain <[email protected]>
achingbrain
pushed a commit
that referenced
this issue
Jan 8, 2024
## [@helia/dag-json-v1.0.1](https://github.com/ipfs/helia-dag-json/compare/@helia/dag-json-v1.0.0...@helia/dag-json-v1.0.1) (2023-08-27) ### Dependencies * **dev:** bump aegir from 39.0.13 to 40.0.11 ([#28](ipfs/helia-dag-json#28)) ([d126e6a](ipfs/helia-dag-json@d126e6a))
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Context
Garbage collection in js-ipfs originally followed the go-ipfs model whereby pins were stored in a big DAG that was traversed to work out which blocks could be deleted and which couldn't while running garbage collection.
ipfs/js-ipfs#2771 changed that to store the pins in the datastore instead of a big DAG which yielded a massive speed up when adding new pins, but garbage collection was still slow because the algorithm has to walk every dag that's pinned to build up a list of blocks in those dags.
Helia gives us an amazing opportunity to solve that slow garbage collection problem, this would be incredibly valuable to pinning services, for example, who typically don't garbage collect anything as their blockstores are so large the time it takes to run gc makes it impractical to do so.
Gotchas
add
operations add blocks that GC then immediately deletes. Coming up with a clever way to not have to do this would be greatly appreciatedfailed
Interface
An interface to the pinning system might look like this (somewhat similar to js-ipfs):
Strategies
Some benchmarking will be required to choose the appropriate pinning strategy. These should store several 100k of pins of varying depths before running gc.
Classic
/pin/${cid.multihash}
object for each pin:Reference counting
/pin/${cid}
object for each pin:multihash
of the block, e.g.'/pinned-block/${cid.multihash}'
pinCount
by 1, creating thePinnedBlock
entry if necessarypinCount
- if it's then zero, remove the/pinned-block/${mh}
key from the datastorePinnedBlock
entry in the datastore - this should be nicely parallelizablehelia.blockstore.delete
method only checking for the presence of aPinnedBlock
entry should be sufficient to prevent a pinned block from being deleted accidentallySomething else?
We are open to suggestions, but all implementations should be benchmarked.
The text was updated successfully, but these errors were encountered: