-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add pinning API #36
Conversation
I've updated the benchmark to omit setup and teardown from the timed sections, though it needs tinylibs/tinybench#33 to be merged before it can be run. The benchmark creates a load of small pinned dags and a load of unpinned blocks, then runs garbage collection. In the results below we start with 1,000 dags/unpinned blocks, up to 10,000, increasing in increments of 1,000, so we start with 4k blocks (1k DAGs of 3 blocks each + 1k unpinned blocks) and end with 40k blocks. It tests helia, js-ipfs and kubo though I've omitted js-ipfs from the graphs because it skewed them so far to make the helia/kubo series be plotted as flat lines at the bottom of the graph. As an indication running gc with 40k blocks took js-ipfs 5.5 hours, it takes helia about 4 seconds(!) The difference is that js-ipfs walks the DAG of each block in the blockstore to build up a list of unpinned blocks whereas helia already knows which blocks are pinned due to reference counting. I ran these benchmarks on my laptop while doing other things which is why there is some wobble in the lines so I've added trend lines to make things a bit clearer. Splitting the setup out of the timed function also allowed me to benchmark the different steps separately for some bonus stats. This is deleting pins (start with deleting 1k pins, end with 10k pins): Adding blocks (kubo adds blocks with the Finally adding new pins to the pinset, (start with adding 1k pins, end with 10k pins, the ms value is how long adding that many pins takes). Helia scales better because it's not mutating a giant DAG that gets bigger with every pin added: |
@achingbrain , this is great. nice work |
Vive le JS! |
## @helia/interface-v1.0.0 (2023-03-23) ### Features * add bitswap progress events ([#50](#50)) ([7460719](7460719)), closes [#27](#27) * add pinning API ([#36](#36)) ([270bb98](270bb98)), closes [#28](#28) [/github.com//pull/36#issuecomment-1441403221](https://github.com/ipfs//github.com/ipfs/helia/pull/36/issues/issuecomment-1441403221) [#28](#28) * initial implementation ([#17](#17)) ([343d360](343d360)) ### Bug Fixes * extend blockstore interface ([#55](#55)) ([42308c0](42308c0)) * make all helia args optional ([#37](#37)) ([d15d76c](d15d76c)) * survive a cid causing an error during gc ([#38](#38)) ([5330188](5330188)) * update block events ([#58](#58)) ([d33be53](d33be53)) * update blocks interface to align with interface-blockstore ([#54](#54)) ([202b966](202b966)) ### Dependencies * update interface-store to 5.x.x ([#63](#63)) ([5bf11d6](5bf11d6)) ### Trivial Changes * add release config ([a1c7ed0](a1c7ed0)) * fix ci badge ([50929c0](50929c0)) * release main ([#62](#62)) ([2bce77c](2bce77c)) * update logo ([654a70c](654a70c)) * update publish config ([913ab6a](913ab6a)) * update release please config ([b52d5e3](b52d5e3))
🎉 This PR is included in version @helia/interface-v1.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
## helia-v1.0.0 (2023-03-23) ### Features * add bitswap progress events ([#50](#50)) ([7460719](7460719)), closes [#27](#27) * add pinning API ([#36](#36)) ([270bb98](270bb98)), closes [#28](#28) [/github.com//pull/36#issuecomment-1441403221](https://github.com/ipfs//github.com/ipfs/helia/pull/36/issues/issuecomment-1441403221) [#28](#28) * initial implementation ([#17](#17)) ([343d360](343d360)) ### Bug Fixes * make all helia args optional ([#37](#37)) ([d15d76c](d15d76c)) * survive a cid causing an error during gc ([#38](#38)) ([5330188](5330188)) * update blocks interface to align with interface-blockstore ([#54](#54)) ([202b966](202b966)) * use release version of libp2p ([#59](#59)) ([a3a7c9c](a3a7c9c)) ### Trivial Changes * add release config ([a1c7ed0](a1c7ed0)) * fix ci badge ([50929c0](50929c0)) * release main ([#62](#62)) ([2bce77c](2bce77c)) * update logo ([654a70c](654a70c)) * update publish config ([913ab6a](913ab6a)) * update release please config ([b52d5e3](b52d5e3)) * use wildcards for interop test deps ([29b4fb0](29b4fb0)) ### Dependencies * update interface-store to 5.x.x ([#63](#63)) ([5bf11d6](5bf11d6)) * update sibling dependencies ([ac28d38](ac28d38))
🎉 This PR is included in version helia-v1.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
How do I use this to pin a CID. I have tried below options and none of them are working,
and
|
Bumps [@libp2p/websockets](https://github.com/libp2p/js-libp2p) from 6.0.3 to 7.0.5. - [Release notes](https://github.com/libp2p/js-libp2p/releases) - [Changelog](https://github.com/libp2p/js-libp2p/blob/master/.release-please.json) - [Commits](libp2p/js-libp2p@peer-record-v6.0.3...websockets-v7.0.5) --- updated-dependencies: - dependency-name: "@libp2p/websockets" dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alex Potsides <[email protected]>
Bumps [aegir](https://github.com/ipfs/aegir) from 40.0.13 to 41.0.0. - [Release notes](https://github.com/ipfs/aegir/releases) - [Changelog](https://github.com/ipfs/aegir/blob/master/CHANGELOG.md) - [Commits](ipfs/aegir@v40.0.13...v41.0.0) --- updated-dependencies: - dependency-name: aegir dependency-type: direct:development update-type: version-update:semver-major ... --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: achingbrain <[email protected]>
Bumps [aegir](https://github.com/ipfs/aegir) from 40.0.13 to 41.0.0. - [Release notes](https://github.com/ipfs/aegir/releases) - [Changelog](https://github.com/ipfs/aegir/blob/master/CHANGELOG.md) - [Commits](ipfs/aegir@v40.0.13...v41.0.0) --- updated-dependencies: - dependency-name: aegir dependency-type: direct:development update-type: version-update:semver-major ... --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: achingbrain <[email protected]>
Bumps [aegir](https://github.com/ipfs/aegir) from 40.0.13 to 41.0.0. - [Release notes](https://github.com/ipfs/aegir/releases) - [Changelog](https://github.com/ipfs/aegir/blob/master/CHANGELOG.md) - [Commits](ipfs/aegir@v40.0.13...v41.0.0) --- updated-dependencies: - dependency-name: aegir dependency-type: direct:development update-type: version-update:semver-major ... --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: achingbrain <[email protected]>
## [@helia/dag-json-v1.0.3](https://github.com/ipfs/helia-dag-json/compare/@helia/dag-json-v1.0.2...@helia/dag-json-v1.0.3) (2023-10-07) ### Dependencies * bump @helia/interface from 1.2.2 to 2.0.0 ([#32](ipfs/helia-dag-json#32)) ([eb836ef](ipfs/helia-dag-json@eb836ef)) * **dev:** bump aegir from 40.0.13 to 41.0.0 ([#36](ipfs/helia-dag-json#36)) ([9f57d11](ipfs/helia-dag-json@9f57d11))
## [@helia/dag-cbor-v1.0.3](https://github.com/ipfs/helia-dag-cbor/compare/@helia/dag-cbor-v1.0.2...@helia/dag-cbor-v1.0.3) (2023-10-07) ### Dependencies * **dev:** bump aegir from 40.0.13 to 41.0.0 ([#36](ipfs/helia-dag-cbor#36)) ([77e29bc](ipfs/helia-dag-cbor@77e29bc))
## [@helia/json-v1.0.3](https://github.com/ipfs/helia-json/compare/@helia/json-v1.0.2...@helia/json-v1.0.3) (2023-10-07) ### Dependencies * **dev:** bump aegir from 40.0.13 to 41.0.0 ([#36](ipfs/helia-json#36)) ([ca3f05a](ipfs/helia-json@ca3f05a))
Adds the pinning API with the reference counting implementation as specified in #28 - see that issue for discussion
Benchmarks prove reference counting is faster than DAG traversal during gc, see this comment for results & discussion.
Closes: #28