Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add pinning API #36

Merged
merged 5 commits into from
Feb 24, 2023
Merged

feat: add pinning API #36

merged 5 commits into from
Feb 24, 2023

Conversation

achingbrain
Copy link
Member

@achingbrain achingbrain commented Feb 20, 2023

Adds the pinning API with the reference counting implementation as specified in #28 - see that issue for discussion

Benchmarks prove reference counting is faster than DAG traversal during gc, see this comment for results & discussion.

Closes: #28

Adds the pinning API as specified in #28 - see that issue for discussion

Benchmarks incoming!

Closes: #28
@achingbrain
Copy link
Member Author

achingbrain commented Feb 23, 2023

I've updated the benchmark to omit setup and teardown from the timed sections, though it needs tinylibs/tinybench#33 to be merged before it can be run.

The benchmark creates a load of small pinned dags and a load of unpinned blocks, then runs garbage collection.

In the results below we start with 1,000 dags/unpinned blocks, up to 10,000, increasing in increments of 1,000, so we start with 4k blocks (1k DAGs of 3 blocks each + 1k unpinned blocks) and end with 40k blocks.

It tests helia, js-ipfs and kubo though I've omitted js-ipfs from the graphs because it skewed them so far to make the helia/kubo series be plotted as flat lines at the bottom of the graph. As an indication running gc with 40k blocks took js-ipfs 5.5 hours, it takes helia about 4 seconds(!) The difference is that js-ipfs walks the DAG of each block in the blockstore to build up a list of unpinned blocks whereas helia already knows which blocks are pinned due to reference counting.

I ran these benchmarks on my laptop while doing other things which is why there is some wobble in the lines so I've added trend lines to make things a bit clearer.

image


Splitting the setup out of the timed function also allowed me to benchmark the different steps separately for some bonus stats.

This is deleting pins (start with deleting 1k pins, end with 10k pins):

image

Adding blocks (kubo adds blocks with the ipfs.block.put RPC command, helia uses the blockstore putMany function, which allows the blockstore to control concurrency):

image

Finally adding new pins to the pinset, (start with adding 1k pins, end with 10k pins, the ms value is how long adding that many pins takes). Helia scales better because it's not mutating a giant DAG that gets bigger with every pin added:

image

@achingbrain achingbrain merged commit 270bb98 into main Feb 24, 2023
@achingbrain achingbrain deleted the feat/add-pinning-api branch February 24, 2023 14:01
@SgtPooki
Copy link
Member

@achingbrain , this is great. nice work

@achingbrain
Copy link
Member Author

Vive le JS!

github-actions bot pushed a commit that referenced this pull request Mar 23, 2023
## @helia/interface-v1.0.0 (2023-03-23)

### Features

* add bitswap progress events ([#50](#50)) ([7460719](7460719)), closes [#27](#27)
* add pinning API ([#36](#36)) ([270bb98](270bb98)), closes [#28](#28) [/github.com//pull/36#issuecomment-1441403221](https://github.com/ipfs//github.com/ipfs/helia/pull/36/issues/issuecomment-1441403221) [#28](#28)
* initial implementation ([#17](#17)) ([343d360](343d360))

### Bug Fixes

* extend blockstore interface ([#55](#55)) ([42308c0](42308c0))
* make all helia args optional ([#37](#37)) ([d15d76c](d15d76c))
* survive a cid causing an error during gc ([#38](#38)) ([5330188](5330188))
* update block events ([#58](#58)) ([d33be53](d33be53))
* update blocks interface to align with interface-blockstore ([#54](#54)) ([202b966](202b966))

### Dependencies

* update interface-store to 5.x.x ([#63](#63)) ([5bf11d6](5bf11d6))

### Trivial Changes

* add release config ([a1c7ed0](a1c7ed0))
* fix ci badge ([50929c0](50929c0))
* release main ([#62](#62)) ([2bce77c](2bce77c))
* update logo ([654a70c](654a70c))
* update publish config ([913ab6a](913ab6a))
* update release please config ([b52d5e3](b52d5e3))
@github-actions
Copy link
Contributor

🎉 This PR is included in version @helia/interface-v1.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

github-actions bot pushed a commit that referenced this pull request Mar 23, 2023
## helia-v1.0.0 (2023-03-23)

### Features

* add bitswap progress events ([#50](#50)) ([7460719](7460719)), closes [#27](#27)
* add pinning API ([#36](#36)) ([270bb98](270bb98)), closes [#28](#28) [/github.com//pull/36#issuecomment-1441403221](https://github.com/ipfs//github.com/ipfs/helia/pull/36/issues/issuecomment-1441403221) [#28](#28)
* initial implementation ([#17](#17)) ([343d360](343d360))

### Bug Fixes

* make all helia args optional ([#37](#37)) ([d15d76c](d15d76c))
* survive a cid causing an error during gc ([#38](#38)) ([5330188](5330188))
* update blocks interface to align with interface-blockstore ([#54](#54)) ([202b966](202b966))
* use release version of libp2p ([#59](#59)) ([a3a7c9c](a3a7c9c))

### Trivial Changes

* add release config ([a1c7ed0](a1c7ed0))
* fix ci badge ([50929c0](50929c0))
* release main ([#62](#62)) ([2bce77c](2bce77c))
* update logo ([654a70c](654a70c))
* update publish config ([913ab6a](913ab6a))
* update release please config ([b52d5e3](b52d5e3))
* use wildcards for interop test deps ([29b4fb0](29b4fb0))

### Dependencies

* update interface-store to 5.x.x ([#63](#63)) ([5bf11d6](5bf11d6))
* update sibling dependencies ([ac28d38](ac28d38))
@github-actions
Copy link
Contributor

🎉 This PR is included in version helia-v1.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

@logesh2496
Copy link

How do I use this to pin a CID. I have tried below options and none of them are working,

helia.pins?.add("QmRrzxiXGefcF9VbbThGrkhiQeuhPFjRbGgqYZmxYXYkHr")

and

import CID from "cids"
const newcid = new CID(cid)
const pin = await helia.pins?.add(newcid);

This was referenced Jan 8, 2024
achingbrain added a commit that referenced this pull request Jan 8, 2024
Bumps [@libp2p/websockets](https://github.com/libp2p/js-libp2p) from 6.0.3 to 7.0.5.
- [Release notes](https://github.com/libp2p/js-libp2p/releases)
- [Changelog](https://github.com/libp2p/js-libp2p/blob/master/.release-please.json)
- [Commits](libp2p/js-libp2p@peer-record-v6.0.3...websockets-v7.0.5)

---
updated-dependencies:
- dependency-name: "@libp2p/websockets"
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alex Potsides <[email protected]>
achingbrain added a commit that referenced this pull request Jan 8, 2024
Bumps [aegir](https://github.com/ipfs/aegir) from 40.0.13 to 41.0.0.
- [Release notes](https://github.com/ipfs/aegir/releases)
- [Changelog](https://github.com/ipfs/aegir/blob/master/CHANGELOG.md)
- [Commits](ipfs/aegir@v40.0.13...v41.0.0)

---
updated-dependencies:
- dependency-name: aegir
  dependency-type: direct:development
  update-type: version-update:semver-major
...

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: achingbrain <[email protected]>
achingbrain added a commit that referenced this pull request Jan 8, 2024
Bumps [aegir](https://github.com/ipfs/aegir) from 40.0.13 to 41.0.0.
- [Release notes](https://github.com/ipfs/aegir/releases)
- [Changelog](https://github.com/ipfs/aegir/blob/master/CHANGELOG.md)
- [Commits](ipfs/aegir@v40.0.13...v41.0.0)

---
updated-dependencies:
- dependency-name: aegir
  dependency-type: direct:development
  update-type: version-update:semver-major
...

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: achingbrain <[email protected]>
achingbrain added a commit that referenced this pull request Jan 8, 2024
Bumps [aegir](https://github.com/ipfs/aegir) from 40.0.13 to 41.0.0.
- [Release notes](https://github.com/ipfs/aegir/releases)
- [Changelog](https://github.com/ipfs/aegir/blob/master/CHANGELOG.md)
- [Commits](ipfs/aegir@v40.0.13...v41.0.0)

---
updated-dependencies:
- dependency-name: aegir
  dependency-type: direct:development
  update-type: version-update:semver-major
...

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: achingbrain <[email protected]>
achingbrain pushed a commit that referenced this pull request Jan 8, 2024
## [@helia/dag-json-v1.0.3](https://github.com/ipfs/helia-dag-json/compare/@helia/dag-json-v1.0.2...@helia/dag-json-v1.0.3) (2023-10-07)

### Dependencies

* bump @helia/interface from 1.2.2 to 2.0.0 ([#32](ipfs/helia-dag-json#32)) ([eb836ef](ipfs/helia-dag-json@eb836ef))
* **dev:** bump aegir from 40.0.13 to 41.0.0 ([#36](ipfs/helia-dag-json#36)) ([9f57d11](ipfs/helia-dag-json@9f57d11))
achingbrain pushed a commit that referenced this pull request Jan 8, 2024
achingbrain pushed a commit that referenced this pull request Jan 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pinning and Garbage Collection
3 participants