Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

ipfs.dag.put pinning performance #2650

Closed
shamb0t opened this issue Dec 3, 2019 · 4 comments
Closed

ipfs.dag.put pinning performance #2650

shamb0t opened this issue Dec 3, 2019 · 4 comments
Labels
exp/expert Having worked on the specific codebase is important P2 Medium: Good to have, but can wait until someone steps up status/ready Ready to be worked topic/perf Performance

Comments

@shamb0t
Copy link

shamb0t commented Dec 3, 2019

Hey folks 👋 over in ipfs-log we're using ipfs.dag.put to write each appended entry to ipfs and can end up with a high number of entries/hashes fairly quickly. We're not currently pinning content by default but are facing some performance degradation when enabling the pin option in the latest release:

$ node benchmarks/benchmark-append.js
Starting benchmark...
1239 queries per second, 1239 queries in 1 seconds (Entry count: 1239)
1894 queries per second, 3133 queries in 2 seconds (Entry count: 3133)
2316 queries per second, 5449 queries in 3 seconds (Entry count: 5449)
2377 queries per second, 7826 queries in 4 seconds (Entry count: 7826)
2442 queries per second, 10268 queries in 5 seconds (Entry count: 10268)
2430 queries per second, 12698 queries in 6 seconds (Entry count: 12698)
2280 queries per second, 14978 queries in 7 seconds (Entry count: 14978)
2389 queries per second, 17367 queries in 8 seconds (Entry count: 17367)
2456 queries per second, 19823 queries in 9 seconds (Entry count: 19823)
--> Average of 2213.6 q/s in the last 10 seconds

with pinning enabled:

$ node benchmarks/benchmark-append.js
Starting benchmark...
42 queries per second, 42 queries in 1 seconds (Entry count: 42)
42 queries per second, 84 queries in 2 seconds (Entry count: 84)
38 queries per second, 122 queries in 3 seconds (Entry count: 122)
33 queries per second, 155 queries in 4 seconds (Entry count: 155)
29 queries per second, 184 queries in 5 seconds (Entry count: 184)
28 queries per second, 212 queries in 6 seconds (Entry count: 212)
24 queries per second, 236 queries in 7 seconds (Entry count: 236)
24 queries per second, 260 queries in 8 seconds (Entry count: 260)
21 queries per second, 281 queries in 9 seconds (Entry count: 281)
--> Average of 30.2 q/s in the last 10 seconds

With a high number of hashes, would you recommend pinning them in batches separately? As far as I understand garbage collection still needs to be manually triggered for the moment, is that right? Thanks!

@achingbrain
Copy link
Member

Pinning performance gets progressively worse over time - as implemented it's the number of pins that are the problem rather than how you create them (see #2197 for more).

I'm planning on addressing this in the very near future - fingers crossed in v0.41.0.

GC does still need to be triggered manually so you are ok to not pin anything for now.

@shamb0t
Copy link
Author

shamb0t commented Dec 4, 2019

Good to know, thanks @achingbrain! The example above begins with 0 hashes pinned but as you said the performance does continue to degrade as that number increases. Will look out for v0.41.0

@alanshaw alanshaw added exp/expert Having worked on the specific codebase is important P2 Medium: Good to have, but can wait until someone steps up topic/perf Performance status/ready Ready to be worked labels Dec 4, 2019
@matheus23
Copy link
Contributor

GC does still need to be triggered manually so you are ok to not pin anything for now.

Is this still the case @achingbrain ?

We're running into performance problems with pinning in js-ipfs (commit 511147b).

@achingbrain
Copy link
Member

Yes, GC still need to be triggered manually. I don't think this will ever change TBH.

Pins were migrated to the datastore last summer - if you are still seeing performance problems could you please open a new issue with some benchmark stats and a repro case?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
exp/expert Having worked on the specific codebase is important P2 Medium: Good to have, but can wait until someone steps up status/ready Ready to be worked topic/perf Performance
Projects
None yet
Development

No branches or pull requests

4 participants