ipfs.dag.put pinning performance

[shamb0t]

Hey folks 👋 over in ipfs-log we're using ipfs.dag.put to write each appended entry to ipfs and can end up with a high number of entries/hashes fairly quickly. We're not currently pinning content by default but are facing some performance degradation when enabling the pin option in the latest release:

$ node benchmarks/benchmark-append.js
Starting benchmark...
1239 queries per second, 1239 queries in 1 seconds (Entry count: 1239)
1894 queries per second, 3133 queries in 2 seconds (Entry count: 3133)
2316 queries per second, 5449 queries in 3 seconds (Entry count: 5449)
2377 queries per second, 7826 queries in 4 seconds (Entry count: 7826)
2442 queries per second, 10268 queries in 5 seconds (Entry count: 10268)
2430 queries per second, 12698 queries in 6 seconds (Entry count: 12698)
2280 queries per second, 14978 queries in 7 seconds (Entry count: 14978)
2389 queries per second, 17367 queries in 8 seconds (Entry count: 17367)
2456 queries per second, 19823 queries in 9 seconds (Entry count: 19823)
--> Average of 2213.6 q/s in the last 10 seconds

with pinning enabled:

$ node benchmarks/benchmark-append.js
Starting benchmark...
42 queries per second, 42 queries in 1 seconds (Entry count: 42)
42 queries per second, 84 queries in 2 seconds (Entry count: 84)
38 queries per second, 122 queries in 3 seconds (Entry count: 122)
33 queries per second, 155 queries in 4 seconds (Entry count: 155)
29 queries per second, 184 queries in 5 seconds (Entry count: 184)
28 queries per second, 212 queries in 6 seconds (Entry count: 212)
24 queries per second, 236 queries in 7 seconds (Entry count: 236)
24 queries per second, 260 queries in 8 seconds (Entry count: 260)
21 queries per second, 281 queries in 9 seconds (Entry count: 281)
--> Average of 30.2 q/s in the last 10 seconds

With a high number of hashes, would you recommend pinning them in batches separately? As far as I understand garbage collection still needs to be manually triggered for the moment, is that right? Thanks!

[achingbrain]

Pinning performance gets progressively worse over time - as implemented it's the number of pins that are the problem rather than how you create them (see #2197 for more).

I'm planning on addressing this in the very near future - fingers crossed in v0.41.0.

GC does still need to be triggered manually so you are ok to not pin anything for now.

[shamb0t]

Good to know, thanks @achingbrain! The example above begins with 0 hashes pinned but as you said the performance does continue to degrade as that number increases. Will look out for v0.41.0

[matheus23]

GC does still need to be triggered manually so you are ok to not pin anything for now.

Is this still the case @achingbrain ?

We're running into performance problems with pinning in js-ipfs (commit 511147b).

[achingbrain]

Yes, GC still need to be triggered manually. I don't think this will ever change TBH.

Pins were migrated to the datastore last summer - if you are still seeing performance problems could you please open a new issue with some benchmark stats and a repro case?