[meta] high IO usage #42
Comments
RubenKelevra
added
help wanted
|
not solving the underlying issue but what about updating the pins only every few half a day or so? that + bumping on security updates seems reasonable to me. I'm also pretty sure that's how often official mirrors update anyway. |
|
@guysv there's no way to tell what a security update is and what isn't. Apart from this that is highly discouraged, as you can tell from the wiki page:
I only run an update when there's an update available - so there's no unnecessary IO. I think I found the issue why it's pretty high in IO load. It does up to 4 tasks concurrently right now. This doesn't reduce the IO usage, but the load, since ipfs does less concurrently. |
|
Wouldn't it be possible to optimise IPFS itself so that for every node in the merle dag, it caches wether or not all nodes below that node are available locally? I would imagine, that this would make a recursive pin, that only has a small difference to another recursive pin, about as fast as a non-recursive pin. This could massively reduce the amount of IO required to recursively pin something if most of the stuff is already available locally. |
|
Not sure if this helps, but we can see if Arch has anything like this: https://wiki.ubuntu.com/Mirrors/PushMirroring |
|
@Luflosi well... yeah, there might be potential to optimize IPFS itself - which would be great. But - there are already several ways to improve the situation which are implemented:
|
|
@FireMasterK wrote:
Well, no technical Arch does not. But we use very frequent polling to archive the same thing: Single-digit minute numbers between a package maintainer pushing an update until it has reached the IPFS-pacman-mirror. But the delay between updates which are available and having them added to the cluster isn't the issue here. It's more a internal cluster issue, where the changes have to be compared to the old version by each cluster member resulting in very large amounts of IO reading the old data from the disk. |
This comment has been minimized.
This comment has been minimized.
|
@Luflosi I've pushed the concurrency fix half an hour ago to ipfs. If you restart your follower you should see a drop in IO load. Please report back. :) |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@RubenKelevra I tried using BadgerDB a couple weeks ago but I think that made performance worse, possibly because my ZFS recordsize is set to 128K, which might be too large for BadgerDB. But I couldn't find any information about the optimal block size for BadgerDB online. Setting the recordsize to a lower value also reduces the possible compression ratio. Setting it to 4k when ashift is 12 effectively disables compression. I was also concerned about fragmentation at such a low recordsize. I also didn't feel like experimenting since my repo has enough blocks that many operations such as the conversion take over a day to complete, so I just went back to an older snapshot with FlatFS. |
|
@Luflosi wrote
I'm currently using the following settings on the import server:
BadgerFS isn't working for me since I need to hold the data outside and inside of IPFS which makes deduplication very reasonable. As Blocksize I would recommend 8K as for all databases and compression / deduplication turned off. I would also recommend to turn off sync in ZFS, which might sound counter-intuitive, but ZFS makes sure that override operations and other atomic changes of data are kept atomic. That's why a database would send a sync, so zfs can ignore the command and do its operations more efficiently.
Well, it's an interesting feature, but you need to tune it depending on the amount of CIDs you store. It replaces a "let's check if we got that data already" with a "let's have a calculation which gives us with 99.xx% certainty the same result". It's definitely faster, but rarely used. That's why I stay away from it in my daily IPFS usage. I don't know what happens when we cannot deliver 0.xx percent of the blocks, because their checksums was too similar, but I guess it wouldn't be pretty. Moved to #47 |
|
Hey @teknomunk, thanks ... I feel like this idea needs a dedicated ticket. Can you move your idea to a new ticket and we discuss there? This one here is more like a meta-ticket and it get's a bit too cluttered with our discussion here :) |
|
Just to give some impression about how many times the cluster gets updated and how it's spaced here's an example how the cluster pinset looks like: |
I restarted my follower a couple days ago and now it finally caught up (pinned everything) but I'm not sure if it's better or worse than before, since I changed quite a lot of things on my end. Since I last ran the follower, I switched Linux distros from Arch Linux to NixOS, put my root FS on the same ZFS pool as everything else, added a SATA SSD as an L2ARC and added the Bloom filter option in the IPFS config. |
|
@RubenKelevra wrote
This has been fixed with IPFS
Still not fixed upstream.
Today I started running the cluster with pin-update. Please report back how the IO is now. |
|
Since no further feedback has come in, I guess this is resolved. |
the current approach to pin on each update the full folder structure recursive requires IPFS to do quite a lot of IO, as @Luflosi mentioned here: #39
There's still the other option that I've implemented on version one of this project, which is to pin each file on it's own, as well as each folder non-recursive.
The advantage is a low amount of IO and a much faster processing time of updates on the cluster nodes, as it's easier to be processed by IPFS. But it requires a lot of disk space on the cluster-database since each changed file and each changed folder is an individual transaction. After just some months the database had a file size of 20 GB while there were 400'000 transactions.
Once ipfs-cluster/ipfs-cluster#1008 and ipfs-cluster/ipfs-cluster#1018 are implemented we could explore this possibility to reduce the IO load again, with one transaction per update, but individually pinned files and folders.
The text was updated successfully, but these errors were encountered: