Garbage Collector doesn't clean up MFS items

[RubenKelevra]

Version information:

go-ipfs version: 0.4.23-6ce9a355f
Repo version: 7
System version: amd64/linux
Golang version: go1.13.7

Description:

I couldn't find any hin in the CLI documentation, that this should happen so it's for me quite a surprising behavior:

When I create a folder or write a file the item isn't pinned (and there's no option to change this behavior). So I expected, that I need to pin the files, to preserve them from the garbage collector.

But while testing I created a folder and added a file to the folder, run GC manually and the folder and the file still exist.

I expected that the file get removed, and since the folder is then empty, it is removed as well.

I understand, that these would lead to changes in the file structure and thus to a different CID of the folder.

So I expected that either the folder gets removed when the last file has been unpinned (with all other files) or the files are just no longer accessible but are still listed with their CID.

This behavior gets especially strange when you add a file with ipfs add, then copy (ipfs files cp /ipfs/$CID /$folder/$filename) it to a folder, and then unpin the file since you don't need it anymore - it will still remain in the repo, but just because it exists in the file structure.

And the other way around too, you pin a file, delete it from the MFS but the GC cannot release the storage since it's somewhere deep in the ipfs pin ls list without a file name and you probably never find it again when you have pinned a lot of data.

[hsanjuan]

I think this is intended behaviour. If your file is in MFS, then it is effectively pinned. If your file is not in MFS, then it is pinned if it's on pin ls.

There were plans to consolidate everything around MFS, but I think things work as intended here. See #4763 for some of that discussion.

[RubenKelevra]

@hsanjuan thanks for the clarification, I really wasn't aware of this and wrote my setup around a different assumption.

I'll now create a HTML file to get the necessary listing of a 'directory' where folders and files will drop automatically if they reach the pin timeout of a cluster - by using a non-recursive pin on the directory CID and pinning each file individually.

If I pin an directory (which isn't stored in the MFS) non-recursive, can individually files be garbage collected when unpinned, or is there a similar constraint?

I think we should at least add this behavoir to the documentation.

Thanks for the fast response!

[hsanjuan]

If I pin an directory (which isn't stored in the MFS) non-recursive, can individually files be garbage collected when unpinned, or is there a similar constraint?

So, if it's not pinned directly or indirectly (ipfs pin ls <cid>) and it's not in mfs, then it should be GC'ed just fine. But note that a directory/file that grows too big might be split in multiple parts.

[RubenKelevra]

If I pin an directory (which isn't stored in the MFS) non-recursive, can individually files be garbage collected when unpinned, or is there a similar constraint?

So, if it's not pinned directly or indirectly (ipfs pin ls <cid>) and it's not in mfs, then it should be GC'ed just fine. But note that a directory/file that grows too big might be split in multiple parts.

I would create just html listings, but I would really like to offer the ability to pin those folders.

So, how do I check that it's not split?

Is 'size: 0' from ipfs files stat /dir the right way to check this? 🤔

And I'm currently just shy of 12 k files.

[hsanjuan]

Hmm, I'm not too sure there, can you ask on discourse? Is it ok to close this one?

[RubenKelevra]

@hsanjuan please keep it open, as reminder to document this behavior better. :)

When I'm done with my pacman.store project I'll take a look at this.

[hsanjuan]

as reminder to document this behavior better

Actually, if it's a docs issue, it should probably go to https://github.com/ipfs/docs :P

[RubenKelevra]

@hsanjuan I was referring to the CLI command output on --help :)

I'm not that into writing articles 😉

[hsanjuan]

Hows this @RubenKelevra #6882 ?

[RubenKelevra]

@hsanjuan thanks!

I've noted a small thing which is still missing IMHO, but the issue is solved for me with this PR :)

Thanks for the work!