-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adder: Fix multi-file add so it works as expected. #3258
Conversation
2d016a6
to
551f540
Compare
…dually Add (currently failing and marked as such) test that each file specified on the command line is pinned individually when using "ipfs add". License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
8355c74
to
31add0f
Compare
Added test case. |
This doesnt handle the case where i I think that instead of skipping using the mfs root, we rework how the 'pin finalize' works. We need to be able to convey what we want eventually pinned to the adder. Maybe we do a logically separate add for each input? (that seems to make the most sense to me) |
You are right it doesn't. Handling that case will be very tricky due to how the request is sent to the server. For now, I think we should detect and disallow adding multiple directories at once. I don't particularly like the idea of using an mfs root when it is not necessary. In a script for the filestore (add-dir, see the README (section Adding all files in a directory) for info on using it) I am adding 1000s of unrelated files (about as many as I can get away with before running into system limits on the size of the command line) at the same time, the mfs root in those cases is completely irrelevant. |
Also, we talked a lot about this in #2811. I tried one request per file in that commit and it turned out to me more code so I went with what I was currently using. To see how an alternative GetLinks() is provided see filestore/support/dagservice.go in #2634. |
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
4033959
to
b831b3c
Compare
If neither "-w" or "-r" is specified don't use an mfs-root. Add and pin each file separately. Closes #2811 License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
b831b3c
to
543f611
Compare
@whyrusleeping I recently pushed another test case to cover the other problem caused by using an MFS Root as summarized in #2811
Due to the way "-r" is handled this will need to be done outside of the add command. The directory content are already expanded and there is no way for the add command to see which ones where specified on the command line. |
@kevina the This needs to be verified, but i think the structure looks something like:
(still need to confirm that, but) If thats the case, we can just iterate over the immediate children of the Files object and run 'add and pin' on each of them individually. |
@whyrusleeping I did some checks to double check. The Files object in the PreRun is as you described. However, when the request is sent to the server the directory contents are expended and there is no way to tell what was specified on the command line. For now the easiest thing would be to detect the situation in the PreRun and abort with an clean error. Agreed? |
Adding multiple directories without "-w" will not produce the expected result, so for now it is best to disallow it. This also added a NumFiles() method to SliceFile. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
@whyrusleeping, okay, as discussed over IRC, I added a test to disallow multiple directories and filed an new issue. Hopefully this is good to go. |
@whyrusleeping I would like you to consider giving this pull request some serious considering. Not using a MFS root for single file adds turns out to required for my filestore to function. Here is what happens in a single file add without this patch:
If the MFS root is not used than 2 and 3 do not happen when adding single files and there is no problem. For now, as a hack, I will ignore a ProtoNode if the block already exists in the filestore. (And in case there is any confusion I can't just blindly accept FilestoreNode and pass on ProtoNode as that will lead to storing the same block twice, once in the filestore and once in the normal datastore.) Steps (2) and (3) seam like a horrible waste of time even without the problems it causes the filestore. |
In which cases will the mfs root actually be a file? I think most cases it will be a directory node, or at least an intermediate node. And thats fine to have in the normal blockstore anyway. You only really care about having leaf nodes (data containing nodes) in the filestore, right? |
|
Also, although not strictly relevant to this problem, I also store non-leaf nodes related to a file in the filestore. They are used for maintenance operations and to be able to easily list the files stored in it by only listing the roots. I do not store directory nodes in the filestore. |
Lets loop back around and see about getting this issue fixed |
This code was rewritten at least once since this PR was opened, feel free to reopen if it's still relevant. |
If neither "-w" or "-r" is specified don't use an mfs-root. Add and
pin each file separately.
Closes #2811
Part of #2634
License: MIT
Signed-off-by: Kevin Atkinson [email protected]