blobfuse2 doesn't list all folders in a directory #942
Comments
|
If your account is a non-HNS account, blobfuse will look for special directory marker files in the container. These are required by blobfuse to identify a directory. If data was uploaded to container through any means which did not create these files, blobfuse will fail to identify them. We have already committed the fix for this as part of #927. |
|
Ok I might try the main branch then. Previously, when using blobfuse v1.4.5 it was able to list the directories on the same account, without HNS enabled. |
|
@vibhansa-msft I tried on the main branch to the config. But I am still not seeing the "missing" directories? |
|
Hi @Mmdixon, could you please describe a minimal way to structure the storage account so we can reproduce your issue? I attempted to replicate the issue with no luck so far. |
|
Can you also share mount command and config you used for 1.4.5 so that we can compare, in case we are missing some options with blobfuse2. |
|
@gapra-msft it's a Premium LRS BlockBlobStorage account. There are multiple folders each with a couple of subfolders each with ~64K files. the v1 config file just has the account/container/key in it $ ./blobfuse2 --version
blobfuse2 version 2.0.0-preview.4
$ ./blobfuse2 mount /blob/cache --config-file=~/code/BlobfuseConfigs/turingitp2Config.yaml
$ ls -l /blob/cache/CC
drwxrwxrwx 2 madixon madixon 4096 Sep 13 2021 2021-25$ blobfuse --version
blobfuse 1.4.5
$ blobfuse /blob/data1 --config-file=/home/madixon/code/BlobfuseConfigs/turingitp2Config.cfg --tmp-path=/tmp/blob/data1/ --file-cache-timeout-in-seconds=86400 --use-attr-cache=true -o attr_timeout=120 -o entry_timeout=120 -o negative_timeout=240
$ ls -l /blob/data1/CC
drwxrwx--- 2 madixon madixon 4096 Sep 13 2021 2021-25
drwxrwx--- 2 madixon madixon 4096 Sep 11 2021 cache
drwxrwx--- 2 madixon madixon 4096 Jul 30 23:02 cache_conv
drwxrwx--- 2 madixon madixon 4096 Apr 20 01:30 dedupe
...
etclisting in the blobfuse v1 mount for the first time does take some time compared to blobfuse2 (many blobs in this container). |
|
updated syslog since switching to click for syslog |
As per above logs, in first iteration we tried to get list of blobs on "cc/" path and we got just one object out. In next iteration we tried to get more, and we did not get any response from server. This is kind of server saying there are no elements on this path. |
|
Kindly share your v1 mount command and v2 config file. |
Did I not do that? v2 config file is in the original post + the |
Can you add this to your v2 config towards the end and recollect the logs. As I mentioned earlier from storage it appears we are not getting the list of blobs. Just want to validate what request we are sending and what response we are getting. |
|
@vibhansa-msft here is the logs with |
|
Can you collect logs for v1 while you are listing and share that too, just to compare on what is going wrong here. |
|
@vibhansa-msft Some v1 logs click to see v1 logs |
|
|
|
Yes I saw that too, let me investigate if that is something playing a role here or just a logging typos. |
|
Ok, this is just a logging issue and not the URL. If there is anything in string with '%' logging is not able to escape that properly. So this is not the root-cause for the issue we are trying to trace. |
|
I have a theory here which v1 log supports. Need to check if v2 does not handle this situation. If we see this snippet from log, we are taking a token and asking backend to return us back with more elements (next block of elements in the listing). Server is returning back an empty list but at the same time it's also saying there are more items left. This means for some reason server did not return back any element and yet asked us to try back. In v2 we might be terminating the listing call if server returns back empty list. Will check the code and update on this. |
|
I have a potential fix for the above-described situation in "vibhansa/v2/streamdiremptylist" branch. If you can pull this branch and build locally and give it a try, we can confirm this theory. As of now I am not able to recreate this situation locally so not able to validate my fix. For some reason your account is hitting this corner case where server does not return back any list item for couple of iterations. |
|
thanks @vibhansa-msft the |
|
Thats great, thanks for trying this out quickly. |
|
Can you try listing once with SDK trace on and share the complete log file. From backend side we want to understand in what condition this happens where server does not return a list but does return back a token. |
|
@vibhansa-msft list on your branch? And include the startup mount logs as well? |
|
yes, on this working code you can just enable sdk traces and it shall list some REST calls going for listing but not returning back with any item. Just want those ids to trace some logs and follow up with backend team on whether this is expected case or not. |
|
@vibhansa-msft here is the full log click for full log |
|
Thanks for sharing the logs. As updated earlier we have the fix ready and merged in main branch, next release shall have it. Closing this issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Which version of the blobfuse was used?
blobfuse2 version 2.0.0-preview.3Which OS (please include version) are you using?
Ubuntu 20.04
What problem was encountered?
Doing
lsin a mount directory doesn't list all the folders. But if Ilson a one of those folders I know exists it works.Notice the
dedupefolder is the hidden folder.Have you found a mitigation/solution?
The files exist but can't list them.
By default, blobfuse logs errors to syslog. If this is relevant, is there anything in the syslog that might be helpful?
Click for syslog
If relevant, please share your mount command.
The text was updated successfully, but these errors were encountered: