dafaframe overwrite failed due to _temporary clearance on Azure

[johnxpan]

We run a data collection app on Ubuntu box, and do the following to save the result on distributed storage:
line 160: df.coalesce(1).write.mode('overwrite').format('csv').save(hdfsPath)

Previously we saved the result to hdfs on local hadoop cluster, no problem. Recently we moved the storage to Azure Data Storage Gen2, mounted with Blobfuse2. We also set up a RAM disk for blobfuse2 file cache.

The data collector runs every 5 minutes and overwrites the previous existing csv file. It works for several rounds, then we got the following excpetion:

_Exception in thread xxx:
Traceback (most recent call last):
File "/usr/local/stow/Python-3.12.1/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/local/stow/Python-3.12.1/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self.kwargs)
File "/a/work/xxx/CollectData.py", line 160, in saveQueryData
df.coalesce(1).write.mode('overwrite').format('csv').save(hdfsPath)
File "/a/work/greenseer/venv/lib/python3.12/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 1463, in save
File "/a/work/greenseer/venv/lib/python3.12/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in call
File "/a/work/greenseer/venv/lib/python3.12/site-packages/pyspark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 179, in deco
File "/a/work/greenseer/venv/lib/python3.12/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o323.save.
: org.apache.spark.SparkException: Unable to clear output directory file:/path/to/mounted/adls/ prior to writing to it.

We changed the blobfuse2 filecache timeout from default 120 seconds to 0, the app runs longer time, but eventually we run into the same issue, the _tempoarary under mounted ramdisk cache can not be cleared, triggering application to crash.

The questions are:
Is there any setting to disable the blobfuse2 file cache altogether? A little performance hit is better than perioidic crash.
is there any setting to force the cleanup of _temporary once write is completed?

[ashruti-msft]

Hello,
There could be several reasons for the crash you’re experiencing:

1. Data Corruption: Manipulating the contents of tempcache while blobfuse2 is mounted may lead to data corruption and subsequent crashes.
2. Filecache Timeout: Even if you’ve set the filecache timeout to 0, data will not be evicted if any handle remains open.

If caching isn’t necessary for your use case, consider using block-cache instead. You can find a configuration example here: sampleBlockCacheConfig.

[johnxpan]

I did stop the application, umount the azure storage, clean up, re-mount storage, then restart the application. The issue still happens, so we can rule out the "Data Corruption". I checked blobfuse README.txt, it seems that we have 3 different kinds of cache: file cache, streaming cache, and block-cache. My questions are:

1. What is the difference between these 3 caching? It seems stream caching is better used for writing large files. What file size is considered large? The largest file size I am writing now is about 100MB.
2. According to the help file, we can not use file cache and Stream cache at the same time. Can I use stream cache and block cache at the same time. What is the difference between stream cache and block cache?

[vibhansa-msft]

File-Cache: This maintains a local copy of the file and serves all read/write requests from local disk until the timeout expires. This means if application needs to process the same file over and over again this will give you a significant boost in performance as there are not network calls involved here.
Steaming: This has some issues right now so we might just deprecate this soon enough. Avoid using this.
Block-cache: This is our latest offering which provides significant boost over streaming. In file-cache model we maintain a local cache. If you do not have sufficient disk space available to hold the file locally (consider a case where you have 1TB blob and you want to read and process it serially) then block-cache is the way. Due to its prefetching logic, it works well even with smaller blobs (anything beyond 1GB). If workflow involves reading a large file only once, then this is best solution. If workflow involves reading large file multiple times, then there is a disk-cache option in block-cache as well where instead of holding entire file on disk we just store fragments of it,

All of these models are mutually exclusive as of now and then cannot co-exist in a running system so based on your workflow choose which suits you best and configure accordingly.

[johnxpan]

Thanks for the detailed explanation! In our case, data collector does not read back data or do any data manipulation from ADLS, the only interaction between our data collector on Linux box and ADLS is to write the collected data into the ADLS every 5 minutes. All data processing are done later on databricks. Our concern is how to write the collected data to ADLS reliably. Can you explain how these caching (or buffering) mechanisms affect writing data into ADLS, and which option is better for us?

[vibhansa-msft]

If the same files, which are being read, are written to then you might have issues with want to maintain a consistent block size. If files being written to are totally new dataset then there is no issue.
File-cache: when you create a file, it gets created in local cache. All writes are redirected to local cache and only on close() file call the file gets uploaded back to storage container. If the size of file is larger then what you have in available disk space this is bound to fail.
Block-Cache: As and when you write the blocks, some of these blocks which are completly written will be upstread to container. At this stage they are not committed so you will not see anything on the container but data is already available at backend. Once you close the file the data will be persisted in backend and you will start seeing the file in portal or other tools. As we are continuously upstreaming the data performance will be higher than file-cache for large files and you can create files much beyond your available disk space as well.

[johnxpan]

I moved from file-cache to block-cache. The problem is gone. In block-cache, will blobfuse2 save a copy of the _temporary on the local machine at all? If so, where the cached copy be saved (memory?)