Clearing the cache / Question 2/2

[gmaze]

Hi folks !

This is 2/2 questions wrt to fsspec cache handling when we want to clear it.

Context:

I develop a python library to work with ocean data collected using a specific type of instruments (Argo floats): https://github.com/euroargodev/argopy
Data can be fetched from local files or from an erddap server online.

I recently tried to use fsspec to implement file/resource management and ease handling of cached data.
I ended up wrapping file, memory and http fsspec file systems into classes that managed custom methods like open_dataset or open_dataframe.

Question 2:

Since the fsspec method wasn't working the way I expected (see issue #324), I implemented my own method in the custom file system wrapper. It works like this:

• Every time a call is made to open, I register the opened path in an internal list (self.cache_registry)
• When users call on the clear_cache() method, I load the cache and remove the corresponding entry from the pickle cache file and I also delete the corresponding hash file.

This goes basically like this:

import pickle
import tempfile
import shutil
import fsspec
import os

class argo_store():
    protocol = 'file'
    
    def __init__(self, cache: bool = False, cachedir: str = "", **kw):
        """ Create a file storage system for argopy

            Parameters
            ----------
            cache : bool (False)
            cachedir : str (from OPTIONS)

        """
        self.cache = cache
        self.cachedir = OPTIONS['cachedir'] if cachedir == '' else cachedir
        self.cache_registry = []  # Will hold uri cached by this store instance
        if not self.cache:
            self.fs = fsspec.filesystem(self.protocol, **kw)
        else:
            self.fs = fsspec.filesystem("filecache",
                                        target_protocol=self.protocol,
                                        target_options={'simple_links': True},
                                        cache_storage=self.cachedir,
                                        expiry_time=86400, cache_check=10, **kw)

    def store_path(self, uri):
        if not uri.startswith(self.fs.target_protocol):
            path = self.fs.target_protocol + "://" + uri
        else:
            path = uri
        return path

    def register(self, uri):
        """ Keep track of files open with this instance """
        if self.cache:
            self.cache_registry.append(self.store_path(uri))
        
    def _clear_cache_item(self, uri):
        """ Open fsspec cache registry (pickle file) and remove entry for an uri """
        # Inspired from the fsspec "save_cache()" method in:
        # https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/cached.html#WholeFileCacheFileSystem
        fn = os.path.join(self.fs.storage[-1], "cache")
        cache = self.fs.cached_files[-1]
        if os.path.exists(fn):
            with open(fn, "rb") as f:
                cached_files = pickle.load(f)
        else:
            cached_files = cache
        cache = {}
        for k, v in cached_files.items():
            if k != uri:
                cache[k] = v.copy()
            else:
                os.remove(os.path.join(self.fs.storage[-1], v['fn']))
        fn2 = tempfile.mktemp()
        with open(fn2, "wb") as f:
            pickle.dump(cache, f)
        shutil.move(fn2, fn)

    def list_cache(self):
        if self.cache:
            fn = os.path.join(self.fs.storage[-1], "cache")
            cache = self.fs.cached_files[-1]        
            if os.path.exists(fn):
                with open(fn, "rb") as f:
                    cached_files = pickle.load(f)
            else:
                cached_files = cache
            cache = {}
            print("\ncached_files items:")
            for k, v in cached_files.items():
                print("\n", k, "\n",v)

    def clear_cache(self):
        """ Remove cached files and cache registry entry from uri opened with this store instance """
        if self.cache:
            for uri in self.cache_registry:
                self._clear_cache_item(uri)
    
    def open(self, url, *args, **kwargs):
        url = os.path.abspath(url)
        self.register(url)
        return self.fs.open(url, *args, **kwargs)               

and here it is in action, trying to show case the effective cleaning of the cache:

with open("dummy_fileA.txt", "w") as fp:
    fp.write('Hello world!')

# Instantiate the file store and list the initial state of the cache:
fs = argo_store(cache=1, cachedir="./tmp")
fs.list_cache()

# Then we read some dummy data from a dummy file to trigger caching
with fs.open(os.path.abspath("dummy_fileA.txt"), "r") as fp:
    txt = fp.read()

# Check on the file store internal cache registry:
print("\nURI in argo_store registry:\n", fs.cache_registry)
# and list the cache folder:
fs.list_cache()

# Now, we can clear the cache:
print("\nCLEAR CACHE !")
fs.clear_cache()

# and check again the cache folder content to see the effect:
print("\nFiles in cache folder:\n", os.listdir("./tmp"))
fs.list_cache()

produces:

cached_files items:

URI in argo_store registry:
 ['file:///Users/gmaze/git/github/euroargodev/argopy/local_work/dummy_fileA.txt']

cached_files items:

 file:///Users/gmaze/git/github/euroargodev/argopy/local_work/dummy_fileA.txt 
 {'fn': 'efedbec0e26404c4c80f7668501cb66bd14a44112161acd9b6b5ce0f7706ce0c', 'blocks': True, 'time': 1592411294.763516, 'uid': '282336a7a94cc0ce955a259e49539c0e'}

CLEAR CACHE !

Files in cache folder:
 ['cache']

cached_files items:

Using this mechanism, the cache folder can be used by other fs instances.
My question is: would this mechanism be robust to other protocols than "file" ? and does it solve #324

[martindurant]

Yes, yes, and I think this approach should be implemented for CachingFileSystem and WhileFileCachedFileSystem. They could either be methods directly on the implementation or, I think cleaner, via an attribute like fs.cache.clear(), where the attribute acts something like a dict, so you can also del/pop just one URL and indeed list like your example.

[gmaze]

Great !
So I'll keep that in my wrapper.
but to implement this in fsspec, I'm not sure where the registry should be located

[martindurant]

If you really want the metadata to be separate for the instances, then it would have to be in memory only. I think the implementation I would/might do, would maintain the one metadata file in the cache directory as before, so perhaps not quite what you want.

[gmaze]

If you really want the metadata to be separate for the instances, then it would have to be in memory only.

Yes, I believe that would be a more complete version of my registry in the above example ?

... would maintain the one metadata file in the cache directory as before...

I agree, only one cache pickle file.

[martindurant]

There is a wrinkle here, in that the cache is designed to have multiple possible locations, with only the last being considered writable - so any "clear" operation would act only on that last location.

[gmaze]

oh, right, I missed this point, good to know then
Does this have to do with distributed file systems ?

[martindurant]

Possibly: you could have a shared network drive with cached content that is not as fast as local. But I can imagine other uses, where you want a more permanent cache shared between processes/users and a writable one for other stuff.

[martindurant]

Sorry that this conversation dried up. I think it might be helpful to include some of your ideas in a PR here, adding cache information functions and maybe dict-like syntax. Are you still interested, @gmaze ? Or perhaps you think that the del/clear methods in #328 are sufficient.

[gmaze]

Thanks for your patience @martindurant
I'm happy with #328 which does what I needed

May be the one last thing that is missing is a clear_cache() method for the case where the cache folder will be shared by several file system instances. In this case, we may need a clear_cache() that will only remove files fetched from one instance.
With the current implementation of #328, the clear_cache() will delete every thing.

I have no idea of how to code this. May be a filter on file prefix could help distinguish files cached with different instance

[martindurant]

There is certainly no tracking of which instance/process wrote which cache file, so I don't know how you would go about this without more rewriting. Indeed, I'm not really sure how that would be different from having separate caches for each.

I suppose you could do something like this now, like

fs1 = fsspec.filesystem(.., cache_storage=[loc2, loc1])
fs2 = fsspec.filesystem(.., cache_storage=[loc1, loc2])

i.e., both instances look at both locations for reading, but writing and clearing only apply to the last element of the locations list.

[gmaze]

Closing for similar reasons to #324
thanks for your help