[v8r0] Implement a recursive listDirectory for the DFC #6682

chaen · 2023-01-12T13:16:33Z

I'd have hoped for a simple flag in the listDirectory method, but it unfortunately can't be done (too much modification of too many methods that are used by too many components).

So I created a new getDirectoryDump method. For a given directory, it returns all the files and sub folders contained at any depth.

For now I've applied it only to the dirac-dms-user-lfn commands. In the case of the LHCb managers, the speedup is substantial.

For example, on the test instance, I can dump 250k files and 100k sub directories in less than 10 seconds. The original command never returned....

time dirac-dms-user-lfns -b /lhcb/user/c/cburr/
Will search for files in /lhcb/user/c/cburr
/lhcb/user/c/cburr: 251681 files, 98570 sub-directories
251681 matched files have been put in lhcb-user-c-cburr.lfns

real    0m9.893s
user    0m4.280s
sys     0m0.248s

BEGINRELEASENOTES

*DMS
NEW: Introduce getDirectoryDump method in the DFC
CHANGE: dirac-dms-user-lfns uses getDirectoryDump method
CHANGE: DataManager getReplicasFromDirectory and __cleanDirectory use getDirectoryDump

ENDRELEASENOTES

tests/Integration/DataManagementSystem/Test_FileCatalogDB.py

src/DIRAC/DataManagementSystem/DB/FileCatalogWithFkAndPsDB.sql

chrisburr · 2023-01-18T16:06:02Z

src/DIRAC/DataManagementSystem/Client/DataManager.py

        if not res["OK"]:
            return res

-        if not res["Value"]:
+        if not (res["Value"]["Files"] or res["Value"]["SubDirs"]):
            # folder is empty, just remove it and return
            res = returnSingleResult(self.fileCatalog.removeDirectory(folder, recursive=True))


I'm surprised that this has recursive=True?

It's maybe not needed indeed, but I'd rather not change it here in the risk of breaking something, and I'd like this to be merged before Monday's release 👼

src/DIRAC/DataManagementSystem/DB/FileCatalogComponents/DirectoryManager/DirectoryClosure.py

src/DIRAC/DataManagementSystem/DB/FileCatalogComponents/DirectoryManager/DirectoryTreeBase.py

src/DIRAC/DataManagementSystem/Service/FileCatalogHandler.py

src/DIRAC/DataManagementSystem/DB/FileCatalogComponents/DirectoryManager/DirectoryTreeBase.py

…c-dms-user-lfns

… use getDirectoryDump; remove __getCatalogDirectoryContents

DIRACGridBot · 2023-01-19T20:00:28Z

Sweep summary

Sweep ran in https://github.com/DIRACGrid/DIRAC/actions/runs/3961922796

Successful:

integration

DIRACGridBot added the alsoTargeting:integration Cherry pick this PR to integration after merge label Jan 12, 2023

chaen force-pushed the rel-v8r0_FEAT_recursiveListDir branch 5 times, most recently from f14ab2c to 8cc2b53 Compare January 18, 2023 14:32

chaen marked this pull request as ready for review January 18, 2023 16:02

chaen requested review from fstagni, andresailer and atsareg as code owners January 18, 2023 16:02

chrisburr reviewed Jan 18, 2023

View reviewed changes