non-trivial directory layout

[fwang2]

As far as directory layout goes, currently LCIO just lay it out on a per-process basis, all files are created under p_rankid. This is overly naive and won't scale when we have billions of files - as it literally put hundreds of millions of file under a SINGLE directory. In short, we need a non-trivial directory layout scheme. Maybe also drawn from a distribution.

[mjbach]

How about a max_files_per_directory setting? Each process gets ceil(num_files/max_files_per_dir) directories, then randomly assigns a file to one of its directories.

[fwang2]

then random is not random, right? say each process is responsible for 1m files, and max_file_per_dir is 1000, then by calculation, you will create 1000 dirs under and put 1000 files in each.

[fwang2]

How about avg_files_per_dir = n, and this will define how many sub-dirs you will create.
However, the # of files you create under each sub-dir will be drawn from a normal distribution, and n is mu.

[mjbach]

That could work. That would also make analysis easier as well since it will add another normalization factor.