Ability to specify multiple final copy destinations

[jasonstjohn]

Want to specify that the final file be written to two destinations: One being archived to tape (but not readily readable), and one which is readily readable (but not so robustly backed up).

[beauremus]

The strategy now, if I remember correctly, is to copy to the destination and remove the local copy.
Does it make sense to support an option to not remove the local copy?

Also, outside of the software, the disk space for L-CAPE has been problematic. Is that a concern?

[jasonstjohn]

These questions zoom out a little bit, so it makes sense to summarize the feature requirements driven by the computing resources:

• The original file is created and edited (repeatedly appended to) in-place in the volume where this is possible cd /accelai/app/, but where total space is very limited (500 GB, exclusive use). So it cannot be left in place.

• The final file is being moved to a directory under /pnfs/ldrd/accelai/ which is very large (1.0 EB, shared used), but where file editing is not permitted.

  • Currently the only copy of the final file goes to /pnfs/ldrd/accelai/l-cape/ which is tape-backed. Files here are available to read only when staged to the cache (DCache), a shared resource managed by fuzzy logic such that files tend to become unavailable for reading after 30 days or so without warning.
  • The use case is to specify a second copy be made under /pnfs/ldrd/accelai/persistent/ which is not tape-backed, where files remain readable. An allotment of 20 TB exclusive has been granted here, more than twice the accumulated 8.5 TB L-CAPE accumulated over 3 years of data collection. It is more precarious but more often readable, so

  Having the two copies is meant to cover the bases (recoverable deep storage and also the ability to re-process all recent files) given these PNFS resources.

[beauremus]

Given these requirements, I'd recommend that the output field be allowed to be a list, and when the file gets copied, it'll iterate over the list of destinations.