Improve or remove lsof file checking

[masaball]

Description

In our batch process we call lsof to make sure the files are fully uploaded to the dropbox before being ingested. This method is also utilized in the regular file upload flow so that only files that are fully processed will be shown in the dropbox listing.

This accounts for as much as 30-40% of the load time when rendering the file-upload-step page. Improving the efficiency of the check, moving the check to happen when opening dropbox instead of at initial page load, or removing the file check entirely would net us a significantly lower load time.

Done Looks Like

• Remove check on page load when clicking on Manage Files
• Can the lsof call be removed completely?

[joncameron]

Could there be a check against modification times on files, and then run lsof only on those? This could be a way to speed things up.

[elynema]

If we remove the lsof check on batch ingest in general, files could be truncated when ingested or could error on encoding. If there is an error on encoding, then that will go out in a report to the user post-ingest. If the file is truncated, there's no error to report. However, we think the moving the original file to ArchiverSpool to get sent to SDA happens post-encoding, so by that time it is much more likely that the entire file has uploaded and the full version is sent to tape for safe storage.