Add filesystem locking on jetstream fileStore instances.

[jevolk]

• Link to issue, e.g. Resolves #NNN
• Documentation added (if applicable)
• Tests added
• Branch rebased on top of current main dev
• Changes squashed to a single commit (described here)
• Build is green in Travis CI
• You have certified that the contribution is your original work and that you license the work to the project

[derekcollison]

Possible to get a test written that shows correct behavior?

Also where to we clear the lock on successful shutdown of the filestore?

[jevolk]

Thanks for taking the time to review this.

Also where to we clear the lock on successful shutdown of the filestore?

The lock is cleared when the file descriptor at fileStore.lfd is garbage collected, or when the process exits. I have since added an explicit Close() call in fileStore.Stop(). I have not added matching unlock fcntl's but can do so at your request.

Possible to get a test written that shows correct behavior?

POSIX record locks used here are "process-associated" which means testing within the same process or thread group will likely be insufficient. The flock(2) alternative is superior but not available on all platforms. AFAICT, the testing infrastructure here is limited to instances within the same address-space, and does not fork into sub-processes. I'm not yet sure how to test this here.

Let me know your thoughts, thanks.

[jevolk]

Also, the CI appears limited to Linux only; I have not verified if the windows overload is effective.

[derekcollison]

If you added the close to Stop() you can do a Stop() and make sure you can recreate the filestore with the same storedir..

[jevolk]

By switching to flock(2) I have been able to add the proper tests, one affirmative (TestFileStoreLockFileSystem), one negative (TestFileStoreLockFileSystemConflict), which will work with the single address-space testing infrastructure here. This approach is superior because it can catch conflicts from two FileStore instances within the same server process as well as between two separate servers processes.

It appears that syscall.Flock() is made available by Go on all relevant targets. The caveat is that the behavior may be different depending on whether it's wrapping fcntl(2). On systems with a wrapper, the negative test might fail, since the POSIX record lock (via fcntl(2)) does not contend within the same process; two FileStores instantiated within the same process will not exclude each other. So while flock(2) is superior if available, it might not actually be available.

Volumes have been written about how UNIX filesystem locking is a notorious shitshow, so please bear with me here.