Fix issue preventing reads after flush on a file handle

[passaro]

Description of change

Fix a regression introduced in 0030b0a where closing (flush) a file descriptor would prevent subsequent reads on the corresponding file handle. The issue was caused by flush setting the internal state of the file handle to Closed and decrementing the number of readers for the inode, which is used to disallow overwrites when there are open read file handles.

This PR includes 2 main changes:

• it removes the Closed state and reverts flush to a no op for read file handles. The number of readers for an inode is decremented only on release.
• it makes open eagerly discriminate between READ and WRITE, rather than on first read or write (or flush or release). This reverts the opposite change in Allow file overwrites #487 and allows to report error conditions on open when appropriate rather than on subsequent operations.

Relevant issues: #749

Does this change impact existing behavior?

Because release is async, there is a race condition when trying to overwrite a file immediately after closing a read file handle on it, which results in the overwrite to occasionally fail on open.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

[dannycjones]

LGTM.

Acknowledging trade-off that we mitigate the regression preventing reads after flush but introducing a race condition into file overwrites (available since v1.4.0 released this month) which could prevent opening a file for writing immediately after closing a file. (@passaro should we add an ignored test for this? I can add one in a follow-up PR.)

[dannycjones]

LGTM!

[vladem]

LGTM:

1. flush() is no-op for read;
2. inode readers counter is decremented on release (accepted behaviour that open for write following a close may fail for short period of time);
3. eager initialisation of file handles to avoid FH being readable when we already have a scheduled work to write to it

[arsh]

LGTM