About file descriptors in lib/wasi & their POSIX compliance

[corwin-of-amber]

Summary

Generally, I am trying to understand the relationship between Inode, InodeVal, and Fd. Perhaps this mechanism can be improved.

Additional details

First, some background: I am running ocamlc on top of Wasmer (in wasm32). One thing that I noticed is that it opens the same file multiple times. This is weird perhaps, but is acceptable POSIX behavior. In Wasmer, I find an fd_map which maps int -> Fd, which contains an Inode, which in turn is mapped through inodes.arena to an InodeVal. The latter has a Kind which in turn holds a handle: Option<Box<dyn VirtualFile>>. As much as I look at it, it seems that the Inode is determined from the path at path_open, and the arena is indexed by the inode.index, such that if I open the same file multiple times, I am bound to get the same Inode and therefore the same InodeVal. This is in line with my observation, which is (finally): if I open the same file twice, and then close one of the file handles, then the other file handle is also closed as a side effect.

What I am asking is then two things: (i) what is the motivation for so many levels of indirection from an int fd to a handle? (ii) would it make sense to have handles keyed by fd instead?

It also strikes as a bit strange that files in the vfs have a FileDescriptor (usize) and that e.g. mem_fs maintains a position cursor for a File in file nodes. Because the position is associated with the fd, and the entire model kind of breaks if a file has multiple active fd's.

[syrusakbary]

Hey @corwin-of-amber , we added vfs as an after-thought to wasi.
Initially wasmer-wasi handled all the filesystem by itself. And then we added support of doing that through the virtual filesystem.

What I am asking is then two things: (i) what is the motivation for so many levels of indirection from an int fd to a handle? (ii) would it make sense to have handles keyed by fd instead?

I believe most of the complexity came from the reasons I described before. I'm sure a lot of this logic could be refactored and improved.
Right now we are in the midst of a big refactor, but once that settles in I think it might be a good idea to start rethinking on how to simplify those layers

[syrusakbary]

It also strikes as a bit strange that files in the vfs have a FileDescriptor (usize) and that e.g. mem_fs maintains a position cursor for a File in file nodes. Because the position is associated with the fd, and the entire model kind of breaks if a file has multiple active fd's.

This might be a bug that we need to address

[corwin-of-amber]

Hi @syrusakbary! Thanks for the incredibly swift reply. I am happy to learn that you are going to take that into consideration. I will stay tuned to see the changes that you are doing in this refactor.

[fschutt]

Thank you for reporting, we are likely to do this after the WASIX merge, the filesystem code is currently frozen.

Yes, the Fd should either have a proper reference count and we need to make sure that the closing one Fd doesn't affect others, this is a serious bug.

The reason for the indirection is probably so that the Fd can be expanded later, which would not be possible with a simple integer.

[syrusakbary]

This requires a bit of refactor on wasi and the filesystem. Will post here the updates

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Feel free to reopen the issue if it has been closed by mistake.