Move File and IO::FileDescriptor's platform-specific parts to Crystal::System by RX14 · Pull Request #5333 · crystal-lang/crystal

RX14 · 2017-11-28T23:58:39Z

Hopefully the third time lucky we'll manage to merge an IO portability refactor.

I've copied the commit messages out below so people actually read them. I also suggest you review each commit in order, instead of reviewing the diff as a whole. it makes the entire thing a lot more readable and understandable.

Make IO::FileDescriptor fcntl private

fcntl() is an extremely low-level platform-specific detail which should only be used internally to IO::FileDescriptor. In this commit, it is removed from the public API and made private.

The only usage of this API was added only recently, in Crystal::Main. It is replaced with a method which calls LibC.fcntl directly, since it is too early in init to create a new IO::FileDescriptor instance directly and ask it if it's blocking.

Separate platform-specific parts of IO::FileDescriptor

In order to add windows support to Crystal, IO::FileDescriptor cannot depend on posix APIs. Here, we refactor out the platform-specific parts of IO::FileDescriptor into a mixin module in Crystal::System. Some platform-specific methods are refactored out of methods which contain both platform-specific and cross-platform behaviour. These methods are prefixed with system_ and are private.

Make Tempfile inherit File

Making Tempfile inherit from File allows the instance to be passed to methods which expect File, and makes Tempfile much more powerful. This requires adding a new private constructor to File which sets @path and calls super.

Separate platform-specific parts of File

Similar to previous patch: Separate platform-specific parts of IO::FileDescriptor.

straight-shoota · 2017-11-29T12:11:10Z

In the second commit, the extracted platform-specific methods are prefixed platform_. However, in the commit message it says to be system_.

I'd prefer system_ because it matches the namespace and is more descriptive.

RX14 · 2017-11-29T12:24:45Z

I'll wait a few days to see if there is consensus on naming. Then I'll pick one and make the commit message and diff match up.

If the only problem you could find is naming, I'm happy!

Papierkorb · 2017-11-29T12:36:24Z

src/crystal/system/unix/file.cr

+          raise "Invalid access mode #{mode}"
+        end
+      else
+        raise "Invalid access mode #{mode}"


Doesn't support a mode like rb+. Also, #{mode.inspect} would make it clearer in the output what comes from the user and what's the error message itself

This is directly copied from the old file.cr if it has issues, it's out of the scope of this PR.

Fair enough

Papierkorb · 2017-11-29T12:39:43Z

src/crystal/system/unix/file.cr

+      path = "#{tmpdir}#{name}.XXXXXX#{extension}"
+
+      if extension
+        fd = LibC.mkstemps(path, extension.bytesize)


mkstemps(3) reads: Since it [the template] will be modified, template must not be a string constant, but should be declared as a character array. That's not obvious from the function itself (At least to a non-C user), there should be atleast a comment noting that the string will be modified (Which isn't "possible" from Crystal), or it could be copied into a temporary Bytes.

Papierkorb · 2017-11-29T12:41:24Z

src/crystal/system/unix/file.cr

+
+    def stat(path)
+      if LibC.stat(path.check_no_null_byte, out stat) != 0
+        raise Errno.new("Unable to get stat for '#{path}'")


path.inspect would give proper visual "escaping"

Papierkorb · 2017-11-29T12:44:36Z

src/crystal/system/unix/file.cr

+
+    def real_path(path)
+      real_path_ptr = LibC.realpath(path, nil)
+      raise Errno.new("Error resolving real path of #{path}") unless real_path_ptr


Same path.inspect, this and following error messages omit the single-quote. (Consistency)

Papierkorb · 2017-11-29T12:57:17Z

I think there needs to be a proper support for platform-specific functionality in the stdlib. IMHO, throwing stuff out like #fcntl (The easy access to it) can be harmful: The wrapper methods are usually much easier and nicer to use; The LibC pendants are harder to get right.

Truth is, programs tend to need platform-specific stuff that's not yet nicely wrapped. I doubt that you'll be able to offer everything that the control functions support in an agnostic manner. And many programs are fine if they don't work on some random platform, as long they work where they're intended to be used. Diminishing all API to the most common denominator will only hurt on the long run.

Outside the scope of this PR, I'd actually like to see a documentation flag to note platform information right in the in-source docs. The docs generator could then mark such a section visually, or add a "Tag badge" to it.

RX14 · 2017-11-29T15:17:02Z

re: removing fcntl, I actually think you're right. I'll probably revert that commit.

RX14 · 2017-11-29T15:19:23Z

I appreciate these comments, but comments (in support of or against) about the approach of using a mixin module from Crystal::System would be most helpful to tell whether this PR is going to get merged at all.

ysbaddaden

I'm not sure mixins are better than composition, but that should really help going forward, so I'm gonna say 👍 ! I guess we'll probably rework it later when there is windows support, but... later. Let's go forward —as long as the public API is good and stable.

I don't think we need the Mixin suffix when naming modules, I mean include Crystal::System::FileDescriptor is just fine, and the filenames don't have the _mixin suffix anyway (crystal/system/file_descriptor.cr).

About platform-specific methods, there are usages of the _impl suffix in different places in the stdlib. Maybe we could stick to that? My point is that "private impl" can englob the "target-specific" meaning.

ysbaddaden · 2017-11-29T00:12:42Z

src/file.cr

+require "crystal/system/file"

 class File < IO::FileDescriptor
+  private alias Sys = Crystal::System::File


I'd avoid the alias and use the long naming.

ysbaddaden · 2017-11-30T09:54:43Z

src/file.cr

+  # TODO: use fcntl/lockf instead of flock (which doesn't lock over NFS)
+  # TODO: always use non-blocking locks, yield fiber until resource becomes available
+
+  def flock_shared(blocking = true)


Would it be opportune to break the file lock API now? The current flock naming and API is tied to the underlying flock POSIX implementation, which is bad. What about abstracting it now, since this patch is meant to support windows, eventually.

enum Lock Shared Exclusive end def lock(operation : Lock, blocking = false, &block) end def lock(operation : Lock, blocking = false) end def unlock end

I'd tentatively say that's easily done in a latter PR - and so it should be.

So be it. Let's stick to extracting posix-specific calls in this PR.

I'd recommend making an issue to track that though, we seem to forget a lot of these refactors once the PR gets merged.

RX14 · 2017-11-30T13:49:52Z

@ysbaddaden if you look at File, there's a Crystal::System::File and a Crystal::System::FileMixin. The Mixin naming comes from there.

ysbaddaden · 2017-11-30T15:22:28Z

Hum, I prefer to use def self. instead of extend self when there is no actual benefit to using the module methods as both includeable and class methods (for example Random::Secure does benefit). This would avoid having System::File and System::FileMixin and but just System::File, with some class methods, and some instance methods to be included in ::File.

Fairly minor, but it looks better to my eyes.

RX14 · 2017-11-30T15:30:56Z

@ysbaddaden sounds good to me. I'll change it.

RX14 · 2017-12-01T12:44:59Z

I'm still undecided about naming. I think a system_ prefix is better than _impl, because there's a difference between just a class-private implementation detail and a platform-specific method. system corresponds to the namespace, so it'd a good name.

ysbaddaden · 2017-12-01T13:54:06Z

To me it boils down to "implementation detail", but that's pedantic nitpicking, no need to spend hours on such details.

RX14 · 2017-12-01T15:35:05Z

Updated this PR. Changed the naming to system_, and removed the Mixin naming. Also re-added fcntl and moved the IO::System-specific parts to the mixin (windows doesn't use it yet because there's no evented IO).

RX14 · 2017-12-01T20:46:15Z

Also, if this gets merged, please make sure to rebase, not just squash. I think it'll make for a better commit history.

ysbaddaden · 2017-12-04T15:52:19Z

src/io/file_descriptor.cr

  private def unbuffered_rewind
-    seek(0, IO::Seek::Set)
-    self
+    self.pos = 0


Why the change?

pos=(value) calls seek(value) and Seek::Set is default. So seek(0) would be the most direct call here.

No reason for the change, it's trivial to see it's the same but I can revert it if you want.

ysbaddaden · 2017-12-04T15:59:23Z

src/crystal/system/unix/file_descriptor.cr

+    end
+  end
+
+  private def system_pos


I'd prefer system_tell over system_pos.

I wish we'd drop IO#pos and IO#pos= and kept IO#seek and IO#tell instead (less duplication, more explicit naming).

We just need 1 call: seek which always returns the current position. I suggested that IO implement pos pos= and tell in the base class to @asterite and simply provide a raising IO#tell by default.

For now, this is how IO works. It's not great, but it is how it is until I refactor.

Personally i'd rather keep pos, pos= and seek. tell is obscure and weird naming to me.

Well, it's "seek and tell". They're well known associated terms, whatever the language. E.g. C, Haskell, Python, Ruby or Rust, all have both.

I don't like the pos naming, it's an abbreviation and inexplicit (inherited from Ruby). At least position would be a more explicit name but I still prefer tell (broadly used). Using a setter (pos=) to perform an action, that is move the cursor in a file, sounds awful to me, in addition to be an exact duplicate for seek, too.

I have to say "seek" is not an unknown term to me, although "tell" is. I agree with you on pos vs position, but not that setters shouldn't have performance side-effects.

As far as I can tell, the terms tell and seek come from working with files. IO is a general class, and to me it doesm't seem logical to transfer this meaning from files to any other stream. But I may be mistaken.
Another thought: pos and pos= is a concept easy to understand. One reads the current stream position, the other sets it (position and position= are a bit longer but also fine). With seek and tell it is more complicated, if your not familiar with these terms. You have to remember two names instead of one and can't simply asseses their meaning.

I'm also for dropping #seek/#tell and using #position/#position= (Not #pos). The latter is really clear in intention. Also, it lets you seek around in a file using normal code:

my_file.position += 5 # This is how you'd write it anywhere else too! Fantastic! my_file.seek(my_file.tell + 5) # Compared to the one above: Meh. my_file.seek(5, File::SEEK_CUR) # I can never remember which SEEK_* I need - this sucks.

I think seek should stay, io.position = io.size - 5 involves 2 syscalls whereas io.seek(-5, IO::Seek::End) requires only one. Similar for io.position += 5. That being said, I think that using io.position is a much nicer interface which is more than adequate for most people. Perhaps some caching for size and position would mean that we can obviate seek entirely.

Regardless of that, we should make a new issue containing all the IO refactor suggestions from this thread. Unless anyone else volunteers to make that issue now, i'll get it done once this PR is merged.

ysbaddaden · 2017-12-04T16:00:24Z

src/crystal/system/unix/file_descriptor.cr

+    r
+  end
+
+  def fcntl(cmd, arg = 0)


IO#fcntl method is supposed to be public, but it won't be documented?

It wasn't documented before, I can add documentation if you want.

No, I meant: will the #fcntl method still appear in the documentation?

Hmm, not sure. Let me check.

They actually break the doc generator by saying they're included from the module, but the module is :nodoc: so that the links don't work.

I'll go and do this method the same as the rest - with the system_ prefix and {% ifdef %} the method out.

Yes, I've noticed this, too. Inclusion and extension of modules that are not documented should probably be hidden in the docs generator.

I pushed a fixup commit which resolves this. If this is OK i'll rebase to allow this to be merged.

RX14 · 2017-12-04T16:37:51Z

There have been loads of great ideas about refactoring IO::FileDescriptor and IO here, but I think that i'll work on a later PR which basically contains all this. I want to really focus on windows this christmas, instead of the IO refactors that work uncovers.

If people feel that some of that work would be better in this PR however, i'm happy to include it.

RX14 · 2017-12-04T16:43:44Z

Plus, a larger scope pf the PR means I have the chance to perform these same refactors on Socket and such. I really feel that these should be seperate PRs for more reasons than wanting to get this merged.

RX14 · 2017-12-12T16:22:35Z

Ping? I think I've solved all the reviews, and all this should need is a tiny review of the fixup commit from @ysbaddaden or a review from another core team member. I think this PR is fine to land in 0.24.1.

In order to add windows support to Crystal, IO::FileDescriptor cannot depend on posix APIs. Here, we refactor out the platform-specific parts of IO::FileDescriptor into a mixin module in Crystal::System. Some platform-specific methods are refactored out of methods which contain both platform-specific and cross-platform behaviour. These methods are prefixed with `system_` and are private.

Making Tempfile inherit from File allows the instance to be passed to methods which expect File, and makes Tempfile much more powerful. This requires adding a new private constructor to File which sets @path and calls super.

Similar to previous patch: Separate platform-specific parts of IO::FileDescriptor.

RX14 · 2017-12-13T15:24:40Z

Build failure is known related to travis's new build images: travis-ci/travis-ci#8891.

RX14 force-pushed the feature/io-crystal-system-2 branch from b0e0631 to fd81b4d Compare November 29, 2017 00:07

Papierkorb reviewed Nov 29, 2017

View reviewed changes

RX14 mentioned this pull request Nov 29, 2017

[WIP] Windows #3582

Closed

24 tasks

ysbaddaden reviewed Nov 30, 2017

View reviewed changes

RX14 mentioned this pull request Nov 30, 2017

Add Crystal::System::FileHandle #4912

Closed

RX14 force-pushed the feature/io-crystal-system-2 branch from fd81b4d to 70bb1e0 Compare December 1, 2017 15:20

RX14 mentioned this pull request Dec 1, 2017

"hello world" windows port #5339

Merged

ysbaddaden reviewed Dec 4, 2017

View reviewed changes

RX14 requested a review from ysbaddaden December 6, 2017 11:24

ysbaddaden approved these changes Dec 13, 2017

View reviewed changes

RX14 added 3 commits December 13, 2017 15:01

Make Tempfile inherit File

66daccf

Making Tempfile inherit from File allows the instance to be passed to methods which expect File, and makes Tempfile much more powerful. This requires adding a new private constructor to File which sets @path and calls super.

Separate platform-specific parts of File

586846d

Similar to previous patch: Separate platform-specific parts of IO::FileDescriptor.

RX14 force-pushed the feature/io-crystal-system-2 branch from ccdd6b7 to 586846d Compare December 13, 2017 15:02

RX14 merged commit 5fd965e into crystal-lang:master Dec 13, 2017

RX14 added this to the Next milestone Dec 24, 2017

RX14 added kind:refactor topic:stdlib platform:windows Windows support based on the MSVC toolchain / Win32 API labels Dec 24, 2017

straight-shoota mentioned this pull request Sep 7, 2018

Prepare IO::FileDescriptor to be platform agnostic #4500

Closed

Uh oh!

Conversation

RX14 commented Nov 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Make IO::FileDescriptor fcntl private

Separate platform-specific parts of IO::FileDescriptor

Make Tempfile inherit File

Separate platform-specific parts of File

Uh oh!

straight-shoota commented Nov 29, 2017

Uh oh!

RX14 commented Nov 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Papierkorb Nov 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Papierkorb commented Nov 29, 2017

Uh oh!

RX14 commented Nov 29, 2017

Uh oh!

RX14 commented Nov 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ysbaddaden left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RX14 commented Nov 30, 2017

Uh oh!

ysbaddaden commented Nov 30, 2017

Uh oh!

RX14 commented Nov 30, 2017

Uh oh!

RX14 commented Dec 1, 2017

Uh oh!

ysbaddaden commented Dec 1, 2017

Uh oh!

RX14 commented Dec 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RX14 commented Dec 1, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RX14 commented Nov 28, 2017 •

edited

Loading

RX14 commented Nov 29, 2017 •

edited

Loading

Papierkorb Nov 29, 2017 •

edited

Loading

RX14 commented Nov 29, 2017 •

edited

Loading

ysbaddaden left a comment •

edited

Loading

RX14 commented Dec 1, 2017 •

edited

Loading

RX14 Dec 5, 2017 •

edited

Loading

ysbaddaden Dec 4, 2017 •

edited

Loading

RX14 Dec 5, 2017 •

edited

Loading