remove build step from serialport

[tmpvar]

Having spent a bunch of time with new comers trying to control their robots with node-serialport (as I'm sure many of you have as well) I've come to realize that this is crazy town and as soon as you add more install steps to the mix people's eyes glaze over and they start drooling. It's hard to recover from that and what we really want here is a ✨ magical ✨ experience.

There have already been few attempts at merging some amount of node-serialport into core, that have fallen flat or been rejected due to being too large a maintenance task to include.

I think what we need to do is find the least amount of functionality that needs to be pushed into core and champion getting it landed.

This particular pull request looks like a decent starting point: libuv/libuv#19

[KarbonDallas]

Thanks for starting this conversation @tmpvar!

What you mention is also my main motivator for wanting to see (at least some of) node-serialport's functionality brought into core. I must admit that it seems silly to have 'out-of-the-box' support for TCP & UDP (not to mention HTTP/HTTPS, DNS, SSL/TLS), but still lack the support for something as basic as serial.

[monteslu]

I think we have to be careful about getting anything other than full serialport API into core.

Aside from connecting at a specified baud rate and moving data, we need full control signaling which was recently added. This provides us the ability to do things like creating programmers to update various hardware with firmware binaries.

Seems like the path forward for serialport apis that get left out could be at least as much work as getting them included into core.

[bnoordhuis]

Refs nodejs/node#1546

[voodootikigod]

There are many ways to handle this, bringing serial port into the magical core is often what people run to. Other valuable and great options include:

• NPM providing a better solution for native modules that is uniform for all native modules (ideally) which includes a build/host infrastructure for all modules (not just serial port).
• A new 3rd party service providing the build environment for all native modules (possibly the node.io foundation)
• Reimplement tty such that the necessary socket options (flow control, baud, parity, XOR, etc.) can be exposed/configured purely in JS.

The ideal case is the third option. @bnoordhuis reference doesn't get us there, but could be the start. There is still a tremendous amount that would need to be built out to reach node-serialport parity.

[tmpvar]

oh, my bad! libuv/libuv#19 is basically nothing to do with serial ports. Sorry for the noise!

[mikeal]

What is the current state of this? I know there was a thread somewhere that identified a little closer what needs to happen but I can't seem to find it.

[nebrius]

Here ya go: #10

[mikeal]

@nebrius should we close this and link to that or is there enough other work/context here that we'd want to keep both open?

[nebrius]

I'm on the fence, although I lean towards keeping this one open.I kinda like this issue as a "god" issue, as it directly states what we want to accomplish, without tying it to how we want to accomplish it.

As a hypothetical: what if, after everything is said and done in #10, that something is missing? There's just one last little thing we didn't think about that we need to go pure JS that isn't accomplished in #10.

[joshperry]

I think we should close one of them and consolidate. If our goal is to commute native code, we should look at a more generalized solution than proposed in #10 (implementing something like setbaud, etc). Perhaps a way to easily communicate required ioctl functions that are needed by the JS code?

Transferring data over a serial port (which to be honest is, usually, really a USB device these days, unless you're running on embedded hardware) is only a very small part of the different kinds of hardware we'd like to be able to communicate with without writing native code.

This issue is probably better off in voodootikigod/node-serialport, keeping #10 open here.

[joshperry]

Actually wouldn't node-ffi would be sufficient for device configuration, while using the fs module's functions for reading/writing to the device, as long as it would work on Windows as well (can you use \\.\COM1 with fs.createReadStream()?.

[joshperry]

libuv/libuv#484 (comment) He basically says the exact same thing. Sorry, but not sorry if you guys hadn't noticed it like I hadn't...

[nebrius]

You definitely nailed the two use cases for native modules. The former should be able to work once we get the libuv patch landed and then make the appropriate changes in core.

Unfortunately node-ffi won't really work for the latter case. Good instincts though, we had the same thought :) #11

[joshperry]

If I understand correctly, node-ffi is not viable because of its speed for doing things like PWM with tight timing and low jitter or interrupt handlers?

[nebrius]

It's not so much PWM (software based PWM tends to be suspect, even when implemented in pure C). It's more that any call has an unacceptable amount of overhead.

When we're running Node.js on slower embedded systems, we sometimes struggle to do even pure JS things with the built in APIs. Our performance margins are razor thin. So even though a single FFI call may not make much different, once we start making multiple function calls in close proximity on these devices, the performance difference becomes noticeable in terms of cpu usage and lag.

[joshperry]

I'm just trying to characterize the needed use case for node-ffi in conjunction with configing streaming hardware. I assume there are other nonstreaming uses for native modules (bit banging PWM was just one example that popped into my head).

If soft PWM/I2C/SPI/Whatever is not realistic and not something we're aiming to provide, even from C++ because of non-deterministic latencies, then I think we will probably be able to get to a point, with uv_device_t, that node-ffi would work for the configuration of hardware devices (baud, etc).

Devices that still need tighter timing than node-ffi can provide could still build a native module, but I think that would be incredibly more rare than it is right now?

I can help with the native code, js layer, perf improvements, etc. but my hardware experience with node is pretty much limited to serial and USB work at this point, so I don't have a great base of experience to draw from as far as native code needs for driving other hardware-level devices.

I'm just hoping to ask the right questions and spur conversation so that we can get a good characterization of user needs, to see if there's anything we can do to meet them.

If we need node-ffi to be as fast as native, that's obviously not going to happen, but if pure-native perf isn't an absolute necessity, then doing things like constraining ourselves to certain types of functions and parameters, and taking advantage of asm.js speedups, we could possibly cut a good chunk of the overhead off in most cases.

[nebrius]

Devices that still need tighter timing than node-ffi can provide could still build a native module, but I think that would be incredibly more rare than it is right now?

A good use case to check out is my own code I wrote for the Raspberry Pi: https://github.com/nebrius/raspi-pwm/. I needed to use a C library to call out to PWM, because Raspbian doesn't provide drivers for PWM, and there aren't any good standalone drivers out there that I can just point users to and say "install this." This C library, however, does provide a driver of sorts (wrapped up in a bunch of other stuff). I would say this is really the other big use case: calling 3rd party C libraries.

If we need node-ffi to be as fast as native, that's obviously not going to happen, but if pure-native perf isn't an absolute necessity, then doing things like constraining ourselves to certain types of functions and parameters, and taking advantage of asm.js speedups, we could possibly cut a good chunk of the overhead off in most cases.

The Node.js core team isn't as pumped on asm.js, and the V8 team hasn't implemented it either. A good read is over at node-forward/discussions#16. I'm somewhat ambivalent myself, as it doesn't do anything to help with our having to cross the bridge to C land.

One of the zanier ideas I've had is to ship a build toolchain with node, which really would solve most of our problems in the general case. It wouldn't help with. e.g. NAN2 breakage, but it would solve what has traditionally been our biggest challenge.

The problem of course is that gcc/llvm/etc are really heavyweight and complicated. However, there is hope in Rust. The Rust compiler is very lightweight (or so I'm told), and I've chatted with some people in the Rust community and they think the idea sounds fun. So maybe, in time?

[nebrius]

I'm just hoping to ask the right questions and spur conversation so that we can get a good characterization of user needs, to see if there's anything we can do to meet them.

So far we've been focused more on user pain points, which I would argue is slightly different than user needs. I would claim that the difference is this: a user pain point is something that exists in the existing system that causes frustration and ultimately leads people to give up, whereas user needs are things in the existing that don't exist. This is, of course, just my definition, I just want to make sure we're on the same page.

For the Johnny-Five (and related) communities, this all comes down to installation. Once everyone has everything installed and working, it's a generally a great experience. But getting everything installed can be really challenging. I would claim that the pain points are:

• Installing node itself and getting a dev environment up and running (you'd be surprised how often people get stuck here)
• Getting the native modules installed
• Getting the system configured properly (this is easier on some systems than others)

So we can't do anything about the first point...it is what it is. We're mostly focused on the second point right now, since it's the most node-like. We're willing to try anything, but so far the perfect general case eludes us. We haven't really done anything for the third point, outside of pie the sky dreaming at conferences.

All of that said, we haven't really focused on user needs, i.e. new functionality that would enable new applications and use cases. Doing user studies, outreach to companies, etc could all be extremely valuable, and I would love to find someone to champion this cause. I barely have the time to be as involved as I do, so unfortunately it can't be me. Without someone to lead such an initiative, I'm afraid it just won't happen. So if you have any ideas for someone to lead it, or if you want to take it on yourself, that would be amazing!

[joshperry]

Sorry, I guess when I said "user" needs I meant the needs of the native module developer "user".

What perf guarantees do we need in order to get rid of the native part of native modules (using uv_device_t for streaming data and node-ffi for configuration, opening, control line twiddling, etc. or maybe even some completely different method).

I understand there is a heavy cost in calling native functions through node-ffi, I'm just trying to suss out what an acceptable level of overhead would be for a majority of the current native module use-cases.

By using asm.js-style code syntax you can execute code at near-native speeds. TurboFan already implements a large portion of these speedups using heuristics instead of (or possibly soon, in addition to the use asm directive).

What I'm curious is, if we had an asm.js compliant function calling out to a native one, if we could get a nice boost in speed when bridging that gap.

Shipping tooling isn't out of the question, but yeah, it puts a huge burden on module devs and the whole node toolchain. I know we want to alleviate pain points for js devs, but I also think that the community would benefit greatly from opening up direct device development to those that don't have the interest or time to learn C.

This is another place that the tooling driving asm.js could be helpful. We could take advantage of the emscripten/llvm work that's been done to create tools to automate the export of structures, constants, function defs, and perhaps even generate code to automate marshalling parameters and calling native functions.

On top of that we may even be able to make a file with annotations or something that sits to the side of the native lib that marks which functions can be compiled to native js and run as much of the native lib's logic in happy jsland as possible.

[nebrius]

What perf guarantees do we need in order to get rid of the native part of native modules (using uv_device_t for streaming data and node-ffi for configuration, opening, control line twiddling, etc. or maybe even some completely different method).

I understand there is a heavy cost in calling native functions through node-ffi, I'm just trying to suss out what an acceptable level of overhead would be for a majority of the current native module use-cases.

I don't think anyone has an answer to that, unfortunately. That said, JS performance is actually good enough for us. We don't use C for performance reasons, just because we need to do something that we can't do in pure JS. It's funny, FFI is so slow, that doing it in pure JS is actually faster in a lot of cases than using FFI to call to C.

I think most of us would be thrilled if we could do everything in JS, but we just can't get there from here, as my old dissertation advisor used to say.

What I'm curious is, if we had an asm.js compliant function calling out to a native one, if we could get a nice boost in speed when bridging that gap.

So the bridge is really fast actually, when doing normal C calls. It's just FFI that's slow. Crossing the bridge from JS to C, and then doing an FFI call from C to C is actually still really slow (in fact, this is what node-ffi does. The actual FFI call is done in C). So the bridge and FFI are two different things.

I guess what I'm getting at is that asm.js doesn't gain us anything in this area.

[joshperry]

@nebrius I really really appreciate you taking the time to engage with me in this conversation. I have enjoyed it quite a bit.

It's interesting how this seems to be very coincident with some things that have been weighing on my mind quite heavily somewhat recently. In some way, I think we're on the cusp of spanning the native/non-native chasm in general, and OSs as an abstraction over hardware specifically.

Going to have to uhhh ruminate a bit on this.

[dtex]

This comment is a duplicate from #10 which goes hand-in-hand with this issue.

Hey gang, a new milestone has been added to libuv for the forthcoming 1.8.0 release and it does not include our favorite PR. I've added a comment requesting its inclusion but if y'all can think of additional, compelling arguments in favor of it, I don't think that would hurt to add them.