Add node.js implementation to std

[waneck]

This issue report is to start a discussion about whether it would be a good idea to implement the node.js library as part of haxe std.

Node.js can be considered a very useful technology, and I see that its use has been growing a lot in the Haxe community. We have some problems with the current state of affairs now, however:

• Synchronization between haxe std files and nodejs libraries - many libraries need to change some std Haxe files to provide e.g. the nodejs way to deal with bytes and streams. These changes into core may get out of sync if we don't maintain it in the same place
• Try to unify all the scattered implementations / forks, in a place where we can be certain to maintain them
• Add a specific travis target for node.js, so we can make sure we don't break anything there as well

Thoughts?

[ncannasse]

How much base Node.js API is stable ? At first it was changing quite often, sometime breaking things so we felt it was more in place into an haxelib. I agree however that having it standard would be nice. We could actually add Node-Webkit API as well if they are stable enough.

[andyli]

For Travis, we do actually run the unit test on nodejs (see this line).
But of course we don't have any nodejs specific test.

[waneck]

The API itself seems very stable. But maybe someone with more experience with Node.js could tell us more precisely about this.
@andyli , I know the tests are run on nodejs, but they are still run without e.g. the Bytes implementation as a NodeBuffer.

[ncannasse]

BTW how does NodeBuffer relates to typed arrays ? We want to change haxe.io.Bytes to use typed arrays in JS.

[waneck]

It seems like node buffers are modelled after the typed array specs. From the docs, however, it says that node.js buffers are more efficient but introduce some subtle incompatibilities with typed arrays. Also converting a node buffer into an ArrayBuffer will require some data copying.
Since most of node's bytes handling use NodeBuffer, I'd say that it's best that we use it when -D nodejs is defined.

[ncannasse]

Maybe we could have this togglable ? If you want to use Node Webkit for instance, you want to keep haxe.io.Bytes as native JS

[ncannasse]

(because it's used to upload to WebGL)

[aduros]

@ncannasse The API is very stable these days.

A big problem is that all of Node's IO methods are async and don't map well to Haxe's std lib. (there are sync versions, but they're a bad idea to use in server code)

Pinging @dionjwa who is the maintainer of https://github.com/dionjwa/nodejs-std

Personally I think we should go ahead and include basic Node externs now, and consider more features later. It would be good to be seen as "officially" supporting Node in the eyes of backend developers evaluating Haxe.

[ncannasse]

I think @clemos might be interested too

[hexonaut]

+1 for this idea.

[Justinfront]

Nicolas Cannasse:
We could actually add Node-Webkit API as well if they are stable enough.

+1

Node-webkit works with haxe javascript ( canvas and dom ) browser code
without any modifications

https://github.com/rogerwang/node-webkit

It would certainly be simpler if hxjavascript or similar haxelib had
node-webkit haxe.sys support built in and it was in the haxefoundation
repository, and improvements actively merged in. Quite keen to use and
contribute if and where I can, but don't really know node well enough to
start the process as the current hxnode git versions do not seem to be
setup to work well with webkit-node.

[frabbit]

Maybe we should consider adding async counterparts to the sync apis for filesystem etc.

[clemos]

It's a complicated question :)

It might be a good idea to add nodejs-std to Haxe's std lib.
This said, I must say I don't really use the std lib anyway...

I tend to believe that most parts of the std lib, being very rarely used by most Haxe users, should exist in a separate project, probably separate haxelibs.
Of course, the most basic/low level and cross-platform parts, such as Array, String, maybe Date, ds.*, etc, should remain there (as well as a cross platform ByteArray implementation :D).
But the rest should split to different "modules", pretty much like what openfl does, something like :

• haxe-sys for PHP, (synchronous) JS/Node, Java, C#, C++
• haxe-flash for the Flash API
• haxe-php with externs for native PHP apis
• haxe-js with native JS externs like Regexp, Error, Console, ... ?
• haxe-nodejs with externs for native NodeJS apis
• haxe-nodejs-webkit ...
• libraries like SPOD, Dispatch, maybe others, should go in separate haxelibs

To me, it would make much more sense.
This might also be easier to maintain.
I think Haxe is much more powerful as a compiler "alone", with different projects plugged on top of it.
IMHO, the std lib doesn't reflect this power, as it's the "minimal common denominator".
The most successful projects (NME, OpenFL, Cocktail) are of this nature: they don't really care about the std lib, but rather invent their own "cross-platform" way with "just" the compiler.

But maybe I'm going too far :D

[clemos]

The model for these different "per-platform" "per-usage" modules being, obviously, my https://github.com/clemos/haxe-js-kit project :D

[delahee]

Well i strongly disagree with Clem, the std lib is well maintained and very
reliable because it is there. And it does not need 5 git pull to freshen...
As to include node to std its very nature ( async & promises ) makes it a
bit awkward to integrate here but it would be fun :)
Le 20 mars 2014 23:22, "Clément Charmet" [email protected] a
écrit :

The model for these different "per-platform" "per-usage" modules being,
obviously, my https://github.com/clemos/haxe-js-kit project :D

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2784#issuecomment-38228383
.

[Simn]

I don't like things like SPOD and Dispatch very much. Every time a bug is reported for these I hide under my bed and hope that it goes away.

[delahee]

Reading these codes tickles my brain. It feels like invoquing an eldritch
horror from a non euclidian plane, dancing half naked on a squid spire
during a full moon eclipse.

Let ftgn=tl in l with x ? Gn:fb let chtgn' chl...

Don t read this aloud. Never. Just like hxods code. Never ever.

Also if you want to summon Nyarlatoteps Brood...there is a macro for that.
Le 20 mars 2014 23:32, "Simon Krajewski" [email protected] a
écrit :

I don't like things like SPOD and Dispatch very much. Every time a bug is
reported for these I hide under my bed and hope that it goes away.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2784#issuecomment-38229207
.

[clemos]

I think haxelib becomes mature enough to handle different "official" modules, and Haxe should rely more on it.
Once again, OpenFL has split to several modules, it works and I think it's a great step (there is a lot to say about it, it could be better/cleaner, but still...).

There are so much examples of "broken" things in the std lib :

• JQuery : it's not up-to-date, JQueryHelper is just weird (this in js should be in js.Lib or something...). To at least keep it up-to-date, the releases of this extern should be desynchronised from Haxe releases...
• SPOD, Dispatch : already discussed, enough said... maybe List could also go in the list :D
• ByteArray / Bytes is incomplete / not cross platform : I used Bytes once in Haxe/Flash, believing it would be cross platform, but unfortunately it's not... Later I found a ByteArray implementation in NME/OpenFL, which turned out to be rather different than Flash's (see my pull request to OpenFL HTML5 merged 2 month ago or so)
• It may be a weird example, but one can want to do server-side AS3 (FMS). This is a pretty similar situation than browser JS vs node.js (vs rhino on Red5, vs...)... I believe js.html.* should also go to a different lib
• Lots of things are not 100% consistent across platform... That's normal, but needs proper documentation, which may be easier in smaller projects, with official maintainers.

Also, I've been thinking for a long time that Haxe for "single platform" things should be promoted. That's what I try to do for node.js.
In order to do this, per-platform implementations/maintenance must be strong, and thus can't possibily be managed by a couple of people alone for all platforms / targets.
I think it also needs desynchronization.
hxcpp is a good example for that matter:
It's "independent" and lives on its own,
but it's still rigorously maintained such as every Haxe release implies a new hxcpp release,
but hxcpp is still free to do releases between Haxe releases.

[clemos]

Forgot to mention SWFObject.... god

[clemos]

Haxe remoting should totally go to a different package as well...

[Justinfront]

Last time I used swfobject I forked Mockey's one because it was out of
date and got it working with very minor changes, looks like he has since
updated his repository to also work.

https://github.com/mockey/swfobject-haxe
https://github.com/Justinfront/swfobject-haxe

Maybe we can remove the one from Haxe foundation and Mockey can upload
to haxelib, but would suggest he adds an example use similar to the one
I added as it saves user some time and is small.

Clément Charmet:

Forgot to mention SWFObject.... god

---

Reply to this email directly or view it on GitHub:
#2784 (comment)

[ncannasse]

@clemos I think that's going quite OT there.

We have no plans to split Haxe std into submodules. The only reason for doing so would be that there have different release cycles and for instance disk access haven't been changed since the 70s

Regarding your other comments:

• JQuery should be definitely improved, please open another issue or even better send a pull request.
• Bytes is cross platform, we have unit tests for it. If there are issues with NME they should be addressed by the project team or discussed with us
• Please report any other cross platform inconsistency. We have many unit tests and have specified most of the std lib, so you can read what is supposed to be cross platform and what is left unspecified on http://api.haxe.org

Back to the topic:

If the base NodeJS API is stable enough, we can add it to std. You're welcome to contribute.

[clemos]

The NodeJS API stability is pretty completely documented :
http://nodejs.org/api/all.html#all_stability_index
There are still some quite important parts of the lib that are considered unstable, such as crypto, streams or cluster.

For all the reasons stated before, I will just continue to contribute via my haxe-js-kit project, because I believe it makes much more sense to keep it separate / desynchronized.

[Deltachaos]

I would like to implement this feature. But for now I don't have any information about how it should be implemented to get accepted by the Haxe Core Developers. In general the sys package should first implement the Synchronous API's of Nodejs. In addition to this we should have an Asynchronous implementation in an other package (e.g. haxe.async.sys.io). This should be an wrapper for the Synchronous method calls to be usable even on platforms that don't allow Asynchronous programming (like php). But this is an other issue to be discussed in #2863

I see several options:

• Make an haxelib
• Add to Code (using externs)

The Problem about the last option is that we have an dependency on node externs. I dont think that this is a good idea to have the next externs for a third party software in the haxe core. But on the other side it would be more user friendly to have the nodejs implementation directly shipped with haxe. In my opinion the Haxe core should get split up into several packages as mentioned by @clemos already. But this is an issue that could get target in an major release. For now the best would be to add those nodejs externs to the Haxe core and implement the Sys package for nodejs with them. But before I start doing this things (and waste my Time if it gets not accepted by the Haxe Core Developers) I want an clear decision by the Haxe Core Team which solution should get implemented.

[Deltachaos]

Started implementing this in https://github.com/Deltachaos/haxe/tree/nodejs-sys

[dionjwa]

Awesome!

On Sat, Apr 12, 2014 at 6:43 AM, Maximilian Ruta
[email protected]:

Started implementing this in
https://github.com/Deltachaos/haxe/tree/nodejs-sys

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2784#issuecomment-40280642
.

[Simn]

Isn't it easier to just create the repository and copy his current state?

[nadako]

just create a repo, add remote and push there

[Simn]

Yes, that's basically what I mean.

[eduardo-costa]

Whatever works for you guys!
Also nodehx have externs for packages commonly used in nodejs (e.g. mongodb)

How haxe will handle them?
Em 05/10/2014 15:53, "Dan Korostelev" [email protected] escreveu:

just create a repo, add remote and push there

—
Reply to this email directly or view it on GitHub
#2784 (comment)
.

[nadako]

i don't think we should have these in std lib

[Simn]

https://github.com/HaxeFoundation/hxnodejs

Team invitations sent to @eduardo-costa and @clemos.

[eduardo-costa]

Agreed!
I can keep them in nodehx.
After removing the nodejs part!
Em 05/10/2014 15:57, "Dan Korostelev" [email protected] escreveu:

i don't think we should have these in std lib

—
Reply to this email directly or view it on GitHub.

[nadako]

nice, let's get it started

[Simn]

If you don't mind I'll go ahead and close this issue. We can keep track of everything in the new repository.

[eduardo-costa]

Cool! I'll start to dump the nodejs modules!

2014-10-05 16:18 GMT-03:00 Simon Krajewski [email protected]:

Closed #2784 #2784.

—
Reply to this email directly or view it on GitHub
#2784 (comment).

[image: Imagem inline 1]

[nadako]

Please push only externs there, without any compiled js, hello worlds and node-modules.

[eduardo-costa]

Got it!
Will push the externs then you tell me if anything is wrong!

2014-10-05 16:46 GMT-03:00 Dan Korostelev [email protected]:

Please push only externs there, without any compiled js, hello worlds and
node-modules.

—
Reply to this email directly or view it on GitHub
#2784 (comment)
.

[image: Imagem inline 1]

[ponticelli]

My 2 cents:

• keep member names with lower case (I've seen a lot of capitalized methods in nodehx)
• only map the NodeJS API and keep express and friends out of it (at least untile the API is very stable).

[eduardo-costa]

We talked about it.

• nodehx will be now a "nodejs third party packages externs"

After the first push we will review all classes to make it standard.
The uppercase stuff can be my own code slipped in some parts.
But all "official" nodejs stuff I think I kept the same as the API.

2014-10-05 16:50 GMT-03:00 Franco Ponticelli [email protected]:

My 2 cents:

• keep member names with lower case (I've seen a lot of capitalized
  methods in nodehx)
• only map the NodeJS API and keep express and friends out of it (at
  least untile the API is very stable).

—
Reply to this email directly or view it on GitHub
#2784 (comment)
.

[image: Imagem inline 1]

[nadako]

HaxeFoundation/hxnodejs@7f6cb00 this looks great structure-wise. One thing I'm not sure of is the package we want to use for this. If we're going to integrate this into haxe stdlib, maybe it should be js.nodejs (or even js.node for shortness). @ncannasse any thoughts?

[ponticelli]

js.node would be awesome!

[ponticelli]

@eduardo-costa perfect, thanks! I'll try to contribute as I can.

[eduardo-costa]

I like js.node!
Do we vote or something to make it official?

2014-10-05 16:58 GMT-03:00 Franco Ponticelli [email protected]:

@eduardo-costa https://github.com/eduardo-costa perfect, thanks! I'll
try to contribute as I can.

—
Reply to this email directly or view it on GitHub
#2784 (comment)
.

[image: Imagem inline 1]

[Simn]

js.node is good.

[fponticelli]

I'd say go for it and we will change it if someone raises some good points against it ;)

[eduardo-costa]

Fair enough!
Will do now!

2014-10-05 17:10 GMT-03:00 Franco Ponticelli [email protected]:

I'd say go for it and we will change it if someone raises some good points
against it ;)

—
Reply to this email directly or view it on GitHub
#2784 (comment)
.

[image: Imagem inline 1]

[eduardo-costa]

Done!
Users will now handle nodejs stuff using the 'js.node' path.

2014-10-05 17:11 GMT-03:00 Eduardo Dias da Costa [email protected]:

Fair enough!
Will do now!

2014-10-05 17:10 GMT-03:00 Franco Ponticelli [email protected]:

I'd say go for it and we will change it if someone raises some good
points against it ;)

—
Reply to this email directly or view it on GitHub
#2784 (comment)
.

[image: Imagem inline 1]

[image: Imagem inline 1]

[nadako]

I've created a couple of issues on the new hxnodejs project which should be a short-term roadmap for cleaning stuff up.

[eduardo-costa]

Thanks!
I appreciate the help!

2014-10-05 20:16 GMT-03:00 Dan Korostelev [email protected]:

I've created a couple of issues on the new hxnodejs project which should
be a short-term roadmap for cleaning stuff up.

—
Reply to this email directly or view it on GitHub
#2784 (comment)
.

[image: Imagem inline 1]

[zaxebo1]

apart from the usual server application, poeple are also writing normal command line utilities too in nodejs
for example:
http://shapeshed.com/command-line-utilities-with-nodejs/
http://www.2ality.com/2011/12/nodejs-shell-scripting.html
http://oscar-mejia.com/blog/how-to-create-a-command-line-program-with-nodejs/

So IMHO the "sync" part like - "complete" implementation of Sys.hx consisting of println(),command() etc should also be done for nodejs. The "complete implementation" of haxe Standard API (even if it is sync ) will facilitate nodejs shell scripting/commandline utility users too

[nadako]

That's for sure, but with basic node.js API it's not possible to implement e.g. Sys.command without using third-party modules, like execSync. So we have to choose the most popular module, write an implementation using it and throw an error if it's not available.

But first we should prepare a quality set of externs anyway, so let's think about implementing sys later.

[Simn]

Locking this issue because it's annoying in my mail reader with thread view. :P