WG: Considering a new HTTP WG

[jasnell]

I am considering the formation of a new working group to take responsibility for the HTTP and HTTPS submodule in the core lib. The HTTP WG would take responsibility for the http API, the http-parser depenency, implementation of HTTP within core, as well as support of HTTP in the larger ecosystem.

I am opening this issue to solicit input on (a) whether such a WG would be considered helpful and (b) put out a call for participation.

/cc @nodejs/tsc @indutny @dougwilson @piscisaureus @mscdex

[Fishrock123]

If so we should also pull in someone from Hapi. @hueniverse any recommendations?

[indutny]

+1

[dougwilson]

I'd participate, for sure.

[hueniverse]

I'll participate at first to see what it will look like.

[jasnell]

@hueniverse .. at this point, what it will look like will depend on the priorities of those who are involved as well as the needs of the community. So given that, I would turn the question around and ask you: from your point of view and from Hapi's point of view... what (if anything) should be the top priorities for a WG looking at supporting / evolving the http support in Node core?

[jasnell]

I mean... I have a number of specific things that I'd like to see happen, but, to be honest, it needs to be driven to a large degree by what the community actually needs, which is why having both Hapi and Express represented in the WG is ideal

[jasnell]

Let's get an initial call spun up to discuss: http://doodle.com/poll/8hy262fu5xn4kp8a

[hueniverse]

I would like HTTP to move out of core into a separate set of npm modules. That would be my ideal setup.

[pmq20]

@hueniverse Why do you want it out of the core?

[hueniverse]

@pmq20 I want everything out of core :-)

I think we are hurting the community by not allowing some healthy competition in offering better or different approaches to HTTP. It is impossible to get anyone to care about it when it comes with core and using something else feels like an exception.

I would like to see an HTTP implementation that goes further in terms of handling headers, status codes, etc. Right now, every framework has to deal with a lot of these things on its own over and over again.

It would be great to offer a few high quality modules like an HTTP parser and TLS constructs that people can use to write their own. For example, I wrote a module called shot which simulates an HTTP request against a live server to avoid making a network call. I would like to be able to implement it without feeling dirty about the hacks I had to do to get it working.

[jakerella]

+1

[Fishrock123]

It is impossible to get anyone to care about it when it comes with core

Maybe during the two-year limbo, but not anymore. :)

@pmq20 I want everything out of core :-)

Also I'd like to just leave a note about "what belongs in core" from the perspective of core:

If it cannot be done outside of core, at all, or without great pain, it probably belongs in core. Also, anything that currently exists in core belongs in core unless it was effectively private.

That is:

• If people can't do it outside and people need it, it belongs in core. Similarly goes for something that is very painful to do outside of core. Stuff in this category is basically core's job.
• If it currently exists in core, it (probably) belongs in core. I.e. we can't break the ecosystem. Our deprecations have to have time on the order of a year or so, at minimum, for even small features.
  • Given that http is used by a huge percent of the ecosystem, we can't remove it without a multi-year timeframe. This is probably something like a hundred of thousand modules, or will be soon. Directly or indirectly.
  • Things that can be removed fall into one category: use of them is almost not impactful. I.e. tops 100 (small) modules use it. If things are private, we may expedite this. See deprecate _linklist for an example.
  • Security issues may bypass all of the above, if necessary.

Again, I'm just try to lay out some prior groundwork here so that everyone's expectations can be reasonably based. :)

[mikeal]

I think we are hurting the community by not allowing some healthy competition in offering better or different approaches to HTTP

Can you be a little more specific. Hapi.js has an API that is quite different from core and has built a rather successful ecosystem around it. What were you not able to do, or would have done differently, if you weren't building on top of the core HTTP API?

[jasnell]

@hueniverse , as @Fishrock123 points out, for the project currently, we definitely do care about this kind of feedback --heck, that's why I'm proposing this new working group! So exactly these kinds of issues and concerns can be looked at and addressed. @mikeal's question is a good one: given the current http stuff that's in core, can you describe the kinds of hacks / work arounds that have been required to get hapi working, what kinds of specific changes in the core http api would have made it easier, and if you had a green field to work with (which unfortunately we don't, but let's pretend), what would you want core to provide so that hapi could do it's job better and make it's users hapi--um--er (sorry, couldn't resist).

[hueniverse]

Here are some things I would love to move to another layer:

• Handle every HTTP header properly based on its specification so that when a request is received, all the headers are already properly parsed and ready for usage. For example, breaking cookies into key-value pairs, figure out accept headers, etc.
• Better handling of setting headers - right now I need to block undefined values, check headers for invalid characters, know when I can set an array or when I need to concat the strings separated by space or comma. I want a much smarter set header API that knows these things.
• Enforce HTTP status code rules. For example, if the status code says no payload, don't allow payload.
• Make components available as lower level public API. Just go and look at shot and tell me this doesn't make you throw up in your mouth a bit.
• Expose HTTP status codes as a module without having to keep core and that module in sync manually. We use core's status codes in boom which in turn adds a node dependency which should be avoided.

And then there are all the things we need to do to handle streams and connection state which is where most of the framework complexity is. I don't know how to fix it but if I was given more granular API and building blocks, I could look into putting them together differently. I don't have a lot of details on this because I never took the time to think how I would build the HTTP module from scratch.

As the world moves to HTTP2, the core HTTP module will become less and less interesting and will require others to constantly keep everything in sync. I would like to see core makes it easier for using another HTTP module. Of course I am not suggesting shipping a version of node anytime soon without HTTP APIs or different ones. That would be absolutely destructive. But I would like to be able to easily offer alternative solutions whether in hapi or a new tier.

Hope this helps.

[mikeal]

Handle every HTTP header properly based on its specification so that when a request is received, all the headers are already properly parsed and ready for usage. For example, breaking cookies into key-value pairs, figure out accept headers, etc.

Do you want this as a hook on the parser or a hook for mutations after headers are parsed and handed to JavaScript?

[mikeal]

Enforce HTTP status code rules. For example, if the status code says no payload, don't allow payload.

I believe this is already the case, the parser ignores payloads for certain status codes and for HEAD request responses. If there are some that aren't handled properly, those are bugs.

[mikeal]

Make components available as lower level public API. Just go and look at shot and tell me this doesn't make you throw up in your mouth a bit.

So, there's talk right now of doing a "libuv.js" which is the low level primitives below most of stdlib being exposed as the "Node.js API" and stdlib being something that just runs on top of that. Given that toolkit, along with the parser, you could create your own HTTP API but I'm wondering what pieces might be missing in that equation.

[jasnell]

I believe this is already the case, the parser ignores payloads for certain status
codes and for HEAD request responses. If there are some that aren't handled
properly, those are bugs.

Unfortunately it's not the case for every status code, @mikeal -- try a 205 Reset, for instance. The payload is passed right along. There are definite improvements that need to be made there.

[mikeal]

@jasnell ah, well, those are just straight up bugs then ;) We've already set the precedent that we will ignore payloads when HTTP doesn't allow (see HEAD responses) so I don't even think this is controversial.

[Qard]

Yeah, I'm hoping API WG has some interest in exposing the lower-level components that support the current http implementation as public APIs. It'd be great if the parser was officially usable directly, among other things.

[hueniverse]

@mikeal this is a detailed as I am going to get on this thread... my point is that the current API and implementation are not great. I would rather solve a lot of these problems in a new separate module but happy to provide feedback in any context. The extent of my personal contribution will depend on how excited I am about the actual changes being made but that's beside the point.

[jasnell]

Yeah, there's no need to go into too much detail here in this thread... that's what WG meetings are for. These are excellent points and echo quite a bit of the work I'd like to see done. Of course, it's going to take a while to get there.

[jasnell]

So far it's looking like Monday October 12th at 9am Pacific for the first call. I'll get the google hangout set up.

[trevnorris]

From the discussions it seems a useful first step would be making it easier to consume a raw TCP stream that HTTP modules could consume. Possibly other things along the lines of exposing the HTTPParser in a more user-friendly way.

[jasnell]

@trevnorris ... yes, precisely. Obviously the devil is in the details tho.

[jasnell]

Initial meeting Hangout on air for participants:

Monday October 12th, 9am-10am pacific

https://plus.google.com/hangouts/_/hoaevent/AP36tYeU5yu0hCNObLgsM6jThX_qzvVuyHcVpXjM3PT-OW2jexTaMQ (I'm hoping this works). @indutny @dougwilson @hueniverse @trevnorris @piscisaureus @mscdex @nodejs/tsc

For observers: https://plus.google.com/u/0/events/c81l0hi96175jkup9ebtcm6rch8

[jasnell]

Canceled the hangout. Only @dougwilson and I showed. Will reset and schedule a new hangout and hopefully get more there.

[hueniverse]

Probably better to send an actual invite... [email protected]

[Fishrock123]

Doh, forgot that was today. :/

[aks-]

Hi folks! I would love to be involved. I've been talking to @Fishrock123

[mscdex]

Since the hangout was cancelled, is there a new doodle for the next one? I had forgotten about the first hangout.

[jasnell]

I'll be setting one up shortly
On Oct 14, 2015 1:34 PM, "Brian White" [email protected] wrote:

Since the hangout was cancelled, is there a new doodle for the next one? I
had forgotten about the first hangout.

—
Reply to this email directly or view it on GitHub
#3214 (comment).

[Fishrock123]

Fwiw: I'd also like to join. (I worked on express prior to Node core)

[ritch]

@jasnell ...I guess I'm late to the party. @mhdawson mentioned that I should be a part of this.

@mikeal @hueniverse this discussion is eerily familiar. IIRC this goes back to some agreements at the first node summer camp:

1. innovate outside of core
2. do as much as possible in JS (instead of C++)

I agree whole heartedly with both ideas (then and now). Especially with this:

... move HTTP out of core into a separate set of npm modules.

This would allow the community to innovate around http. That might mean different things to different people: but I think thats the point.

The reality is, its going to be a fairly difficult process (its not just a matter of hacking the modules together, we need good documentation, need to educate people/frameworks to use it instead of core, etc), and people that normally don't work together are going to need to.

If you guys think it makes sense for me to be involved, I'd be glad to help. But I've noticed these working groups get pretty large pretty quickly, and that doesn't sound so practical IMO.

[jasnell]

@ritch ... doodle poll for scheduling here: http://doodle.com/poll/xbqfpqv7hfcriwin#table

[rvagg]

fwiw I've filled in the doodle, my availability is a bit spotty next week cause of travel and beyond that will be awkward because of timezones. While I'd like to be involved I don't want to be a blocker to getting new folks involved so count me out if it doesn't work.

[aks-]

all the meetings are after midnight for me, but i will try my best to attend the first ones.

[jasnell]

FYI: Meeting is scheduled for Thursday, Oct 22 at 1:00pm pacific time.

[rvagg]

keeping this on tsc-agenda cause a quick report from the initial meeting might be helpful