Node.js support

[kirill-konshin]

Do you have anything in your roadmap regarding Node.js support?

The purpose is to allow other cross-env libraries to use fetch as the client.

From what I see now in master there are couple of things to be done:

1. Use XMLHttpRequest polyfill (for example https://github.com/pwnall/node-xhr2)
2. Wrap the code in something like UMD to make sure CommonJS and AMD compatibility
3. Export all classes and fetch() function instead of always writing to global
4. Add optional polyfill() method just like in es6-promise for AMD users
5. Take over NPM package name fetch, current one is unmaintained for 2 years

Is it worth investigation and making a PR? Does not seem to be too much effort.

[matthew-andrews]

@kirill-konshin have you seen the node-fetch project? If you want a single library that will pull both through you might even like to consider using mine as well.

[kirill-konshin]

Github's polyfill is the most popular and well known, so I thought maybe it is reasonable to make it isomorphic...

I personally prefer single library approach so that I won't have to deal with bugs in 2-3 libraries which potentially may not catch up with the spec/bleeding edge implementations :)

Your version is not UMD-wrapped (will not work with RequireJS at all) and it is not a one-file dependency as this repo (yours has own sub-dependencies, which means slightly trickier install for non-AMD users and front end developers in general).

As I said earlier, those 5 items does not seem to be hard to do.

[matthew-andrews]

You make some really good points that I hadn't thought about before because I don't really use RequireJS.

I remember point 5 was considered but rejected because in order not to break backwards compatibility this repo would have to jump to v2.0.0… which didn't make sense… Once you know it's whatwg-fetch though it's not too difficult to find, is it?

I can't speak for the GitHub team but much of what you're asking for seems to take this project away from what I think it is defined as:- a client side implementation of the whatwg fetch specification and nothing more.

I don't think it would be very difficult to make a little requirejs-isomorphic-fetch wrapper similar to my one (perhaps mine ought to have been called commonjs-isomorphic-fetch). Maybe you could do it? :)

Also having used variants of fetch on the server for over half a year in production (including initially one that used the node XMLHttpRequest polyfill, which is abandoned too*) I really must recommend node-fetch. It is far far far more robust and is actively maintained!

* Here's a pull request I made to it in January.

[kirill-konshin]

I was talking about https://github.com/pwnall/node-xhr2 which I've been using in production for about a year too :) but anyway, native NodeJS http(s) modules will do a much better job.

If another isomorphic library uses fetch for browser (even wrapped in order to be compatible with AMD/Browserify/Webpack) and node-fetch for NodeJS then it 100% leads to trickery around package names in UMD definition which leads to issues with Browserify/Webpack (they tend to try to resolve both packages). I had such issues in the past, maybe it's not true anymore, I will check it.

I think we've got 2 options:

1. Only wrap fetch in UMD declaration in order to make it compatible with AMD/Browserify/Webpack (items 2, 3, 4 in my first list)
2. First option + use NodeJS http and https

Since they are sequential, I can make a PR for option 1 and then take a look on option 2.

[kirill-konshin]

I think that it's OK to deviate a bit from the original course in order to make someone's life easier :)

[mislav]

Hi, the request to make this library isomorphic comes in every now and then and we always refuse it. You should have searched past issues, like: #31 (comment)

To reiterate our reasons for refusal:

1. The goal of this library is to create a polyfill for an upcoming web browser standard.
2. We're not aware that there's any attempt to standardize fetch with the same API within Node.js.
3. We don't use this library on the server and therefore there's no incentive for us to maintain isomorphic support.
4. There are alternative libraries for fetch in Node environment that probably do a better job of maintaining that functionality than we would have done.

[kirill-konshin]

Ok, than what about at least making it AMD-compatible? So that I can use the same interface like:

var {fetch, Request, Headers, Response} = (typeof window !== 'undefined') 
    ? require('fetch') 
    : require('node-fetch');

Right now fetch polyfill only patches the global scope and does not export anything.

[kirill-konshin]

Taking into account that currently in Chrome there is no way to track fetch requests in network tab, I would like to use polyfill even if native implementation is available. Which can be achieved with appropriate exports or polyfill() method.

[dgraham]

Chrome's Network tab displays all requests, including those made with window.fetch. You can force the polyfill by removing the browser's native implementation before loading the fetch.js file.

<script>window.fetch = null</script>
<script src="fetch.js"></script>

I don't recommend doing this, but it will work.

[dgraham]

Right now fetch polyfill only patches the global scope and does not export anything.

The Fetch specification defines window as implementing the fetch method. There is nothing to export because fetch is not defined as a module that may be imported.

Our goal is to faithfully implement the polyfill in accordance with the specification so that it interoperates well with browsers' native fetch implementations.

[kirill-konshin]

Sorry for slightly wrong description, here is what I meant: https://code.google.com/p/chromium/issues/detail?id=457484, Chrome does not show POST bodies as it does for XHR.

The solution with deleting window.fetch was first thing that came into my mind too :) unfortunately, it will make a global change which should be avoided.

As an example, https://github.com/jakearchibald/es6-promise is an AMD compatible module, which can be loaded in scope and will not poison the global scope if not asked to do so.

[mislav]

@kirill-konshin You can easily obtain a reference to this fetch under another name.

<script>
originalFetch = window.fetch
window.fetch = null
</script>
<script src="fetch.js"></script>
<script>
realPolyfillFetch9000 = window.fetch
window.fetch = originalFetch
</script>

We're not responsible for the Chrome inspector bug and we will not change our code to work around your debbuging troubles. Charles Proxy is a great HTTP debugging tool that works regardless of browser inspector support.

[kirill-konshin]

I am using Charles all the time.

All that you said does not mean that you can't take the approach of es6-promose though. Not because of my "debbuging troubles" but for the sake of better implementation and integration. Don't you think so? Mangling with window does not seem to be a good solution if fetch is used in AMD environment where such trickery is not needed at all.

[mislav]

We don't want to maintain modular support for fetch right now because we don't want to support people loading it in a modular fashion just yet (or never).

[kirill-konshin]

Any justification why not? We're talking about ~20 lines of code at the end of file:

function polyfill(obj){
    if (!obj) obj = self; // fill window if nothing provided
    obj.Header = Header;
    obj.Body = Body;
    obj.Request = Request;
    obj.Response = Response;
    obj.fetch = fetch;
    return obj;
}
if (typeof define === 'function' && define['amd']) {
    define(function() {
        var res = polyfill({});
        res.polyfill = polyfill;
        return res;
    });
} else {
    polyfill();
}

[mislav]

We don't use AMD at GitHub so we have no incentive to maintain support for AMD. It's just 20 lines of code from your perspective, but for us it means committing to AMD support and maintaining this boilerplate code by updating it with future AMD updates. Also, if we supported AMD then we would have to support similar loaders that people ask us about, which means adding even more boilerplate code or various .json files to our project.

So you see, from your perspective it makes perfect sense for us to add these 20 lines of code to our project, and from ours it doesn't so much. From our perspective it makes sense to continue maintaining this perfectly fine polyfill that does its job as intended.

[kirill-konshin]

AMD boilerplate has not been changed for last 5 years and I doubt that anything will change in future according to RequireJS author.

Almost every other loader is compatible either with CommonJS or with AMD. Just use full UMD declaration (+2 lines of code) instead of short one (which I posted) and you're good to go, this code will never change and will not require any .json files in future.

Again, even es6-promise, the library that you recommend in README.md, use this approach.

[mislav]

Again, even es6-promise, the library that you recommend in README.md, use this approach.

Oh you mean this es6-promise which when adding loader support broke its functionality as a polyfill for months, which prevented us from upgrading it on GitHub.com?

[kirill-konshin]

According to semver it was a backward-incompatible change, this is why it is in 2.0. Stay with 1.0, if you use Bower or NPM it is easy.

My proposal is fully backward compatible with what you have now.

[mislav]

According to semver it was a backward-incompatible change

Fair point. However just reiterating on your previous arguments won't necessarily convince us any faster. I'd like to hear what my coworkers think. @dgraham some backup here please? 😅

[kirill-konshin]

I really doubt that such a small piece functionality, which by the way has value for your consumers, worth such a long discussion. I can make a PR and make sure all existing tests are passing.

[mislav]

https://twitter.com/mislav/status/623240201205256196

[kirill-konshin]

Life is tough ;)

[dgraham]

We receive a couple common requests, so I think it’s useful to summarize why we reject them. I spend lot of time considering these issues, and I don’t feel like I’m any closer to understanding why they recur. Here’s my perspective.

Background

Before looking at the requests, let’s cover some background. A polyfill is, by definition, a piece of JavaScript code that mimics a native web browser API that is still in development. It fills in missing functionality until the point where the browser supports the API natively, and then the polyfill is removed.

Browser APIs are not modules—they are all attached to the window global object. There is no support for optionally loading them, or loading some other version of them. They are native functionality that is built-in to the browser.

A polyfill is a somewhat unique piece of code, in the sense that it is intended to not be used at all in most browsers. A majority of visitors do not run this fetch polyfill. Their browser natively supports both window.fetch and the window.Promise global on which fetch depends.

Polyfills are valuable not because of their reach, but because they allow us to use new browser features before all major browsers have shipped them natively. This is code that is intended to be deleted within 2-3 years, once all browsers have built-in support.

Ok, so let’s discuss the two recurring requests.

1. Node.js support

The dream of Write Once Run Anywhere is being repeated under the banner of “isomorphic” JavaScript. The goal is to support both web browser runtime environments and Node.js server environments with a single piece of code. This is admirable, but having lived this dream once before with Java, it is not a reasonable long-term strategy. It does not work.

JavaScript must be optimized for its runtime: either a browser or a server process. Attempting to support both runtimes makes each of them worse. Some concrete examples follow.

Complexity

Fetch is a very well designed modeling of headers, requests, and responses, using asynchronous promises. This polyfill treats that design as a beautiful layer above an XMLHttpRequest implementation. We use XMLHttpRequest so you don’t have to!

But a Node.js server does not have XMLHttpRequest—it doesn’t make sense in a server environment. Node is not constrained by a standards process or several competing implementations. Anyone is free to write an HTTP library for Node.js, and subsequently, there are many great HTTP packages.

In order for fetch to support both a browser and a server we must introduce a plugin system where the HTTP subsystem may be swapped out for something else. We must then test the plugin system on all major browsers and then again with all popular HTTP libraries inside a Node.js server. This is complexity that we just don’t have time to support.

Lowest common denominator features

When designing for both a browser and a server, features that exist in only one runtime must not be used, and then we end up with something that works poorly in both environments.

The bitinn/node-fetch package is a great implementation of a fetch-like API optimized for Node.js. Browsing through the code reveals many non-trivial code changes required to optimize for servers:

1. URL protocol restrictions not present in a browser. A browser can fetch relative URLs because the request occurs within the context of a web page. Servers don’t have that context, so a restriction is needed to allow only absolute URLs.
2. Request compression, timeouts, custom user-agent, redirects. We would love to have timeout support in the browser, but it isn’t available. Because this is optimized for Node, we get to use timeouts on the server.
3. Headers access to internal data members. This would never be added to a browser implementation because it’s out of spec.
4. Chunked response body streaming. This is terrific and something that the fetch browser polyfill is unlikely to ever support because of limitations with XMLHttpRequest.
5. Custom response decoding. This is required by a server but handled by a browser natively.
6. Response is missing arrayBuffer() and blob() handlers. Headers is missing a forEach() method. These are required by the Fetch browser specification, but clearly not required for a server.

This list demonstrates why optimizing for the JavaScript runtime is important and extremely difficult to do in a single codebase. On the server, we get to use some of Node's best features even though web browsers don’t support them.

2. Module loader support

We are occasionally asked to add support for one of several module loaders. These requests are usually accompanied by dramatic language like “poison the global scope”, meaning this line of code is considered offensive: if (!window.fetch) window.fetch = fetch;. This is precisely how a polyfill works! It provides a global API only if it’s missing in the browser.

JavaScript has four different module systems: AMD, Common.js, UMD, and standard ES6 modules. In the absence of a standard module system, the community built several of it own, which is great! For library authors, however, this is a disaster. Each of the competing module formats must be supported and tested with each new release. This is complexity that we are not interested in supporting for a polyfill. It is certainly reasonable to expect a library to support modules, but this isn’t a library.

If it were as simple as adding the standardized export default fetch; to the end of the file, we would absolutely provide that. But standard modules aren’t well supported yet, and we have no interest in supporting the legacy module loaders.

Summary

When receiving these requests, we point people to supported Node.js versions of fetch or try to provide debugging tips related to the user’s browser tools. We won’t be providing direct support for either Node.js or modules.

This polyfill provides excellent, well-tested, cross-browser fetch support that’s used in production. When we open source something like this, that’s the level of commitment we’re making. We just can’t make that commitment to tools we don’t use in production ourselves.

Hopefully this long-winded answer provides context around how we’re thinking about these issues.

<3

[mislav]

When we open source something like this, that’s the level of commitment we’re making. We just can’t make that commitment to tools we don’t use in production ourselves.

Excellent, @dgraham. Thanks 🙇

[kirill-konshin]

@dgraham thanks for the perfect explanation. This should be a well-advertised blog article posted everywhere 👍

[niftylettuce]

why isn't it clearly documented to npm install --save node-fetch instead of using this in the Readme? it should be made clear that this package at github/fetch and the whatwg-fetch package does not work out of the box with node runtime.

whatwg-fetch/fetch.js:4
  if (self.fetch) {
      ^

ReferenceError: self is not defined

Perhaps you should instead say in the Readme.md of this repo "For use in Node.js projects, use the node-fetch package, as self is not defined error would otherwise be thrown... or something?

[xgqfrms-GitHub]

node-fetch

https://www.npmjs.com/package/node-fetch