fix: should not destroy streams #612

dead-horse · 2015-12-02T05:43:48Z

use black-hole-stream to make sure stream's data has been read.

since user may set this.body to a http IncomingMessage, if we destroy this stream, it will destroy the http socket that make keep-alive failed, and may affect other http requests reusing this socket.
I think maybe we shouldn't destroy streams, only need to insure streams' data will be read and don't leak.

JacksonTian · 2015-12-02T05:57:25Z

package.json

  },
  "engines": {
-    "node": ">= 0.12.0",
-    "iojs": ">= 1.0.0"
+    "node": ">= 0.12.0"


can we drop the support of node < 4 in koa@1 ? /cc @tj @jonathanong

oh, master is v1.

this should be its own pull request

dead-horse · 2015-12-03T02:25:57Z

ping @jonathanong @fengmk2

fengmk2 · 2015-12-04T09:09:03Z

LGTM

juliangruber · 2015-12-04T09:19:12Z

.travis.yml

  - "4"
+  - "5"


this should be its own pull request

dead-horse · 2015-12-04T10:23:54Z

@juliangruber all fixed

juliangruber · 2015-12-04T12:10:55Z

test/application.js

+      var app = koa();
+      let stream1 = fs.createReadStream(__filename);
+      let stream2 = fs.createReadStream(__filename);
+      stream1.once('end', done);


i don't understand why this one would end as well? it's never read from

we'll destroy(consume) all stream's that set to this.body here https://github.com/koajs/koa/pull/612/files#diff-d86a59ede7d999db4b7bc43cb25a1c11R166

juliangruber · 2015-12-04T12:15:05Z

hmmm, this seems more like an edge case to me. are there other cases that would benefit from this change?

dead-horse · 2015-12-04T16:43:04Z

The most serious problem now is http keepalive sockets will be destroyed, and we must fix this issue. Not sure any other side effect if we destroy the streams.

juliangruber · 2015-12-04T16:58:58Z

are you sure destroying a http stream will affect other requests on the same socket? sounds more like a node bug to me

dead-horse · 2015-12-04T17:09:48Z

If you destroy an IncommingMessage, it will destroy the socket behind: https://github.com/nodejs/node/blob/master/lib/_http_incoming.js#L92

That will affect other IncommingMessage's that reusing the same socket and will cause socket hang up error. This is reported in ali-sdk/ali-oss#40 (sorry this issue is in chinese).

dead-horse · 2015-12-05T04:57:40Z

any suggestions? I'll merge this PR if no response in the next day. :)

juliangruber · 2015-12-05T16:05:44Z

Hey man, no need to put a deadline on it. If you need it urgently use a fork or find a userland solution.

dead-horse · 2015-12-06T09:45:50Z

I think this is just a bug fix. so if anyone doubt this patch, I'll leave it here and fix in our own framework for now. :)

juliangruber · 2015-12-06T10:32:57Z

this patch fails if the body stream is endless, like for example:

this.body = fs.createReadStream('/dev/urandom');

or more real-worldy

this.body = serverSentEvents(createUpdateStream());

in the cases above it's required to destroy the stream.

juliangruber · 2015-12-06T10:33:23Z

and trying to read it all out will never finish

dead-horse · 2015-12-06T11:02:43Z

although i don't think pipe an endless stream to response meaningful. but maybe there is a solution has less impact. detect the stream in destroy and don't destroy IncommingMessage(a breaking change in destroy).

use black-hole-stream to make sure stream's data has been read

dead-horse · 2015-12-09T17:17:45Z

@juliangruber how about this?

juliangruber · 2015-12-09T19:20:43Z

too edge-casy. koa should not need to worry about this.

Just to be sure, this would work for you, right?

httpStream.destroy = blackHole.bind(null, httpStream);
this.body = httpStream;

dead-horse · 2015-12-10T03:08:57Z

it is a common usage that use koa as a http proxy, and anybody who use this.body = yield request('http://foo.bar/'); //keep-alive will hit this "edge case", and it is really hard to figure out why. so I think we should fix it in framework level, because it is a bug in koa, and we can't let everyone to use httpStream.destroy = blackHole.bind(null, httpStream); hack.

jonathanong · 2015-12-21T06:16:23Z

maybe a better solution is to just wrap the http stream in another stream so you never set the http stream as the actual .body=.

const PassThrough = require('stream').PassThrough

app.use(function * (next) {
  this.body = httpStream.on('error', this.onerror).pipe(PassThrough())
})

would need docs on this though... super edge case

juliangruber · 2015-12-22T09:38:32Z

would need docs on this though... super edge case

what about documenting that a body stream can be .destroyed and showing how to guard against that

tejasmanohar · 2016-01-18T06:04:45Z

what about documenting that a body stream can be .destroyed and showing how to guard against that
Show all checks

+1 @juliangruber

jonathanong · 2016-02-25T22:46:53Z

lib/response.js

+      onFinish(this.res, function(){
+        // don't destroy http IncomingMessage, keep `keep-alive` conncetion alive.
+        if (val instanceof http.IncomingMessage) {
+          if (val.readable) val.pipe(new BlackHoleStream());


why do you need black hole stream? just val.resume() to dump it

jonathanong · 2016-02-25T22:47:43Z

anyways prefer going the documentation route. will close this when i or someone else adds documentation

dead-horse · 2016-02-26T01:58:51Z

It's ok to add some warning in documentation.

catamphetamine · 2017-03-06T04:21:00Z

Didn't understand a thing from the documentation on what's happening here.
It's just like "warning, never set a stream as ctx.body".

jiajianrong · 2018-08-01T12:37:21Z

@catamphetamine, they were warning not to set a http-request-alike-stream which has keep-alive enabled. Because the framework will close the stream by default, which will break keep-alive feature.
Keep-alive means to reuse tcp socket connection for two or more http connections.

Generally, it's OK with ctx.body = fs-read-stream.
Just be careful with ctx.body = request('host:port/path').

rhinoceros · 2018-11-12T03:07:11Z

@dead-horse
cnpmjs.org中目前用的koa的版本是"koa": "^1.2.0", 如果使用oss作为nfs，在从oss上get的时候会有一堆的sockets hang up。

有什么好的办法能避免这类错误吗？

cnpm/cnpmjs.org#1401

谢谢

zhangwilling · 2023-08-31T07:49:10Z

@dead-horse 马哥好，昨天夜里见到这个 issue 忘了跟你打招呼了，白天来踩一下。

----- 骄傲的用中文交流

dead-horse added bug 1.x labels Dec 2, 2015

JacksonTian reviewed Dec 2, 2015
View reviewed changes

dead-horse force-pushed the fix-destroy-everty-stream branch 2 times, most recently from 0da33f7 to 42382b4 Compare December 2, 2015 06:37

dead-horse mentioned this pull request Dec 2, 2015

getStream dont use keepalive feature ali-sdk/ali-oss#40

Closed

dead-horse force-pushed the fix-destroy-everty-stream branch from 42382b4 to 65f13fb Compare December 2, 2015 06:45

juliangruber reviewed Dec 4, 2015
View reviewed changes

.travis.yml

- "4"

- "5"

Copy link

Contributor

juliangruber Dec 4, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be its own pull request

dead-horse force-pushed the fix-destroy-everty-stream branch from 65f13fb to bbf850b Compare December 4, 2015 10:23

dead-horse force-pushed the fix-destroy-everty-stream branch from bbf850b to 078cac0 Compare December 4, 2015 10:42

juliangruber reviewed Dec 4, 2015
View reviewed changes

dead-horse self-assigned this Dec 5, 2015

dead-horse mentioned this pull request Dec 6, 2015

do not destroy IncommingMessage stream-utils/destroy#2

Closed

dead-horse force-pushed the fix-destroy-everty-stream branch from 078cac0 to a949d79 Compare December 9, 2015 17:10

fix: should not destroy streams

d9c1ea3

use black-hole-stream to make sure stream's data has been read

dead-horse force-pushed the fix-destroy-everty-stream branch from a949d79 to d9c1ea3 Compare December 9, 2015 17:11

tejasmanohar mentioned this pull request Feb 25, 2016

Koa 2.0.0 #533

Closed

jonathanong reviewed Feb 25, 2016
View reviewed changes

dead-horse closed this Feb 26, 2016

dead-horse deleted the fix-destroy-everty-stream branch February 26, 2016 01:58

dead-horse added documentation and removed bug labels Feb 26, 2016

jonathanong added a commit that referenced this pull request Feb 29, 2016

docs: note stream error handling and destruction. ref: #612

97c34d1

jonathanong added a commit that referenced this pull request Mar 12, 2016

docs: note stream error handling and destruction. ref: #612

d768ed8

JasonBoy mentioned this pull request Mar 27, 2019

接口代理数据，数据被截断会是什么原因？ JasonBoy/koa-web-kit#31

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: should not destroy streams #612

fix: should not destroy streams #612

dead-horse commented Dec 2, 2015

JacksonTian Dec 2, 2015

dead-horse Dec 2, 2015

JacksonTian Dec 2, 2015

juliangruber Dec 4, 2015

dead-horse commented Dec 3, 2015

fengmk2 commented Dec 4, 2015

juliangruber Dec 4, 2015

dead-horse commented Dec 4, 2015

juliangruber Dec 4, 2015

dead-horse Dec 4, 2015

juliangruber commented Dec 4, 2015

dead-horse commented Dec 4, 2015

juliangruber commented Dec 4, 2015

dead-horse commented Dec 4, 2015

dead-horse commented Dec 5, 2015

juliangruber commented Dec 5, 2015

dead-horse commented Dec 6, 2015

juliangruber commented Dec 6, 2015

juliangruber commented Dec 6, 2015

dead-horse commented Dec 6, 2015

dead-horse commented Dec 9, 2015

juliangruber commented Dec 9, 2015

dead-horse commented Dec 10, 2015

jonathanong commented Dec 21, 2015

juliangruber commented Dec 22, 2015

tejasmanohar commented Jan 18, 2016

jonathanong Feb 25, 2016

jonathanong commented Feb 25, 2016

dead-horse commented Feb 26, 2016

catamphetamine commented Mar 6, 2017

jiajianrong commented Aug 1, 2018

rhinoceros commented Nov 12, 2018

zhangwilling commented Aug 31, 2023

fix: should not destroy streams #612

fix: should not destroy streams #612

Conversation

dead-horse commented Dec 2, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dead-horse commented Dec 3, 2015

fengmk2 commented Dec 4, 2015

Choose a reason for hiding this comment

dead-horse commented Dec 4, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juliangruber commented Dec 4, 2015

dead-horse commented Dec 4, 2015

juliangruber commented Dec 4, 2015

dead-horse commented Dec 4, 2015

dead-horse commented Dec 5, 2015

juliangruber commented Dec 5, 2015

dead-horse commented Dec 6, 2015

juliangruber commented Dec 6, 2015

juliangruber commented Dec 6, 2015

dead-horse commented Dec 6, 2015

dead-horse commented Dec 9, 2015

juliangruber commented Dec 9, 2015

dead-horse commented Dec 10, 2015

jonathanong commented Dec 21, 2015

juliangruber commented Dec 22, 2015

tejasmanohar commented Jan 18, 2016

Choose a reason for hiding this comment

jonathanong commented Feb 25, 2016

dead-horse commented Feb 26, 2016

catamphetamine commented Mar 6, 2017

jiajianrong commented Aug 1, 2018

rhinoceros commented Nov 12, 2018

zhangwilling commented Aug 31, 2023