http: use `autoDestroy: true` in incoming message #33035

dnlup · 2020-04-24T07:08:04Z

Enable the default autoDestroy: true option in IncomingMessage.

Refactor _http_client and _http_server to remove any manual destroying/closing of IncomingMessage.
Refactor IncomingMessage destroy method to use the standard implementation of the stream module and move the abort logic
inside of it.

Ref:

#30625

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
commit message follows commit guidelines

ronag · 2020-04-30T20:44:36Z

IncomingMessage overrides destroy() which kind of blocks any effort in this regards. That needs to IMO be fixed first.

dnlup · 2020-05-01T09:46:48Z

IncomingMessage overrides destroy() which kind of blocks any effort in this regards. That needs to IMO be fixed first.

Thanks, @ronag, for the suggestion. I'll look into it. There is an issue with destroy for sure. When using keep-alive, we don't want to destroy the socket, and that's what's happening by just setting the option to true.

dnlup · 2020-05-06T05:32:38Z

Sorry if this is taking a long time, but I am trying to figure out what's wrong 🙏

Removing the override of destroy() resolves the EECONRESET errors, fewer tests are failing, but request is emitting close twice, and some connections seem to hang.

dnlup · 2020-05-20T07:13:55Z

The changes made here are overriding the ones you made recently, @ronag . I don't know if this approach is the best.

PS: just rebased against master on the last forced push.

BridgeAR · 2020-05-23T23:36:24Z

@ronag PTAL

ronag

Please also try to remove all cases of manually managing destroyed and emitting 'close' on IncomingMessage. Having some kind of hybrid solution will make things hard to maintain IMO.

lib/_http_incoming.js

dnlup · 2020-05-28T07:36:07Z

Please also try to remove all cases of manually managing destroyed and emitting 'close' on IncomingMessage. Having some kind of hybrid solution will make things hard to maintain IMO.

You mean like in here? I agree, that is also the reason why the callback is not called in all cases in _destroy. I also agree that it is ugly, sorry. I wasn't sure I could change the _http_server.js parts.

Thanks for the pointers.

dnlup · 2020-06-03T06:50:33Z

Do we need to keep this behavior? If we do, I think it would be better to implement the destroy logic directly in IncomingMessage and not reusing the general lib/internal/streams/destroy.js implementation as I did. Overriding that implementation causes a lot of tests to fail because they are expecting an unhandled error. Also, do you have anything against adding an abort method on IncomingMessage?

ronag · 2020-06-03T07:11:26Z

Do we need to keep this behavior?

Yes, just call destroy and move the logic there?

If we do, I think it would be better to implement the destroy logic directly in IncomingMessage and not reusing the general lib/internal/streams/destroy.js implementation as I did.

I disagree. We should implement it as a regular destroy.

Overriding that implementation causes a lot of tests to fail because they are expecting an unhandled error.

Can still be an unhandeld error?

Also, do you have anything against adding an abort method on IncomingMessage?

Yes, destroy should be abort. We just deprecated abort on ClientRequest in favor of destroy.

dnlup · 2020-06-03T07:44:54Z

Yes, just call destroy and move the logic there?

Ok.

I disagree. We should implement it as a regular destroy.

I agree. The only difference I have seen with a regular destroy is that IncomingMessage emits an error only if there are listeners attached, at least when aborting.

Yes, destroy should be abort. We just deprecated abort on ClientRequest in favor of destroy.

Got it.

ronag · 2020-06-03T07:49:32Z

The only difference I have seen with a regular destroy is that IncomingMessage emits an error only if there are listeners attached, at least when aborting.

I can help with sorting this out. If you have any test that depends on this behavior just comment them out and we can take a look together when that's the only thing remaining.

lib/_http_incoming.js

lib/_http_client.js

test/parallel/test-http-aborted.js

lib/_http_incoming.js

Add to the history table that the `destroyed` value returns `true` after the incoming data is consumed. Refs: #36617 Refs: #33035 PR-URL: #36641 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Robert Nagy <[email protected]>

Since 55e83cb has changed the ordering of the close event, add a test case. IncomingMessage will emit close before the response is sent in case the server is consuming data from it. Refs: #33035 (comment) PR-URL: #36645 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Matteo Collina <[email protected]>

This fixes a major memory leak I encountered usign this on Node v16. Doing a binary search of node versions, the problem appears to originate in v15.5.0 as a result of nodejs/node#33035, however later changes have quietly completely removed the 'aborted' event that `http-proxy` relies on, and later added a deprecation note about it (which seems to actually be incorrect). Despite what the notes about [DEP0156](https://nodejs.org/dist/latest-v16.x/docs/api/deprecations.html#DEP0156) say, the only way I could get this module working reliably again was to replace `req.on('aborted')` with instead checking `res.on('close')` and looking at `res.writeableFinished`.

PR-URL: #42521 Fixes: #38924 Refs: #33035 Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Beth Griggs <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: nodejs#42521 Fixes: nodejs#38924 Refs: nodejs#33035 Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Beth Griggs <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: #42521 Fixes: #38924 Refs: #33035 Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Beth Griggs <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: nodejs#42521 Fixes: nodejs#38924 Refs: nodejs#33035 Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Beth Griggs <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: #42521 Fixes: #38924 Refs: #33035 Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Beth Griggs <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

PR-URL: nodejs/node#42521 Fixes: nodejs/node#38924 Refs: nodejs/node#33035 Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Beth Griggs <[email protected]> Reviewed-By: Antoine du Hamel <[email protected]>

This fixes a major memory leak I encountered usign this on Node v16. Doing a binary search of node versions, the problem appears to originate in v15.5.0 as a result of nodejs/node#33035, however later changes have quietly completely removed the 'aborted' event that `http-proxy` relies on, and later added a deprecation note about it (which seems to actually be incorrect). Despite what the notes about [DEP0156](https://nodejs.org/dist/latest-v16.x/docs/api/deprecations.html#DEP0156) say, the only way I could get this module working reliably again was to replace `req.on('aborted')` with instead checking `res.on('close')` and looking at `res.writeableFinished`.

This fixes a major memory leak I encountered usign this on Node v16. Doing a binary search of node versions, the problem appears to originate in v15.5.0 as a result of nodejs/node#33035, however later changes have quietly completely removed the 'aborted' event that `http-proxy` relies on, and later added a deprecation note about it (which seems to actually be incorrect). Despite what the notes about [DEP0156](https://nodejs.org/dist/latest-v16.x/docs/api/deprecations.html#DEP0156) say, the only way I could get this module working reliably again was to replace `req.on('aborted')` with instead checking `res.on('close')` and looking at `res.writeableFinished`. The original commit is [here](http-party@058182a) by Jimbly.

This fixes a major memory leak I encountered usign this on Node v16. Doing a binary search of node versions, the problem appears to originate in v15.5.0 as a result of nodejs/node#33035, however later changes have quietly completely removed the 'aborted' event that `http-proxy` relies on, and later added a deprecation note about it (which seems to actually be incorrect). Despite what the notes about [DEP0156](https://nodejs.org/dist/latest-v16.x/docs/api/deprecations.html#DEP0156) say, the only way I could get this module working reliably again was to replace `req.on('aborted')` with instead checking `res.on('close')` and looking at `res.writeableFinished`.

nodejs-github-bot added the http Issues or PRs related to the http subsystem. label Apr 24, 2020

dnlup force-pushed the _http_incoming_autodestroy branch from f4960e4 to 95d8354 Compare April 29, 2020 05:14

dnlup force-pushed the _http_incoming_autodestroy branch from 95d8354 to 28da814 Compare May 4, 2020 08:28

dnlup force-pushed the _http_incoming_autodestroy branch 3 times, most recently from 336478c to 62e8aec Compare May 20, 2020 06:36

dnlup marked this pull request as ready for review May 20, 2020 06:36

dnlup force-pushed the _http_incoming_autodestroy branch from 62e8aec to 26a6e60 Compare May 20, 2020 07:09

BridgeAR requested a review from ronag May 27, 2020 23:56

ronag requested changes May 28, 2020

View reviewed changes

lib/_http_incoming.js Show resolved Hide resolved

BridgeAR force-pushed the master branch 2 times, most recently from 8ae28ff to 2935f72 Compare May 31, 2020 12:18

dnlup force-pushed the _http_incoming_autodestroy branch 4 times, most recently from 872f751 to 3231c42 Compare June 9, 2020 06:23

dnlup commented Jun 9, 2020

View reviewed changes

lib/_http_incoming.js Show resolved Hide resolved

dnlup commented Jun 9, 2020

View reviewed changes

lib/_http_client.js Outdated Show resolved Hide resolved

dnlup commented Jun 9, 2020

View reviewed changes

test/parallel/test-http-aborted.js Outdated Show resolved Hide resolved

dnlup commented Jun 9, 2020

View reviewed changes

lib/_http_incoming.js Show resolved Hide resolved

devinivy mentioned this pull request Jan 23, 2021

Future-proof hapi for node v16, rely on res close rather than req hapijs/hapi#4225

Merged

devinivy mentioned this pull request Feb 2, 2021

lookup: add hapi nodejs/citgm#436

Merged

1 task

Linkgoron mentioned this pull request May 13, 2021

Stream finished does not always work with http incoming message #38657

Open

bizob2828 mentioned this pull request May 19, 2021

Run apollo-server-hapi versioned tests on Node 16 newrelic/newrelic-node-apollo-server-plugin#81

Closed

targos mentioned this pull request Jun 5, 2021

Undocumented breaking change on v16.0.0 #38924

Closed

devinivy mentioned this pull request Jun 23, 2021

Fix req end during response transmission hapijs/hapi#4264

Merged

flotwig mentioned this pull request Nov 9, 2021

feat(deps): update dependency electron to v15 🌟 cypress-io/cypress#18317

Merged

kanongil mentioned this pull request Nov 10, 2021

Missing 'aborted' on IncomingMessage (v16 regression) #40775

Closed

This was referenced Dec 8, 2021

Fix 'aborted' detection on Node v15.5.0+ http-party/node-http-proxy#1559

Open

http.IncomingMessage 'aborted' issues #41117

Closed

josephharrington mentioned this pull request Jan 31, 2022

fix: use close event on response instead of socket restify/node-restify#1892

Merged

aduh95 mentioned this pull request Mar 30, 2022

doc: document breaking change in http.IncomingMessage 'close' event #42521

Merged

Flarna mentioned this pull request Apr 1, 2022

ECONNRESET at TLSWrap.onStreamRead issue impacting Azure extensions zip deploy (related to Node v16?) microsoft/vscode#146577

Closed

nodech mentioned this pull request Jun 8, 2022

server: destroy socket. bcoin-org/bmocha#11

Merged

jmartin4563 mentioned this pull request Feb 16, 2023

fix: Replace request.abort with response.close in HTTP instrumentation newrelic/node-newrelic#1510

Merged

wh0 mentioned this pull request Jan 14, 2024

doc: http event order correction #51464

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http: use `autoDestroy: true` in incoming message #33035

http: use `autoDestroy: true` in incoming message #33035

dnlup commented Apr 24, 2020 •

edited

Loading

ronag commented Apr 30, 2020 •

edited

Loading

dnlup commented May 1, 2020

dnlup commented May 6, 2020

dnlup commented May 20, 2020 •

edited

Loading

BridgeAR commented May 23, 2020

ronag left a comment

dnlup commented May 28, 2020

dnlup commented Jun 3, 2020

ronag commented Jun 3, 2020

dnlup commented Jun 3, 2020

ronag commented Jun 3, 2020

http: use autoDestroy: true in incoming message #33035

http: use autoDestroy: true in incoming message #33035

Conversation

dnlup commented Apr 24, 2020 • edited Loading

Checklist

ronag commented Apr 30, 2020 • edited Loading

dnlup commented May 1, 2020

dnlup commented May 6, 2020

dnlup commented May 20, 2020 • edited Loading

BridgeAR commented May 23, 2020

ronag left a comment

Choose a reason for hiding this comment

dnlup commented May 28, 2020

dnlup commented Jun 3, 2020

ronag commented Jun 3, 2020

dnlup commented Jun 3, 2020

ronag commented Jun 3, 2020

http: use `autoDestroy: true` in incoming message #33035

http: use `autoDestroy: true` in incoming message #33035

dnlup commented Apr 24, 2020 •

edited

Loading

ronag commented Apr 30, 2020 •

edited

Loading

dnlup commented May 20, 2020 •

edited

Loading