Regular schedule for Security releases?

[mhdawson]

I think @sam-github suggested this in an issue, but I think it would be good to have a specific issue to discuss.

I'm +1. It's understood that we will still have to have some emergency security releases based on disclosed vulnerabilities or OpenSSL updates but it seems like planning for a regular cadence would be good both for the project and the consumers or Node.js.

@nodejs/security-wg

[Trott]

Seems sensible to me. Two thoughts:

• Maybe we can automate that addition and removal of the banner announcing security release availability to the website.

• Maybe we should try to align our schedule with some de facto industry standard like Patch Tuesday.

[mcollina]

What would be a good schedule? How about monthly if there is anything to release?

[mhdawson]

In terms of schedule, I think it depends on how much we have to get in. I think we want the smallest number (assuming less release is less overall work) that gets fixes out in a reasonable timeframe. I was thinking maybe every 2 months?

[sam-github]

Sec releases take a reasonable amount of work, and block CI for days, monthly seems too fast to me. We seem to be doing about 4 a year, maybe keeping to that pace, but with a more predictable cadence, would work.

[Trott]

Sec releases take a reasonable amount of work, and block CI for days

Fixing (or at least mitigating) both of those problems would seem to be decent strategic initiatives, if we have someone on TSC willing to do the work and championing them. (I might be able to take on the CI one at some point, but right now I'm awaiting a decision on my request to re-join TSC.)

[ljharb]

I’m a bit confused; when would you have a security fix, but not release it more-or-less immediately, or at the soonest opportunity?

[sam-github]

Lots of security issues are things that have existed since the early days of node, can't really be prevented (like a DoS), but the sensitivity to them can be adjusted. Take the last round of security fixes. IMO, they could have been done mostly (entirely?) as open PRs in node, not in private. Sure, people could be watching our PRs, and see the DoS opportunity and exploit... but none of them allowed node servers to be smashed, and the fix just adjusts a knob, making the DoS linerally less effective. Anyone who has heard of slowloris and wanted to play around with node could already have found them, and with a large enough bot farm, been exploiting them. Another category of recent vulns that required web servers to use specific coding patterns for the issue to be exploited. Its hard to argue about how common the patterns are, as they do exist, but most most node servers aren't affected.

A heartbleed level attack would be fixed ASAP. But those aren't the common security issues we release in security updates, its just bugs/features, that have some credible possibility of security implications, to someone, somewhere, so we have to advertise them via CVE, just in case.

[ljharb]

Sure, it makes sense that not everything is a "stop the world, release now" situation - but presumably a normal release happens pretty regularly whenever there's anything landed and unreleased, so any non-urgent security fixes would just get naturally shipped with those, no?

[sam-github]

We don't mix security and non-security updates in a single release. Not as far as I know. I'm not sure of the history of why this is, but I assume its to make updates as safe as possible for people who don't intend to make updates to their deployed-and-working apps unless they have to. Also, security updates get released simultaneously across all release lines, so that old release lines aren't "vulnerable" while new lines are protected. This is unlike normal features and bugs, which bake a while in current before they are considered stable enough to be backported to LTS.

@MylesBorins @rvagg perhaps can comment on this.

[mhdawson]

You are correct that we specifically ensure that in security releases the only changes are for the security-related fixes. And we do it that way as you said to make updates as safe as possible for people who don't intend to make updates to their deployed-and-working apps unless they have to.

[MylesBorins]

I have very mixed feelings on this

If we were to do it I think a cadence that would make sense would be quarterly. With that being said the timing that we have might be a bit rigid... we don't know when very serious vulnerabilities come in... and while we do end up having a backlog of "non critical" sercurity bugs I'm unsure that we want to be giving up 1 in 3 LTS releases to security releases.

Alternatively, perhaps we can focus on improving our ability to test and prepare security releases with a smaller sub-set of CI... that seems to be the bigger issue than cadence to me. Please correct me if I am wrong

[mcollina]

We have a concept of a "missed target" in HackerOne. I have not found this defined anywhere in our policies, but maybe I missed it. Have we defined a max time between a vulnerability is sent and a fix is received?

[lirantal]

It's a HackerOne setting that applies.

Defaults (and maximum) that apply to projects are:

• Time to first response: 5 days
• Time to triage: 10 days
• Time to resolution: 40 days

[mcollina]

Basically with a quarterly schedule we are almost 100% certain to go over that maximum time to resolution.

[lirantal]

That's true but sometimes you can't rush a fix.. though? we might be able to opt-in for an extended period but I tried that before for the ecosystem program and seems like this is a max they have site-wide but perhaps things have changed.

Also, more info on the response targets at https://docs.hackerone.com/programs/response-target-metrics.html

[sam-github]

The problem I see with the ad-hoc release schedule is it take a fair amount of work, and we can't plan ahead very well to find available people. Also, consumers of node can't plan very far ahead to receive the update. Those aren't very large problems, though, and arguably aren't problems at all. We don't have releases triggered by a reported vuln very often, AFAICT. Even the javascript url one, though reported externally, had its release delayed so we could batch it up with the HTTP DoS work that was in progress. This makes it seem to me that the issues are not so urgent that we are working to to get them turned around quickly, and a regular cadence might work.

On the other hand, I agree that if preping the releases was less work and less disruptive, doing them on-demand would be easier, too.

This may be anathema to some, but one way to do this would be to do more of the vulnerability fixing in public github. I know that we have a reflex to keep anything that could have any security implication private, but for example, nodejs/node#8081 was already reported (and fixed) publically in 8.x, but we decided to issue a CVE and backport the fix to 6.x. This got it classified as a vulnerability, and the fix secluded into nodejs-private. IMO it coulc have been fixed and released faster through the regular LTS process.

We have a fairly rapid and well-oiled cadence for non-security release maintenance, if we could funnel fixes that make node more secure into that process things might be smoother.

PR review in particular would be improved, and lack of feedback from users was an issue in the last set of security releases.

[mcollina]

I think we should:

a. produce some good guidelines on what we consider a security vuln and what does not. As an example, if the vulnerability is already public we could speed up a fix by developing it in the open. As an example, we do not have a documented threat model.
b. see if we could run a subset of CI for security fixes, that would help significantly in developing those fixes.
c. decide if we could bundle low-level security vulnerabilities (low to medium?) with the normal LTS releases, litterally pulling them in from node-private.

[sam-github]

I'd like to kick this again, especially after the recent sec release that was motivated by the 6.x deadline (or so it appeared to me). Regular deadlines might get low severity sec issues a bit more priority.

Following on to @mcollina 's comments:

a. If publically known vulns can be fixed in the open I too think they'd get fixed and released faster.
c. We try to release fixes to vulns (even well-known ones, or low severity ones) in all release lines simultaneously, which isn't how LTS does things. If below a certain threshold we could fix things in public, and let them follow the normal course of backporting, it would be easier. The window in which users are "vulnerable" would be longer, but, its not clear to me how many people drop everything, and upgrade all their 6.x machines with the latest sec release when we announce one. Anybody using 6.x in production is clearly averse to updating.

[sam-github]

And we know have a concrete example of a CVE being addressed as a public PR: nodejs/node#26537, so the above two comments are no longer hypothetical, but actual.

/to @nodejs/tsc @nodejs/security @nodejs/security-wg

[mhdawson]

We could also look at past security releases to understand the ratio between what we might have been able to do in public versus where that won't work. Past does not predict the future but might provide some insight into how much benefit we would get from the suggestion.

[sam-github]

In my very personal opinion, the June 2018 release had some genuninely remote-exploitable vulns, most of the others require carefully written user-code ot obscure scenarios (malicious servers, running debuggers on untrusted networks, timing attacks from the same machine, etc.). DoS is always possible, with sufficient attacker resource, mostly in the DoS fixes we provide user-control to limit size or time for various protocol elements, it looks like most of the DoS could have been fixed in the open.

# Security Release Summary for 2018 and 2019 so far

## https://nodejs.org/en/blog/vulnerability/march-2018-security-releases/

- Node.js Inspector DNS rebinding vulnerability

Requires the inspector to be running, and someone on the inspector's network to
connect to a remote malicious website.

- 'path' module regular expression denial of service

Requires user's code to take malicious data and pass it to the path module.

- Spaces in HTTP Content-Length header values are ignored

No known way to exploit.


## https://nodejs.org/en/blog/vulnerability/june-2018-security-releases/

- Denial of Service Vulnerability in HTTP/2

Remotely exploitable.

- Denial of Service, nghttp2 dependency

Remotely exploitable.

- Denial of Service Vulnerability in TLS

Remotely exploitable.

- Memory exhaustion DoS on v9.x

Remotely exploitable.

- Calls to Buffer.fill() and/or Buffer.alloc() may hang

Requires user's code to deliberately make nonsensical Buffer calls, or to pass
malicious user-provided lenghts to Buffer.


## https://nodejs.org/en/blog/vulnerability/august-2018-security-releases/

- OpenSSL: Client DoS due to large DH parameter

Requires a malicious server wanting to cause a client's CPU to work harder than
normal.

- OpenSSL: ECDSA key extraction via local side-channel

Local-only, and so theoretical OpenSSL didn't give it a CVE.

- Unintentional exposure of uninitialized memory

Requires user coding errors, though of a class that has been demoed in the wild.

- Out of bounds (OOB) write

ucs-2 encoding is abusable to do OOB writes, and possibly crashes.


## https://nodejs.org/en/blog/vulnerability/november-2018-security-releases/

- Debugger port 5858 listens on any interface by default

Requires debugger to be run on untrused network.

- Denial of Service with large HTTP headers

Remotely exploitable with care.

- "Slowloris" HTTP Denial of Service

Decreases the time allowed to send HTTP headers, but it was already limited.

- Hostname spoofing in URL parser for javascript protocol

Requires user-code to be using URL parser to make security decisions.

- HTTP request splitting

Requires user-code to put unsanitized user-provided unicode data in an HTTP
request's path option.

- OpenSSL Timing vulnerability in ECDSA signature generation

Low enough severity OpenSSL didn't do a security release for it.

- OpenSSL Timing vulnerability in DSA signature generation

Low enough severity OpenSSL didn't do a security release for it.

- OpenSSL Microarchitecture timing vulnerability in ECC scalar multiplication 

Low enough severity OpenSSL didn't do a security release for it.


## https://nodejs.org/en/blog/release/v6.15.1/

"Slowloris" HTTP Denial of Service got insufficient review, and caused a
regression that needed patching.


## https://nodejs.org/en/blog/vulnerability/february-2019-security-releases/

- Node.js: Slowloris HTTP Denial of Service with keep-alive

"Slowloris" HTTP Denial of Service wasn't working with keep-alive, and needed
patching.

- Node.js: Denial of Service with keep-alive HTTP connections

DoS protection, reported and worked on publically: https://github.com/nodejs/node/pull/2534

- OpenSSL: 0-byte record padding oracle

Requires Node C++ to use the OpenSSL API in very specific ways under very
specific conditions, and we were unable to convince ourselves it was impossible
(or that it was possible).

[mhdawson]

I think we should have this be discussed by the security wg and the Release wg to see if we can get any consensus there. Adding agenda tags

[sam-github]

I don't want to displace any scheduled releases with a planned security release (unplanned high severity releases may disrupt scheduled releases, but that's just what happens).

Last sec release was Feb, 2019, how about another in June? Sometime between June 4th and 21st looks free: https://github.com/nodejs/Release/wiki. How about we plan for a release then, and another 4 months later, and see if knowing before hand when we'll need help from releasers to do a release, and planning the time so it doesn't bump any other scheduled releases is helpful, and whether having a deadline helps get movement on any of the slowly being worked on open issues.

@nodejs/releasers thoughts on what would be a non-disruptive time?

[sam-github]

https://github.com/nodejs/Release/wiki#q2-2019-dubnium-10x-release-schedule

We have a June 25th date available (if we want) for doing a security release. I'll open an issue in the private repo to see if there is anything we want to release then.

[mhdawson]

@sam-github would it make sense to close this issue now?