Skip to content

security: branch policy and availability clarifications. #6940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
htuch merged 6 commits into from
May 23, 2019
Merged

security: branch policy and availability clarifications. #6940

Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6a43d03
security: branch policy and availability clarifications.
htuch May 14, 2019
3f82d82
Typo.
htuch May 14, 2019
83ddbac
Merge remote-tracking branch 'upstream/master' into sec-policy-master
htuch May 22, 2019
342aa3a
Clarify availability some more.
htuch May 22, 2019
1bd7c09
wip
htuch May 23, 2019
977b30b
Rewording suggested by Piotr.
htuch May 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions SECURITY_RELEASE_PROCESS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,42 @@ score >= 4; see below). If the fix relies on another upstream project's disclosu
will adjust the process as well. We will work with the upstream project to fit their timeline and
best protect our users.

### Released versions and master branch

If the vulnerability affects the last point release version, e.g. 1.10, then the full security
release process described in this document will be activated. A security point release will be
created for 1.10, e.g. 1.10.1, together with a fix to master if necessary. Older point releases,
e.g. 1.9, are not supported by the Envoy project and will not have any security release created.

If a security vulnerability affects only these older versions but not master or the last supported
point release, the Envoy security team will share this information with the private distributor
list, following the standard embargo process, but not create a security release. After the embargo
expires, the vulnerability will be described as a GitHub issue. A CVE will be filed if warranted by
severity.

If a vulnerability does not affect any point release but only master, additional caveats apply:

* If the issue is detected and a fix is available within 5 days of the introduction of the
vulnerability, the fix will be publicly reviewed and landed on master. A courtesy e-mail will be
sent to envoy-users@googlegroups.com, envoy-dev@googlegroups.com and
cncf-envoy-distributors-announce@lists.cncf.io if the severity is medium or greater.
* If the vulnerability has been in existence for more than 5 days, we will activate the security
release process for any medium or higher vulnerabilities. Low severity vulnerabilities will still
be merged onto master as soon as a fix is available.

We advise distributors and operators working from the master branch to allow at least 3 days soak
time after cutting a binary release before distribution or rollout, to allow time for our fuzzers to
detect issues during their execution on ClusterFuzz.

### Confidentiality, integrity and availability

We consider vulnerabilities leading to the compromise of data confidentiality or integrity to be our
highest priority concerns. Availability, in particular in areas relating to DoS and resource
exhaustion, is a serious security concern that we are making a best effort attempt to address in
Envoy today. We will not activate the security release process for vulnerabilities that only affect
availability until we are confident that Envoy is production hardened with respect to availability.

@mattklein123 mattklein123 May 14, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really sure how we are ever going to be confident in this, and this is also very dependent on proper configuration and deployment details. I might just drop or slightly loosen this statement but up to you.

@htuch htuch May 14, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a good example is something like Slowloris. Would we go through the CVE and security release process for a report of an attack like this? Do operators placing Envoy in edge ingress configurations expect guarantees around this and released versions?

My intuition is we need to incorporate availability into our threat model given edge use of Envoy, but at the same time, if we did this today, we would be doing half-a-dozen or so security releases a quarter, which is unsustainable to both the Envoy security team and our distributors; we'd be on a continuous CVE treadmill.

@mattklein123 mattklein123 May 14, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's true, my point though is that everyone has a different definition of what "availability" means, and I don't know there will ever be consensus here. Today, I think in general we have all the gross knobs in place to prevent most basic attacks (if they are properly configured). We can spend a ton of time doing better and better, and likely better than most other publicly available solutions, but when are we done? What's the bar? That's my main point here.

@htuch htuch May 14, 2019 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's another concrete example. Let's say we hear that sending a header "foobar: baz" from a client causes Envoy to segfault. This is ordinarily a functional bug, but when Envoy is placed at the edge of a network for ingress purposes, it comes a major availability one, as the network ingress becomes trivially DoS-able. What's the correct response from a security perspective?

@mattklein123 mattklein123 May 14, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the correct response from a security perspective?

We invoke the security procedure IMO.

@htuch htuch May 15, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to agenda for 5/21/2019 meeting, let's continue the discussion there.

@PiotrSikora PiotrSikora May 15, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see how a data plane "packet of death" or trivial resource exhaustion leading to OOM could be treated with anything else than the highest priority, and stating that such types of attacks won't even trigger the security release process cannot possibly make our users or distributors happy.

Yes, this could result in a non-zero amount of point releases, but those are growing pains, unfortunately, and the situation will get better over time.

I do agree that we need to be somehow reasonable, and if the attack requires some less popular extension to be configured, then perhaps we need to treat it with less urgency than attacks on HTTP or TCP proxy, but the boundary should be defined somewhere, perhaps by defining different tiers of support for various extensions?

We could also state that a control plane "packet of death" doesn't trigger the security release process, since the control plane should be trusted, but "packet of death" from either client and/or backend (including health-check responses) should be a fair game, IMHO.

As for the attacks reported publicly on GitHub, and not to envoy-security@, we could potentially treat them as "actively exploited issues", and trigger the security release process without embargo / waiting period.

@mattklein123 mattklein123 May 15, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to what @PiotrSikora said. This is my exact thinking also.

@htuch htuch May 16, 2019 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fundamentally agree that availability is an important consideration, but let me play the role of devil's advocate in order to allow us to better hash out these issues and try and surface some of the underlying tensions that might make this less clear cut.

I think it's reasonable to treat attacks that result in process compromise or leak of data as distinct and far more serious than availability attacks. The whole point of priorities is.. you prioritize. I.e. a CVSS score of "critical" has a higher priority than a "medium" vulnerability. So, clearly something like Heartbleed would be higher priority than an availability attack, and it's not helpful to claim that all security issues are the highest priority.

At the end of the day, security is a practice that must consider economics. There is a finite amount of bandwidth we have in each quarter for security releases in the Envoy projects and for distributors. Doing a security release results in high cost to organizations performing this, I've seen this first hand. Also, you don't want to be the child who cried Wolf, or your signal disappears. So, given that you have the bandwidth for N security releases in a quarter (5 at most perhaps?), the question is do you want to burn this on lower priority or higher priority issues?

I think we have to be very careful about how we express the policy in terms of predicating it on configuration. My main concern is that Envoy in its "default configuration" state is pretty terrible for resource exhaustion attacks. Should this be improved? Maybe, but we have to balance concerns with backwards compatibility of configuration and Envoy having a performant configuration with minimal surprising behavior out-of-the-box.

Leaving aside extensions, which are admittedly something we can treat with different tiered support (which will need to be explicit), we'll need to be clear on what configuration options need to be set to make this work.

If a resource or QoD uses only core features (no extensions), but requires some configuration that seems uncommon, do we reduce priority? I think the answer is yes, but then how do we ascertain objectively what is and isn't common? The Envoy community is large and we have no objective data to make these decisions on.

Also, I'm curious if the following question can be answered today: I am an Envoy user with X MB of RAM. What configuration options must I set in Envoy to guarantee the absence of resource DoS?

If we can't answer ^^ in a precise way, I think we have a trivial resource exhaustion issue inherent in Envoy, so creating a policy that ignores this reality is not practical at this point.

I'm hoping that we can get to this kind of detail in our policy, this is the objective of this PR.

@htuch htuch May 22, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Availability wording updated based on the community call and various conversations I've had 1:1 with folks. I'm hoping that what we have now addresses @PiotrSikora and @alyssawilk (and probably other) concerns, while balancing with the reality of Envoy's availability status quo. PTAL.

Operators deploying Envoy at the edge should keep this caveat in mind.

### Fix Team Organization

These steps should be completed within the first 24 hours of disclosure.
Expand Down