-
Notifications
You must be signed in to change notification settings - Fork 507
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
First cut at security audit guidelines #125
Changes from all commits
4a66450
ee8befc
f576e82
95c9efe
050c64b
42fc444
5fb123a
3bfb1e9
14b5b12
eb76909
48c90f9
b7fb720
8527f02
78c23e6
6d22b37
d758788
40e40fa
8b1ffb0
275a5f0
91d8e07
b4b29ed
8e066ef
3887158
875eb40
3d8dbd6
aad11c1
e490eaf
52221a1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
# Security Assessments | ||
|
||
## Goals | ||
The [security assessment process](guide) is designed to accelerate the adoption | ||
of cloud native technologies, based on the following goals and assumptions: | ||
|
||
### 1) Reduce risk across the ecosystem | ||
|
||
The primary goal is to reduce the risk from malicious attacks and accidental breaches of privacy. This process supports that goal in two ways: | ||
|
||
* Clear and consistent process for communication increases detection & | ||
reduces time to resolve known or suspected vulnerability issues | ||
* A collaborative evaluation process increases domain expertise | ||
within each participating project. | ||
|
||
### 2) Accelerate adoption of cloud native technologies | ||
|
||
Security reviews are a necessary, yet time consuming process, where each | ||
company, organization and project must perform its own reviews to ensure | ||
that it meets its unique commitments to its own users and stakeholders. | ||
In open source, simply finding security-related information can be a very | ||
time consuming part of the the process. The process is designed to enable improved discovery of security information & streamlined security reviews in multiple ways: | ||
|
||
* Consistent documentation reduces review time | ||
* Established baseline of security-relevant information reduces Q&A | ||
* Clear rubric for security profile enables organizations to align their | ||
risk profile with project’s risk profile and effectively allocate resources | ||
(for review and needed project contribution) | ||
* Structured metadata allows for navigation, grouping and cross-linking | ||
|
||
We expect that this process will raise awareness of how specific open source | ||
projects affect the security of a cloud native system; however, separate | ||
activities may be needed to achieve that purpose using materials generated by | ||
the assessements. | ||
|
||
## Outcome | ||
|
||
Each project assessment will: | ||
1. ensure a clear description of the project's design goals with respect to | ||
security | ||
2. uncover design flaws and document known limitations | ||
3. document next steps toward increasing security of the project itself and/or | ||
increasing the applications of the project toward increasing security of the | ||
cloud native ecosystem | ||
|
||
Due to the nature and timeframe for the analysis, *this review is not meant | ||
to subsume the need for a professional security audit of the code*. Audits | ||
of implementation vulnerabilities and similar issues at that level are not | ||
intended to be covered by this assessment. The purpose of this effort is to | ||
uncover design flaws and to obtain a clear articulation of what the project's | ||
design goals and security properties are intended to be. | ||
|
||
## Process | ||
|
||
The project assessment is intended to be a collaborative process for | ||
the benefit of the project and the community, where the primary content is | ||
generated by the [project lead](project-lead.md) and revised based on feedback | ||
from [security reviewers](security-reviewer.md) and other members of the | ||
working group (WG). | ||
|
||
See [security assessment guide](guide) for more details. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
To provide the CNCF’s TOC with effective | ||
information about the security of different projects, | ||
this document outlines the procedure by which a project should be evaluated. | ||
|
||
## Roles | ||
|
||
* [project lead](project-lead.md) | ||
* [security reviewers](security-reviewer.md) | ||
|
||
## Steps | ||
|
||
1. Create tracking issue | ||
* Issue may be a request from TOC liason or project itself | ||
2. Project provides self-assessment | ||
* [project lead](project-lead.md) responds to the issue with draft document (see [outline](outline.md)) | ||
* issue assigned to lead [security reviewer](security-reviewer.md) who | ||
will recruit a second reviewer, if one not already assigned, and facilitate | ||
the process | ||
3. Inital review | ||
* Upon request by the project, security reviewer may do an inital review to | ||
verify completeness, ask for clarifications and provide quick feedback | ||
* Project posts their document to the group mailing list, allowing at least | ||
one week for review before presenting to the WG | ||
4. Presentation | ||
* Project lead presents to WG | ||
* This will be recorded as part of standard WG process | ||
5. Final assessment | ||
* Reviewers provide final feedback with recommendations | ||
* Either project lead or reviewers may request further WG discussion | ||
* Project lead prepares a PR to /assessments/project-docs/project-name/ | ||
|
||
## Additional Process Notes | ||
|
||
Iteration is expected; however, we expect quick turnaround (at most a week). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there a desire to have an explicit followup process, ie an annual review? I think the notion that a single point in time review ignores the reality that threats change over time. someone using this review process as evidence of "trustworthiness" can be lulled into thinking something is "safe" since it was reviewed N years ago. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes! added as open issue: #152 (so we can get this PR merged for initial reviews that are currently blocked, then add more detail on annual review process) |
||
In rare cases unrelated issues can unexpectedly interrupt the process and | ||
it may be appropriate to address specific concerns rather than continuing with | ||
the assessment. We encourage open communication between project lead and s | ||
ecurity reviewers: | ||
|
||
* At any time, the project lead may request additional time to respond | ||
to feedback from security reviewers | ||
* Project lead or lead security reviewer may pause the process where a delay | ||
of over a week cannot be accomodated by the review team. Simply close | ||
the github issue with a note to SIG co-chairs. | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,114 @@ | ||
# Outline | ||
|
||
First of all, the burden is primarily on the proposing project to | ||
demonstrate it is secure in a manner that is understandable to the broader | ||
community. The [reviewers](security-reviewer.md) will help to assess and probe the design. | ||
|
||
The proposing project must provide a written document that describes the | ||
project and its security. The document must contain the following | ||
information, at a minimum. Where security considerations do not fit into | ||
the outline below, if possible, add a sub-section when possible such that the | ||
additional content conforms to the general flow of the review. | ||
|
||
|
||
## Metadata | ||
A table at the top for quick reference information, later used for indexing. | ||
|
||
| | | | ||
| -- | -- | | ||
| Software | A link to the software’s repository. | ||
| Security Provider | Yes or No. Is the primary function of the project to support the security of an integrating system? | ||
||| | ||
|
||
## Overview | ||
|
||
One or two sentences describing the project -- something memorable and accurate | ||
that distinguishes your project to quickly orient readers who may be | ||
reviewing multiple assessments. | ||
|
||
* Background. Provide information for reviewers who may not be familiar with | ||
your project's domain or problem area. | ||
* Goal. The intended goal of the projects including the security guarantees | ||
the project is meant to provide (e.g., Flibble only allows parties with an | ||
authorization key to change data it stores) | ||
* Non-goals. Non-goals that a reasonable reader of the project’s literature | ||
could believe may be in scope (e.g., Flibble does not intend to stop a | ||
party with a key from storing an arbitrarily large amount of data, possibly | ||
incurring financial cost or overwhelming the servers) | ||
|
||
## Intended use | ||
|
||
* Target users and use cases. Provide a mapping from [standard personas](../../usecases.md) to the | ||
nomenclature used in your project docs (which you should then use consistently | ||
for the remaider of this document). Describe the scenarios in which the project is expected to be used. This must be specific enough to provide context for analysis. For example: | ||
|
||
Flibble can be used in any cloud environment. Three diverse examples are | ||
as follows: | ||
1. when a Flibble server is used by legacy servers as a | ||
database for salted password hashes. | ||
2. a Flibble cloudlet may be run | ||
on virtualized fog hardware near smartphone users. | ||
3. a Flibble distributed service may serve as a backend for a Notary image registry.) | ||
* Operation. A description of the operational aspects of the system, such as | ||
how keys are likely to be managed and stored. | ||
|
||
## Project design | ||
* Design. A description of the system design that discusses how it works. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is there any interest in promoting/recommending formal design specification ,eg TLA+ and such? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This seems great if a project has it. I don't think we want to require this at the outset. |
||
This is especially critical for any mechanisms that relate to the security | ||
of the system. | ||
|
||
## Security analysis | ||
* Attacker motivations. A discussion about the likely goals of an attacker. | ||
This likely relates closely to the impact of different attacks in the | ||
scenarios. (e.g., In the password hash case, the attacker wants to expose | ||
those hashes on the Flibble server. However, a Flibble cloudlet attacker | ||
may find it more interesting to bring down the service.) | ||
* Expected attacker capabilities. A description of likely capabilities that | ||
the attacker has in these scenarios should be described. Both assumptions | ||
about the strength and limitations of attackers should be described. | ||
(e.g., We assume that an attacker may be able to exploit implementation | ||
errors in some set of the servers to take control of them. However, we | ||
assume the attacker cannot break AES or SHA256.) | ||
* Attack risks and effects. A rough estimation of the risk posed by different attacks, and potential negative consequences. | ||
(e.g., The master Flibble server only communicates with Flibble servers | ||
using a minimalistic API that is formally verified and written in Rust.) | ||
* Security degradation. A discussion about the resulting security when | ||
various attacks are launched. Note, that no system is secure in all | ||
scenarios, hence it is expected that this will include areas where attacks | ||
compromise all meaningful security. (e.g., If an attacker is able to | ||
compromise the “master” Flibble server, they may read, write, or delete any | ||
content stored on any system) | ||
|
||
## Secure development practices | ||
* Development pipeline. A description of the testing and review processes | ||
that the software undergoes as it is developed and built. | ||
* Communication Channels. Reference where you document how to reach your | ||
team or describe in corresponding section. | ||
* Internal. How do team members communicate with each other? | ||
* Inbound. How do users or prospective users communicate with the team? | ||
* Outbound. How do you communicate with your users? (e.g. flibble-announce@ mailing list) | ||
* Vulnerability Response Process. Who is responsible for responding to a report. What is the reporting process? How would you respond? | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. at a minimum shouldn't this SIG be informed "formally"? i suggest there is a mechanism (GHI, commit to some markdown file, etc) for explicit capture of all known vulns over time. |
||
* Ecosystem. How does your software fit into the cloud native ecosystem? (e.g. Flibber is integrated with both Flocker and Noodles which covers virtualization for 80% of cloud users. So, our small number of "users" actually represents very | ||
wide usage across the ecosytem since every virtual instance uses Flibber | ||
encryption by default.) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Within the ecosystem, is there a concern of delegation of security responsibilities from one project to another and vice versa? Perhaps a project is relying on security mechanisms of another CNCF technology which has not been reviewed yet? How would this affect the security posture of the project? I suppose that this would be a rare case that can be handled whenever it comes up. Side comment: Along the lines of a point mentioned in the earlier section about AES/sha256, I like the idea of a list of technologies and algorithms/libraries/projects with their security assumptions accepted. This could help in getting project-leads to reduce the assumptions they are making and/or default to more well established security libraries/components. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. +1 on explicitly listing the technologies and assumptions. This would also inform mapping which domains and subject matter experts are needed. even more important it might highlight areas where stuff should NOT have been invented (we rolled our own symmetric encryption algorithm instead of using X cuz we thought X was too slow...oops we never considered side channel attacks) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is covered by the assessment, but could benefit from more detailed documentation of effective techniques. I've opened an issue on that, if anyone wants to elaborate further on the need: #155 |
||
|
||
## Roadmap | ||
|
||
* Project next steps. Link to your general roadmap, if available, then list | ||
prioritized next steps that may have an impact on the risk profile of your | ||
project, including anything that was identified as part of this review. | ||
* CNCF requests. In the initial draft, please include whatever you believe | ||
the CNCF could assist with that would increase security of the ecosystem. | ||
|
||
## Appendix | ||
|
||
* Known Issues over time. List or sumarize statistics of past vulerabilities | ||
with links. If none have been reported, provide data, if any, about your | ||
track record in catching issues in code review or automated testing. | ||
* CII Best Practices. A brief discussion of where the project is at with | ||
respect to CII best practices and what it would need to achieve the badge. | ||
* Case Studies. Provide context for reviewers by detailing 2-3 scenarios of | ||
real-world use cases. | ||
* Related Projects / Vendors. Reflect on times prospective users have asked | ||
about the diffeerences between your project and projectX. Reviewers will have | ||
the same questions. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
# Project Lead | ||
|
||
In the context of the security assessment, the "project lead" should be | ||
someone on the security team for the project. For new or smaller projects | ||
without an established security team, this could be a project maintainer | ||
or they may delegate to a regular contributor who has an interest in security. | ||
|
||
## Expected time / effort | ||
|
||
The level of effort for the team providing the information is expected to | ||
be around 80 hours of work. Note, that projects that have already | ||
performed a security analysis internally are expected to have much lower | ||
requirements. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
# Security Reviewers | ||
|
||
Reviewers will try to understand the | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I recommend we have at least N (N>=2, 3?) reviewers to ensure diverse and possibly even conflicting opinions, lest there is too much dependence on one or two reviewers? it also encourages expansion of the network of reviewers so we cultivate a large, diverse experience base. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. specified in outline.md " assigned to lead security reviewer who |
||
system and probe its security. Specifically design level issues are meant | ||
to be addressed as well as high level problems with the project’s setup and | ||
operation. This is meant to provide an independent analysis and estimation | ||
of the above information. The goal is to ask questions to understand if | ||
there are hidden assumptions, underestimated risk, or design issues that | ||
harm the security of the project. It may be useful to reach out to | ||
community members to understand the answers to some questions, especially | ||
involving deployment scenarios and the impact of attacks. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Perhaps there should be a formal "request for comment" process where domain experts are solicited and there is a waiting period to allow those experts to weigh in? eg 10 days? perhaps we should put the burden on the project under review to nominate 2-3 non-affiliated subject matter experts to participate and advise the reviewer team? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This makes sense but I think might be a little too heavyweight for the initial iterations of this process. We will grow and add to this over time in the most helpful ways possible and this certainly could be one of those directions. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. that is specified in outline.md "Project posts their document to the group mailing list, allowing at least |
||
|
||
## Qualifications | ||
|
||
Unless approved by the SIG chairs, at least one of the reviewers will | ||
have previously performed a SAFE WG audit. (Exemptions are expected to be | ||
granted bootstrap the process but only in extreme cases later on.) In | ||
general, a reviewer should have performed security audits for diverse | ||
organizations. The ideal reviewer should also have been the recipient | ||
of security audits for a software project they manage. Note that it is | ||
encouraged to have participation (shadowing) from participants that are not | ||
yet qualified to help them gain the necessary skills to be a reviewer | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Perhaps we should explicitly catalog what those skills are? Not only is that useful for those hoping to participate, but it lends itself to segregating the work and parallel processing. And not all experts will be experts in everything. for example I have deep crypto review expertise, but very little linux kernel review expertise. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There's not a clear consensus on what this is, so we decided to do a number of assessments before adding more detail -- which i think we set as 6-8 assessments, but could revisit earlier if it causes friction in the process or there are specific recommendations that help the process @rficcaglia -- thanks for chiming in on #142 -- feel free to add more thoughts / ideas there on this topic, and we'll revisit at a future checkpoint |
||
in the future. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we add something about managing conflict of interest of the reviewer/project? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
|
||
## Time and effort | ||
|
||
The level of effort for the reviewers is expected to be 10 hours. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we require (encourage) folks to log their time so we can actually benchmark this and track and normalize (ie # hours per SLOC or feature count, etc?) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this is mostly to set expectations about the level of commitment. Some reviewers will want to spend more time, based on their interest in the project or level of experience. If reviewers find that the time estimate is way off, then I think running an experiment where people track time could be helpful. For now, I think we need a bit more experience with the process first. |
||
Despite the fact that there may be some back and forth to get clarification | ||
on a few points, it is expected analysis can usually be concluded in a few | ||
weeks. This will primarily involve carefully reading the written | ||
document and analyzing the security assertions and assumptions. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with separating the concerns here, but I think there would be huge benefit in providing open source code audits that do not rely on vendors, and encourage transparent and diverse methodologies. Perhaps an open code audit sub-SIG? maybe put the burden on the USERS to volunteer to participate and collectively encourage more active engagement with the user community instead of just passively harvesting value from open source?
I think open source code audit would also have the side benefit of providing real world education to a larger group of engineers on how to code securely (and how to review code for security). this is a skill that should really be part of every developer's forte, and the more we can model and inculcate secure coding skills, the better for all of us.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a different CNCF audit process -- we should link that here (or open an issue if it's not yet documented).