first 5 security assessments (before process review/improvement)

[ultrasaurus]

We have a target of 5 security assessments, before doing a retrospective on process and addressing (most of) open issues/questions

This was the initial set of security-related projects, initially identified by TOC for our SIG: SPIFFE, SPIRE, Open Policy Agent, Notary, TUF, Falco. We omitted those that had already had a formal audit or TOC contributor assessment.

Done:

• Harbor
• in-toto
• OPA
• SPIFFE / SPIRE - [Assessment] SPIFFE/SPIRE #308
• Keycloak - presented to TOC, 9 Apr '18

In Progress

• Cloud Custodian - [Assessment] Cloud Custodian #307

Considered:

• non-security provider, such as Strimz, that sets a pattern that others might follow
• ISTIO
• Network Service Mesh
• INACTIVE: Falco - [Assessment] Falco #273 sandbox

Note: of the CNCF security-related projects identified by the TOC, some have already received a prior assessment or audit

• Justin Cappos reviewed SPIFFE, SPIRE, per TOC request
• TUF and Notary have been audited

Project lead: @JustinCappos
SIG Chair: @ultrasaurus

[JustinCappos]

I'll be on the Harbor assessment (non-lead).

Daniel Iziourov volunteers to participate in the Falco assessment.

[lumjjb]

I can help with keycloak assessment - I can lead this as well.

[rficcaglia]

happy to assist with Harbor.

[jonmuk]

I would be happy to partner with another member to perform an audit (Jonathan Meadows)

[harche]

I would be happy to assist with keycloak assessment

[ultrasaurus]

@JustinCappos I see you added Istio. I had assumed that our "first five" would all be CNCF projects. What's your thinking on adding Istio to the list?

[mhausenblas]

@ultrasaurus you beat me to it, exactly my thinking as well. Suggest to do Envoy instead, which is a CNCF project and used in service mesh data planes, including Istio.

[lumjjb]

Did we have discussions with anyone from keycloak about security reviews yet? if not i will reach out to them to gauge interest and timeline.

[ultrasaurus]

@lumjjb looped you into an email thread from a few months ago (cc'd @JustinCappos) -- we said we'd get back to them when we were further along in our process... so I think it would be a great time to reconnect. Thank you!

[edwarnicke]

@nickolaev @fkautz @haiodo - Adding the other Network Service Mesh maintainers here. Happy to engage when you guys are ready :)

[randomvariable]

Hi, is this the right place to say we would like and are ready for a security audit for kubeadm? kubeadm is the recommended bootstrapper for Kubernetes and is consumed by a lot of other Kubernetes infrastructure bootstrappers such as Kubespray and Kubernetes Cluster API. cc @timothysc

[ficcaglia]

Just an aside wrt projects that had a prior audit... I feel previous audits are good inputs but are just one of many inputs into what should be a consistently applied process. Certainly for practical reasons deferring those until later makes sense. But they should be scheduled for a later assessment IMO. In short every CNCF project should go through the same assessment process eventually and consistently.

[fkautz]

I'm looking forward to the security audit for NSM. Once we are more established, I am also going to approach multiple companies who are depending on NSM to see if we can get them to provide independent security audits. The more eyes the better.

[ficcaglia]

@fkautz I see you have mostly completed the CII best practices:
https://bestpractices.coreinfrastructure.org/en/projects/2725

Any particular reason this could not be pushed to 100%? Happy to help.

[w8mej]

I would be happy to assist with KeyCloak.

[lumjjb]

@cloudsriseup @harche Awesome, I think we almost have a full group for KeyCloak. We had a chat with the maintainers and this is slated for sometime in August!

[danmx]

if you still need an extra person for KeyCloak, feel free to add me to it.

[qnetter]

Not sure if I am helpful or less than, but I would be happy to be at least a ride-along on any remaining assessment.

[fkautz]

@ficcaglia we're a relatively new project, we're getting to these tasks. :)

If you have any suggestions or would like to help, let me know. Ping us on slack.cncf.io on #nsm or me directly under the fkautz username.

Cheers!

[rficcaglia]

@fkautz sure! it would be a good example of "assessment readiness" since you are engaging so soon! I'll ping you on slack next week.

@randomvariable I'm not speaking "officially", but my guess is that given the load of CNCF projects queued up...it might be awhile before we have bandwidth to cover non-CNCF projects...I defer to @ultrasaurus on that. (note also the discussion around Istio above) However, I think a huge benefit of the open process here is that individual (non-CNCF) projects can take all the work product and process templates here, then reach out to their own community and organize a "self-assessment" following the exact same methodology. (once it's defined sufficiently!) Then assuming the project is ultimately accepted for CNCF Sandbox, you would be that much more ahead of the curve! It is also a good test of whether the methodology and process is usable by others...is it well defined? does it make sense across domains? do other assessment teams hit the same or different bottlenecks? are the results similar? etc.

Just to codify the project queue, I have added a draft overview table here: #241

[joshuagl]

I'd like to volunteer to observe and learn more about the process so that I can
be more active in future assessments. I'm particularly keen to observe the Notary
assessment, but would be happy to observe on any/all of the assessments.

[lumjjb]

@danmx most definitely, I think we will have space for keycloak.

[chubirka]

Happy to join this effort regarding assessment process improvement.

[itaysk]

Happy to join the security assessment improvement process

[bdaw]

Interested to help in the WG

[lumjjb]

Comment on this thread if you'd like to be part of the working group to improve security assessment process! Just comment on the thread and we will invite everyone to a new slack channel for this discussion!

[knowlengr]

Count me in. My day job constraints (unforecasted day job Outlook bookings) are considerable, hence noon (except Wed) is good and I'm willing to do earlier or later outside the normal workday if needed. Having a specific day of the month, or biweekly, for instance, that I can block out would help facilitate my regular participation. (Apologies for this self-centered tone here.) In fact, if we have a straw man slot, I'd love to block it today. I'm on eastern time.

[ashutosh-narkar]

I would be interested to join this effort. Thanks.

[sunstonesecure-robert]

Falco is not active - we should take that off the list.

[lumjjb]

@sunstonesecure-robert done.

I will create a slack channel shortly! Look out for it and get on slack (if you aren't already on the CNCF slack!)

[chubirka]

Can someone add me? Thanks. Regards, Michele

…

On Wed, Sep 23, 2020 at 1:45 PM Brandon Lum ***@***.***> wrote: Comment on this thread if you'd like to be part of the working group to improve security assessment process! Just comment on the thread and we will invite everyone to a new slack channel for this discussion! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#167 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHR3BAILJVWAQ5C75ZIWMH3SHIX4LANCNFSM4HOBWDTQ> .

[lumjjb]

At the kick-off meeting, we went through the process of improving the security assessment process! We are trying something new that some of us have had success with in the past. We are going to use a Mural board to brainstorm on the ideas. This will allow collaboration in a fairly asynchronous way.

Also a remind we are on CNCF slack channel: #sig-security-secassess-wg

The phases of the process is going to be:

1. Brainstorming improvement ideas (Week 1)
2. Categorizing/clustering ideas to form themes, which develop into PRs (Week 2)
3. Open issues for identified improvement areas and solicit extra help for PRs (Week 2)
4. Work on the PRs for improvement (Week 3/4)

Next Steps & Action Items: Let's fill this up by Oct 15 to stay on schedule!

0. Please slack me your email (or email me at [email protected]), this is to help scheduling in the future so I can send direct calendar invites. We will schedule another session on Oct 15/16.

1. Go onto the Mural board, you will see something like the following:
   

The boxes on the top and bottom are the individual brainstorming boxes. The task for now is to:

1. Pick a box which doesn't have a name, and double click to edit and enter your name
2. Fill in the sticky notes with your ideas
3. For now, ignore the circles in the middle, we will use those in week 2 to cluster similar ideas.

Before:

After:

You can create more sticky notes by right clicking on a sticky note and duplicating it!

[lumjjb]

Next meeting is currently scheduled for 15 Oct 10am-11am PST. details are in the slack channel.

[lumjjb]

Hi All! Reminder that we are meeting today at 10-11am PST! Looking forward to seeing everyone!

[lumjjb]

Hi All! Last Thursday, we went through the mural board and grouped up some of the suggestions that people wrote into different categories, I will be going through them and defining some issues for folks to take the lead on. If you haven't had the chance to put in any suggestions, please do so now! Put them in one of the sub-categories or create a new category (circle) if it doesn't fit!

I will be going through these tomorrow to start organizing them.. so please put in your suggestions by tomorrow (10/20)!

https://app.mural.co/t/sigsecurityassesswg9089/m/sigsecurityassesswg9089/1602018028189/0429893063587a307d2368e09f9b844d282820ee

[lumjjb]

Next Steps: Synthesized Issues

Hi All! The following are the list of subcategories that we came up with, as well as the ideas and premises organized accordingly! I think it looks like we have a pretty good list of things with pretty well defined scopes.

The cochairs and TLs would like to encourage members to take the lead on some of these issues. Please comment if you find a topic interesting that you'd like to work on!

Provide some consistency across reviews

Premise:

• Auditing imperative systems is more of an art than a science
• Threat modeling is important for projects, we should figure out how to help them be consistent with it
• Different projects have a variety in the level of detail for different sections and emphasis on different aspects

Ideas:

• Make adjustments to assessment doc structure
• Give recommendations of word length for sections
• Map assessment findings to MIRTE ATTACK or similar
• Use an existing assessment framework + template - can be tweaked for CNCF but start with that so there is a consistent process and vocabulary
• Provide qualitative ranking for projects i.e. scoring between 1 - 10
• Provide ideas/guidelines for how one should perform a review

Naming and Scope of assessments

Premise:

• Assessment is an overloaded term, and can lead to confusion

Ideas:

• Have a better articulation of what is a sec assess.
• Include scope to include additional aspect of code audit related checks/certification
• Add mapping aspects of assessments to compliance frameworks

Additional suggestion of scope to include related to security testing

• Implement hands-on security testing [Suggestion] Perform hands-on security testing during security assessments #394
• Define requirements for hands-on security testing / best practices
• Must be a process for onboarding and approving new "hands-on" testers

Benefits of a Security Assessment for Projects

Premise:

• It is not entirely clear that why a project should be incentivized to participate in an assessment

Ideas:

• Add "benefit for the project" to security assessment guide
• Independent evaluation provides a primer to the CNCFs security audit team
• It provides a SECURITY.md
• Self evaluation should be reflective of their software development practices
• Provide badging: As a badge, reference material for security aspect of the project

Time and Effort

Premise:

• The result time span of assessments tend to stretch
• There is little awareness or lack of clarity of the current scheduling aspects of assessments
• Fairly time consuming on project side - barrier for project without strong corp sponsorship

Ideas:

• Assessments tend to drag on over >2-3 weeks
• Assesment timeline should be capped, or with a hard time limit
• Give recommendations of word length for sections
• Part of the review should be automated
• It makes it easy for people to report issues
• Templates for project and reviewers

Additional Context:

• https://github.com/cncf/sig-security/tree/master/assessments/guide
• https://github.com/cncf/sig-security/blob/master/assessments/guide/security-reviewer.md#time-and-effort
• https://github.com/cncf/sig-security/blob/master/assessments/guide/project-lead.md#time-and-effort

Getting more reviewers

Premise:

• Challenge of assembling a team for each review

Ideas:

• what are the reasons that people want to participate? can we incentivize more?
• Provide swag/recognition
• For issues found they would get discount for courses and conferences
• actively reach out to past reviewers (This is currently done by co-chairs and TLs informally)
• Create a more concrete list of the expectations/requirements of a reviewer
• Find new ways to engage new reviewers including in-experienced ones
• Reach out to researchers to review the projects
• Recommend the CNCF provide training/skills to community members to be able to perform assessments and audits

Mapping to TOC Process

Premise

• The security assessment aims to tie into the CNCF project process, but it is not clear how
• There is no current agreed upon requirement for the process for what is asked of a SIG

Ideas

• better document the Due Diligence process and document
• Need more detail on when the SIG recommends for the next CNCF phase
• not tie it too directly to TOC process
• CNCF needs to explicitly define the requirement for projects at each level to go through assessment
• Map security assessment process to process of CNCF
• Are assessments necessary/mandated by the TOC to move from one stage to another ? We need to make projects realize it's in their interest to invest time in going through the assessment process.

Process Improvement/Changes

Process documentation

• Define the new process(es) then define the roles for each part
• Break the process into 3 parts, one for each part of the CNCF cycle: Self, Joint, Independent
• user community should determine who is assessed first and what assessment goals are highest priority since they are the ones using a project IRL - require user "sponsors" who participate and review assessment before presentation to SIG and TOC

Tracking issue:

• Keep conflict of interest statements on the issues.

Self-assessment:

• Self-Assessment table of contents (like harbor)
• For concerns of self-evaluating self, Sign off by project lead for integrity of self-assessment
• Joint evaluation should review the previous self evaluation, maybe do a small hands-on assessment, and highlight 'security features' of the project

Reviewing process

• Come up with recommendations for security reviewers (i.e. here are 10 things you can do for a review)

Feedback and post review

• Review survey - Fresher feedback from review participants post review
• Integrate 1 page summary slides into repo

Misc:

• (needs clarification) Projects should be provided "summer of security" support for initial assessment and ongoing assessment efforts

[lumjjb]

Hi All - the issues for each of the topics have been created! Please go through and sign up for ones that you would like to be involved with!

[ultrasaurus]

First five are complete -- closing this issue!