Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CIS benchmark checks for generating a Kubernetes compliance report #3239

Merged
merged 23 commits into from
Sep 5, 2023

Conversation

stmcginnis
Copy link
Contributor

@stmcginnis stmcginnis commented Jun 28, 2023

Issue number:

Closes #2852

Description of changes:

This adds a set of CIS benchmark checks to bloodhound for the Kubernetes CIS benchmark.

Still needs exposure through apiserver and apiclient. The plumbing for CIS reports in general are still being worked on for the CIS Bottlerocket benchmark. Depending on timing, the wiring up of this benchmark will either be added to this PR in additional commits, or added as a follow up PR after this one merges.

Testing done:

Ran new checks on aws-k8s-1.26 node:

[ssm-user@control]$ apiclient report cis-k8s
Benchmark name:  CIS Kubernetes Benchmark (Worker Node)
Version:         v1.7.1
Reference:       https://www.cisecurity.org/benchmark/kubernetes
Benchmark level: 1
Start time:      2023-07-18T15:09:51.451720655Z

[PASS] 4.1.1     Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automatic)
[PASS] 4.1.2     Ensure that the kubelet service file ownership is set to root:root (Automatic)
[SKIP] 4.1.3     If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual)
[SKIP] 4.1.4     If proxy kubeconfig file exists ensure ownership is set to root:root (Manual)
[PASS] 4.1.5     Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automatic)
[PASS] 4.1.6     Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automatic)
[PASS] 4.1.7     Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automatic)
[PASS] 4.1.8     Ensure that the client certificate authorities file ownership is set to root:root (Automatic)
[PASS] 4.1.9     If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automatic)
[PASS] 4.1.10    If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automatic)
[PASS] 4.2.1     Ensure that the --anonymous-auth argument is set to false (Automatic)
[PASS] 4.2.2     Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automatic)
[PASS] 4.2.3     Ensure that the --client-ca-file argument is set as appropriate (Automatic)
[PASS] 4.2.4     Verify that the --read-only-port argument is set to 0 (Automatic)
[PASS] 4.2.5     Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automatic)
[PASS] 4.2.6     Ensure that the --make-iptables-util-chains argument is set to true (Automatic)
[SKIP] 4.2.7     Ensure that the --hostname-override argument is not set (not valid for Bottlerocket) (Manual)
[PASS] 4.2.9     Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automatic)
[SKIP] 4.2.10    Ensure that the --rotate-certificates argument is not set to false (not valid for Bottlerocket) (Manual)
[PASS] 4.2.11    Verify that the RotateKubeletServerCertificate argument is set to true (Automatic)
[PASS] 4.2.12    Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automatic)
[PASS] 4.2.13    Ensure that a limit is set on pod PIDs (Automatic)

Passed:          18
Failed:          0
Skipped:         4
Total checks:    22

Compliance check result: PASS
[ssm-user@control]$ apiclient report cis-k8s -l 2
Benchmark name:  CIS Kubernetes Benchmark (Worker Node)
Version:         v1.7.1
Reference:       https://www.cisecurity.org/benchmark/kubernetes
Benchmark level: 2
Start time:      2023-07-18T15:10:15.191556058Z

[PASS] 4.1.1     Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automatic)
...
[ssm-user@control]$ apiclient report cis-k8sh
Missing or unknown subcommand for 'report'
...
apiclient report cis-k8s -l 2 -f json
{
  "level": 2,
  "total": 23,
  "passed": 18,
  "skipped": 5,
  "failed": 0,
  "status": "PASS",
  "timestamp": "2023-07-18T15:10:32.352887194Z",
  "name": "CIS Kubernetes Benchmark (Worker Node)",
  "version": "v1.7.1",
  "url": "https://www.cisecurity.org/benchmark/kubernetes",
  "results": {
    "k8s04010100": {
      "name": "k8s04010100",
      "id": "4.1.1",
      "level": 1,
      "title": "Ensure that the kubelet service file permissions are set to 644 or more restrictive",
      "mode": "Automatic",
      "status": "PASS",
      "error": ""
    },
   ...
}

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@stmcginnis stmcginnis changed the title Cis agent k8s Add CIS benchmark checks for generating a Kubernetes compliance report Jun 28, 2023
@maxres-ch
Copy link

This is great to see! I wrote a daemonset to run the bash script outlined here: https://github.com/aws-samples/containers-blog-maelstrom/tree/main/cis-bottlerocket-benchmark-eks/

Will there be an operator to accompany this or will it be up to users to invoke this client and report the status individually?

@stmcginnis
Copy link
Contributor Author

Will there be an operator to accompany this or will it be up to users to invoke this client and report the status individually?

Hey! Great questions. The plan is for the Bottlerocket and Kubernetes benchmarks to ultimately be exposed via apiclient. That work is still in progress, with the apiserver updates happening here, and the last step of wiring in to apiclient coming once that is done.

So while it's possible to execute this command directly to get the output, the actual end user experience should be a little cleaner through apiclient report cis-k8s (or something like that).

sources/bloodhound/src/bin/kubernetes-checks/main.rs Outdated Show resolved Hide resolved
sources/bloodhound/src/bin/kubernetes-checks/checks.rs Outdated Show resolved Hide resolved
sources/bloodhound/src/lib.rs Show resolved Hide resolved
sources/Cargo.lock Show resolved Hide resolved
sources/bloodhound/src/bin/kubernetes-checks/checks.rs Outdated Show resolved Hide resolved
sources/bloodhound/src/bin/kubernetes-checks/checks.rs Outdated Show resolved Hide resolved
sources/bloodhound/src/bin/kubernetes-checks/checks.rs Outdated Show resolved Hide resolved
sources/bloodhound/src/bin/kubernetes-checks/checks.rs Outdated Show resolved Hide resolved
@stmcginnis stmcginnis force-pushed the cis-agent-k8s branch 4 times, most recently from 91eccbe to 8c2e4cb Compare July 18, 2023 15:54
@stmcginnis
Copy link
Contributor Author

Added apiserver and apiclient parts of the complete implementation and rebased on develop to resolve conflicts.

@stmcginnis stmcginnis force-pushed the cis-agent-k8s branch 3 times, most recently from 4e8d702 to 41b4803 Compare August 18, 2023 14:24
sources/api/apiclient/README.md Outdated Show resolved Hide resolved
sources/api/apiclient/README.md Outdated Show resolved Hide resolved
sources/api/apiclient/README.tpl Outdated Show resolved Hide resolved
sources/api/apiclient/src/main.rs Show resolved Hide resolved
@stmcginnis stmcginnis force-pushed the cis-agent-k8s branch 2 times, most recently from 1437e3c to 3428e07 Compare August 22, 2023 19:49
sources/api/apiclient/README.tpl Outdated Show resolved Hide resolved
sources/bloodhound/src/lib.rs Outdated Show resolved Hide resolved
sources/bloodhound/src/lib.rs Outdated Show resolved Hide resolved
This adds a multicall binary to be used as the common entry point for
all Kubernetes CIS benchmark checks.

Signed-off-by: Sean McGinnis <[email protected]>
This adds the 4.1.1 check to verify the kubelet service file has
restrictive permissions.

Signed-off-by: Sean McGinnis <[email protected]>
This adds the 4.1.2 check to make sure the kubelet service file
ownership is root:root.

Signed-off-by: Sean McGinnis <[email protected]>
This adds the 4.1.5 check to ensure the kubelet.conf file has
restrictive permissions.

Signed-off-by: Sean McGinnis <[email protected]>
This adds the 4.1.6 check to verify kubelet.conf is owned by root:root.

Signed-off-by: Sean McGinnis <[email protected]>
This adds the 4.1.7 check to verify the kubelet CA file has restrictive
permissions.

Signed-off-by: Sean McGinnis <[email protected]>
This adds check 4.1.8 to verify the kubelet CA file is owned by
root:root.

Signed-off-by: Sean McGinnis <[email protected]>
This adds the 4.1.9 check to verify the kubelet configuration file has
restrictive permissions.

Signed-off-by: Sean McGinnis <[email protected]>
This adds check 4.1.10 to verify the kubelet config file is owned by
root:root.

Signed-off-by: Sean McGinnis <[email protected]>
This adds check 4.2.1 to verify anonymous auth is not enabled.

Signed-off-by: Sean McGinnis <[email protected]>
This adds check 4.2.2 to verify authorization mode is not set to
AlwaysAllow.

Signed-off-by: Sean McGinnis <[email protected]>
This adds check 4.2.3 to verify clientCAFile is configured and points to
an actual file.

Signed-off-by: Sean McGinnis <[email protected]>
This adds the 4.2.4 to verify the readOnlyPort is not set to 0.

Signed-off-by: Sean McGinnis <[email protected]>
This adds check 4.2.5 to verify streamingConnectionIdleTimeout is not
set to 0.

Signed-off-by: Sean McGinnis <[email protected]>
This adds check 4.2.6 to verify makeIPTablesUtilChains is true.

Signed-off-by: Sean McGinnis <[email protected]>
This adds the check to make sure the server TLS certificates are
configured correctly.

Signed-off-by: Sean McGinnis <[email protected]>
This adds check 4.2.10 to verify rotateCertificates is not disabled.

Signed-off-by: Sean McGinnis <[email protected]>
This adds check 4.2.11 to verify RotateKubeletServerCertificate is not
disabled.

Signed-off-by: Sean McGinnis <[email protected]>
This adds check 4.2.12 to check that kubelet is only configured to use
secure cryptographic ciphers.

Signed-off-by: Sean McGinnis <[email protected]>
This adds verification that podPidsLimit is configured to restrict the
maximum number of PIDs allowed per-pod.

Signed-off-by: Sean McGinnis <[email protected]>
This changes the 4.2.10 check to be a manual test response. This check
does not apply to EKS or EKS-D since external IAM auth is used.

This leaves the validation logic in place as we may want to use it in
the future. For not EKS-related use cases, it may make sense to enable
configuration of certificate rotation. If/when that is done, this check
can be reenabled and updated to validate under the proper conditions.

Signed-off-by: Sean McGinnis <[email protected]>
This adds handling for a "type" parameter to be passed along to the
`/report/cis` endpoint to control what type of CIS report to generate.
The default type is the Bottlerocket CIS benchmark report, with
"kubernetes" being the only recognized other option.

Signed-off-by: Sean McGinnis <[email protected]>
This adds the `cis-k8s` report subcommand to access the Kubernetes CIS
report. It uses the standard CIS arguments of `-l` for level and `-f`
for format.

Signed-off-by: Sean McGinnis <[email protected]>
@stmcginnis stmcginnis merged commit d1766a1 into bottlerocket-os:develop Sep 5, 2023
48 checks passed
@stmcginnis stmcginnis deleted the cis-agent-k8s branch September 5, 2023 19:32
@stmcginnis stmcginnis mentioned this pull request Sep 6, 2023
14 tasks
@grodriguezl
Copy link

This is beautiful. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consider adding Kubernetes CIS Benchmark checks
7 participants