FIPS Support / Validation

[vennemp]

This is related to my comment in #1297

Will you adding support for FIPS 140-2 standards? Please note: I am NOT asking about FIPS validation with regards to the containers themselves - I understand that is handled, at least technically speaking, during the build phase of the container image. FIPS 140-2 compliance for other OSes (Amazon Linux 2, RHEL) configures the crypto module to operate in compliance (proper crypto algorithms, entropy, etc) - these would be done in container image; however, there are other changes at the kernel-level that occur as well. I don't believe full on validation is in scope for Bottlerocket since crypto modules themselves get validated (not necessarily an OS) but perhaps guidance and/or documentation on meeting compliance requirements would be helpful.

[zmrow]

Thanks for opening this! We'll look into it.

[technotaff-nbs]

A lack of compliance statement for CIS is preventing our company (and i'm sure many others) from adopting Bottlerocket.

[JohnTrevorBurke]

Any updates around FIPS compliance and the usage of Bottlerocket OS? It's a blocker for our team to be able to use for Fedramp compliant EKS nodes. Any updates would be very much appreciated!

[vennemp]

Hey @JohnTrevorBurke i believe I just added you in linked in. I may be able to help.

[geekmuse]

In the same boat here re: FedRAMP. Any info/updates here?

[FringeCase]

Bumping @geekmuse comment; FedRAMP path

[walesb]

Hi Team, I have the same query here, I just want to confirm to know if AWS BottleRocket AMI's are FIPS-140-2 compliant by default or not?

Looking forward to your reply.

[stevehipwell]

Is there any news on this?

[bcressey]

FIPS compliance is our second most requested feature, behind CIS (which is in progress), and I'm planning to focus on it once the CIS benchmark is complete.

There are (at least) two interpretations of what this means, and different auditors may have different takes:

1. Bottlerocket's kernel uses compliant algorithms when fips=1 is passed on the kernel command line, and its userspace tries to arrange for containers to do the same (by mounting in /etc/system-fips and such).
2. In addition to the above, Bottlerocket's kernel and userspace are themselves FIPS certified, meaning that they only use validated, FIPS-compliant crypto.

The first should be somewhat lower hanging fruit, and it's how I understand @vennemp's initial request. But I've also heard interest in the second, stronger version. That's a bigger lift because it likely means switching away from rustls and Go's crypto library, using OpenSSL for everything instead, and adopting something like Go FIPS rather than the regular Go toolchain.

[bputt-e]

Is this planned to be done in early 2023?

[stmcginnis]

Hi @bputt-e - the hope is someone can work on this in 2023, but there is no definite time set at this point.

[oded-dd]

Hey. Any update on that? Same path of "Fedramp"

[voidlily]

Also looking for this for fedramp purposes

[amandagalligan]

Hi @stmcginnis wanted to check in on updates or any rough timeline for Bottlerocket FIPS 140-2 support in 2023?

[stmcginnis]

Hi @amandagalligan! No timeline yet, but FIPS compliance is an important thing on the roadmap. Unfortunately it will take some time to get everything in place. But there is work being done towards that end. I just can't give any kind of time estimate at this point, unfortunately.

[bcressey]

One fairly widespread example of a container dependency on host crypto is the kernel PRNG, accessible via /dev/random, /dev/urandom, or the getrandom() syscall. Pretty much every standard library or runtime for popular languages will default to these mechanisms for getting random bytes on Linux.

[bcressey]

I've owed this issue a more substantial update for a while so here goes.

In the current plan, FIPS for Bottlerocket requires two modules:

• AWS-LC Cryptographic module (in "Coordination" state)
• Amazon Linux 2023 Kernel Crypto API Cryptographic Module (in "Implementation Under Test" state)

NIST doesn't share the expected timeline but based on historical data scraped from those links, it looks like validation may take up to 650 days - meaning December 2024 before the kernel will be validated.

The remaining implementation work required is:

• Create a golang build that swaps in aws-lc for the Go crypto module, using the upstream GOEXPERIMENT=boringcrypto.
• Figure out how to plug in aws-lc-rs instead of ring at build time.
• Adjust templates and helpers to automatically use FIPS API endpoints instead of non-FIPS

Given that FIPS entails some functional and cryptographic trade-offs - no Wireguard support in the kernel, none of the newer, well-regarded non-FIPS algorithms, more overhead in crypto operations in Go, etc - my plan is to release a parallel set of FIPS variants.

For example:

• aws-k8s-1.28 would be non-FIPS, use native Rust and Go crypto, and provide the full set of kernel functionality, same as today.
• aws-k8s-1.28-fips would be FIPS enabled out of the box, use aws-lc backed crypto, have fips=1 on the kernel command line, mount /etc/system-fips into containers, and use FIPS API endpoints for communicating with EKS, ECR, and EC2.

[bcressey]

Some exciting news from last week - AWS-LC is now FIPS 140-3 certified - so it's just the 6.1 kernel that's left.

[bcressey]

The 6.1 kernel has reached the interim validation stage - so we're getting closer.

[bcressey]

For those interested in FIPS AMIs - I'm curious whether there's any use case for publishing them in regions other than the ones listed here:

• us-east-1
• us-east-2
• us-west-1
• us-west-2
• us-gov-east-1
• us-gov-west-1
• ca-central-1
• ca-west-1

@arnaldo2792 is evaluating whether to set AWS_USE_FIPS_ENDPOINT=true to force the use of FIPS endpoints, which has this property:

If this setting is enabled and a FIPS endpoint does not exist for the service in your AWS Region, the AWS call may fail.

That in turn would make the FIPS AMIs rather useless outside of regions where EC2, EKS, ECS, and ECR FIPS endpoints were available. (edit to add: this apparently includes the Canada regions also, except for EC2).

[jxn]

@arnaldo2792 is evaluating whether to set AWS_USE_FIPS_ENDPOINT=true to force the use of FIPS endpoints, which has this property:

If this setting is enabled and a FIPS endpoint does not exist for the service in your AWS Region, the AWS call may fail.

That in turn would make the FIPS AMIs rather useless outside of regions where EC2, EKS, ECS, and ECR FIPS endpoints were available.

To be honest, I have use-cases for both behaviors. There are definitely instances where, if a FIPS endpoint is not available in a region or even for a service as a whole, I strongly prefer a failure and error message to a silent failover. This, again, would be best if it works this way for both a region that does not support FIPS endpoints AND for a service that does not support it at all. (Ideally, this would be the same as AWS_USE_DUALSTACK_ENDPOINT. And, if I specify both USE_FIPS_ENDPOINT AND USE_DUALSTACK_ENDPOINT to true, I'd hope to fail with an error if said endpoint doesn't support BOTH FIPS and ipv6). AWS_REQUIRE_FIPS_ENDPOINT would seem like a fair new var if you don't want to break configurations that currently exist.

I also have a number of container resources and IAC which are only different in that one should use FIPS endpoints and the other is agnostic. For these items, it's helpful that USE_FIPS_ENDPOINT=true falls back to a non-fips endpoint, so I don't have to build two different versions for both scenarios. If both behaviors were available, this would be helpful for when I have to deal with a mix of services in the same context, some of which support FIPS and some of which do not. I'd consider this the less important, as I can work around it ... having to know every service and region with support and buildling checks into the software and IAC

[stevehipwell]

We have a requirement to run FIPS nodes in regions without FIPS ECR endpoints (e.g. Canada). I think defaulting to the FIPS ECR endpoint for the FIPS version across the board would be best, with the ability to change the ECR endpoint to non-FIPS for the regions where it's not available. This would support all use cases but make the use of a non-FIPS ECR endpoint require explicit configuration.

[vennemp]

The 6.1 kernel has reached the interim validation stage - so we're getting closer.

@bcressey
Maybe I'm missing something in the Security Policy but is there a reason why AWS is being so restrictive in what it defines in the Vendor Affirmed Operational Environments for this module? I've read the security policy a couple times now, and I cannot find anything that mitigates this issue.

Check page 9 of CMVP #4808.

For Amazon Linux 2023 and Bottle Rocket you are basically requiring us to use c7g.metal or c6i.metal instances - since those are what you tested in or are affirming. I get you want to be explicit in the testing operational environment - but other Vendors typically use the Vendor affirmed Operational environment to be more open so more configurations are covered by the certificate.

For example:
Boring Crypto CMVP Policy basically lists any version of Linux as an affirmed operational environment.

Trend Micro's Policy basically does the same. They are explicit in the testing environment and more open on vendor affirmed operational environments.

While I admit it is unlikely an auditor will enforce this level of granularity - they are well within their rights to mark it as a finding for not using FIPS 140-3 Validated Cryptography if the configuration does not match the policy. I've heard from other AWS employees that in internal documentation regarding previously validated modules, the modules were affirmed to work on any EC2 instance type - but it was NOT listed in the CMVP security policy it was only mentioned in internal documentation. FWIW, I believe it was this module Why omit this information from the security policy - since that is what auditors refer to if they have questions?

[ginglis13]

Hi @stevehipwell, I wanted to clarify on your suggested approach to fulfill your requirement:

We have a requirement to run FIPS nodes in regions without FIPS ECR endpoints (e.g. Canada). I think defaulting to the FIPS ECR endpoint for the FIPS version across the board would be best, with the ability to change the ECR endpoint to non-FIPS for the regions where it's not available. This would support all use cases but make the use of a non-FIPS ECR endpoint require explicit configuration.

For Bottlerocket's host containers, and host-ctr, I have opened bottlerocket-os/bottlerocket-core-kit#204. This will add functionality to host-ctr such that it resolve FIPS ECR endpoints if FIPS is supported in the region specified by the image URI, and it will set the URIs for host containers when run in FIPS mode.

We're also planning to set use_fips_endpoint=true in AWS config via Bottlerocket's settings.aws.config by default for FIPS variants, which will address your requirement to use FIPS AWS service endpoints by default, when available.

Bottlerocket will default to the FIPS service endpoint for AWS services in FIPS variants, and be intentional about FIPS ECR endpoints in host containers / host-ctr. Will that still work for your requirements?

[stevehipwell]

@ginglis13 that looks to be a very comprehensive solution for our requirements.

[adamcrosby]

For those interested in FIPS AMIs - I'm curious whether there's any use case for publishing them in regions other than the ones listed here:

• us-east-1
• us-east-2
• us-west-1
• us-west-2
• us-gov-east-1
• us-gov-west-1
• ca-central-1
• ca-west-1

us-iso* and the other 'special' partitions would be super useful.