Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Signed Migrations #905

Closed
webern opened this issue Apr 14, 2020 · 2 comments · Fixed by #930
Closed

RFC: Signed Migrations #905

webern opened this issue Apr 14, 2020 · 2 comments · Fixed by #930
Assignees

Comments

@webern
Copy link
Contributor

webern commented Apr 14, 2020

Feature: Signed Migrations

Summary

During an upgrade or downgrade sequence, migrator runs migrations that have not been cryptographically verified since the prior boot. We propose a change to Bottlerocket that would improve security by verifying migrations using The Update Framework (TUF) at the time that they are executed. Originally documented in #91, the current issue serves as an RFC for the proposed solution. (Formalizing an RFC process is #36.)

Motivation

Security is a primary tenet of Bottlerocket. Signed Migrations will make Bottlerocket more secure by cryptographically verifying migrations at boot time.

Background

When a Bottlerocket upgrade or downgrade takes place, if the data store needs to be changed, executable binaries called migrations are used to alter the data store.

updog determines what migrations will be needed based on data found in the manifest.json file (the manifest) stored in the TUF repo. updog downloads the needed migrations and uncompresses them into /var/lib/bottlerocket-migrations Upon boot, a program named migrator runs these binaries to make the necessary alterations to the data store.

updog uses TUF (via the tough library), which means that the manifest and migrations are cryptographically verified by updog. migrator, on the other hand, lists the contents of /var/lib/bottlerocket-migrations to discover migrations and runs them based on a file naming convention.

This introduces a vulnerability because migrator is not aware if the migrations have been manipulated since the time that updog verified them.

Proposed Changes

We propose a change such that migrator will no longer discover and run unverified binaries. Instead migrator will use tough and the manifest to determine which migrations are needed and load them into memory thereby ensuring that migrations are verified.

Proposed migration process

  • updog uses tough to obtain the manifest (as before).
  • updog uses the manifest to determine which migrations are needed.
  • updog saves
    • the manifest.
    • the migrations in compressed format and with tuf-friendly filenames.
    • The tuf repo metadata needed for migrator to verify them. (timestamp.json, targets.json, snapshot.json, and all versions of root.json.)
  • A reboot occurs into the new version of Bottlerocket (as before).
  • migrator uses tough to load the manifest.
  • migrator uses the manifest to determine which migrations are needed.
  • migrator loads the migrations via tough which ensures they are cryptographically secure.
  • migrator decompresses the migrations.
  • migrator runs the migrations from memory using pentacle.

Shared Code

Because tough and migrator will both use the manifest to determine which migrations are needed, the code that does this will need to be shared by both updog and migrator.

root.json

migrator works the same way during a downgrade as it does during an upgrade with the following caveat: once booted into the older partition, migrator may be referencing a root.json file older than the one used to sign the manifest and migrations. In that case, migrator relies on the fact that tough has all the intermediate root.json files needed to work properly.

Drawbacks, Rationale and Alternatives

Backward Compatibility

The proposal does not handle the case of downgrading into a version of Bottlerocket that existed prior to the roll-out of signed migrations. We do not take breaking of downgrades lightly, however we believe this change is needed to better secure Bottlerocket.

The issue of backward compatibility could arise during a failed upgrade boot when Bottlerocket would fallback to the previous partition. The backward compatibility issue could also arise during an explicit attempt to downgrade. In either case, if the manner in which migrations have been saved by updog is different from what an old version of migrator expects, the boot will fail.

To mitigate this issue we could have updog continue to write the uncompressed migrations where old versions of migrator expect them (in addition to all the files described in the changes above). Then either an old migrator version or a new one can find what it needs. A new migrator version would only run signed migrations and an old version would only run the unsigned migrations.

However, leaving this downgrade path in place also leaves in place the possibility of unsigned binaries running at boot. In other words, to completely prevent unsigned binaries from running at boot we need to prevent hosts from being downgraded to an old version of migrator, and thus we propose a change that breaks downgrades to prevent it.

Additional migrator Complexity

Because migrator is crucial to the Bottlerocket boot process, we want to keep it as simple as possible. This proposal increases the complexity of migrator by adding tough, decompression, and usage of the manifest.

LZ4

We considered retaining the responsibility for uncompressing the binaries in updog. However this proves to be unwieldy as we would need to add the uncompressed migration checksums into the TUF repo. This is far more complicated than adding an LZ4 deflate routine to migrator.

The Manifest

We considered having updog write out a list of migrations to a file, or having migrator rely on the list of files that updog has written (as it does now) in order to avoid having migrator run logic against the manifest. We rejected these ideas because they leave a vulnerability where an attacker could affect the list or order of migrations to be run.

TUF Timestamps

TUF uses expiration timestamps to limit a replay attack window. However, in the case of Signed Migrations it is possible for a TUF repo timestamp to expire between the time that updog downloads the information and the time the migrator runs. In this case, we would rather ignore the timestamp expiration than fail the boot.

The rationale is that the TUF repo timestamps serve their purpose when updog downloads the information (updog would fail on an expired timestamp). Once updog has downloaded the data, we explicitly want it to remain valid until migrator has had a chance to use it.

We propose to have migrator ignore TUF timestamps. Using the TUF framework in an ‘ignore timestamps’ mode will serve our use case without affecting other uses of tough.

Open Questions

  • Is the decision for a breaking change the right one?
  • Should TUF repo metadata be re-usable on the system? If so, should we store is at /var/lib/bottlerocket-metadata, for example, instead of e.g. /var/lib/bottlerocket-migrations/metadata?
@bcressey
Copy link
Contributor

bcressey commented Apr 20, 2020

I'd like to get the API labeling part of #764 done before this lands, so we can ensure the local repository and the migrations have the right labels on disk. That would act as a further mitigation for the replay scenario.

EDIT: having thought about it more, I don't think we want updog to carry an API label, but we could usefully give it a label of its own to better protect this content.

@webern
Copy link
Contributor Author

webern commented Jun 16, 2020

Edit: We have decided to support a compatibility migration path. We plan to release a version that can support both signed and unsigned migrations (a 'pivot' version). After that, a version that does not support unsigned migrations will be released, and the repo URL will be changed to ensure no instances can skip the 'pivot' version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants