RFD 17: Supply chain security hardening

[klizhentas]

No description provided.

[webvictim]

I'm not sure we can do this, I think it's automatic on their end. We can definitely publish source artifacts with checksums though.

[klizhentas]

We can remove the tar.gz files after we create a release and update with our own. Then github will not alter and generate the tarballs.

[a-palchikov]

Suggested change

	Once pull request from external contributor has been approved, C/CD can amend the commit, sign it using our CI/CD key
	Once pull request from external contributor has been approved, CI/CD can amend the commit, sign it using our CI/CD key

[webvictim]

Suggested change

	Establish Teleport-specific repository of supported public keys used by core team developers and have an
	Establish Teleport-specific repository of supported GPG public keys used by core team developers and have an

[webvictim]

Suggested change

	To prevent supply chain attacks take control over all software distribution channels for Teleport and
	To prevent supply chain attacks, take control over all software distribution channels for Teleport and

[webvictim]

Suggested change

	Have a cryptographicaly-verifiable audit trail of all software changes and releases
	Have a cryptographically verifiable audit trail of all software changes and releases,

[webvictim]

If we used self-hosted Gitlab, it looks like we could use SSH certificates issued by Teleport's user CA to authenticate users to the repo (see https://docs.gitlab.com/ee/administration/operations/ssh_certificates.html). I believe BitBucket also supports SSH CAs.

We'd probably still need to use a GPG key to sign commits, but it this would be an extra security step and would also prevent users from needing to upload their own public keys. Granting/revoking access to the repo would become an easier process (given that we can't just use HTTPS authentication because our repos use submodules)

[sskousen]

We're doing exactly this and it works pretty well, though there are a couple minor gotchas. Happy to give more details in slack or something more private than github. :)

[knisbet]

I left a few thoughts since you asked for review.

[knisbet]

I suspect this problem statement might be a bit overly broad. Also I believe there is alot more to supply chain security than this spec really covers.

For example brandon phillips I think is on atleast his second attempt at setting up an asset transparency system: https://www.transparencylog.com/ although I still have lots of concerns about this approach. So might be too immature but would be an important capability as part of any customer verification of provided assets (basically a customer can verify they've been provided the same assets as some trusted third party).

One of the frequenters at CNCF Sig Security also did a presentation about https://in-toto.io/ awhile back which is some tooling and specification targetting software supply chain, but I haven't dove in too deeply.

For docker there is also content trust that can be opted into, but I have no idea if it's widely used and definitely introduces new key handling requirements.

[awly]

Agree, this spec addresses our CI and asset hosting dependencies.
It does not cover code dependencies. The latter is largely an unsolved problem in software engineering today.
It's probably OK to focus on CI and assets for now, but the title and description should reflect that.

[knisbet]

Sorry, I don't understand how this works as described and why you can't use github as a source of truth?

[klizhentas]

I have been worried about their ability to alter content and slow response times. It seems that they have modified the content on the generated tarball and our ticket is waiting for 3+ days for any response. If there were a real a security incident, that will put is in dangerous territory.

To mitigate that, I was thinking to keep a read only mirror that is used by folks to fork the content, submit PRs, but it won't be used by Teleport to source any content for builds.

To make this happen, we can have a separate flow for processing pull requests. Folks can still have branches and PRs, but merges on protected branches will be happening outside the github flow by our infrastructure.

May be I'm overthinking this.

[knisbet]

Well would GPG signing all commits that make it to any release basically prevent githubs ability to modify any of the sources? They basically become a repository, but wouldn't have any ability to change anything without breaking signatures.

For the generated assets, I'm not sure how keeping a duplicate mirror of git somewhere else helps with that problem though. If it's more for backup, it could just be any git client that's keeping a copy instead of having to piece things together from a devs machine.

[klizhentas]

Isn't it the same problem as with log integrity we had? Any actor can remove or rearrange commits and update the branches, unless you sign the entire history?. I'd need to dig deeper to see how github solves this problem with signatures, so maybe I'm wrong.

[knisbet]

I don't believe it's exactly the same, but I too don't have as good a mental model as needed to be confident in the git properties.

It's possible a signed commit in git also represents the history in some cryptographic secure way, such as chaining the hashes of the commit history. If that's the case, then an adversary hosting the repository has some fairly limited options. They can't change history because that breaks the cryptograghic hashes leading up to the signed commit (if that's a property, I don't know for sure). If they can edit history it might not be a big deal if tools like drone just won't build a commit that isn't signed.

This leaves some fairly limited attack vectors for who's hosting the repo, because the security model is in the data, not the host. So the host could selectively not deliver the last commit to clients, but wouldn't be able to otherwise modify history or files. This would presumably be detectable in a couple of ways, if someone noticed they were missing commits, or there were diverging histories that could no longer become in sync. However it might be a little more subtle, if the release engineer and merge engineer were the same person they may need to notice missing commits in the history.

Where this gets especially brutal for the host though, is say tools like drone would only work with a signed commit, and if the host like github selectively tried to not return the tag... this would be very detectable as the release engineer wouldn't be able to tell drone to build assets that github as withheld.

There's lots of ifs and gaps in understanding though and assumptions that we would want to verify. But I believe this is the model signed commits are meant to protect against, as it is the security model developed for distributed development of the linux kernel, and the trust is totally minimized on the host and transfer of the repository itself.

[knisbet]

I'm not saying don't sign the tag, why should drone specifically be the signer? Why not the release engineer for example?

[klizhentas]

One idea is that before signing the tag, Drone could verify that signatures of all the commits in the release are from trusted public keys.

[knisbet]

I would expect no matter who is signing the release tag, that they should be responsible for verifying the contents.

[klizhentas]

That's fair. However, devs could forget to verify the commits or make a mistake.

I thought we might automate the process to avoid such mistakes. If ask Drone to do verification as a part of release process before creating a signed tag, we can state the invariant: that tag is only created if all commits since last signed tag are from trusted parties.

[knisbet]

Well, that's where having good rock solid processes and tooling come into play. If there is a role within the team for doing release engineering, with the tools required to verify signatures (and even contents) of releases, with checklists and other related disciplines to reduce the risk of someone making a mistake. It might also possible with release engineers to eventually get to an m:n model, with multiple individuals making checks on each other with multiple signatures required (I don't know how this works in git).

From a threat modelling perspective it seems like we're trying to solve a problem statement something along the lines of:

1. Asset release tools/systems should have a way to validate the sources provided to them are from designated individuals.
2. An individual (internal or external) should have a way to copy the repository from public hosts and inspect / validate the tag they received is published by Teleport.
3. Both of the above to be delivered by not trusting the repository host.

By automating this through drone, to me implies a release process that looks something like:

1. A release engineer tells drone we need a new release on a specific branch.
   2. Note: I'm still a bit unclear on this, the release engineer tells drone to tag a specific commit by hash? If the specification is by branch, a release engineer will need to later validate that drone did tag the latest commit.
2. Drone checks out the code from the public host, verifies signatures of individual commits, and publishes the signed tag back to the repository host(s).
3. (Optional) Based on above, the release engineer needs to validate the repository host didn't drop the commit when provided to drone.
4. Drone still needs to build the assets, so based on the new tag, drone checks out the code, verifies the signature against it's own key, and proceeds to run normal build and asset publishing to various locations we make the release assets available.

To me having drone sign that tags implies a number of implications:

1. Drone will need additional key handling, with all the normal key handling constraints, in order to tag and publish back to the public repo.
   1. How are those keys stored, backed up, etc.
   2. Who has access to those keys, what happens when an employee leaves, can the keys even be revoked and rotated.
2. Drone itself becomes an attack vector, is there a way to introduce additional configuration to trick drone into using the signing secret in some way it's not supposed to (release it outright, signing an asset or tag it shouldn't). This might be the situation of other secrets as part of the release process, but for this spec I think it's still a consideration from a surface area perspective.
   1. A breach of drones secrets also creates a property, that a particular validator could be the target by selectively changed secrets, especially if we consider a host like github not to be secure. So the signed tag x could be delivered to drone for build verification while all other users receive an alternate tag. I suspect this would be fairly difficult to detect based on what's described.
3. Drone needs to be kept in sync with all active developers for the company
   1. How should drone react to a developer who has left, has merged commits, that haven't been tagged yet?
4. How do we communicate and configure the public keys of the developers securely?
5. How do we communicate externally the public keys for an individual who wants to checkout and verify a tag?
6. I suspect all developers must be handling their GPG keys securely, as drone will trust a merged commit by any developer???
7. Who has access to drone and the relevant secrets? etc.

[knisbet]

What does drone do as verification?

[webvictim]

Presumably Drone would validate that each commit is signed by one of the authorised GPG keys which we maintain a list of.

[klizhentas]

Drone should only allow merges with commits signed by trusted public keys.

[knisbet]

So all this key handling presumably comes into scope and should be specified should it not?

[knisbet]

Sorry I don't get this, what is the source of truth drone will be trusting to choose to amend and sign commits?

[klizhentas]

Good point. May be engineers should review and sign-off the commits then to let Drone merge them?

[knisbet]

Sorry, it's not clear to me what this does or the security properties that change?

[webvictim]

The context is this issue where it appears that the behaviour of Github's automatic source tarball generation code has changed, resulting in the SHA256 hashes of the source archives changing without commits from our end. Controlling the process of generating the source tarballs ourselves removes a dependency on Github's code (which we have no control of)

[knisbet]

resulting in the SHA256 hashes of the source archives changing without commits from our end

Are we actually sure about this, or just assuming it's the case because we don't know why it changed on that particular asset. And don't have a copy of the old asset to compare against, but presumably compared to the sources and the asset appears legitimate? And we're sure the asset actually did change, and not that the reporter accidentally hashed a partial download or something.

I think my advice would be if we're looking at signing the commits anyways, is to shallow copy of the repo and validate the commits with GPG. Because otherwise we're introducing new key handling to sign source files, and methods to communicate those keys to customers who want to validate signatures, and then need to be concerned with rotating those secrets, storing them, etc, etc. Not that GPG signed commits don't have all those same problems, it's just double the work to do it for the sources in git, and to do it again with CI/CD and the assets tarball.

Although I suppose it would work to just have drone shallow copy the tag and overwrite the github generates releases, so all the signature are still in place. I don't know if you can fully validate the git history with just a shallow copy though. No new key handling or security.

[klizhentas]

I downloaded the archive, unpacked it and the contents are a bit different:

$ diff -r teleport-5.0.2 teleport

diff -r teleport-5.0.2/vendor/k8s.io/client-go/pkg/version/base.go teleport/vendor/k8s.io/client-go/pkg/version/base.go
58,59c58,59
< gitVersion   string = "v0.0.0-master+17679dcec2"
< gitCommit    string = "17679dcec24796955b96485d5922b6d695735a3b" // sha1 from git, output of $(git rev-parse HEAD)
---
> gitVersion   string = "v0.0.0-master+$Format:%h$"
> gitCommit    string = "$Format:%H$" // sha1 from git, output of $(git rev-parse HEAD)

It may as well be user/CI/CD or some other error. I'm not as concerned about this particular use-case though. I'm concerned about the fact that we have no way of verifying the integrity of the sources against the git commits.

[webvictim]

The fact that it's happened for many of our source archives at once (we checked 4.4.4, 4.4.5, 5.0.0, 5.0.1 and 5.0.2 - all are affected) points to an upstream issue.

There are just two differences in the tarball versus the tag checkout - %h and %H in a vendor file were replaced with actual commit hashes. It looks a lot like a bug.

[awly]

I'll write an RFD for RFD format.
For now, let's keep it consistent to existing RFDs.

Suggested change

	# Supply Chain Hardening
	---
	authors: Alexander Klizhentas ([email protected])
	state: draft
	---
	# RFD 17 - Supply Chain Hardening

[awly]

The idea is to replace N 3rd parties (quay, homebrew, etc) with 1 3rd party (AWS) for easier auditing, right?

[awly]

Agree, this spec addresses our CI and asset hosting dependencies.
It does not cover code dependencies. The latter is largely an unsolved problem in software engineering today.
It's probably OK to focus on CI and assets for now, but the title and description should reflect that.

[awly]

This spec could benefit from a few examples: a supply chain compromise attempt and which control stops it.

[klizhentas]

@webvictim another idea to consider - print a warning if teleport was not build using our official supply chain build process to protect from possible docker images or hacked systems on brew. This will require a tool hosted by us to verify the signature of the build as well.

[awly]

any check that a binary performs itself can be disabled by an attacker, but it does raise the difficulty bar a bit.
plus, there are legitimate cases where teleport binaries could be compiled outside of our infra (e.g. using nix).

we could consider signing all release artifacts (we only sign .rpm and .deb right now) and instructing security-concious users to verify the signatures.

[webvictim]

We also sign our tsh binary and .pkg package on MacOS. We will sign the teleport binary and .pkg too once #3158/#5595 is done.

Homebrew is another case that comes to mind where the binaries would be compiled and distributed outside of our infrastructure. They do not allow binary artifacts to be used; all formulas must contain a recipe which can build the appropriate binaries from source - Homebrew then package up the built binaries as "bottles" and distribute them via Bintray.

While the idea of having the Teleport binary check its signature against a central server is a reasonable idea in principle (and is actually what MacOS now does with all signed binaries) it can only really be a best-effort solution - see the early details in Cloudflare's summary of how they implemented OCSP stapling. In particular a quote from a blog post by Adam Langley stands out to me: "So soft-fail revocation checks are like a seat-belt that snaps when you crash. Even though it works 99% of the time, it's worthless because it only works when you don't need it."

It'd be worse for us because the chances of people having Teleport installed on machines which don't have outbound internet connectivity is quite high.

[awly]

@klizhentas @knisbet can we add a threat model to this RFD?
There's an interesting mix of solutions, would be good to map them to actual problems.

[knisbet]

@klizhentas @knisbet can we add a threat model to this RFD?
There's an interesting mix of solutions, would be good to map them to actual problems.

Yea I think this is sort of what I was getting at with some of the back and fourth, it's not clear to me what I like to call problem statements are. So I would sort of supersede this with some targeted designs for specific problem statements.

I think based on the back and fourth I can see a couple threats this might be trying to address that I would try and break down into separate but related problems:

• Limiting the trust in the repository host.
  • This is all the stuff about how we sign code, how CI/CD authenticates the code it has been given is trustworthy and should be built, etc. IE don't trust github to produce the correct code.
• Supply chain of upstream binary assets
  • I imagine this is mostly docker base images and related. What protection mechanisms do we have in place that we're pulling assets from verified sources, ensuring they're void of known vulnerabilities, etc.
  • And specific to teleport, teleport cloud has the same set of problems but probably different solutions.
• Supply chain of upstream source assets
  • Similar to binary, but for vendored sources, or go cache, etc, etc.
• Downstream supply chain (docker)
  • Probably just signing images for docker content trust, and publishing images for anyone who opts in. Could also just be publishing sha hashes.
• Downstream supply chain (rpm/apt/etc)
  • All the signing and key handling for common packaging systems.
  • Probably something like minisign for any other type of assets for an opt-in verification: https://github.com/jedisct1/minisign
• Asset Transparency
  • This is very raw, but if we want to be at the forefront we should be at the forefront. Both for upstream and downstream, use a methodology to ensure assets also exist in a public register for transparency protections.

Almost all of these have key and secret handling requirements, so maybe a common RFD for how to tackle all the common key handling stuff, who has access, how secrets are rotated, etc, etc.

[awly]

@knisbet thanks, that's a great breakdown.
I'd also add "developer identity" to the list - how do we prevent an attacker from stealing @klizhentas's (admin-level) SSH keys for github and pushing malicious commits.

[knisbet]

@knisbet thanks, that's a great breakdown.
I'd also add "developer identity" to the list - how do we prevent an attacker from stealing @klizhentas's (admin-level) SSH keys for github and pushing malicious commits.

Yea great point, I forgot about this one although it might sort of be covered by not trusting github based on how that topic gets approached. Some of the discussion around release engineering, and how many checks are done on the release engineering side, which could be signatures from multiple release engineers for example before automation will make a release.

[knisbet]

Came across this today, Linux Foundation project targeting supply chain security: https://sigstore.dev/what_is_sigstore/