Skip to content

enhancements/x509-trust: Propose a new enhancement #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wking wants to merge 1 commit into from
Closed

enhancements/x509-trust: Propose a new enhancement #115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions enhancements/proxy/global-cluster-egress-proxy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ approvers:
- "@eparis"
- "@knobunc"
creation-date: 2019-10-04
last-updated: 2019-10-29
last-updated: 2019-11-24
status: implemented
see-also:
- "https://github.com/openshift/enhancements/pull/22"
Expand Down Expand Up @@ -150,7 +150,7 @@ configmap after confirming the validation endpoints can be accessed using the ne

Additional behaviors:

* configmaps labeled with `config.openshift.io/inject-trusted-cabundle: "true"` will have the current
* configmaps labeled with `config.openshift.io/inject-proxy-cabundle: "true"` will have the current
set of additional CAs injected into them by logic in the cluster network operator.
* deployments with the `config.openshift.io/inject-proxy: <container-name>` will get the current proxy
environment variables injected (HTTP_PROXY, HTTPS_PROXY, NO_PROXY) by the cluster version operator.
Expand Down
226 changes: 226 additions & 0 deletions enhancements/security/x509-trust.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
---
title: x.509-trust
authors:
- "@wking"
reviewers:
- "@abhinavdahiya"
- "@bparees"
- "@dhansen"
- "@sdodson"
approvers:
- TBD
creation-date: 2019-11-14
last-updated: 2019-11-24
status: provisional
see-also:
- "/enhancements/automated-service-ca-rotation.md"
- "/enhancements/kube-apiserver/certificates.md"
- "/enhancements/kube-apiserver/tls-config.md"
- "/enhancements/proxy/global-cluster-egress-proxy.md"
---

# X.509 Trust

This enhancement provides a big-picture overview of X.509 trust management for cluster components.
This enhancement provides for both cluster-scoped defaults and per-component overrides.

## Release Signoff Checklist

- [ ] Enhancement is `implementable`
- [ ] Design details are appropriately documented from clear requirements
- [ ] Test plan is defined
- [ ] Graduation criteria for dev preview, tech preview, GA
- [ ] User-facing documentation is created in [openshift/docs]

## Summary

Several cluster components may make TLS requests to external services.
For example, [the cluster-version operator][cluster-version-operator] may request available updates, [the machine-API operator][machine-api-operator] may call platform APIs to manage machines, [CRI-O][cri-o] may contact container image registries when downloading images for pods, etc.
Cluster components may also perform other X.509 verification besides TLS requests.
For example, user mail processing components may perform [S/MIME][smime] verification.
This enhancement provides a framework for managing X.509 trust for these situations.

Currently the default trust for CRI-O, the kubelet's cloud provider, and other node-namespace egress is the node's X.509 certificate authority (CA) store, which depends on the operating system ([Red Hat Enterprise Linux CoreOS (RHCOS)][rhcos], etc.) and any subsequent configuration ([the machine-config operator][machine-config-operator], etc.).
The default trust for containers like the machine-API operator is that container's CA store, which depends on the base layer ([Red Hat Universal Base Image (UBI)][ubi], etc.).
Many external components will have X.509 certificates signed by one of those default CAs, but this is not always the case.

There may also be internal communication between cluster components that relies on X.509 trust but lacks a custom trust configuration mechanism.
The framework from this enhancement can also be used to configure trust for that communication as well.

This enhancement provides a mechanism for configuring additional or alternative trust bundles for all cluster services that do not already have existing trust configuration mechanisms.

## Motivation

### Goals

Administrators will have a single location to configure a default additional trust bundle to be used by all cluster components.
They will also be able to specify whether this default is to be use in addition to, or instead of, the component's default trust bundle.

Administrators will have per-component configuration locations to adjust the cluster-scoped default.
Copy link
Contributor

@smarterclayton smarterclayton Nov 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

Copy link
Member Author

@wking wking Nov 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

Because you don't need to have a global trust store for the whole cluster. You get better isolation against compromised CAs if each component only trusts the CAs it needs to trust (e.g. the CVO scoping I sketched out here). But while that scoping is nice for the paranoid, it's also a pain, which is why I think we also want to make it easy for folks to mix and match cluster-scoped defaults and per-component overrides to match their security stance.

As for the cluster-scoped default, they will be able to specify whether the per-component trust bundle is to be used in addition to, or instead of, the cluster-scoped default.

### Non-Goals

This enhancement does not address components with existing X.509 trust-distribution mechanisms.
Trust for those requests is covered by [the Kubernetes API-server Certificates enhancement](../kube-apiserver/certificates.md) and [the Automated Service CA Rotation enhancement](../automated-service-ca-rotation.md).

This enhancement does not address configuration for non-CA TLS settings like ciphers and allowed TLS versions.
[The ingress TLS configuration enhancement](../kube-apiserver/tls-config.md) covers those settings for cluster-hosted services.
Outgoing TLS probably needs similar configuration, but that configuration is decoupled from X.509 trust and should be addressed in a separate enhancement.

## Proposal

### Cluster-scoped Default Trust

There will be a ConfigMap (FIXME: or should this be a new CRD under openshift/api?) named `default-ca-bundle` in the `openshift-config` namespace.
The ConfigMap will have the following keys:

* `ca-bundle.crt`, containing a PEM-encoded X.509 certificate bundle.
* `composition`, configuring how the bundle in `ca-bundle.crt` is composed with the component's default trust store.
Valid values are:
* `union`, in which case components should use the union of the `ca-bundle.crt` and their default trust store.
This is convenient for cluster administrators who need to extend the default trust store to support additional CAs, but who wish to delegate default trust store maintenance to the cluster components.
* `override`, in which case components should use only `ca-bundle.crt` and ignore their default trust store.
Copy link
Contributor

@bparees bparees Nov 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are really two places to make this decision:

  1. does the admin want to supply the total set of cluster CAs, or do they want to supply additionalCAs that should be appended to the cluster's system trust (as determined by the ubi image, basically)
  2. does the each component accept the provided cluster CA bundle as the total set of CAs it trusts, or does it append the provided set to its own system trust bundle (it will always append its own component-level CAs, if any)

We'd really only talked about (1) so far, but you seem to be talking about (2).

Copy link
Member Author

@wking wking Nov 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd really only talked about (1) so far, but you seem to be talking about (2).

This is basically "union with UBI defaults". I don't really care if that happens Kube-side (1) or container-side (2). I just assumed (2), because that's what we do now. Do you have a reason to pivot to (1)?

Copy link
Contributor

@bparees bparees Nov 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually we do it somewhat kube side today and component side.

  1. the proxy controller takes the additional CAs and appends the UBI system bundle to it
  2. the proxy controller injects that new bundle into the labeled CMs
  3. components consume that CM in whatever manner they choose. They may use it as the only set of CAs, they might union it with their own system trusts, they might union it with their own other sources of CAs, etc.

Copy link
Member Author

@wking wking Nov 19, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually we do it somewhat kube side today and component side.

Do you have a reason to preserve this, instead of pivoting to explicit append or override semantics?

  1. the proxy controller takes the additional CAs and appends the UBI system bundle to it

Is this documented anywhere? I don't see anything about it here or here.

Copy link
Contributor

@bparees bparees Nov 19, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have a reason to preserve this, instead of pivoting to explicit append or override semantics?

we aren't going to (successfully) dictate what components choose to do with the content... so i think on a component consumption perspective you will always have:

  1. some that use what is provided as the only CAs
  2. some that combine it w/ their own image system CAs
  3. some that combine it with their own component-specific CA configuration

Plus (3) will probably always make sense for some components and not for others. meaning at least (1) and (3) are likely to always need to be allowed/implemented. (1) vs (2) we can potentially declare a best practice rule about..I think we'd choose (1) there.

Is this documented anywhere? I don't see anything about it here or here.

Today the additional CA bundle is a combination of user provided CAs plus the system CAs from the network operator image. In the future adding the system trusts to the bundle should be a configurable optional so customers who want to explicitly control the trusted CAs can do so.

Copy link
Member Author

@wking wking Nov 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we aren't going to (successfully) dictate what components choose to do with the content...

We aren't? If I gave the CVO a ConfigMap with my company CA and override, and someone MitMed my upstream with a cert signed by a sloppy/compromised big CA, and the CVO accepted the connection to the attacker, I would be really mad at the CVO and file a strongly worded bug ;).

Copy link
Contributor

@bparees bparees Nov 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not just CVO, OLM operators will also consume this. which will include ISV operators, community operators, etc.

Copy link
Contributor

@smarterclayton smarterclayton Nov 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I expected a new trust bundle system to expand the existing additionalTrustedCA. It’s not clear to me why we would have a second system?

https://github.com/openshift/api/blob/master/config/v1/types_proxy.go#L65

I expect to cover everything described here. Why would we have a second one?

Or, if it’s not sufficient, how does something new interact? How do they conflict?

Copy link
Member Author

@wking wking Nov 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s not clear to me why we would have a second system?

Because you might have X.509 trust consumers that do not expect to be speaking to a proxy. This enhancement spun off from openshift/installer#2658 where I proposed setting TrustedCA on the proxy config in the absence of other proxy settings to support the network-routed-proxy case, and the pushback was around the proxy config only being used for the ask-apps-explicitly-to-use-a-proxy case. And at least the CVO currently only uses the proxy-config's TrustedCA when HTTPSProxy is set. This proposal works around that pushback by defining a higher-level default trust config (the default-ca-bundle ConfigMap) that components should be clearly falling back to even if they feel like the proxy config doesn't apply to them. So the CVO logic would be:

  1. If HTTPSProxy is set (potentially also if HTTPProxy is set to an https:// URI) and TrustedCA is set, use TrustedCA when validating TLS connections. Otherwise...
  2. If default-ca-bundle exists, use it when validating TLS connections. Otherwise...
  3. Use the container's default (UBI) trust store when validating TLS connections.

That would support use cases like:

I'm not using a proxy, but I don't want my nodes trusting all the UBI CAs. The CVO needs to talk to api.openshift.com (for Cincy, signed by Let's Encrypt) and mirror.openshift.com or storage.googleapis.com (for signatures, signed by DigiCert and Google Trust Services respectively). Limit the CVO trust to those three CAs.

Copy link
Contributor

@danehans danehans Nov 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

potentially also if HTTPProxy is set to an https:// URI

Note that if HTTPSProxy is unset, HTTPProxy is used to proxy https requests regardless of the url scheme.

This is allows cluster admistrators to take complete control of the trust store for situations where delegating trust maintenance is not permissible.
For example, a given component may not need to trust all 100+ CAs in [the default certificate store][mozilla-ca-certificates] if it only uses X.509 trust to verify TLS connections to a few services with stable CAs.
Excluding unnecessary CAs from the trust store reduces the risk of being exploited via a compromised CA.

This behavior is also the default when a component recieves an unrecognized `composition` value.

### Per-component Trust

Each component will define its own well-known ConfigMap(s) in its own namespace, with the same `ca-bundle.crt` and `composition` keys as [the cluster-scoped ConfigMap](#cluster-scoped-default-trust).
For example, the registry operator uses [`trusted-ca`][registry-configmap-name] in [the `openshift-image-registry` namespace][registry-configmap-namespace].
Copy link
Contributor

@danehans danehans Nov 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each component will define its own well-known ConfigMap(s) in its own namespace, with the same ca-bundle.crt and composition keys as the cluster-scoped ConfigMap.

Operators currently define their own configmap statically. See the registry operator as an example. If I understand correctly, operators will need to get the value of composition from the cluster-scoped configmap and create the operator-scoped configmap programatically. Is my understanding correct?

Copy link
Member Author

@wking wking Nov 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, operators will need to get the value of composition from the cluster-scoped configmap and create the operator-scoped configmap programatically

Nope, just make a create-only ConfigMap, as described here. Setting composition locally will get clobbered by the network operator using this logic if you have any of the network-operator labels set, but admins could also unset the network-operator labels and set composition manually on the per-component ConfigMap as they see fit.

Components that have distinct trust domains may define multiple ConfigMaps for each domain.
For example, there may be one ConfigMap for proxy/egress TLS, and another ConfigMap for [S/MIME][smime] verification.
To populate that trust bundle, the cluster administrator can set labels on the ConfigMap(s).
For example, [the `config.openshift.io/inject-proxy-cabundle` label](../proxy/global-cluster-egress-proxy.md#implementation-detailsnotesconstraints-optional) asks for the current proxy trust bundle.

The cluster-version operator (CVO) [merges ConfigMaps][cluster-version-operator-EnsureConfigMap] by [clobbering any manifest-defined data][cluster-version-operator-mergeMap], [labels, and annotations][cluster-version-operator-EnsureObjectMeta].
Data keys, labels, and annotations not defined in the manifest are ignored.
This means that components which decide to provide their per-component ConfigMaps via the CVO should not set labels like `config.openshift.io/inject-default-cabundle` unless they also set [the `release.openshift.io/create-only` annotation to `true`][cluster-version-operator-create-only], to allow cluster administrators to manipulate the per-component ConfigMap without the CVO stomping on their updates.
For example, the registry operator [sets `create-only`][registry-configmap-create-only].

#### Universal Base Image Containers

FIXME: UBI-specific docs for this [RHEL behavior](rhel-cert-injection) that show that you don't need to install some non-UBI RHEL stuff to make it work.

Components can mount `ca-bundle.crt` from their ConfigMap(s) under `/usr/share/pki/ca-trust-source/anchors/`.
Components are then responsible for running `update-ca-trust` (FIXME: do we need `extract` or not?) at start.
They must also run `update-ca-trust` periodically thereafter to pull in any [updates from the mounted ConfigMap][mounted-configmap-updates].
Copy link
Contributor

@bparees bparees Nov 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

running update-ca-trust requires root. not all containers can run as root.

Copy link
Member Author

@wking wking Nov 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not all containers can run as root.

What's the alternative? Holding your trust under ~? Or do you have to union keys in Kube-space and use the single-bundle /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem workaround?

Copy link
Contributor

@bparees bparees Nov 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the latter is more or less what we did for proxy.

or you need to load your CAs from additional locations, if you have that flexibility in your transport libs.

Copy link
Member

@cgwalters cgwalters Nov 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What we really want of course is user namespaces so that "as root" is really only in the container. xref https://lwn.net/Articles/796700/


Components that need a single trust bundle injected can bypass `update-ca-trust` and mount their `ca-bundle.crt` directly into `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` (FIXME: RHEL/UBI docs confirming this as a supported workflow).
Copy link

@iamemilio iamemilio Nov 20, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we automate the creation of environment variables containing the cert bundles, or create a volume using the same mechanism as proxy. It would streamline how we use the injection process.

Copy link
Member Author

@wking wking Nov 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we automate the creation of environment variables containing the cert bundles...

It's not clear to me how this would work. Can you give example Deployment YAML and talk us through this?

Copy link

@iamemilio iamemilio Nov 20, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I was just extrapolating from this excerpt from the global cluster egress policy enhancement.

deployments with the config.openshift.io/inject-proxy: will get the current proxy environment variables injected (HTTP_PROXY, HTTPS_PROXY, NO_PROXY) by the cluster version operator.

So in our case we could add config.openshift.io/add-trusted-ca-bundle: <container name> to our deployments and the cluster-version-operator would add a volume that mounts the trusted-ca-bundle configmap, and an inline command to add it to the container's trusted CAs.

Copy link
Contributor

@danehans danehans Nov 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iamemilio I'm happy that I came across your comment. I'm working on adding proxy support to the ingress operator and was considering a similar approach. I feel like we need a better solution to mounting the trust bundle than what currently exists (see image registry operator).

@wking let's use ingress operator as an example. The deployment would be updated with:

  annotations:
    config.openshift.io/add-trusted-ca-bundle: "ingress-operator" # container name to inject bundle

CVO watches deployments with the above annotation and creates a volume for the operator's trust bundle configmap and mounts the volume in the container(s) specified by the annotation. The ingress operator's deployment would look like the following after CVO finishes reconciliation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-operator
  namespace: openshift-ingress-operator
  annotations:
    config.openshift.io/add-trusted-ca-bundle: ingress-operator
spec:
<SNIP>
      containers:
        - name: ingress-operator
<SNIP>
          volumeMounts:
            - name: trusted-ca
              mountPath: /var/run/configmaps/trusted-ca/
<SNIP>
        configMap:
          name: trusted-ca
          optional: true
          items:
            - key: ca-bundle.crt
              path: tls-ca-bundle.pem

Copy link
Contributor

@bparees bparees Nov 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think you will have a hard time making the case that the CVO should do this.

the only reason the CVO took ownership of setting env vars on the operator deployments is because it is impossible for the operator to set env vars on its own deployment(the cvo would stomp away the values). The same restriction doesn't apply to the configmap volume mounting, since the operator can define its deployment with a CM volume mount, and define the CM to be mounted.

Copy link
Contributor

@danehans danehans Nov 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bparees thanks for the feedback. I'll look into having the ingress operator programmatically manage it's own trust bundle volume/volumeMount. However, having a single controller that manages this would provide consistency across operators.

Copy link

@iamemilio iamemilio Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bparees what would you suggest as a solution then? Should an operator be given the responsibility for injecting this into container deployments, or do you think we should drop it an just add it manually to all the services.

Copy link
Contributor

@bparees bparees Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iamemilio today operators that need proxy CAs in their operator deployment, configure themselves accordingly:

  1. they define a configmap that will have the CAs injected into it:
    https://github.com/openshift/cluster-image-registry-operator/blob/a0efd48f8bc2c8d18d53fc5a9e5b8515b94bc2a4/manifests/04-ca-trusted.yaml#L8

  2. they mount that configmap in their deployment:
    https://github.com/openshift/cluster-image-registry-operator/blob/a0efd48f8bc2c8d18d53fc5a9e5b8515b94bc2a4/manifests/07-operator.yaml#L93

  3. they watch the mounted content for changes and restart their process as needed:
    https://github.com/openshift/cluster-image-registry-operator/blob/a0efd48f8bc2c8d18d53fc5a9e5b8515b94bc2a4/manifests/07-operator.yaml#L62-L73

That is what i would expect here

For example, the registry operator [does this][registry-configmap-mount].

FIXME: We need to grow a way to handle `override` `composition`.

### Node Trust

[Red Hat Enterprise Linux CoreOS (RHCOS)][rhcos], and nodes more broadly, have a separate trust-management framework getting hashed out [here][machine-config-operator-trust].

### Container Image Registry Trust

FIXME: fill in how this works

### User Stories

#### Cloud-provider trust

Several cloud components such as [machine-API][machine-api-operator] providers may call platform APIs to manage cluster infrastructure.
Some cloud-providers may provide those services over HTTPS with X.509 certificates signed by CAs that are not included in default trust stores (e.g. [OpenShift behind a custom CA][openshift-custom-ca]).
By including the cloud provider's CA in the [cluster-scoped default trust bundle](#cluster-scoped-default-trust) or per-component overrides, a cluster administrator could enable cluster-managed infrastructure on those platforms.

#### Transparent proxies

FIXME

### Implementation Details/Notes/Constraints

The network operator supports copying trust bundles between ConfigMaps based on labels on the target ConfigMaps.

* [The `config.openshift.io/inject-default-cabundle` label](../proxy/global-cluster-egress-proxy.md#implementation-detailsnotesconstraints-optional) asks for the current [cluster-scoped default trust bundle](#cluster-scoped-default-trust).
* [The `config.openshift.io/inject-proxy-cabundle` label](../proxy/global-cluster-egress-proxy.md#implementation-detailsnotesconstraints-optional) asks for the current proxy trust bundle.
The network operator will fall back to [the cluster-scoped default trust bundle](#cluster-scoped-default-trust) if no proxy-specific trust bundle has been configured.

If multiple labels are set `true`, the network operator will provide the union of the requested trust bundles.

If any input ConfigMap has the `union` `composition`, the network operator will set that composition on the target ConfigMap.
If no input ConfigMap has `composition` set, the network operator will clear `composition` on the target ConfigMap.
Otherwise, the network operator will set the `override` `composition` on the target ConfigMap.

All components involved in X.509 verification should allow trust configuration by some mechanism, defaulting to the mechanism described in this enhancement if they have no more-specific mechanism.
If a component's only X.509 verification is for outgoing TLS connections, it can include only the proxy-specific trust bundle and does not need to directly request the cluster-scoped default trust bundle.

### Risks and Mitigations

Administrators could break some cluster components by setting `override` `composition` for a trust bundle which does not include an important CA (e.g. a proxy CA).
This is mitigated by the Kubernetes API server and etcd having existing X.509 trust-distribution mechanisms, so the administrator is unlikey to break the core cluster enough to prevent recovery.

Administrators could misconfigure their cluster by setting an unknown `composition` (e.g. typoing `onion` instead of `union`).
The `override` default protects from a accidental promiscuity, but accidental `override` behavior might result in the unsufficient-trust situation discussed in the previous paragraph.
We could protect against these typos by using a custom resource definition, but only ConfigMaps can be mounted into containers, so that protection would be incomplete.

## Design Details

### Test Plan

FIXME

### Graduation Criteria

FIXME

#### Removing a deprecated feature

Currently cluster components set `config.openshift.io/inject-trusted-cabundle` to receive the _proxy_ bundle, not [the cluster-scoped default trust bundle](#cluster-scoped-default-trust).
And currently nothing populates `default-ca-bundle` (the installer uses [`user-ca-bundle`][installer-configmap-name]).
So a hard cut to the approach described in this enhancement would remove configured additional trust from proxy-consuming components.
This enhancement deprecates the `config.openshift.io/inject-trusted-cabundle` label in favor of the [new labels](#implementation-detailsnotesconstraints).
The expected migration path is:

1. The network operator learns about the new labels.
`config.openshift.io/inject-trusted-cabundle` is treated as a synonym for `config.openshift.io/inject-proxy-cabundle`.
2. Existing proxy consumers migrate to `config.openshift.io/inject-proxy-cabundle`.
3. The network operator adds an alert on any ConfigMaps with the `config.openshift.io/inject-trusted-cabundle` label, to notify cluster administrators about the deprecation.
Copy link
Contributor

@smarterclayton smarterclayton Nov 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not deprecate features like this. It is forever.

This doesn't have to be an alert, it could happen through [the insights operator][insights-operator] instead.
4. After a suitable deprecation period, the `config.openshift.io/inject-trusted-cabundle` handling is removed from the network operator.

### Version Skew Strategy

FIXME

## Implementation History

None yet :).

## Drawbacks

Attempting to fix this with a single architecture is nice for consistency, but risks trying to wedge everyone into the same bucket.
Some components like [build tooling](#container-image-registry-trust) have established systems for trust distribution, and we will either leave them separate (which may be annoying for cluster adminstrators) or try to port them to the generic system (which may be annoying for their maintainers and existing users).

## Alternatives

Let every cluster component figure out their own approach for this.

[cluster-version-operator]: https://github.com/openshift/cluster-version-operator/
[cluster-version-operator-create-only]: https://github.com/openshift/cluster-version-operator/blob/751c6d0c872e05f218f01d2a9f20293b4dfcca88/docs/dev/operators.md#what-if-i-only-want-the-cvo-to-create-my-resource-but-never-update-it
[cluster-version-operator-EnsureConfigMap]: https://github.com/openshift/cluster-version-operator/blob/751c6d0c872e05f218f01d2a9f20293b4dfcca88/lib/resourcemerge/core.go#L10-L16
[cluster-version-operator-EnsureObjectMeta]: https://github.com/openshift/cluster-version-operator/blob/751c6d0c872e05f218f01d2a9f20293b4dfcca88/lib/resourcemerge/meta.go#L8-L16
[cluster-version-operator-mergeMap]: https://github.com/openshift/cluster-version-operator/blob/751c6d0c872e05f218f01d2a9f20293b4dfcca88/lib/resourcemerge/meta.go#L28-L41
[cri-o]: https://github.com/cri-o/cri-o/
[insights-operator]: https://github.com/openshift/insights-operator
[installer-configmap-name]: https://github.com/openshift/installer/blob/de10297f9158c16f36471c91a3d48be2fb2938e1/pkg/asset/manifests/additionaltrustbundleconfig.go#L26
[machine-api-operator]: https://github.com/openshift/machine-api-operator/
[machine-config-operator]: https://github.com/openshift/machine-config-operator/
[machine-config-operator-trust]: https://github.com/openshift/machine-config-operator/issues/528
[mounted-configmap-updates]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically
[mozilla-ca-certificates]: https://ccadb-public.secure.force.com/mozilla/IncludedCACertificateReport
[openshift-custom-ca]: https://bugzilla.redhat.com/show_bug.cgi?id=1608888#c0
[registry-configmap-create-only]: https://github.com/openshift/cluster-image-registry-operator/blob/75e8e851700add9c847190fb228d2e702b2af2e8/manifests/04-ca-trusted.yaml#L6
[registry-configmap-mount]: https://github.com/openshift/cluster-image-registry-operator/blob/75e8e851700add9c847190fb228d2e702b2af2e8/manifests/07-operator.yaml#L87-L97
[registry-configmap-name]: https://github.com/openshift/cluster-image-registry-operator/blob/75e8e851700add9c847190fb228d2e702b2af2e8/manifests/07-operator.yaml#L92-L93
[registry-configmap-namespace]: https://github.com/openshift/cluster-image-registry-operator/blob/75e8e851700add9c847190fb228d2e702b2af2e8/manifests/07-operator.yaml#L6
[rhcos]: https://docs.openshift.com/container-platform/4.2/architecture/architecture-rhcos.html
[rhel-cert-injection]: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-shared-system-certificates_security-hardening#adding-new-certificates_using-shared-system-certificates
[smime]: https://tools.ietf.org/html/rfc5751
[ubi]: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/building_running_and_managing_containers/using_red_hat_universal_base_images_standard_minimal_and_runtimes