Skip to content

docs/dev/operators: Document a dynamic-object approach #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wking wants to merge 3 commits into from
Closed

docs/dev/operators: Document a dynamic-object approach #59

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1bfc4a0
docs/dev/operators: /manifests backticks, etc.
wking Nov 27, 2018
c4b47a6
docs/dev/operators: Convert image-references procedure to a list
wking Nov 27, 2018
e24bc29
docs/dev/operators: Document a dynamic-object approach
wking Nov 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 37 additions & 8 deletions docs/dev/operators.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ LABEL io.openshift.release.operator=true
Ensure your image is published into the cluster release tag by ci-operator
Wait for a new release payload to be created (usually once you push to master in your operator).
Copy link

@DanyC97 DanyC97 May 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: will be nice to add a period - tag by ci-operator. Wait for a new release...


## What do I put in /manifests?
## What do I put in `/manifests`?

You need the following:

Expand All @@ -26,7 +26,7 @@ You need the following:
- Deployment for your operator
- A ClusterOperator CR [more info here](clusteroperator.md)
- Any other config objects your operator might need
- An image-references file (See below)
- An `image-references` file (see [below](#how-do-i-ensure-the-right-images-get-used-by-my-manifests)).

In your deployment you can reference the latest development version of your operator image (quay.io/openshift/origin-machine-api-operator:latest). If you have other hard-coded image strings, try to put them as environment variables on your deployment or as a config map.

Expand Down Expand Up @@ -84,7 +84,7 @@ Your manifests can contain a tag to the latest development image published by Or

Assume you have two images in your manifests - `quay.io/openshift/origin-ingress-operator:latest` and `quay.io/openshift/origin-haproxy-router:latest`. Those correspond to the following tags `ingress-operator` and `haproxy-router` when the CI runs.

Create a file `image-references` in the /manifests dir with the following contents:
Create a file `image-references` in the `/manifests` directory with the following contents:

```yaml
kind: ImageStream
Expand All @@ -101,13 +101,42 @@ spec:
Name: quay.io/openshift/origin-haproxy-router
```

The release tooling will read image-references and do the following operations:
The release tooling will read `image-references` and do the following operations:

Verify that the tags `ingress-operator` and `haproxy-router` exist from the release / CI tooling (in the image stream `openshift/origin-v4.0` on api.ci). If they don’t exist, you’ll get a build error.
Do a find and replace in your manifests (effectively a sed) that replaces `quay.io/openshift/origin-haproxy-router(:.*|@:.*)` with `registry.svc.ci.openshift.org/openshift/origin-v4.0@sha256:<latest SHA for :haproxy-router>`
Store the fact that operator ingress-operator uses both of those images in a metadata file alongside the manifests
Bundle up your manifests and the metadata file as a docker image and push them to a registry
1. Verify that the tags `ingress-operator` and `haproxy-router` exist from the release / CI tooling (in the image stream `openshift/origin-v4.0` on api.ci). If they don’t exist, you’ll get a build error.
2. Do a find and replace in your manifests (effectively a `sed`) that replaces `quay.io/openshift/origin-haproxy-router(:.*|@:.*)` with `registry.svc.ci.openshift.org/openshift/origin-v4.0@sha256:<latest SHA for :haproxy-router>`
3. Store the fact that operator ingress-operator uses both of those images in a metadata file alongside the manifests.
4. Bundle up your manifests and the metadata file as a docker image and push them to a registry.

Later on, when someone wants to mirror a particular release, there will be tooling that can take the list of all images used by operators and mirror them to a new repo.

This pattern tries to balance between having the manifests in your source repo be able to deploy your latest upstream code *and* allowing us to get a full listing of all images used by various operators.

## Dynamic objects

Some objects are not static.
For example, an operator configuration could depend on configuration passed to the installer (cluster name, cluster domain, service CIDR, etc.).
An operator should auto-discover the dependencies and build the dynamic objects when it first comes up.
For example:

1. An operator provides `/manifests/deployment.yaml` with a static deployment.
2. The cluster-version operator processes the operator's [`/manifests`](#what-do-i-put-in-manifests) and pushes the deployment into the cluster.
3. The cluster schedules the operator for a particular node.
4. That node launches the operator pod and its constituent containers.
5. The operator container tries to pull its configuration.
* If found, the operator uses that configuration.
* If not found, the operator:
1. Pulls the dependency resources (e.g. the `cluster-config-v1` config-map from the `kube-system` namespace).
Documentation for connecting to the Kubernetes API from pod containers is [here][API-from-pod].
Operators written in Go can use [`InClusterConfig`][InClusterConfig] to configure their client.
2. Generates the dynamic configuration based on the dependencies.
3. Pushes the generated configuration into the cluster, where future operator runs and other consumers will be able to find it.

For an example, see [the machine-API operator's `getOperatorConfig`][getOperatorConfig], although they don't need the "push the generated configuration into the cluster" step.
Copy link

@rajatchopra rajatchopra Nov 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should inline the example in a code block here.

Copy link
Member Author

@wking wking Nov 27, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should inline the example in a code block here.

How much of the code? e24bc29 currently links to the main portion of this, and I mention InClusterConfig further up but don't link to the operator's InClusterConfig call or client construction. Putting that all together into one example would be something like:

include (
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
  "k8s.io/client-go/kubernetes"
  corev1 "k8s.io/client-go/kubernetes/typed/core/v1"
  "k8s.io/client-go/rest"
)

func example() {
  config, err := rest.InClusterConfig() // FIXME: error handling
  client := kubernetes.NewForConfigOrDie(rest.AddUserAgent(config, "your-operator")).CoreV1().ConfigMaps("kube-system")
  for {
    config, err := getConfig(client) // FIXME: error handling
    // FIXME: do whatever your operator does
  }
}

func getConfig(client corev1.ConfigMapInterface) (FIXMESomeType, error) {
  clusterConfig, err := client.Get("cluster-config-v1", metav1.GetOptions{}) // FIXME: error handling
  return generateYourConfig(clusterConfig)
}

That's a lot of FIXMEs for things I've skipped, and it's still fairly involved. Is it particularly helpful when we're already linking folks to the not-much-more-complicated real-world machine-API operator implementation?

Their process is similar to the above example, except that they have no external consumers for their configuration.
To avoid confusion, they just pull `cluster-config-v1` and generate their configuration in memory [for each round of their poll loop][getOperatorConfig-call-site].

[API-from-pod]: https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod
[getOperatorConfig]: https://github.com/openshift/machine-api-operator/blob/v0.1.0/pkg/operator/operator.go#L302-L310
[getOperatorConfig-call-site]: https://github.com/openshift/machine-api-operator/blob/v0.1.0/pkg/operator/operator.go#L157
[InClusterConfig]: https://godoc.org/k8s.io/client-go/rest#InClusterConfig