Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Advanced Topics Doc #2999

Merged
merged 19 commits into from
May 27, 2020
Merged

Update Advanced Topics Doc #2999

Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
fe448f5
update 3rd party resource docs
May 8, 2020
aff09d1
update init functions
May 8, 2020
fe97142
Update website/content/en/docs/kubebuilder/advanced-topics.md
May 11, 2020
3a4c9b2
Update website/content/en/docs/kubebuilder/advanced-topics.md
May 11, 2020
220dd04
addressing nits
May 11, 2020
2c1a297
Update website/content/en/docs/kubebuilder/advanced-topics.md
May 11, 2020
0284e15
remove err check
May 11, 2020
4f6516f
Update website/content/en/docs/kubebuilder/advanced-topics.md
May 12, 2020
ac343c6
Update website/content/en/docs/kubebuilder/advanced-topics.md
May 12, 2020
15a1768
remove todo, ref to main.go
May 14, 2020
b150a8f
update example reconcile function
May 20, 2020
2fbe547
remove todo
May 20, 2020
8faedb9
func sig and vars
May 20, 2020
e5542b9
fix tabs and filepath
May 27, 2020
7013dad
fix recon to ctrl
May 27, 2020
54f5f94
fix all tabs
May 27, 2020
d9f7676
align 4sp tabs
May 27, 2020
262e40d
dots spacing
May 27, 2020
e16b490
final nits
May 27, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
282 changes: 135 additions & 147 deletions website/content/en/docs/kubebuilder/advanced-topics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,48 +28,44 @@ Then, in your controller, you can use [`Conditions`][godoc-conditions] methods t

### Adding 3rd Party Resources To Your Operator

> **// TODO:** Update the `main.go` code in these sections to use the `init()` func to register the scheme instead of doing it in `main()`.

The operator's Manager supports the core Kubernetes resource types as found in the client-go [scheme][scheme_package] package and will also register the schemes of all custom resource types defined in your project.

```Go
import (
"github.com/example-inc/memcached-operator/pkg/apis"
...
cachev1alpha1 "github.com/example-inc/memcached-operator/api/v1alpha1
...
)

estroz marked this conversation as resolved.
Show resolved Hide resolved
// Setup Scheme for all resources
if err := apis.AddToScheme(mgr.GetScheme()); err != nil {
log.Error(err, "")
os.Exit(1)
func init() {

// Setup Scheme for all resources
utilruntime.Must(cachev1alpha1.AddToScheme(mgr.GetScheme()))
// +kubebuilder:scaffold:scheme
}
```

To add a 3rd party resource to an operator, you must add it to the Manager's scheme. By creating an `AddToScheme()` method or reusing one you can easily add a resource to your scheme. An [example][deployments_register] shows that you define a function and then use the [runtime][runtime_package] package to create a `SchemeBuilder`.

#### Register with the Manager's scheme

Call the `AddToScheme()` function for your 3rd party resource and pass it the Manager's scheme via `mgr.GetScheme()`
in `cmd/manager/main.go`.

Call the `AddToScheme()` function for your 3rd party resource and pass it the Manager's scheme via `mgr.GetScheme()` in `main.go`.
Example:
```go
import (
....

routev1 "github.com/openshift/api/route/v1"
routev1 "github.com/openshift/api/route/v1"
)

func main() {
...
func init() {
estroz marked this conversation as resolved.
Show resolved Hide resolved
...

// Adding the routev1
if err := routev1.AddToScheme(mgr.GetScheme()); err != nil {
log.Error(err, "")
os.Exit(1)
}
// Adding the routev1
utilruntime.Must(clientgoscheme.AddToScheme(mgr.GetScheme()))

...
utilruntime.Must(routev1.AddToScheme(mgr.GetScheme()))
// +kubebuilder:scaffold:scheme

...
}
```

Expand All @@ -81,28 +77,28 @@ Example of registering `DNSEndpoints` 3rd party resource from `external-dns`:

```go
import (
...
...
"k8s.io/apimachinery/pkg/runtime/schema"
"sigs.k8s.io/controller-runtime/pkg/scheme"
...
// DNSEndoints
externaldns "github.com/kubernetes-incubator/external-dns/endpoint"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
...
// DNSEndoints
externaldns "github.com/kubernetes-incubator/external-dns/endpoint"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func main() {
...
func init() {
...

log.Info("Registering Components.")
log.Info("Registering Components.")

schemeBuilder := &scheme.Builder{GroupVersion: schema.GroupVersion{Group: "externaldns.k8s.io", Version: "v1alpha1"}}
schemeBuilder.Register(&externaldns.DNSEndpoint{}, &externaldns.DNSEndpointList{})
if err := schemeBuilder.AddToScheme(mgr.GetScheme()); err != nil {
log.Error(err, "")
os.Exit(1)
}
schemeBuilder := &scheme.Builder{GroupVersion: schema.GroupVersion{Group: "externaldns.k8s.io", Version: "v1alpha1"}}
schemeBuilder.Register(&externaldns.DNSEndpoint{}, &externaldns.DNSEndpointList{})
if err := schemeBuilder.AddToScheme(mgr.GetScheme()); err != nil {
log.Error(err, "")
os.Exit(1)
}

...
...
}
```

Expand All @@ -115,18 +111,11 @@ func main() {

### Metrics

> **// TODO:** Update the [metrics doc](https://github.com/operator-framework/operator-sdk/blob/master/website/content/en/docs/golang/metrics/operator-sdk-monitoring.md) since it doesn't match the default scaffolded by kubebuilder anymore.

To learn about how metrics work in the Operator SDK read the [metrics section][metrics_doc] of the user documentation.

#### Default Metrics exported with 3rd party resource
To learn about how metrics work in the Operator SDK read the [metrics section][metrics_doc] of the Kubebuilder documentation.

> **// TODO:** Remove this section since we're no longer scaffolding main.go to use the SDK's `GenerateAndServeCRMetrics()` util in `pkg/kube-metrics`.

### Handle Cleanup on Deletion

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@theishshah There's still a TODO here in the finalizer section for updating the finalizer section of this doc with the new var names and signatures of the kubebuilder controller.

// TODO: Update finalizer reconcile code for kubebuilder's default reconciler imports and variable names

> **// TODO:** Update finalizer reconcile code for kubebuilder's default reconciler imports and variable names

To implement complex deletion logic, you can add a finalizer to your Custom Resource. This will prevent your Custom Resource from being
deleted until you remove the finalizer (ie, after your cleanup logic has successfully run). For more information, see the
[official Kubernetes documentation on finalizers](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#finalizers).
Expand All @@ -137,105 +126,104 @@ The following is a snippet from the controller file under `pkg/controller/memcac

```Go
import (
...
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
...
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
)

const memcachedFinalizer = "finalizer.cache.example.com"

func (r *ReconcileMemcached) Reconcile(request reconcile.Request) (reconcile.Result, error) {
reqLogger := log.WithValues("Request.Namespace", request.Namespace, "Request.Name", request.Name)
reqLogger.Info("Reconciling Memcached")

// Fetch the Memcached instance
memcached := &cachev1alpha1.Memcached{}
err := r.client.Get(context.TODO(), request.NamespacedName, memcached)
if err != nil {
if errors.IsNotFound(err) {
// Request object not found, could have been deleted after reconcile request.
// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
// Return and don't requeue
reqLogger.Info("Memcached resource not found. Ignoring since object must be deleted.")
return reconcile.Result{}, nil
}
// Error reading the object - requeue the request.
reqLogger.Error(err, "Failed to get Memcached.")
return reconcile.Result{}, err
}

...

// Check if the Memcached instance is marked to be deleted, which is
// indicated by the deletion timestamp being set.
isMemcachedMarkedToBeDeleted := memcached.GetDeletionTimestamp() != nil
if isMemcachedMarkedToBeDeleted {
if contains(memcached.GetFinalizers(), memcachedFinalizer) {
// Run finalization logic for memcachedFinalizer. If the
// finalization logic fails, don't remove the finalizer so
// that we can retry during the next reconciliation.
if err := r.finalizeMemcached(reqLogger, memcached); err != nil {
return reconcile.Result{}, err
}

// Remove memcachedFinalizer. Once all finalizers have been
// removed, the object will be deleted.
controllerutil.RemoveFinalizer(memcached, memcachedFinalizer)
err := r.client.Update(context.TODO(), memcached)
if err != nil {
return reconcile.Result{}, err
}
}
return reconcile.Result{}, nil
}

// Add finalizer for this CR
if !contains(memcached.GetFinalizers(), memcachedFinalizer) {
if err := r.addFinalizer(reqLogger, memcached); err != nil {
return reconcile.Result{}, err
}
}

...

return reconcile.Result{}, nil
func (r *MemcachedReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
ctx := context.Background()
reqLogger := r.log.WithValues("memcached", req.NamespacedName)
reqLogger.Info("Reconciling Memcached")

// Fetch the Memcached instance
memcached := &cachev1alpha1.Memcached{}
err := r.Get(ctx, req.NamespacedName, memcached)
if err != nil {
if errors.IsNotFound(err) {
// Request object not found, could have been deleted after reconcile request.
// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
// Return and don't requeue
reqLogger.Info("Memcached resource not found. Ignoring since object must be deleted.")
return ctrl.Result{}, nil
}
// Error reading the object - requeue the request.
reqLogger.Error(err, "Failed to get Memcached.")
return ctrl.Result{}, err
}

...

// Check if the Memcached instance is marked to be deleted, which is
// indicated by the deletion timestamp being set.
isMemcachedMarkedToBeDeleted := memcached.GetDeletionTimestamp() != nil
if isMemcachedMarkedToBeDeleted {
if contains(memcached.GetFinalizers(), memcachedFinalizer) {
// Run finalization logic for memcachedFinalizer. If the
// finalization logic fails, don't remove the finalizer so
// that we can retry during the next reconciliation.
if err := r.finalizeMemcached(reqLogger, memcached); err != nil {
return ctrl.Result{}, err
}

// Remove memcachedFinalizer. Once all finalizers have been
// removed, the object will be deleted.
controllerutil.RemoveFinalizer(memcached, memcachedFinalizer)
err := r.Update(ctx, memcached)
if err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{}, nil
}

// Add finalizer for this CR
if !contains(memcached.GetFinalizers(), memcachedFinalizer) {
if err := r.addFinalizer(reqLogger, memcached); err != nil {
return ctrl.Result{}, err
}
}

...

return ctrl.Result{}, nil
}

func (r *ReconcileMemcached) finalizeMemcached(reqLogger logr.Logger, m *cachev1alpha1.Memcached) error {
// TODO(user): Add the cleanup steps that the operator
// needs to do before the CR can be deleted. Examples
// of finalizers include performing backups and deleting
// resources that are not owned by this CR, like a PVC.
reqLogger.Info("Successfully finalized memcached")
return nil
func (r *MemcachedReconciler) finalizeMemcached(reqLogger logr.Logger, m *cachev1alpha1.Memcached) error {
// TODO(user): Add the cleanup steps that the operator
// needs to do before the CR can be deleted. Examples
// of finalizers include performing backups and deleting
// resources that are not owned by this CR, like a PVC.
reqLogger.Info("Successfully finalized memcached")
return nil
}

func (r *ReconcileMemcached) addFinalizer(reqLogger logr.Logger, m *cachev1alpha1.Memcached) error {
reqLogger.Info("Adding Finalizer for the Memcached")
controllerutil.AddFinalizer(m, memcachedFinalizer)

// Update CR
err := r.client.Update(context.TODO(), m)
if err != nil {
reqLogger.Error(err, "Failed to update Memcached with finalizer")
return err
}
return nil
func (r *MemcachedReconciler) addFinalizer(reqLogger logr.Logger, m *cachev1alpha1.Memcached) error {
reqLogger.Info("Adding Finalizer for the Memcached")
controllerutil.AddFinalizer(m, memcachedFinalizer)

// Update CR
err := r.Update(context.TODO(), m)
if err != nil {
reqLogger.Error(err, "Failed to update Memcached with finalizer")
return err
}
return nil
}

func contains(list []string, s string) bool {
for _, v := range list {
if v == s {
return true
}
}
return false
for _, v := range list {
if v == s {
return true
}
}
return false
}
```

### Leader election

> **// TODO:** Update this section to remove `leader-for-life` option? Since it's no longer the default.

During the lifecycle of an operator it's possible that there may be more than 1 instance running at any given time e.g when rolling out an upgrade for the operator.
In such a scenario it is necessary to avoid contention between multiple operator instances via leader election so that only one leader instance handles the reconciliation while the other instances are inactive but ready to take over when the leader steps down.

Expand All @@ -254,18 +242,18 @@ A call to `leader.Become()` will block the operator as it retries until it can b

```Go
import (
...
"github.com/operator-framework/operator-sdk/pkg/leader"
...
"github.com/operator-framework/operator-sdk/pkg/leader"
)

func main() {
...
err = leader.Become(context.TODO(), "memcached-operator-lock")
if err != nil {
log.Error(err, "Failed to retry for leader lock")
os.Exit(1)
}
...
...
err = leader.Become(context.TODO(), "memcached-operator-lock")
if err != nil {
log.Error(err, "Failed to retry for leader lock")
os.Exit(1)
}
...
}
```
If the operator is not running inside a cluster `leader.Become()` will simply return without error to skip the leader election since it can't detect the operator's namespace.
Expand All @@ -276,19 +264,19 @@ The leader-with-lease approach can be enabled via the [Manager Options][manager_

```Go
import (
...
"sigs.k8s.io/controller-runtime/pkg/manager"
...
"sigs.k8s.io/controller-runtime/pkg/manager"
)

func main() {
...
opts := manager.Options{
...
LeaderElection: true,
LeaderElectionID: "memcached-operator-lock"
}
mgr, err := manager.New(cfg, opts)
...
opts := manager.Options{
...
LeaderElection: true,
LeaderElectionID: "memcached-operator-lock"
}
mgr, err := manager.New(cfg, opts)
...
}
```

Expand All @@ -300,9 +288,9 @@ When the operator is not running in a cluster, the Manager will return an error
[deployments_register]: https://github.com/kubernetes/api/blob/master/apps/v1/register.go#L41
[runtime_package]: https://godoc.org/k8s.io/apimachinery/pkg/runtime
[scheme_builder]: https://godoc.org/sigs.k8s.io/controller-runtime/pkg/scheme#Builder
[metrics_doc]: /docs/golang/metrics/operator-sdk-monitoring/
[metrics_doc]: https://book.kubebuilder.io/reference/metrics.html
[lease_split_brain]: https://github.com/kubernetes/client-go/blob/30b06a83d67458700a5378239df6b96948cb9160/tools/leaderelection/leaderelection.go#L21-L24
[leader_for_life]: https://godoc.org/github.com/operator-framework/operator-sdk/pkg/leader
[leader_with_lease]: https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg/leaderelection
[pod_eviction_timeout]: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/#options
[manager_options]: https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg/manager#Options
[manager_options]: https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg/manager#Options