feat: Support Reconciliation Tracing #8143

free6om · 2024-09-12T10:32:10Z

Background

"Reconciliation Tracing" refers to the process of tracking the reconciliation workflow of an Operator or Controller and presenting it in a task flow format. This allows for a more intuitive observation of the reconciliation process and helps in problem identification. This feature can be considered an observability tool for Operators or Controllers.

Goals and Motivation

The tracking of the reconciliation process has two extremes. One is to simply observe whether the current status (usually the Status section) of the primary resource aligns with the desired state (typically the Spec section), which represents the highest-level observation.

The other direction is to observe changes related to the primary resource and all associated secondary resources to gain a comprehensive understanding of the entire reconciliation process. The former aspect has been sufficiently considered in the design of each API or Custom Resource Definition (CRD), while this proposal focuses on the latter.

The goal of this proposal is to track and display the reconciliation process of KubeBlocks to facilitate a detailed understanding of the reconciliation process or to identify issues. Specifically, it includes the following objectives:

Track and display the ongoing reconciliation process to understand detailed steps and identify bottlenecks.
Predict and display upcoming reconciliation processes to grasp the future reconciliation flow.
Support a dry-run feature, which answers the question: If a specific update is made to the Cluster Spec, what will the subsequent reconciliation process and results look like?

Proposal Design

Mental Model

Object Tree

The reconciliation process revolves around the object tree and the changes within it. An object tree consists of a primary object and all its associated secondary objects. For a KB Cluster, the structure of its object tree is as follows:

Object Change

The reconciliation process ultimately manifests as the addition, removal, or state change of nodes within the object tree. By recording these changes and presenting them along a timeline, we can obtain a clear and detailed view of the reconciliation process.

The recorded content includes two types: one is the changes to the objects themselves within the object tree, and the other is the events associated with those objects in the object tree.

Reconciliation Cycle

Throughout the entire lifecycle of a Cluster, from creation to eventual destruction, it undergoes multiple reconciliation loops. At the end of each reconciliation loop, the Cluster either reaches a terminal state or does not. We refer to all reconciliations occurring between two instances of the terminal state as a reconciliation cycle.

Metaphorically, the object tree can be viewed as a tree in nature, and what we aim to describe is the growth and changes of this tree over the course of a year (or one growth cycle).

API Design

For detailed design, please refer to the pull request (PR). Below is an example:

apiVersion: trace.kubeblocks.io/v1
kind: ReconciliationTrace
metadata:
  finalizers:
  - trace.kubeblocks.io/finalizer
  name: redis-cluster-tracing
  namespace: default
spec:
  targetObject:
    name: redis-cluster
    namespace: default
  dryRun:
    desiredSpec: |
      componentSpecs:
      - name: redis
        replicas: 3
  stateEvaluationExpression:
    celExpression:
      expression: "has(object.status.phase) && object.status.phase == \"Running\""
  locale: zh_CN
status:
  currentState:
    changes:
    - changeType: Creation
      description: Creation
      objectReference:
        apiVersion: apps.kubeblocks.io/v1alpha1
        kind: Configuration
        name: redis-cluster-redis-sentinel
        namespace: default
        resourceVersion: "1020694"
        uid: 8d522d4d-8bcd-41a7-96f3-1e71af6d70d4
      revision: 1020694
      timestamp: "2024-09-29T05:59:06Z"
    - changeType: Event
      description: no available pod to execute action postProvision
      eventAttributes:
        reason: Warning
        type: Warning
      objectReference:
        apiVersion: apps.kubeblocks.io/v1
        kind: Component
        name: redis-cluster-redis
        namespace: default
        resourceVersion: "1020816"
        uid: 225108bb-7a86-4c94-ae74-6639a1c30e8e
      revision: 1072157
      timestamp: "2024-09-29T13:46:53Z"
    objectTree:
      primary:
        apiVersion: apps.kubeblocks.io/v1
        kind: Cluster
        name: redis-cluster
        namespace: default
        resourceVersion: "1020985"
        uid: b6293a70-cd42-4a03-b4ec-95268683495d
      secondaries:
      - primary:
          apiVersion: apps.kubeblocks.io/v1
          kind: Component
          name: redis-cluster-redis
          namespace: default
          resourceVersion: "1020816"
          uid: 225108bb-7a86-4c94-ae74-6639a1c30e8e
        secondaries:
        - primary:
            apiVersion: workloads.kubeblocks.io/v1
            kind: InstanceSet
            name: redis-cluster-redis
            namespace: default
            resourceVersion: "1040780"
            uid: 95adb8be-326a-44f1-807f-cbfed6f67ec5
          secondaries:
          - primary:
              apiVersion: v1
              kind: Pod
              name: redis-cluster-redis-0
              namespace: default
              resourceVersion: "1040377"
              uid: da071f7b-04e7-423c-975e-b81c7df9f063
          - primary:
              apiVersion: v1
              kind: Pod
              name: redis-cluster-redis-1
              namespace: default
              resourceVersion: "1021064"
              uid: 6dc16aae-bfcd-45f3-b991-7f58338dd84d
          - primary:
              apiVersion: v1
              kind: Service
              name: redis-cluster-redis-headless
              namespace: default
              resourceVersion: "1020732"
              uid: 0b02a3ef-4b50-4963-9c91-69556d47555e
    summary:
      objectSummaries:
      - changeSummary: {}
        objectType:
          apiVersion: apps.kubeblocks.io/v1
          kind: Cluster
        total: 1
      - changeSummary:
          added: 2
        objectType:
          apiVersion: apps.kubeblocks.io/v1
          kind: Component
        total: 2
  dryRunResult:
    desiredSpecRevision: 557db4f88f
    message: no available pod to execute action postProvision
    observedTargetGeneration: 2
    phase: Failed
    plan:
      changes:
      - changeType: Update
        description: Update
        objectReference:
          apiVersion: apps.kubeblocks.io/v1
          kind: Cluster
          name: redis-cluster
          namespace: default
          resourceVersion: "1051770"
          uid: b6293a70-cd42-4a03-b4ec-95268683495d
        revision: 1051770
        timestamp: "2024-09-29T05:59:07Z"
      - changeType: Update
        description: Update
        objectReference:
          apiVersion: apps.kubeblocks.io/v1
          kind: Component
          name: redis-cluster-redis
          namespace: default
          resourceVersion: "1051778"
          uid: 225108bb-7a86-4c94-ae74-6639a1c30e8e
        revision: 1051778
        timestamp: "2024-09-29T05:59:09Z"
      objectTree:
        primary:
          apiVersion: apps.kubeblocks.io/v1
          kind: Cluster
          name: redis-cluster
          namespace: default
          resourceVersion: "1051771"
          uid: b6293a70-cd42-4a03-b4ec-95268683495d
        secondaries:
        - primary:
            apiVersion: apps.kubeblocks.io/v1
            kind: Component
            name: redis-cluster-redis-sentinel
            namespace: default
            resourceVersion: "1020987"
            uid: caa95b87-c2d6-428b-b0c7-c4083a20d1e1
      summary:
        objectSummaries:
        - changeSummary:
            updated: 1
          objectType:
            apiVersion: apps.kubeblocks.io/v1
            kind: Cluster
          total: 1
        - changeSummary:
            updated: 1
          objectType:
            apiVersion: apps.kubeblocks.io/v1
            kind: Component
          total: 2
    reason: ReconcileError
    specDiff: "  map[string]any{\n  \t\"componentSpecs\": []any{\n- \t\tmap[string]any{\n-
      \t\t\t\"componentDef\":    string(\"redis-7-1.0.0-alpha.0\"),\n- \t\t\t\"disableExporter\":
      bool(true),\n- \t\t\t\"name\":            string(\"redis\"),\n- \t\t\t\"replicas\":
      \       int64(2),\n- \t\t\t\"resources\": map[string]any{\n- \t\t\t\t\"limits\":
      \  map[string]any{\"cpu\": string(\"500m\"), \"memory\": string(\"512Mi\")},\n-
      \t\t\t\t\"requests\": map[string]any{\"cpu\": string(\"500m\"), \"memory\":
      string(\"512Mi\")},\n- \t\t\t},\n- \t\t\t\"serviceAccountName\":   string(\"kb-redis-cluster\"),\n-
      \t\t\t\"serviceVersion\":       string(\"7.2.4\"),\n- \t\t\t\"volumeClaimTemplates\":
      []any{map[string]any{\"name\": string(\"data\"), \"spec\": map[string]any{...}}},\n-
      \t\t},\n- \t\tmap[string]any{\n- \t\t\t\"componentDef\":    string(\"redis-sentinel-7-1.0.0-alpha.0\"),\n-
      \t\t\t\"disableExporter\": bool(false),\n- \t\t\t\"name\":            string(\"redis-sentinel\"),\n-
      \t\t\t\"replicas\":        int64(3),\n- \t\t\t\"resources\": map[string]any{\n-
      \t\t\t\t\"limits\":   map[string]any{\"cpu\": string(\"500m\"), \"memory\":
      string(\"512Mi\")},\n- \t\t\t\t\"requests\": map[string]any{\"cpu\": string(\"500m\"),
      \"memory\": string(\"512Mi\")},\n- \t\t\t},\n- \t\t\t\"serviceAccountName\":
      \  string(\"kb-redis-cluster\"),\n- \t\t\t\"serviceVersion\":       string(\"7.2.4\"),\n-
      \t\t\t\"volumeClaimTemplates\": []any{map[string]any{\"name\": string(\"data\"),
      \"spec\": map[string]any{...}}},\n- \t\t},\n+ \t\tmap[string]any{\"name\": string(\"redis\"),
      \"replicas\": float64(3)},\n  \t},\n- \t\"terminationPolicy\": string(\"Delete\"),\n
      \ }\n"
  initialObjectTree:
    primary:
      apiVersion: apps.kubeblocks.io/v1
      kind: Cluster
      name: redis-cluster
      namespace: default
      resourceVersion: "1020985"
      uid: b6293a70-cd42-4a03-b4ec-95268683495d

kbcli Interaction Design

Command Line

Command Path: kbcli trace
Subcommands:

$ kbcli trace --help
list - list all traces
create <trace-name> --cluster-name [cluster-name] --depth [depth] --locale [locale] --cel-state-evaluation-expression [expression]
update <trace-name> --depth [depth] --locale [locale] --cel-state-evaluation-expression [expression]
delete <trace-name>
watch <trace-name>

$ kbcli cluster create/update --dry-run=trace

TUI Display

In the Trace subcommand, the watch command provides real-time tracking and display of the reconciliation process, presented in a TUI format within kbcli. The layout is as follows:

Implementation of the Proposal

Reconciliation Process Tracking

Tracking the reconciliation process involves monitoring all primary and secondary resources related to a KB Cluster, as well as associated events, and meticulously recording each change. The basic flow is as follows:

Reconciliation Process Prediction

Prediction refers to forecasting all reconciliation processes that will occur from the current state until the terminal state is reached. The format and approach for this are similar to process tracking, described through changes in the object tree.

The key difference is that the prediction mechanism builds a simulated environment, where the current state (the Cluster and all its secondary objects) serves as input until the simulation reaches a terminal state. All object changes that occur during this period are recorded as part of the reconciliation process prediction. The basic flow is as follows:

Predictions occur in two scenarios:

When there is a change in the Cluster Spec, initiating a new reconciliation cycle, the upcoming reconciliation process is predicted and updated in trace.status.desiredState.
During a dry run, the desired specification from the dry run is first applied to the current Cluster object in the simulated environment, after which the subsequent reconciliation process is predicted and updated in trace.status.dryRunResult.

kubectl Display

In kubectl, the current reconciliation progress can be viewed in real-time using the command:

kubectl get trace <name> -w

For example:

$ kubectl get trace redis -w
NAME    AGE     TARGET_NS   TARGET_NAME     API_VERSION             KIND        NAMESPACE   NAME                  CHANGE
redis   5d21h   default     redis-cluster   apps.kubeblocks.io/v1   Component   default     redis-cluster-redis   no available pod to execute action postProvision

Screenshot:

kbcli Display

Through kbcli, more detailed information can be displayed. For example, the following shows the tracking of the reconciliation process for a Redis cluster:

free6om · 2024-09-12T11:07:31Z

why use these two markers:

// +kubebuilder:pruning:PreserveUnknownFields
// +kubebuilder:validation:Schemaless

codecov · 2024-09-12T11:10:36Z

Codecov Report

Attention: Patch coverage is 63.01887% with 980 lines in your changes missing coverage. Please review.

Project coverage is 60.63%. Comparing base (fe24a3a) to head (0ee3972).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
controllers/trace/reconciler_tree.go	63.79%	168 Missing and 46 partials ⚠️
controllers/trace/util.go	74.16%	110 Missing and 52 partials ⚠️
controllers/trace/informer_manager.go	0.00%	119 Missing ⚠️
controllers/trace/mock_client.go	59.42%	71 Missing and 28 partials ⚠️
controllers/trace/desired_state_handler.go	53.62%	46 Missing and 18 partials ⚠️
controllers/trace/current_state_handler.go	57.25%	39 Missing and 14 partials ⚠️
controllers/trace/plan_generator.go	58.18%	33 Missing and 13 partials ⚠️
controllers/trace/dry_run_handler.go	57.94%	31 Missing and 14 partials ⚠️
controllers/trace/object_tree_root_finder.go	61.29%	28 Missing and 8 partials ⚠️
controllers/trace/type.go	66.19%	16 Missing and 8 partials ⚠️
... and 12 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8143      +/-   ##
==========================================
+ Coverage   60.35%   60.63%   +0.28%     
==========================================
  Files         357      376      +19     
  Lines       42555    45195    +2640     
==========================================
+ Hits        25684    27404    +1720     
- Misses      14559    15241     +682     
- Partials     2312     2550     +238

Flag	Coverage Δ
unittests	`60.63% <63.01%> (+0.28%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

zjx20

To be continued.

controllers/trace/type.go

controllers/trace/util.go

controllers/trace/change_capture_store.go

controllers/trace/object_revision_store.go

controllers/trace/informer_manager.go

controllers/trace/mock_client.go

controllers/trace/reconciler_tree.go

controllers/trace/desired_state_handler.go

apis/trace/v1/reconciliationtrace_types.go

zjx20 · 2024-11-25T01:58:56Z

controllers/trace/mock_client.go

-	// Unmarshal the patched JSON back into the object
-	return json.Unmarshal(patchedJSON, obj)
+	// Unmarshal the patched JSON into a new clean object
+	newObj := obj.DeepCopyObject().(client.Object)


Can use reflection to create a new zeroed object directly, to avoid this deep-copying.

zjx20 · 2024-11-25T02:48:06Z

deploy/helm/values.yaml

@@ -1865,6 +1865,8 @@ controllers:
    enabled: true
  experimental:
    enabled: false
+  trace:
+    enabled: true


How about setting it to false for now and re-enabling it when the performance issue is solved?

free6om added 5 commits September 10, 2024 17:16

api scaffold

edcd798

embed changes into object tree

c355207

workflow style WIP

d57e381

workflow style done

0e67c96

make manifests

ee8f5ce

free6om added this to the Release 1.0.0 milestone Sep 12, 2024

free6om self-assigned this Sep 12, 2024

github-actions bot added the size/XXL Denotes a PR that changes 1000+ lines. label Sep 12, 2024

free6om added 3 commits September 12, 2024 18:47

gpt suggests that

de61b49

make generate

5ceeb23

fix CRD validation error

294cd9d

free6om added 17 commits September 13, 2024 18:48

view controller WIP

b37bc81

resources loader, validator done, informer manager WIP

88cfe52

informer manager poc done

7d6d2b2

root finder poc done

2b8efcf

view calculation WIP

f28798a

view calculation done

b25d5de

view state evaluation WIP

0a8c3ae

view state evaluation done

188bbdb

object store done

91dc8d5

debug view controller

fe89ded

debug view controller

6b848e2

viewResourses() works

f791e64

viewResourcesValidation() works

f289b55

updateInformerManager() works

202e83b

viewCalculation() works

22cfaa5

viewStateEvaluation() works

fb576e3

view.status update works

0c164b3

free6om added 3 commits October 21, 2024 14:28

object_tree_root_finder ut done

ce09dc3

fix k8sMock related issue

0dadc9c

fix panic error

c138c43

free6om marked this pull request as ready for review October 21, 2024 07:35

free6om requested review from JashBook, ahjing99 and a team as code owners October 21, 2024 07:35

free6om mentioned this pull request Oct 21, 2024

feat: Support Reconciliation Tracing apecloud/kbcli#454

Open

free6om changed the title ~~feat: Support Reconciliation Trace~~ feat: Support Reconciliation Tracing Oct 21, 2024

free6om added 4 commits October 22, 2024 13:33

fix event loss issue

d511335

fix deletion issue

9f4f798

fix panic on trace creation

c8c2ec7

remove Depth

87e4684

zjx20 reviewed Nov 20, 2024

View reviewed changes

zjx20 reviewed Nov 21, 2024

View reviewed changes

free6om added 2 commits November 22, 2024 11:22

Merge branch 'main' into support/reconciliation-view

734d317

refine based on review comments

79de046

free6om requested review from sophon-zt, leon-inf and Y-Rookie as code owners November 22, 2024 10:10

free6om added 9 commits November 22, 2024 18:20

refine based on review comments

626ee7a

refine based on review comments

5260710

refine based on review comments

2557e0a

refine based on review comments

9695815

refine based on review comments

09bb3c6

refine based on review comments

eb1e5e5

return a copy

3578ce1

set latestReconciliationCycleStart default to -1

71735bc

using logger instead of fmt.Printf

0ee3972

zjx20 reviewed Nov 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support Reconciliation Tracing #8143

feat: Support Reconciliation Tracing #8143

free6om commented Sep 12, 2024 •

edited

Loading

free6om commented Sep 12, 2024 •

edited

Loading

codecov bot commented Sep 12, 2024 •

edited

Loading

zjx20 left a comment

zjx20 Nov 25, 2024

zjx20 Nov 25, 2024

feat: Support Reconciliation Tracing #8143

Are you sure you want to change the base?

feat: Support Reconciliation Tracing #8143

Conversation

free6om commented Sep 12, 2024 • edited Loading

Background

Goals and Motivation

Proposal Design

Mental Model

Object Tree

Object Change

Reconciliation Cycle

API Design

kbcli Interaction Design

Command Line

TUI Display

Implementation of the Proposal

Reconciliation Process Tracking

Reconciliation Process Prediction

kubectl Display

kbcli Display

free6om commented Sep 12, 2024 • edited Loading

codecov bot commented Sep 12, 2024 • edited Loading

Codecov Report

zjx20 left a comment

Choose a reason for hiding this comment

zjx20 Nov 25, 2024

Choose a reason for hiding this comment

zjx20 Nov 25, 2024

Choose a reason for hiding this comment

free6om commented Sep 12, 2024 •

edited

Loading

free6om commented Sep 12, 2024 •

edited

Loading

codecov bot commented Sep 12, 2024 •

edited

Loading