Bug 1903290: kubelet: refactor KUBELET_LOG_LEVEL into KubeletConfig field #2262

rphillips · 2020-11-30T18:52:31Z

- What I did
Takes Ben's suggestion and refactors the Kubelet log level into a systemd environment file.

Obsoletes: #2261

This also changes the default of the log level to 2, since stale logs are printed into the kubelet log at level 3: BZ1903290.

Allows the logLevel to be configured on the KubeletConfig CRD:

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: set-kubelet-loglevel
spec:
  logLevel: 2
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""

- How to verify it

- Description for the changelog

sjenning · 2020-11-30T18:55:07Z

/cc @darkmuggle

darkmuggle

Other than indentation, this looks good.

templates/common/_base/files/kubelet-log-level.yaml

kikisdeliveryservice · 2020-11-30T19:38:46Z

vpc exceeded error, reported in 4-dev

kikisdeliveryservice · 2020-11-30T21:07:47Z

/retest

darkmuggle · 2020-11-30T21:37:13Z

/approve
But I'll leave the LGTM to a member of the MCO team.

rphillips · 2020-12-01T20:37:04Z

/retest

kikisdeliveryservice · 2020-12-01T20:54:48Z

aws/gcp are having issues across the board today would like to see passing runs before approving

openshift-ci-robot · 2020-12-02T00:09:06Z

@rphillips: This pull request references Bugzilla bug 1903290, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.0) matches configured target release for branch (4.7.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Details

In response to this:

Bug 1903290: kubelet: refactor KUBELET_LOG_LEVEL into systemd environment file

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kikisdeliveryservice · 2020-12-02T00:05:49Z

templates/common/_base/files/kubelet-log-level.yaml

i thought level 2 was not sufficient for debugging?

Problem is that level 3 is logging dead pod logs into the kubelet log. Instead of chasing all level 3 logs... I'm proposing just setting it to level 2. This new approach with a kubelog.conf would allow us to manage the kubelet log level within the kubeletconfig perhaps.

Log Line

I'm open to changing the log line to be 4, but it seems like whack-a-mole.

If there's consensus that level work works between the experts (ie you and @sjenning) I'm ok with it, just want to make sure this isn't going to hurt us later.

@sjenning wdyt?

@kikisdeliveryservice I'm on board with this

openshift-ci-robot · 2020-12-02T19:06:06Z

@rphillips: This pull request references Bugzilla bug 1903290, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.0) matches configured target release for branch (4.7.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Details

In response to this:

Bug 1903290: kubelet: refactor KUBELET_LOG_LEVEL into systemd environment file

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kikisdeliveryservice · 2020-12-02T19:15:24Z

uh since ci is still underwater could you update: https://github.com/openshift/machine-config-operator/blob/master/docs/KubeletConfigDesign.md

kikisdeliveryservice · 2020-12-02T19:46:06Z

unit and verify failures don't look to be flakes..

rphillips · 2020-12-02T23:32:42Z

Updated to fix the unit tests and allow support for only setting the logLevel

openshift-ci-robot · 2020-12-02T23:33:15Z

@rphillips: This pull request references Bugzilla bug 1903290, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.0) matches configured target release for branch (4.7.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Details

In response to this:

Bug 1903290: kubelet: refactor KUBELET_LOG_LEVEL into KubeletConfig field

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rphillips · 2020-12-03T02:33:11Z

Vpc limit

/retest

kikisdeliveryservice · 2020-12-03T02:58:45Z

@rphillips brought this up again in 4-dev no aws resolution yet ☹️

kikisdeliveryservice · 2020-12-03T20:16:50Z

/skip

kikisdeliveryservice

Tested & seems to work ok - made kubelet config applied, then changed kubelet logging seemed to change appropriately.

openshift-bot · 2020-12-03T23:21:15Z

/retest

Please review the full test history for this PR and help us cut down flakes.

sjenning · 2020-12-04T00:51:52Z

still hitting VPC limits issues 😞

openshift-bot · 2020-12-04T00:52:08Z

/retest

Please review the full test history for this PR and help us cut down flakes.

pkg/controller/kubelet-config/helpers.go

kikisdeliveryservice · 2020-12-04T07:19:39Z

Since Seth found an issue, I think we should fix it first then merge

/hold

rphillips · 2020-12-04T15:03:27Z

/hold cancel

fixed the issue.

sjenning · 2020-12-04T15:58:36Z

Trying it out now. Will lgtm when verified.

sjenning · 2020-12-04T17:28:42Z

/lgtm

openshift-ci-robot · 2020-12-04T17:29:06Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: darkmuggle, kikisdeliveryservice, rphillips, sjenning

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [kikisdeliveryservice]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-12-04T19:42:59Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-12-04T23:11:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-12-05T00:03:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

kikisdeliveryservice · 2020-12-05T00:37:44Z

/skip

openshift-bot · 2020-12-05T05:01:58Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-12-05T05:14:57Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-12-05T08:37:35Z

@rphillips: All pull requests linked via external trackers have merged:

openshift/machine-config-operator#2262

Bugzilla bug 1903290 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1903290: kubelet: refactor KUBELET_LOG_LEVEL into KubeletConfig field

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rphillips · 2020-12-07T14:30:48Z

/cherry-pick release-4.6

openshift-cherrypick-robot · 2020-12-07T14:30:55Z

@rphillips: #2262 failed to apply on top of branch "release-4.6":

Applying: kubelet: refactor KUBELET_LOG_LEVEL into systemd environment file
Using index info to reconstruct a base tree...
M	install/0000_80_machine-config-operator_01_kubeletconfig.crd.yaml
M	pkg/apis/machineconfiguration.openshift.io/v1/types.go
M	pkg/apis/machineconfiguration.openshift.io/v1/zz_generated.deepcopy.go
M	pkg/controller/kubelet-config/helpers.go
M	pkg/controller/kubelet-config/kubelet_config_controller.go
M	pkg/controller/kubelet-config/kubelet_config_controller_test.go
A	templates/master/01-master-kubelet/on-prem/units/kubelet.service.yaml
A	templates/worker/01-worker-kubelet/on-prem/units/kubelet.service.yaml
Falling back to patching base and 3-way merge...
Auto-merging pkg/controller/kubelet-config/kubelet_config_controller_test.go
Auto-merging pkg/controller/kubelet-config/kubelet_config_controller.go
CONFLICT (content): Merge conflict in pkg/controller/kubelet-config/kubelet_config_controller.go
Auto-merging pkg/controller/kubelet-config/helpers.go
Auto-merging pkg/apis/machineconfiguration.openshift.io/v1/zz_generated.deepcopy.go
Auto-merging pkg/apis/machineconfiguration.openshift.io/v1/types.go
Auto-merging install/0000_80_machine-config-operator_01_kubeletconfig.crd.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 kubelet: refactor KUBELET_LOG_LEVEL into systemd environment file
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Details

In response to this:

/cherry-pick release-4.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot requested review from kikisdeliveryservice and mrunalp November 30, 2020 18:52

sjenning mentioned this pull request Nov 30, 2020

templates: change additional kubelet service files with new log-level #2261

Closed

openshift-ci-robot requested a review from darkmuggle November 30, 2020 18:55

darkmuggle reviewed Nov 30, 2020

View reviewed changes

templates/common/_base/files/kubelet-log-level.yaml Outdated Show resolved Hide resolved

rphillips force-pushed the kubelet_log_level branch from 54d6c83 to 8bd8497 Compare November 30, 2020 19:01

rphillips changed the title ~~kubelet: refactor KUBELET_LOG_LEVEL into systemd environment file~~ Bug 1903290: kubelet: refactor KUBELET_LOG_LEVEL into systemd environment file Dec 2, 2020

openshift-ci-robot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Dec 2, 2020

rphillips force-pushed the kubelet_log_level branch from 1388e8d to 5a1f653 Compare December 2, 2020 00:09

kikisdeliveryservice reviewed Dec 2, 2020

View reviewed changes

rphillips force-pushed the kubelet_log_level branch from 5a1f653 to 2a47bdc Compare December 2, 2020 19:04

rphillips force-pushed the kubelet_log_level branch from 36fe533 to 2a47bdc Compare December 2, 2020 19:14

sjenning changed the title ~~Bug 1903290: kubelet: refactor KUBELET_LOG_LEVEL into systemd environment file~~ Bug 1903290: kubelet: refactor KUBELET_LOG_LEVEL into KubeletConfig field Dec 2, 2020

rphillips force-pushed the kubelet_log_level branch from 443daca to 5d6b4d1 Compare December 2, 2020 23:32

rphillips force-pushed the kubelet_log_level branch from 5d6b4d1 to a39d9de Compare December 2, 2020 23:37

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 3, 2020

kikisdeliveryservice approved these changes Dec 3, 2020

View reviewed changes

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 3, 2020

sjenning reviewed Dec 4, 2020

View reviewed changes

pkg/controller/kubelet-config/helpers.go Outdated Show resolved Hide resolved

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 4, 2020

rphillips force-pushed the kubelet_log_level branch from a39d9de to c6cec50 Compare December 4, 2020 15:02

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Dec 4, 2020

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 4, 2020

kubelet: refactor KUBELET_LOG_LEVEL into systemd environment file

1030b2c

rphillips force-pushed the kubelet_log_level branch from c6cec50 to 1030b2c Compare December 4, 2020 16:08

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 4, 2020

openshift-merge-robot merged commit 19b904d into openshift:master Dec 5, 2020

Bug 1903290: kubelet: refactor KUBELET_LOG_LEVEL into KubeletConfig field #2262

Bug 1903290: kubelet: refactor KUBELET_LOG_LEVEL into KubeletConfig field #2262

Uh oh!

Conversation

rphillips commented Nov 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sjenning commented Nov 30, 2020

Uh oh!

darkmuggle left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kikisdeliveryservice commented Nov 30, 2020

Uh oh!

kikisdeliveryservice commented Nov 30, 2020

Uh oh!

darkmuggle commented Nov 30, 2020

Uh oh!

rphillips commented Dec 1, 2020

Uh oh!

kikisdeliveryservice commented Dec 1, 2020

Uh oh!

openshift-ci-robot commented Dec 2, 2020

Uh oh!

kikisdeliveryservice Dec 2, 2020

Choose a reason for hiding this comment

Uh oh!

rphillips Dec 2, 2020

Choose a reason for hiding this comment

Uh oh!

rphillips Dec 2, 2020

Choose a reason for hiding this comment

Uh oh!

kikisdeliveryservice Dec 2, 2020

Choose a reason for hiding this comment

Uh oh!

sjenning Dec 3, 2020

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Dec 2, 2020

Uh oh!

kikisdeliveryservice commented Dec 2, 2020

Uh oh!

kikisdeliveryservice commented Dec 2, 2020

Uh oh!

rphillips commented Dec 2, 2020

Uh oh!

openshift-ci-robot commented Dec 2, 2020

Uh oh!

rphillips commented Dec 3, 2020

Uh oh!

kikisdeliveryservice commented Dec 3, 2020

Uh oh!

kikisdeliveryservice commented Dec 3, 2020

Uh oh!

kikisdeliveryservice left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-bot commented Dec 3, 2020

Uh oh!

sjenning commented Dec 4, 2020

Uh oh!

openshift-bot commented Dec 4, 2020

Uh oh!

Uh oh!

kikisdeliveryservice commented Dec 4, 2020

Uh oh!

rphillips commented Dec 4, 2020

Uh oh!

sjenning commented Dec 4, 2020

Uh oh!

sjenning commented Dec 4, 2020

Uh oh!

openshift-ci-robot commented Dec 4, 2020

Uh oh!

openshift-bot commented Dec 4, 2020

Uh oh!

openshift-bot commented Dec 4, 2020

Uh oh!

rphillips commented Nov 30, 2020 •

edited

Loading