Add kernelType to MachineConfig #1330

sinnykumari · 2019-12-12T14:32:17Z

By default we use traditional kernel on RHCOS system. With this,
one can create a MachineConfig specifying kernelType: realtime .
Once this MachineConfig is applied to OpenShift cluster,
targetted RHCOS node should switch to using realtime kernel.

Currently, realtime kernel packages are shipped as rpm files in
RHCOS OSContainer. MCO fetches those rpms from OSContainer and
uses rpm-ostree to layer realtime kernel on host.

pkg/daemon/update.go

manifests/machineconfig.crd.yaml

pkg/apis/machineconfiguration.openshift.io/v1/helpers.go

pkg/daemon/update.go

kikisdeliveryservice

@sinnykumari is there any background on this change?

Also I would do an e2e on this when you are finished.

pkg/apis/machineconfiguration.openshift.io/v1/helpers.go

We should provide the real time kernel install and update pathes until the openshift/machine-config-operator#1330 will be merged. Signed-off-by: Artyom Lukianov <[email protected]>

sinnykumari · 2020-01-16T14:01:00Z

There were lot of changes since the original implementation were made. To avoid confusion and get a fresh review, have done force push. This should be good for review now.

pkg/daemon/update.go

runcom · 2020-01-16T14:10:04Z

just a super minor nit but the rest looks super sane
/approve

will let others to review again before getting this in

templates/common/_base/units/machine-config-daemon-host.service

pkg/daemon/update.go

cgwalters · 2020-01-16T16:20:22Z

Looking at this, I can definitely see the cost to the MCO of shipping kernel-rt the way we are; it would be simplest with two osImageURL, a bit less simple with two refs in the same container. But there'd be a lot of tech debt to pay down to implement that buildsystem side.

Also, while updating the RHCOS machine with realtime kernel, check if new kernel-rt packages are available. Update rt-kernel on host only when OS update contains updated kernel-rt.

sinnykumari · 2020-01-22T18:06:48Z

Retrying as it says level=error msg="Error: ResourceLimitExceeded: Task limit exceeded"

/test e2e-aws
/test e2e-aws-scaleup-rhel7

runcom · 2020-01-22T19:05:01Z

/skip

runcom · 2020-01-22T19:22:07Z

/skip

pkg/daemon/update.go

sinnykumari · 2020-01-23T06:37:37Z

I moved following rpm-ostree command execution back in the respective if condition so that we never accidentally run it.

if err := exec.Command("rpm-ostree", args...).Run(); err != nil {
    return fmt.Errorf("Failed to execute rpm-ostree %+q : %v", args, err)
}

runcom · 2020-01-23T08:43:01Z

/skip

pkg/daemon/update.go

runcom · 2020-01-23T09:52:36Z

op e2e failing for the usual flakes

level=error msg="Error: Error applying IAM policy to project \"openshift-gce-devel-ci\": Too many conflicts.  Latest error: Error setting IAM policy for project \"openshift-gce-devel-ci\": googleapi: Error 409: There were concurrent policy changes. Please retry the whole read-modify-write with exponential backoff., aborted"

sinnykumari · 2020-01-23T10:10:06Z

/retest e2e-gcp-op

sinnykumari · 2020-01-23T10:41:43Z

/test e2e-gcp-op

MarSik · 2020-01-23T13:14:37Z

/retest

runcom · 2020-01-23T13:17:58Z

/lgtm
/skip

openshift-ci-robot · 2020-01-23T13:18:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: runcom, sinnykumari

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [runcom,sinnykumari]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2020-01-23T13:26:18Z

@sinnykumari: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-scaleup-rhel7	`8ee18d9`	link	`/test e2e-aws-scaleup-rhel7`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

runcom · 2020-01-23T13:30:38Z

/skip

openshift-ci-robot requested review from ericavonb and kikisdeliveryservice December 12, 2019 14:33

cgwalters reviewed Dec 12, 2019

View reviewed changes

pkg/daemon/update.go Outdated Show resolved Hide resolved

ericavonb reviewed Dec 12, 2019

View reviewed changes

manifests/machineconfig.crd.yaml Outdated Show resolved Hide resolved

ericavonb reviewed Dec 12, 2019

View reviewed changes

pkg/apis/machineconfiguration.openshift.io/v1/helpers.go Outdated Show resolved Hide resolved

ericavonb reviewed Dec 12, 2019

View reviewed changes

pkg/daemon/update.go Outdated Show resolved Hide resolved

kikisdeliveryservice reviewed Dec 12, 2019

View reviewed changes

sinnykumari force-pushed the realtime-machineconfig branch from 83f571d to b24edd0 Compare December 17, 2019 11:27

sinnykumari changed the title ~~WIP: Add realtime to MachineConfig~~ WIP: Add kernelType to MachineConfig Dec 17, 2019

miabbott reviewed Dec 19, 2019

View reviewed changes

pkg/apis/machineconfiguration.openshift.io/v1/helpers.go Outdated Show resolved Hide resolved

sinnykumari force-pushed the realtime-machineconfig branch from b24edd0 to e3cbdd5 Compare December 20, 2019 15:18

openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 20, 2019

sinnykumari force-pushed the realtime-machineconfig branch from e3cbdd5 to 1eacdea Compare January 8, 2020 06:51

slintes mentioned this pull request Jan 10, 2020

Install RT kernel from RPMs included in RHCOS openshift-kni/performance-addon-operators#34

Merged

3 tasks

sinnykumari force-pushed the realtime-machineconfig branch from 1eacdea to 18d108e Compare January 16, 2020 11:46

sinnykumari changed the title ~~WIP: Add kernelType to MachineConfig~~ Add kernelType to MachineConfig Jan 16, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 16, 2020

sinnykumari force-pushed the realtime-machineconfig branch from 18d108e to 5db5a59 Compare January 16, 2020 14:00

runcom reviewed Jan 16, 2020

View reviewed changes

pkg/daemon/update.go Outdated Show resolved Hide resolved

ashcrow reviewed Jan 16, 2020

View reviewed changes

templates/common/_base/units/machine-config-daemon-host.service Outdated Show resolved Hide resolved

cgwalters reviewed Jan 16, 2020

View reviewed changes

pkg/controller: Validate whatever we can in MachineConfigs

5e6b3dc

Also, while updating the RHCOS machine with realtime kernel, check if new kernel-rt packages are available. Update rt-kernel on host only when OS update contains updated kernel-rt.

sinnykumari force-pushed the realtime-machineconfig branch from e048de0 to 36ccb1c Compare January 22, 2020 17:07

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Jan 22, 2020

runcom reviewed Jan 22, 2020

View reviewed changes

pkg/daemon/update.go Outdated Show resolved Hide resolved

sinnykumari force-pushed the realtime-machineconfig branch 3 times, most recently from b74cd5c to 67f3f11 Compare January 23, 2020 06:32

sinnykumari force-pushed the realtime-machineconfig branch from 67f3f11 to d9df159 Compare January 23, 2020 06:42

runcom reviewed Jan 23, 2020

View reviewed changes

pkg/daemon/update.go Outdated Show resolved Hide resolved

pkg/daemon: Fixup and error handling in switchKernel()

8ee18d9

sinnykumari force-pushed the realtime-machineconfig branch from d9df159 to 8ee18d9 Compare January 23, 2020 09:07

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 23, 2020

runcom removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 23, 2020

openshift-merge-robot merged commit d27a05d into openshift:master Jan 23, 2020

slintes mentioned this pull request Jan 23, 2020

Remove RT kernel workaround and use new MC config instead openshift-kni/performance-addon-operators#47

Merged

slintes mentioned this pull request Feb 13, 2020

Add periodic job for OCP with realtime workers openshift/release#7154

Merged

4 tasks

miabbott mentioned this pull request Feb 18, 2020

enhancements: support real-time kernels openshift/enhancements#166

Merged

Add kernelType to MachineConfig #1330

Add kernelType to MachineConfig #1330

Uh oh!

Conversation

sinnykumari commented Dec 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kikisdeliveryservice left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sinnykumari commented Jan 16, 2020

Uh oh!

Uh oh!

runcom commented Jan 16, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgwalters commented Jan 16, 2020

Uh oh!

sinnykumari commented Jan 22, 2020

Uh oh!

runcom commented Jan 22, 2020

Uh oh!

runcom commented Jan 22, 2020

Uh oh!

Uh oh!

sinnykumari commented Jan 23, 2020

Uh oh!

runcom commented Jan 23, 2020

Uh oh!

Uh oh!

runcom commented Jan 23, 2020

Uh oh!

sinnykumari commented Jan 23, 2020

Uh oh!

sinnykumari commented Jan 23, 2020

Uh oh!

MarSik commented Jan 23, 2020

Uh oh!

runcom commented Jan 23, 2020

Uh oh!

openshift-ci-robot commented Jan 23, 2020

Uh oh!

openshift-ci-robot commented Jan 23, 2020

Uh oh!

runcom commented Jan 23, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

sinnykumari commented Dec 12, 2019 •

edited

Loading