types: option to disable hyperthreading #1392

staebler · 2019-03-08T18:23:45Z

Add a hyperthreading field to machine pools with options of Enabled or Disabled. The default is Enabled.

When a machine pool has hyperthreading disabled, the Machines asset for that pool creates a MachineConfig manifest for that pool. The ignition config lays down a file at /etc/default/rhcos/karg/nosmt with no contents. The machine-config-daemon is responsible for reading that file and setting the nosmt karg for the machine.

runcom · 2019-03-12T14:15:29Z

is there a design doc for this and/or has this been discussed somewhere with the MCO team? Could you also CC us for work related to the MCO

staebler · 2019-03-12T15:13:54Z

is there a design doc for this and/or has this been discussed somewhere with the MCO team? Could you also CC us for work related to the MCO

There is a diagram that Steven Milner created after the design meeting last week with Ian McLeod and Abhinav Dahiya. You received and responded to the email with that diagram. If there are comments on the design, I suppose that that diagram would be the best place to put them. I will adjust this PR to reflect any changes made to that design.

runcom · 2019-03-12T15:22:24Z

There is a diagram that Steven Milner created after the design meeting last week with Ian McLeod and Abhinav Dahiya. You received and responded to the email with that diagram. If there are comments on the design, I suppose that that diagram would be the best place to put them. I will adjust this PR to reflect any changes made to that design.

Thanks! I have that in my backlog still...

cgwalters · 2019-03-12T17:06:48Z

We have an MCD issue here openshift/machine-config-operator#388 - I would have expected any work on this to show up there personally.

ashcrow · 2019-03-22T19:04:10Z

pkg/asset/machines/machineconfig/karg.go

~~Should be /etc/default/rhcos/kargs ... where kargs is a file with requested changes to operate on.~~

Out of date.

ashcrow · 2019-03-22T19:07:13Z

pkg/asset/machines/machineconfig/hyperthreading.go

~~PTAL https://url.corp.redhat.com/0d85a22 for what's expected in terms of the local service. There's a POC there that's turning into a usable service.~~

Out of date

Users can push manifests during bootstrap that of the form: ```yaml --- ``` Especially for the installer: setting authorizes_keys [1] and setting hyperthreading [2] will push a manifest that includes multiple machineconfig objects for control-plane (master) and compute (worker) roles. Single file with multiple k8s objects separated by `---` is also a supported structure for `oc create|apply` ie. there is a high chance that users trying to push machineconfigs at install time might create such files. This commit allows bootstrap controller to read all k8s objects, even ones described above to find all the `machineconfiguration.openshift.io` Objects. [1]: openshift/installer#1150 [2]: openshift/installer#1392

abhinavdahiya · 2019-03-26T00:43:59Z

rebase around Remove cluster-api cluster object dependency #1449
to restructure so that I can reuse the plumbing you have setup to rebase machines: add the authorized keys for a pool using a machine config #1150
and update the file location for kargs based on https://jira.coreos.com/browse/RHCOS-6?focusedCommentId=78493&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-78493

ashcrow · 2019-03-26T18:04:39Z

See: openshift/pivot#47

abhinavdahiya · 2019-04-03T00:17:24Z

@staebler @wking @ashcrow rebased and ready for review.

wking

To naming nits, otherwise this looks good to me.

pkg/asset/machines/machineconfig/hyperthreading.go

ashcrow · 2019-04-03T14:02:25Z

pkg/asset/machines/master_test.go

Decodes to: ADD nosmt

👍

ashcrow · 2019-04-03T14:03:00Z

pkg/asset/machines/master_test.go

Decodes to: ADD nosmt

👍

ashcrow · 2019-04-03T14:04:05Z

pkg/asset/machines/worker_test.go

Decodes to: ADD nosmt

👍

ashcrow · 2019-04-03T14:04:21Z

pkg/asset/machines/worker_test.go

Decodes to: ADD nosmt

👍

ashcrow · 2019-04-03T14:06:29Z

pkg/asset/machines/machineconfig/hyperthreading.go

How are these assembled if other folks add MachineConfigs with their own kernel options? Do we need to set Append on this?

/etc/pivot/kernel-args is a RHCOS impl and it only accepts the ADD nosmt so no append.

/etc/pivot/kernel-args is a RHCOS impl and it only accepts the ADD nosmt so no append.

Oh, so there's no ability for ADD some-other-flag at the moment?

If someone attempts to ADD or DELETE any other argument it will skip it and log. Only items in the whitelist inside pivot are allowed.

https://github.com/openshift/pivot/blob/master/cmd/root.go#L43-L45

ashcrow

File writing and content looks good to me!

ashcrow · 2019-04-03T16:10:47Z

--- FAIL: TestMasterGenerateMachineConfigs (0.04s)
    --- FAIL: TestMasterGenerateMachineConfigs/no_key_hyperthreading_disabled (0.00s)

wking · 2019-04-04T23:58:58Z

/lgtm

abhinavdahiya · 2019-04-05T00:03:53Z

/hold

need to get approval on the hyperthreading exception.

eparis · 2019-04-08T20:32:03Z

When this PR is ready, @abhinavdahiya you can remove the hold.

wking · 2019-04-08T21:01:54Z

I'm still happy with this as it stands. But I'll leave hold-pulling to @abhinavdahiya ;).

abhinavdahiya · 2019-04-08T23:32:18Z

Waiting on RHCOS bootimage that contains the pivot changes mentioned here https://github.com/openshift/pivot/blob/master/cmd/root.go#L43-L45

This PR only works if we bump rhcos.json to correct RHCOS version that includes the changes mentioned ^^

Waiting on OS team
cc @ashcrow @cgwalters

ashcrow · 2019-04-09T13:23:57Z

cc @darkmuggle

ashcrow · 2019-04-09T14:07:40Z

In chat I passed over an updated rhcos.json from ART builds that contain the latest pivot, rpm-ostree, and cri-o.

cgwalters · 2019-04-09T16:01:22Z

We must have a publicly-accessible location for the images (currently libvirt but really we need all of the openstack, bare metal etc.), i.e. something like this patch:

git diff
diff --git a/data/data/rhcos.json b/data/data/rhcos.json
index 60878c249..04224bb09 100644
--- a/data/data/rhcos.json
+++ b/data/data/rhcos.json
@@ -46,7 +46,7 @@
             "hvm": "ami-0eac581fbaa9fa9c6"
         }
     },
-    "baseURI": "https://releases-rhcos.svc.ci.openshift.org/storage/releases/ootpa/410.8.20190325.0/",
+    "baseURI": "https://download.redhat.com/rhel/coreos/410.8.20190325.0/",
     "buildid": "410.8.20190325.0",
     "images": {
         "metal-bios": {

Or replace download.redhat.com with try.openshift.com/rhcos-bootimages/ or whatever.

ashcrow · 2019-04-09T18:43:13Z

@imcleod is helping make this available.

ashcrow · 2019-04-09T18:51:42Z

FWIW AMI's can be used to test now, and the payload has the proper content if one wants to test manually.

enable or disable hyperthreading for machines. The default is for hyperthreading to be enabled. RHCOS ships with pivot.service that uses the `/etc/pivot/kernel-args` to override the kernel arguments for hosts. Adding `nosmt` kernel argument switches hyperthreading off. Add MachineConfig to disable hyperthreading for control plane and compute that have the hyperthreading option disabled.

this new version of RHCOS include the latest pivot `0.0.4-2.el8` that includes the changes required to act on the hyperthreading disabled kernel arg at /etc/pivot/kernel-args

sdodson · 2019-04-11T19:00:29Z

/lgtm
only change was the addition of RHCOS data

openshift-ci-robot · 2019-04-11T19:02:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sdodson, staebler, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [staebler,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

abhinavdahiya · 2019-04-11T21:26:36Z

created an AWS cluster with:

apiVersion: v1beta4
baseDomain: devcluster.openshift.com
controlPlane:
  name: master
  hyperthreading: Disabled
  replicas: 3
  platform:
    aws:
      zones:
      - us-east-1a
      - us-east-1c
      - us-east-1d
compute:
- name: worker
  replicas: 3
  platform:
    aws:
      zones:
      - us-east-1a
      - us-east-1c
      - us-east-1d
metadata:
  name: adahiya-1
networking:
  clusterNetworks:
  - cidr: 10.128.0.0/14
    hostSubnetLength: 9
  machineCIDR: 10.0.0.0/16
  serviceCIDR: 172.30.0.0/16
  type: OpenShiftSDN
platform:
  aws:
    region: us-east-1

on master machine:

[core@ip-10-0-130-101 ~]$ sudo cat /etc/pivot/kernel-args
ADD nosmt[core@ip-10-0-130-101 ~]$ cat /proc/cmdline
BOOT_IMAGE=/ostree/rhcos-59aadfa277d7d06047d323e53ccf6e937589fc3412e3d1ef7a52c2f2086ac29d/vmlinuz-4.18.0-80.el8.x86_64 no_timer_check console=tty0 console=ttyS0,115200n8 rootflags=defaults,prjquota rw root=UUID=0dc164db-5750-41e7-ba19-b9489acc69b6 ostree=/ostree/boot.1/rhcos/59aadfa277d7d06047d323e53ccf6e937589fc3412e3d1ef7a52c2f2086ac29d/0 coreos.oem.id=ec2 ignition.platform.id=ec2 nosmt
[core@ip-10-0-130-101 ~]$

seems to me as working :yay:

/hold cancel

abhinavdahiya · 2019-04-11T23:32:47Z

e2e-failure

FLAG: --version=\"false\"\nI0411 23:01:04.426031       1 flags.go:33] FLAG: --vmodule=\"\"\nI0411 23:01:04.426038       1 flags.go:33] FLAG: --write-config-to=\"\"\nI0411 23:01:05.621527       1 serving.go:312] Generated self-signed cert (/var/run/kubernetes/kube-scheduler.crt, /var/run/kubernetes/kube-scheduler.key)\nfailed to create listener: failed to listen on 0.0.0.0:10251: listen tcp 0.0.0.0:10251: bind: address already in use\n"

Failing tests:

[Feature:Platform] Managed cluster should have no crashlooping pods in core namespaces over two minutes [Suite:openshift/conformance/parallel]

Writing JUnit report to /tmp/artifacts/junit/junit_e2e_20190411-230420.xml

error: 2 fail, 569 pass, 548 skip (26m28s)

/retest

openshift-bot · 2019-04-12T01:05:56Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-04-12T03:54:49Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-04-12T05:38:51Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot requested review from crawford and russellb March 8, 2019 18:23

openshift-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Mar 8, 2019

ashcrow reviewed Mar 22, 2019

View reviewed changes

This was referenced Mar 25, 2019

pkg/daemon: allow reboots from external componenents openshift/machine-config-operator#576

Closed

machines: add the authorized keys for a pool using a machine config #1150

Merged

abhinavdahiya mentioned this pull request Mar 26, 2019

controller/bootstrap: use files with multiple yaml documents openshift/machine-config-operator#577

Merged

wking reviewed Apr 3, 2019

View reviewed changes

pkg/asset/machines/machineconfig/hyperthreading.go Outdated Show resolved Hide resolved

pkg/asset/machines/machineconfig/hyperthreading.go Outdated Show resolved Hide resolved