[testing] Add coreos-keep-cgroup-v1.service

[jlebon]

In f31, the default cgroup changed to v2. However, we've decided to stay
on v1 for the time being. Thus, we don't want older nodes upgrading to
f31 to be forced into v2.

Add a tiny service which just scans the BLS configs and injects the
systemd.unified_cgroup_hierarchy karg as needed.

For more information, see:
coreos/fedora-coreos-tracker#292
coreos/fedora-coreos-streams#26 (comment)

[jlebon]

(Note you'll want to look at the commit itself since the overall PR diff includes the revert from #237.)

[dustymabe]

testing now

[jlebon]

OK, so one issue here is:

Removed:
  brotli-1.0.7-3.fc30.x86_64
  curl-7.65.3-4.fc30.x86_64
  libcurl-7.65.3-4.fc30.x86_64
  libmetalink-0.1.3-8.fc30.x86_64
  libssh-0.9.0-5.fc30.x86_64
  libssh-config-0.9.0-5.fc30.noarch
Added:
  curl-minimal-7.66.0-1.fc31.x86_64
  libcurl-minimal-7.66.0-1.fc31.x86_64

I think the issue is similar to what I described in coreos/coreos-assembler#553 (comment) (i.e. the lockfile not being canonical means libsolv is still free to pick newer packages with different names which meet all the same constraints), which would be fixed by coreos/rpm-ostree#1858.

A hack for now though (since we're going to nuke all this right after) is to just add all those NEVRAs directly in the manifest.

[dustymabe]

OK, so one issue here is:

Removed:
  brotli-1.0.7-3.fc30.x86_64
  curl-7.65.3-4.fc30.x86_64
  libcurl-7.65.3-4.fc30.x86_64
  libmetalink-0.1.3-8.fc30.x86_64
  libssh-0.9.0-5.fc30.x86_64
  libssh-config-0.9.0-5.fc30.noarch
Added:
  curl-minimal-7.66.0-1.fc31.x86_64
  libcurl-minimal-7.66.0-1.fc31.x86_64

I see this too.

I think the issue is similar to what I described in coreos/coreos-assembler#553 (comment) (i.e. the lockfile not being canonical means libsolv is still free to pick newer packages with different names which meet all the same constraints), which would be fixed by coreos/rpm-ostree#1858.

So because curl-minimal provides curl but is a newer version, curl-minimal-7.66.0-1.fc31.x86_64 chosen over curl-7.66.0-1.fc31.x86_64?

A hack for now though (since we're going to nuke all this right after) is to just add all those NEVRAs directly in the manifest.

+1

[jlebon]

OK, I just tested an upgrade from 30.20191014.0 to this and then to whatever the latest f31 I had locally and it seems like it worked:

[root@coreos repo]# systemctl status coreos-keep-cgroup-v1.service
...
Nov 26 16:43:47 localhost systemd[1]: Starting CoreOS Keep CGroups v1...
Nov 26 16:43:47 localhost coreos-keep-cgroup-v1[744]: ostree-1-fedora-coreos.conf: injected systemd.unified_cgroup_hierarchy=0
Nov 26 16:43:47 localhost coreos-keep-cgroup-v1[744]: ostree-2-fedora-coreos.conf: injected systemd.unified_cgroup_hierarchy=0
Nov 26 16:43:47 localhost systemd[1]: Started CoreOS Keep CGroups v1.
...
[root@coreos repo]# rpm-ostree rebase $PWD:fedora/x86_64/coreos/testing-devel --experimental
...
[root@coreos repo]# reboot
...
[root@coreos ~]# grep -o systemd.unified_cgroup_hierarchy=0 /proc/cmdline
systemd.unified_cgroup_hierarchy=0

So because curl-minimal provides curl but is a newer version, curl-minimal-7.66.0-1.fc31.x86_64 chosen over curl-7.66.0-1.fc31.x86_64?

Right, exactly. (That said, we should probably evaluate whether we should switch to [lib]curl-minimal -- it's not clear offhand what features we lose).

[dustymabe]

The idea here being that coreos-keep-cgroup-v1.service would get removed right after we do this next release and thus it will be removed when we update to F31 FCOS as well? i.e. we don't have to worry about this service running forever and possibly causing side effects down the road?

[dustymabe]

in my testing this seems to work!

[jlebon]

The idea here being that coreos-keep-cgroup-v1.service would get removed right after we do this next release and thus it will be removed when we update to F31 FCOS as well?

Yeah, exactly. It'll only be in that one OSTree commit we build. In fact, would be nice if we could do like RHCOS and have the ability to ship only the update (I mean, we have all the pieces in place, we just need to develop the SOP for it).

[jlebon]

OK, this is rebased now!

[dustymabe]

LGTM