[21.05] extract osds into separate units #778
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
osnyx
force-pushed
the
PL-131646-extract-osds-into-separate-units
branch
5 times, most recently
from
53a52a4
to
3e9d5c9
Compare
osnyx
approved these changes
Sep 19, 2023
osnyx
changed the title
We had hosts run out of memory and the OSDs were all subsumed in the same cgroup which caused them to be killed together because systemd values consistency in a unit's state ... There might be interesting cases here if NixOS decides to activate/deactivate anything, but hopefully the current setup will cause the "group unit" to catch this. I can imagine cases where this might not be the case, but I can't show that they actually exist currently. Re PL-131646 co-authored by [email protected]
Before moving individual OSDs to their own units, the old `fc-ceph-osds` unit with all osds needs to be stopped to allow the start of the separated OSD units again. This is achieved by renaming that service, making the NixOS switch-to-configuration logic stop the old service and start the new service at system update time. This has been tested in our dev cluster: stopping the following units: fc-blockdev.service, fc-ceph-mgr.service, fc-ceph-mon.service, fc-ceph-osds.service, fc-ceph-rgw.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket, systemd-udevd.service, telegraf.service NOT restarting the following changed units: systemd-fsck@dev-disk-by\x2dlabel-boot.service activating the configuration... setting up /etc... reloading user units for os... setting up tmpfiles reloading the following units: dbus.service restarting the following units: sensu-client.service starting the following units: fc-blockdev.service, fc-ceph-mgr.service, fc-ceph-mon.service, fc-ceph-rgw.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket, telegraf.service the following new units were started: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], fc-ceph-osds-all.service, fc-update-channel.timer
follow-up to 5d9e656 where both packages had only been removed as references from the overlay, but not as files from the repo
osnyx
force-pushed
the
PL-131646-extract-osds-into-separate-units
branch
from
fe798ae
to
f7c0f31
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We had hosts run out of memory and the OSDs were all subsumed in the same cgroup which caused them to be killed together because systemd values consistency in a unit's state ...
There might be interesting cases here if NixOS decides to activate/deactivate anything, but hopefully the current setup will cause the "group unit" to catch this. I can imagine cases where this might not be the case, but I can't show that they actually exist currently.
Re PL-131646
@flyingcircusio/release-managers
Release process
Impact: internal only
Changelog:
fc-cephaccordinglyfc-ceph-osds.service->fc-ceph-osds-all.serviceSecurity implications
fc-cephoperations tests in dev cluster