Label BOTTLEROCKET-DATA at runtime
#2807
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
etungsten
commented
Feb 15, 2023
etungsten
changed the title
bcressey
reviewed
Feb 15, 2023
|
Push above addresses @bcressey 's comments. Changes a approach a bit. We don't append the fallback data partition if the image layout is set to "unified" since it's redundant. Update PR description with new testing. |
|
Push removes This way it doesn't look like the boot is hanging when the oneshots are waiting for their data partitions. |
bcressey
reviewed
Feb 15, 2023
etungsten
changed the title
BOTTLEROCKET-DATA and create /local fs at runtime
|
Pushes above addresses @bcressey's comments. Also adds a commit with the changes described in #2807 (comment) that moves the filesystem creation for /local from image build time to host runtime. Re-did the tests as described in the PR description and the results are still as expected with the only difference being the new |
|
Push above renames |
zmrow
reviewed
Feb 16, 2023
|
Push above adds a comment to explain the wait in `label-data-alternative'. Also adds a dependency constraint between |
|
Added boot performance impact information to the PR description. |
etungsten
changed the title
BOTTLEROCKET-DATA and create /local fs at runtimeBOTTLEROCKET-DATA at runtime
|
Push above rebases onto develop. edit: Oops, I did push all my changes along with the rebase. |
|
Pushes above (including the rebase push) addresses @bcressey 's comments.
Testing:Split partition/ normal setup:DATA-A fails to grow as expected because there's not enough free space on the root volume. DATA-B successfully labels and grows: repart-local grows On reboot, the lsblk: Unified:
lsblk: Split but with data volume missing, root volume increased to 12GiB:
lsblk: blkid: |
|
Push above fixes commit message |
|
Push above fixes some inaccurate comments. |
|
Push above is just a rebase to get github actions going again. |
|
Push above rebases onto develop and resolves conflict. |
bcressey
approved these changes
Mar 8, 2023
Changes partitioning to always keep a data partition on the os image. We grow and label data partition at runtime. We will still create a separate data partiion on the second volume/disk for "split" partition layouts. We add two new services that run as part of 'local-fs.target' and before 'local.mount' and 'repart-local.service': * 'label-data-a.service' * 'label-data-b.service' 'label-data-a' and 'label-data-b' will "compete" and both try to label 'BOTTLEROCKET-DATA' first with the partition they're each waiting for. 'label-data-a' waits for the data partition that resides on the OS disk image. Once that device is ready, we call 'systemd-repart' to relabel it as 'BOTTLEROCKET-DATA' and grow it as much as possible. 'label-data-b' calls 'systemd-repart' to label the data partition on the data image as 'BOTTLEROCKET-DATA' and grow the partition to fill the remainder of the disk. All of this lets the host to boot if data partition on the data image doesn't exist and the root filesystem disk has leftover extra space to accommodate a reasonably-sized back-up data partition.
'mark-successful-boot' now also updates BOTTLEROCKET_PRIVATE's GPT table attribute to indicate whether boot has ever succeeded before. Add a new 'has-boot-ever-succeeded' subcommand for checking if boot has ever succeeded before
This adds additional safeguards to prevent a different data partition from being labeled and resized if the host has ever booted successfully in the past with an original data partition.
|
Push above removes the redundant repart conf and now just keeps the single Tested things and they still work as expected. |
bcressey
approved these changes
Mar 8, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue number:
Resolves #2822
Description of changes:
Testing done:
Instance with the
aws-k8s-1.24AMI as is without modification.The host comes up as expected and system logs doesn't show any abnormalities:
The status
label-data-preferred,label-data-alternative,repart-localshows the data partition gets labelled and the local fs gets created and grows successfully.bash-5.1# systemctl status label-data-preferred label-data-alternative repart-local ● label-data-preferred.service - Label preferred data partition Loaded: loaded (/aarch64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/label-data-preferred.service; enabled; vendor preset: enabled) Active: active (exited) since Fri 2023-02-17 23:16:16 UTC; 2h 26min ago Main PID: 881 (code=exited, status=0/SUCCESS) Feb 17 23:16:16 localhost systemd[1]: Starting Label preferred data partition... Feb 17 23:16:16 localhost systemd-repart[881]: Applying changes. Feb 17 23:16:16 localhost systemd-repart[881]: Storage does not support discard, not discarding gap at beginning of disk. Feb 17 23:16:16 localhost systemd-repart[881]: Growing existing partition 0. Feb 17 23:16:16 localhost systemd-repart[881]: Setting partition label of existing partition 0. Feb 17 23:16:16 localhost systemd-repart[881]: Writing new partition table. Feb 17 23:16:16 localhost systemd-repart[881]: Telling kernel to reread partition table. Feb 17 23:16:16 localhost systemd-repart[881]: All done. Feb 17 23:16:16 localhost systemd[1]: Finished Label preferred data partition. ● label-data-alternative.service - Label alternative data partition Loaded: loaded (/aarch64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/label-data-alternative.service; enabled; vendor preset: enabled) Active: active (exited) since Fri 2023-02-17 23:16:16 UTC; 2h 26min ago Main PID: 880 (code=killed, signal=ABRT) Feb 17 23:16:16 localhost systemd[1]: Starting Label alternative data partition... Feb 17 23:16:16 localhost systemd-repart[880]: Assertion 'amount <= total' failed at src/partition/repart.c:533, function charge_size(). Aborting. Feb 17 23:16:16 localhost systemd[1]: Finished Label alternative data partition. ● repart-local.service - Resize Data Partition Loaded: loaded (/aarch64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/repart-local.service; enabled; vendor preset: enabled) Active: active (exited) since Fri 2023-02-17 23:16:17 UTC; 2h 26min ago Main PID: 927 (code=exited, status=0/SUCCESS) Feb 17 23:16:17 localhost systemd[1]: Starting Resize Data Partition... Feb 17 23:16:17 localhost systemd-growfs[927]: Successfully resized "/local" to 19.9G bytes. Feb 17 23:16:17 localhost systemd[1]: Finished Resize Data Partition.blkidshows the expected new partition UUIDs for the data partitionslsblklooks normalUpon reboot,
label-data-*oneshot service units do not run since a partition labelledBOTTLEROCKET-DATAalready exists:bash-5.1# systemctl status label-data-pref label-data-alt ○ label-data-pref.service Loaded: masked (Reason: Unit label-data-pref.service is masked.) Active: inactive (dead) Condition: start condition failed at Wed 2023-02-15 23:10:42 UTC; 3min 11s ago Feb 15 23:10:42 localhost systemd[1]: Label preferred data partition was skipped because of a failed condition check (ConditionPathIsSymbolicLink=!/dev/disk/by-partlabel/BOTTLEROCKET-DATA). ○ label-data-alt.service Loaded: masked (Reason: Unit label-data-alt.service is masked.) Active: inactive (dead) Condition: start condition failed at Wed 2023-02-15 23:10:42 UTC; 3min 11s ago Feb 15 23:10:42 localhost systemd[1]: Label alternative data partition was skipped because of a failed condition check (ConditionPathIsSymbolicLink=!/dev/disk/by-partlabel/BOTTLEROCKET-DATA).Instance with the
aws-k8s-1.24AMI but with the default 20 GB data EBS volume removed. The single root EBS volume size increased from 2GB to 12GBThe host boots and comes up even though it's missing the data EBS volume:
e (dead)
blkidshows the expected static partition UUIDs and no second NVMe disklsblklooks normal.AMI with "unified" image layout
The host comes up as fine. There is no change to the image layout from before this change.
The status
label-data-preferred,label-data-alternative,repart-localshows the data partition gets labelled and the local fs gets created and grows successfully:bash-5.1# systemctl status label-data-preferred label-data-alternative repart-local ● label-data-preferred.service - Label preferred data partition Loaded: loaded (/aarch64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/label-data-preferred.service; enabled; vendor preset: enabled) Active: active (exited) since Fri 2023-02-17 23:58:23 UTC; 1h 53min ago Main PID: 867 (code=exited, status=0/SUCCESS) Feb 17 23:58:23 localhost systemd[1]: Starting Label preferred data partition... Feb 17 23:58:23 localhost systemd-repart[867]: Applying changes. Feb 17 23:58:23 localhost systemd-repart[867]: Storage does not support discard, not discarding gap at beginning of disk. Feb 17 23:58:23 localhost systemd-repart[867]: Growing existing partition 12. Feb 17 23:58:23 localhost systemd-repart[867]: Setting partition label of existing partition 12. Feb 17 23:58:23 localhost systemd-repart[867]: Writing new partition table. Feb 17 23:58:23 localhost systemd-repart[867]: Telling kernel to reread partition table. Feb 17 23:58:23 localhost systemd-repart[867]: All done. Feb 17 23:58:23 localhost systemd[1]: Finished Label preferred data partition. ○ label-data-alternative.service - Label alternative data partition Loaded: loaded (/aarch64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/label-data-alternative.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Fri 2023-02-17 23:59:53 UTC; 1h 52min ago └─ ConditionPathIsSymbolicLink=!/dev/disk/by-partlabel/BOTTLEROCKET-DATA was not met Feb 17 23:59:53 ip-192-168-9-148.us-west-2.compute.internal systemd[1]: Label alternative data partition was skipped because of a failed condition check (ConditionPathIsSymbolicLink=!/dev/disk/by-partlabel/BOTTLEROCKET-DATA). ● repart-local.service - Resize Data Partition Loaded: loaded (/aarch64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/repart-local.service; enabled; vendor preset: enabled) Active: active (exited) since Fri 2023-02-17 23:58:24 UTC; 1h 53min ago Main PID: 895 (code=exited, status=0/SUCCESS) Feb 17 23:58:24 localhost systemd[1]: Starting Resize Data Partition... Feb 17 23:58:24 localhost systemd-repart[894]: No changes. Feb 17 23:58:24 localhost systemd-growfs[895]: Successfully resized "/local" to 20.0G bytes. Feb 17 23:58:24 localhost systemd[1]: Finished Resize Data Partition.blkidlooks fine:lsblklooks normalSingle EBS volume with default 2 GB size and no additional space
The host fails boot as expected during
label-data-altsince there is not enough space to label and grow the alternative data partition.Testing at scale
Launched 1000 instances with "split" layout AMI with both EBS attached. All hosts come up and join the cluster
Launched 1500 instance with split image AMI with a single 12 GB EBS volume. All hosts comes up and joins the cluster.
Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.