-
Notifications
You must be signed in to change notification settings - Fork 510
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
1.21.0 Status 500 when PATCHing /settings?tx=bottlerocket-launch #4135
Comments
We've just seen issues on some of our clusters trying to update to 1.21.0 too - seems a similar issue so pasting here - but if not let me know and I'll raise a separate ticket:
[ 7.882539] pluto[1498]: Unable to retrieve cluster name and AWS region from Bottlerocket API: Deserialization of configuration file failed: invalid type: sequence, expected a string at line 16 column 18 See 'systemctl status pluto.service' for details. [DEPEND] Dependency failed for Applies settings to create config files. [DEPEND] Dependency failed for Sets the hostname. [DEPEND] Dependency failed for Send signal to CloudFormation Stack. [DEPEND] Dependency failed for Bottlerocket initial configuration complete. [DEPEND] Dependency failed for Isolates configured.target. |
Example launch template to reproduce
|
Thank you @EthanKane-FD, @ramseymcgrathfd, and @patkinson01 for reporting this! We are looking at this now and will provide an update as soon as possible. |
For folks that have seen this issue, if you can include the userdata to reproduce, similar to @ramseymcgrathfd, that would help a ton, if you don't want to post to GitHub but can open an AWS Support case and provide it there, that would help too. |
Hey @yeazelm, thanks for checking. Me and @ramseymcgrathfd are on the same team so that's our user data config. |
Hi @yeazelm, please find below our userdata: `[settings.network] [[settings.container-registry.credentials]] [settings.kernel.sysctl] [settings.kubernetes.node-labels] [settings.bootstrap-containers.bottle] [settings.updates] [settings.kubernetes] |
@ramseymcgrathfd Do you by any chance have the rendered userdata? I tried apply some value to the template and failed to reproduce. Here is my userdata.
I was able to upgrade from v1.20.0 to v1.21.0. Using variant
|
I was able to reproduce this issue mentioned in - #4135 (comment) My userdata
The failure
|
This is happening because |
Hi @bcressey , we’ve see the error during a BRUPOP initiated update and haven’t made any changes to our userdata or no_proxy value which is a string. Presumably this is something which has changed in this latest AMI then? |
The bug is in the newer version of |
@sam-berning tracked this down to an issue with optional fields in the CredentialProvider structure. Omitting a field marked as optional will cause it to serialize to "null" which is then rejected by the datastore serializer.
Fully specifying the user data for the credential provider, by passing in a no-op environment variable, would avoid the issue:
|
@bcressey yeah good catch, it does reckon it'll need
|
Yup, that's indeed the right fix. Should be addressed as of bottlerocket-os/bottlerocket-settings-sdk#51. We've also updated the datastore serializer to handle null values correctly in bottlerocket-os/bottlerocket-core-kit#80, which should protect against this sort of bug moving forward |
We have released 1.21.1 that should allow a good upgrade from 1.20.5. Please let us know that it solves your problem! |
All good, thanks for a quick turnaround!! |
Hey thanks @yeazelm , have rolled this out on a few lab clusters and everything seems to be in order. Thanks again |
Hey there, we noticed an issue today with the latest version of bottlerocket. Any help would be greatly appreciated. Our new builds picked up the latest version and our nodes are failing to boot.
Image I'm using:
Bottlerocket OS 1.21.0
What I expected to happen:
What actually happened:
Bottlerocket AMI updated last night to (Bottlerocket OS 1.21.0 (aws-k8s-1.30)!) bottlerocket userdata configuration is failing.
Seeing the following in the system logs
How to reproduce the problem:
Upgrade from 1.20.5
The text was updated successfully, but these errors were encountered: