Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Add bootCommands to cloud-init file generation #11271

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
31406da
Add bootCommands cloud-init file generation
davidumea Sep 26, 2024
97311d6
fixup: go lint
davidumea Oct 18, 2024
4367074
fixup: lint: added missing newline
davidumea Oct 18, 2024
0c23af8
fixup: add changes from verify-gen
davidumea Oct 18, 2024
504dfd5
fixup: filter tests
davidumea Oct 18, 2024
19d52df
fixup: clearer documentation for BootCommands, PreKubeadmCommands and…
davidumea Oct 18, 2024
51068a2
fixup: verify: reflect new documentation in the crds
davidumea Oct 18, 2024
8ed0171
fixup: restore v1alpha3,v1alpha4 and add conversions
davidumea Oct 22, 2024
332e78d
fixup: non-nil values for bootcommands in cloudinit tests
davidumea Oct 22, 2024
28edd6c
fixup: lint
davidumea Oct 22, 2024
8918ff3
fixup: verify crds
davidumea Nov 5, 2024
94835dc
fixup: use yaml names in go doc comments
davidumea Nov 25, 2024
84e06c6
fixup: removed extra linebreak
davidumea Nov 25, 2024
bd63a05
fixup: remove bootcommands from testing AdditionalFileEncodings
davidumea Nov 26, 2024
a4f3bdb
fixup: make sure bootcommands show up in generated output
davidumea Nov 26, 2024
4673fb0
fixup: add test for join cp commands
davidumea Nov 26, 2024
c059138
fixup: add test for join node commands
davidumea Nov 26, 2024
6fb468d
fixup: ensure commands show up in the right section
davidumea Dec 3, 2024
810f7d6
fixup: fix webhook
davidumea Jan 10, 2025
03cbf01
fixup: update copyright year
davidumea Jan 10, 2025
b7c575a
fixup: make sure bootCommands is not configured when ignition format …
davidumea Jan 13, 2025
0484cdc
fixup: use constant when checking kubeadmconfig format
davidumea Jan 24, 2025
3db48fa
fixup: add bootcommands check to validateIgnition function
davidumea Jan 24, 2025
23a608c
fixup: remove webhook check
davidumea Jan 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 24 additions & 2 deletions bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,20 @@ type KubeadmConfigSpec struct {
// +optional
Mounts []MountPoints `json:"mounts,omitempty"`

// preKubeadmCommands specifies extra commands to run before kubeadm runs
// bootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd
// module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is no equivalent for the ignition format, it is good to prevent user confusion by introducing a validation webhook check on kubeadmConfig.spec.format == “cloud-init” for this field to be set.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unfamiliar with ignition so I don't know if there is an equivalent format 😕. Where would you suggest to put this validation webhook check?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bootstrap/kubeadm/internal/webhooks/kubeadmconfig.go seems to be the place for validation webhooks code

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean to validate that format == "clout-init" only if ignition is not used? Is that based on feature.KubeadmBootstrapFormatIgnition or is there another way of checking if ignition should be used?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Format field on the config designates the type of configuration used. If it is cloud-init, then it is not ignition :)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay thanks, is this how you meant? b7c575a

Also, I believe the default value for the format is "cloud-config", but please let me know if I misunderstood and it should be "cloud-init".

// +optional
BootCommands []BootCommand `json:"bootCommands,omitempty"`

// preKubeadmCommands specifies extra commands to run before kubeadm runs.
// With cloud-init, this is prepended to the runcmd module configuration, and is typically executed in
// the cloud-final.service systemd unit. In Ignition, this is prepended to /etc/kubeadm.sh.
// +optional
PreKubeadmCommands []string `json:"preKubeadmCommands,omitempty"`

// postKubeadmCommands specifies extra commands to run after kubeadm runs
// postKubeadmCommands specifies extra commands to run after kubeadm runs.
// With cloud-init, this is appended to the runcmd module configuration, and is typically executed in
// the cloud-final.service systemd unit. In Ignition, this is appended to /etc/kubeadm.sh.
// +optional
PostKubeadmCommands []string `json:"postKubeadmCommands,omitempty"`

Expand Down Expand Up @@ -347,6 +356,16 @@ func (c *KubeadmConfigSpec) validateIgnition(pathPrefix *field.Path) field.Error
}
}

if c.BootCommands != nil {
allErrs = append(
allErrs,
field.Forbidden(
pathPrefix.Child("bootCommands"),
cannotUseWithIgnition,
),
)
}

if c.DiskSetup == nil {
return allErrs
}
Expand Down Expand Up @@ -720,3 +739,6 @@ type Filesystem struct {

// MountPoints defines input for generated mounts in cloud-init.
type MountPoints []string

// BootCommand defines input for each bootcmd command in cloud-init.
type BootCommand []string
30 changes: 30 additions & 0 deletions bootstrap/kubeadm/api/v1beta1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 29 additions & 0 deletions bootstrap/kubeadm/internal/cloudinit/boot_commands.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
/*
Copyright 2025 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package cloudinit

const (
bootCommandsTemplate = `{{ define "boot_commands" -}}
{{- if . }}
bootcmd:{{ range . }}
- {{ range . }}- {{ . }}
{{ end -}}
{{- end -}}
{{- end -}}
{{- end -}}
`
)
5 changes: 5 additions & 0 deletions bootstrap/kubeadm/internal/cloudinit/cloudinit.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ const (
// BaseUserData is shared across all the various types of files written to disk.
type BaseUserData struct {
Header string
BootCommands []bootstrapv1.BootCommand
PreKubeadmCommands []string
PostKubeadmCommands []string
AdditionalFiles []bootstrapv1.File
Expand Down Expand Up @@ -83,6 +84,10 @@ func generate(kind string, tpl string, data interface{}) ([]byte, error) {
return nil, errors.Wrap(err, "failed to parse files template")
}

if _, err := tm.Parse(bootCommandsTemplate); err != nil {
return nil, errors.Wrap(err, "failed to parse boot commands template")
}

if _, err := tm.Parse(commandsTemplate); err != nil {
return nil, errors.Wrap(err, "failed to parse commands template")
}
Expand Down
110 changes: 101 additions & 9 deletions bootstrap/kubeadm/internal/cloudinit/cloudinit_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ func TestNewInitControlPlaneAdditionalFileEncodings(t *testing.T) {
cpinput := &ControlPlaneInput{
BaseUserData: BaseUserData{
Header: "test",
BootCommands: nil,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a test for BootCommands not nil for init, join CP, join workers?

Copy link
Author

@davidumea davidumea Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added non-nil values for BootCommands in TestNewInitControlPlaneCommands and TestNewJoinControlPlaneAdditionalFileEncodings, did you want me to add more tests or is this enough? I couldn't find one for "join workers" daf4f97

Copy link
Member

@fabriziopandini fabriziopandini Nov 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WRT to changes in TestNewInitControlPlaneCommands, TestNewJoinControlPlaneAdditionalFileEncodings I have dedicated comments.

We still need test coverage for join and join CP (if they are missing, kindly add them so we pay down some tech debt before adding more on top)

PreKubeadmCommands: nil,
PostKubeadmCommands: nil,
AdditionalFiles: []bootstrapv1.File{
Expand Down Expand Up @@ -94,9 +95,12 @@ func TestNewInitControlPlaneCommands(t *testing.T) {

cpinput := &ControlPlaneInput{
BaseUserData: BaseUserData{
BootCommands: []bootstrapv1.BootCommand{
{"echo", "$(date) hello BootCommands!"},
},
Header: "test",
PreKubeadmCommands: []string{`"echo $(date) ': hello world!'"`},
PostKubeadmCommands: []string{"echo $(date) ': hello world!'"},
PreKubeadmCommands: []string{`"echo $(date) ': hello PreKubeadmCommands!'"`},
PostKubeadmCommands: []string{"echo $(date) ': hello PostKubeadmCommands!'"},
AdditionalFiles: nil,
WriteFiles: nil,
Users: nil,
Expand All @@ -117,13 +121,18 @@ func TestNewInitControlPlaneCommands(t *testing.T) {
out, err := NewInitControlPlane(cpinput)
g.Expect(err).ToNot(HaveOccurred())

expectedCommands := []string{
`"\"echo $(date) ': hello world!'\""`,
`"echo $(date) ': hello world!'"`,
}
for _, f := range expectedCommands {
g.Expect(out).To(ContainSubstring(f))
}
expectedBootCmd := `bootcmd:
- - echo
- $(date) hello BootCommands!`

g.Expect(out).To(ContainSubstring(expectedBootCmd))

expectedRunCmd := `runcmd:
- "\"echo $(date) ': hello PreKubeadmCommands!'\""
- 'kubeadm init --config /run/kubeadm/kubeadm.yaml && echo success > /run/cluster-api/bootstrap-success.complete'
- "echo $(date) ': hello PostKubeadmCommands!'"`

g.Expect(out).To(ContainSubstring(expectedRunCmd))
}

func TestNewInitControlPlaneDiskMounts(t *testing.T) {
Expand All @@ -132,6 +141,7 @@ func TestNewInitControlPlaneDiskMounts(t *testing.T) {
cpinput := &ControlPlaneInput{
BaseUserData: BaseUserData{
Header: "test",
BootCommands: nil,
PreKubeadmCommands: nil,
PostKubeadmCommands: nil,
WriteFiles: nil,
Expand Down Expand Up @@ -194,6 +204,7 @@ func TestNewJoinControlPlaneAdditionalFileEncodings(t *testing.T) {

cpinput := &ControlPlaneJoinInput{
BaseUserData: BaseUserData{
BootCommands: nil,
Header: "test",
PreKubeadmCommands: nil,
PostKubeadmCommands: nil,
Expand Down Expand Up @@ -247,6 +258,7 @@ func TestNewJoinControlPlaneExperimentalRetry(t *testing.T) {
cpinput := &ControlPlaneJoinInput{
BaseUserData: BaseUserData{
Header: "test",
BootCommands: nil,
PreKubeadmCommands: nil,
PostKubeadmCommands: nil,
UseExperimentalRetry: true,
Expand Down Expand Up @@ -315,3 +327,83 @@ func Test_useKubeadmBootstrapScriptPre1_31(t *testing.T) {
})
}
}

func TestNewJoinControlPlaneCommands(t *testing.T) {
g := NewWithT(t)

cpinput := &ControlPlaneJoinInput{
BaseUserData: BaseUserData{
BootCommands: []bootstrapv1.BootCommand{
{"echo", "$(date) hello BootCommands!"},
},
Header: "test",
PreKubeadmCommands: []string{`"echo $(date) ': hello PreKubeadmCommands!'"`},
PostKubeadmCommands: []string{"echo $(date) ': hello PostKubeadmCommands!'"},
AdditionalFiles: nil,
WriteFiles: nil,
Users: nil,
NTP: nil,
},
Certificates: secret.Certificates{},
JoinConfiguration: "my-join-config",
}

for _, certificate := range cpinput.Certificates {
certificate.KeyPair = &certs.KeyPair{
Cert: []byte("some certificate"),
Key: []byte("some key"),
}
}

out, err := NewJoinControlPlane(cpinput)
g.Expect(err).ToNot(HaveOccurred())

expectedBootCmd := `bootcmd:
- - echo
- $(date) hello BootCommands!`

g.Expect(out).To(ContainSubstring(expectedBootCmd))

expectedRunCmd := `runcmd:
- "\"echo $(date) ': hello PreKubeadmCommands!'\""
- kubeadm join --config /run/kubeadm/kubeadm-join-config.yaml && echo success > /run/cluster-api/bootstrap-success.complete
- "echo $(date) ': hello PostKubeadmCommands!'"`

g.Expect(out).To(ContainSubstring(expectedRunCmd))
}

func TestNewJoinNodeCommands(t *testing.T) {
g := NewWithT(t)

nodeinput := &NodeInput{
BaseUserData: BaseUserData{
BootCommands: []bootstrapv1.BootCommand{
{"echo", "$(date) hello BootCommands!"},
},
Header: "test",
PreKubeadmCommands: []string{`"echo $(date) ': hello PreKubeadmCommands!'"`},
PostKubeadmCommands: []string{"echo $(date) ': hello PostKubeadmCommands!'"},
AdditionalFiles: nil,
WriteFiles: nil,
Users: nil,
NTP: nil,
},
JoinConfiguration: "my-join-config",
}

out, err := NewNode(nodeinput)
g.Expect(err).ToNot(HaveOccurred())

expectedBootCmd := `bootcmd:
- - echo
- $(date) hello BootCommands!`

g.Expect(out).To(ContainSubstring(expectedBootCmd))

expectedRunCmd := `runcmd:
- "\"echo $(date) ': hello PreKubeadmCommands!'\""
- kubeadm join --config /run/kubeadm/kubeadm-join-config.yaml && echo success > /run/cluster-api/bootstrap-success.complete
- "echo $(date) ': hello PostKubeadmCommands!'"`

g.Expect(out).To(ContainSubstring(expectedRunCmd))
}
1 change: 1 addition & 0 deletions bootstrap/kubeadm/internal/cloudinit/controlplane_init.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ const (
owner: root:root
permissions: '0640'
content: "This placeholder file is used to create the /run/cluster-api sub directory in a way that is compatible with both Linux and Windows (mkdir -p /run/cluster-api does not work with Windows)"
{{- template "boot_commands" .BootCommands }}
runcmd:
{{- template "commands" .PreKubeadmCommands }}
- 'kubeadm init --config /run/kubeadm/kubeadm.yaml {{.KubeadmVerbosity}} && {{ .SentinelFileCommand }}'
Expand Down
1 change: 1 addition & 0 deletions bootstrap/kubeadm/internal/cloudinit/controlplane_join.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ const (
owner: root:root
permissions: '0640'
content: "This placeholder file is used to create the /run/cluster-api sub directory in a way that is compatible with both Linux and Windows (mkdir -p /run/cluster-api does not work with Windows)"
{{- template "boot_commands" .BootCommands }}
runcmd:
{{- template "commands" .PreKubeadmCommands }}
- {{ .KubeadmCommand }} && {{ .SentinelFileCommand }}
Expand Down
Loading
Loading