Skip to content

Add Linux VM extension bootstrap script and conditions #1232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
k8s-ci-robot merged 3 commits into from
Mar 25, 2021
Merged

Add Linux VM extension bootstrap script and conditions #1232

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9027dde
Set AzureMachine and AzureMachinePool BootstrapSucceeded condition us…
Mar 11, 2021
342ef15
:warning: Rename VMState to ProvisioningState
Mar 16, 2021
638c14f
Update troubleshooting docs
Mar 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions api/v1alpha3/zz_generated.conversion.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion api/v1alpha4/azuremachine_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ type AzureMachineStatus struct {

// VMState is the provisioning state of the Azure virtual machine.
// +optional
VMState *VMState `json:"vmState,omitempty"`
VMState *ProvisioningState `json:"vmState,omitempty"`

// ErrorReason will be set in the event that there is a terminal problem
// reconciling the Machine and will contain a succinct value suitable
Expand Down
40 changes: 26 additions & 14 deletions api/v1alpha4/conditions_consts.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,7 @@ import clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha4"
// AzureCluster Conditions and Reasons
const (
// NetworkInfrastructureReadyCondition reports of current status of cluster infrastructure
NetworkInfrastructureReadyCondition = "NetworkInfrastructureReady"
// LoadBalancerProvisioningReason API Server endpoint for the loadbalancer
LoadBalancerProvisioningReason = "LoadBalancerProvisioning"
// LoadBalancerProvisioningFailedReason used for failure during provisioning of loadbalancer.
LoadBalancerProvisioningFailedReason = "LoadBalancerProvisioningFailed"
NetworkInfrastructureReadyCondition clusterv1.ConditionType = "NetworkInfrastructureReady"
// NamespaceNotAllowedByIdentity used to indicate cluster in a namespace not allowed by identity
NamespaceNotAllowedByIdentity = "NamespaceNotAllowedByIdentity"
)
Expand All @@ -34,20 +30,36 @@ const (
const (
// VMRunningCondition reports on current status of the Azure VM.
VMRunningCondition clusterv1.ConditionType = "VMRunning"
// VMNCreatingReason used when the vm creation is in progress.
VMNCreatingReason = "VMCreating"
// VMNUpdatingReason used when the vm updating is in progress.
VMNUpdatingReason = "VMUpdating"
// VMNotFoundReason used when the vm couldn't be retrieved.
VMNotFoundReason = "VMNotFound"
// VMCreatingReason used when the vm creation is in progress.
VMCreatingReason = "VMCreating"
// VMUpdatingReason used when the vm updating is in progress.
VMUpdatingReason = "VMUpdating"
// VMDeletingReason used when the vm is in a deleting state.
VMDDeletingReason = "VMDeleting"
// VMStoppedReason vm is in a stopped state.
VMStoppedReason = "VMStopped"
VMDeletingReason = "VMDeleting"
// VMProvisionFailedReason used for failures during vm provisioning.
VMProvisionFailedReason = "VMProvisionFailed"
// WaitingForClusterInfrastructureReason used when machine is waiting for cluster infrastructure to be ready before proceeding.
WaitingForClusterInfrastructureReason = "WaitingForClusterInfrastructure"
// WaitingForBootstrapDataReason used when machine is waiting for bootstrap data to be ready before proceeding.
WaitingForBootstrapDataReason = "WaitingForBootstrapData"
// BootstrapSucceededCondition reports the result of the execution of the boostrap data on the machine.
BootstrapSucceededCondition = "BoostrapSucceeded"
// BootstrapInProgressReason is used to indicate the bootstrap data has not finished executing.
BootstrapInProgressReason = "BootstrapInProgress"
// BootstrapFailedReason is used to indicate the bootstrap process ran into an error.
BootstrapFailedReason = "BootstrapFailed"
)

// AzureMachinePool Conditions and Reasons
const (
// ScaleSetRunningCondition reports on current status of the Azure Scale Set.
ScaleSetRunningCondition clusterv1.ConditionType = "ScaleSetRunning"
// ScaleSetCreatingReason used when the scale set creation is in progress.
ScaleSetCreatingReason = "ScaleSetCreating"
// ScaleSetUpdatingReason used when the scale set updating is in progress.
ScaleSetUpdatingReason = "ScaleSetUpdating"
// ScaleSetDeletingReason used when the scale set is in a deleting state.
ScaleSetDeletingReason = "ScaleSetDeleting"
// ScaleSetProvisionFailedReason used for failures during scale set provisioning.
ScaleSetProvisionFailedReason = "ScaleSetProvisionFailed"
)
40 changes: 22 additions & 18 deletions api/v1alpha4/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,24 +203,28 @@ type PublicIPSpec struct {
}

// VMState describes the state of an Azure virtual machine.
// DEPRECATED: use ProvisioningState
type VMState string

// ProvisioningState describes the provisioning state of an Azure resource.
type ProvisioningState string

const (
// VMStateCreating ...
VMStateCreating VMState = "Creating"
// VMStateDeleting ...
VMStateDeleting VMState = "Deleting"
// VMStateFailed ...
VMStateFailed VMState = "Failed"
// VMStateMigrating ...
VMStateMigrating VMState = "Migrating"
// VMStateSucceeded ...
VMStateSucceeded VMState = "Succeeded"
// VMStateUpdating ...
VMStateUpdating VMState = "Updating"
// VMStateDeleted represents a deleted VM
// NOTE: This state is specific to capz, and does not have corresponding mapping in Azure API (https://docs.microsoft.com/en-us/azure/virtual-machines/states-lifecycle#provisioning-states)
VMStateDeleted VMState = "Deleted"
// Creating ...
Creating ProvisioningState = "Creating"
// Deleting ...
Deleting ProvisioningState = "Deleting"
// Failed ...
Failed ProvisioningState = "Failed"
// Migrating ...
Migrating ProvisioningState = "Migrating"
// Succeeded ...
Succeeded ProvisioningState = "Succeeded"
// Updating ...
Updating ProvisioningState = "Updating"
// Deleted represents a deleted VM
// NOTE: This state is specific to capz, and does not have corresponding mapping in Azure API (https://docs.microsoft.com/en-us/azure/virtual-machines/states-billing#provisioning-states)
Deleted ProvisioningState = "Deleted"
)

// VM describes an Azure virtual machine.
Expand All @@ -235,9 +239,9 @@ type VM struct {
OSDisk OSDisk `json:"osDisk,omitempty"`
StartupScript string `json:"startupScript,omitempty"`
// State - The provisioning state, which only appears in the response.
State VMState `json:"vmState,omitempty"`
Identity VMIdentity `json:"identity,omitempty"`
Tags Tags `json:"tags,omitempty"`
State ProvisioningState `json:"vmState,omitempty"`
Identity VMIdentity `json:"identity,omitempty"`
Tags Tags `json:"tags,omitempty"`

// Addresses contains the addresses associated with the Azure VM.
Addresses []corev1.NodeAddress `json:"addresses,omitempty"`
Expand Down
2 changes: 1 addition & 1 deletion api/v1alpha4/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion azure/converters/vm.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ func SDKToVM(v compute.VirtualMachine) (*infrav1.VM, error) {
vm := &infrav1.VM{
ID: to.String(v.ID),
Name: to.String(v.Name),
State: infrav1.VMState(to.String(v.ProvisioningState)),
State: infrav1.ProvisioningState(to.String(v.ProvisioningState)),
}

if v.VirtualMachineProperties != nil && v.VirtualMachineProperties.HardwareProfile != nil {
Expand Down
4 changes: 2 additions & 2 deletions azure/converters/vmss.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ func SDKToVMSS(sdkvmss compute.VirtualMachineScaleSet, sdkinstances []compute.Vi
vmss := &infrav1exp.VMSS{
ID: to.String(sdkvmss.ID),
Name: to.String(sdkvmss.Name),
State: infrav1.VMState(to.String(sdkvmss.ProvisioningState)),
State: infrav1.ProvisioningState(to.String(sdkvmss.ProvisioningState)),
}

if sdkvmss.Sku != nil {
Expand All @@ -52,7 +52,7 @@ func SDKToVMSS(sdkvmss compute.VirtualMachineScaleSet, sdkinstances []compute.Vi
ID: to.String(vm.ID),
InstanceID: to.String(vm.InstanceID),
Name: to.String(vm.OsProfile.ComputerName),
State: infrav1.VMState(to.String(vm.ProvisioningState)),
State: infrav1.ProvisioningState(to.String(vm.ProvisioningState)),
}

if vm.LatestModelApplied != nil {
Expand Down
17 changes: 17 additions & 0 deletions azure/defaults.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,17 @@ const (
ControlPlaneNodeGroup = "control-plane"
)

const (
// bootstrapExtensionRetries is the number of retries in the BootstrapExtensionCommand.
// NOTE: the overall timeout will be number of retries * retry sleep, in this case 240 * 5s = 1200s.
bootstrapExtensionRetries = 240
// bootstrapExtensionSleep is the duration in seconds to sleep before each retry in the BootstrapExtensionCommand.
bootstrapExtensionSleep = 5
// bootstrapSentinelFile is the file written by bootstrap provider on machines to indicate successful bootstrapping,
// as defined by the Cluster API Bootstrap Provider contract (https://cluster-api.sigs.k8s.io/developer/providers/bootstrap.html).
bootstrapSentinelFile = "/run/cluster-api/bootstrap-success.complete"
)

// GenerateBackendAddressPoolName generates a load balancer backend address pool name.
func GenerateBackendAddressPoolName(lbName string) string {
return fmt.Sprintf("%s-%s", lbName, "backendPool")
Expand Down Expand Up @@ -262,6 +273,12 @@ func GetBootstrappingVMExtension(osType string, cloud string) (name, publisher,
return "", "", ""
}

// BootstrapExtensionCommand is the command that runs on the Boostrap VM extension to check for bootstrap success.
// The command checks for the existence of the bootstrapSentinelFile on the machine, with retries and sleep between retries.
func BootstrapExtensionCommand() string {
return fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then return 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
}

// UserAgent specifies a string to append to the agent identifier.
func UserAgent() string {
return fmt.Sprintf("cluster-api-provider-azure/%s", version.Get().String())
Expand Down
54 changes: 52 additions & 2 deletions azure/scope/machine.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import (
"encoding/base64"
"encoding/json"
"strings"
"time"

"github.com/Azure/go-autorest/autorest/to"
"github.com/go-logr/logr"
Expand Down Expand Up @@ -241,6 +242,9 @@ func (m *MachineScope) VMExtensionSpecs() []azure.VMExtensionSpec {
VMName: m.Name(),
Publisher: publisher,
Version: version,
ProtectedSettings: map[string]string{
"commandToExecute": azure.BootstrapExtensionCommand(),
},
},
}
}
Expand Down Expand Up @@ -350,15 +354,15 @@ func (m *MachineScope) SetProviderID(v string) {
}

// VMState returns the AzureMachine VM state.
func (m *MachineScope) VMState() infrav1.VMState {
func (m *MachineScope) VMState() infrav1.ProvisioningState {
if m.AzureMachine.Status.VMState != nil {
return *m.AzureMachine.Status.VMState
}
return ""
}

// SetVMState sets the AzureMachine VM state.
func (m *MachineScope) SetVMState(v infrav1.VMState) {
func (m *MachineScope) SetVMState(v infrav1.ProvisioningState) {
m.AzureMachine.Status.VMState = &v
}

Expand All @@ -382,6 +386,52 @@ func (m *MachineScope) SetFailureReason(v capierrors.MachineStatusError) {
m.AzureMachine.Status.FailureReason = &v
}

// SetBootstrapConditions sets the AzureMachine BootstrapSucceeded condition based on the extension provisioning states.
func (m *MachineScope) SetBootstrapConditions(provisioningState string, extensionName string) error {
switch infrav1.ProvisioningState(provisioningState) {
case infrav1.Succeeded:
m.V(4).Info("extension provisioning state is succeeded", "vm extension", extensionName, "virtual machine", m.Name())
conditions.MarkTrue(m.AzureMachine, infrav1.BootstrapSucceededCondition)
return nil
case infrav1.Creating:
m.V(4).Info("extension provisioning state is creating", "vm extension", extensionName, "virtual machine", m.Name())
conditions.MarkFalse(m.AzureMachine, infrav1.BootstrapSucceededCondition, infrav1.BootstrapInProgressReason, clusterv1.ConditionSeverityInfo, "")
return azure.WithTransientError(errors.New("extension still provisioning"), 30*time.Second)
case infrav1.Failed:
m.V(4).Info("extension provisioning state is failed", "vm extension", extensionName, "virtual machine", m.Name())
conditions.MarkFalse(m.AzureMachine, infrav1.BootstrapSucceededCondition, infrav1.BootstrapFailedReason, clusterv1.ConditionSeverityError, "")
return azure.WithTerminalError(errors.New("extension state failed"))
default:
return nil
}
}

// UpdateStatus updates the AzureMachine status.
func (m *MachineScope) UpdateStatus() {
switch m.VMState() {
case infrav1.Succeeded:
m.V(2).Info("VM is running", "id", m.GetVMID())
conditions.MarkTrue(m.AzureMachine, infrav1.VMRunningCondition)
case infrav1.Creating:
m.V(2).Info("VM is creating", "id", m.GetVMID())
conditions.MarkFalse(m.AzureMachine, infrav1.VMRunningCondition, infrav1.VMCreatingReason, clusterv1.ConditionSeverityInfo, "")
case infrav1.Updating:
m.V(2).Info("VM is updating", "id", m.GetVMID())
conditions.MarkFalse(m.AzureMachine, infrav1.VMRunningCondition, infrav1.VMUpdatingReason, clusterv1.ConditionSeverityInfo, "")
case infrav1.Deleting:
m.Info("Unexpected VM deletion", "id", m.GetVMID())
conditions.MarkFalse(m.AzureMachine, infrav1.VMRunningCondition, infrav1.VMDeletingReason, clusterv1.ConditionSeverityWarning, "")
case infrav1.Failed:
m.Error(errors.New("Failed to create or update VM"), "VM is in failed state", "id", m.GetVMID())
m.SetFailureReason(capierrors.UpdateMachineError)
m.SetFailureMessage(errors.Errorf("Azure VM state is %s", m.VMState()))
conditions.MarkFalse(m.AzureMachine, infrav1.VMRunningCondition, infrav1.VMProvisionFailedReason, clusterv1.ConditionSeverityError, "")
default:
m.V(2).Info("VM state is undefined", "id", m.GetVMID())
conditions.MarkUnknown(m.AzureMachine, infrav1.VMRunningCondition, "", "")
}
}

// SetAnnotation sets a key value annotation on the AzureMachine.
func (m *MachineScope) SetAnnotation(key, value string) {
if m.AzureMachine.Annotations == nil {
Expand Down
37 changes: 32 additions & 5 deletions azure/scope/machinepool.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ import (
"context"
"encoding/base64"
"fmt"
"time"

clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha4"
"sigs.k8s.io/cluster-api/util/conditions"

"github.com/Azure/go-autorest/autorest/to"
"github.com/go-logr/logr"
Expand Down Expand Up @@ -148,7 +152,7 @@ func (m *MachinePoolScope) SetProviderID(v string) {
}

// ProvisioningState returns the AzureMachinePool provisioning state.
func (m *MachinePoolScope) ProvisioningState() infrav1.VMState {
func (m *MachinePoolScope) ProvisioningState() infrav1.ProvisioningState {
if m.AzureMachinePool.Status.ProvisioningState != nil {
return *m.AzureMachinePool.Status.ProvisioningState
}
Expand Down Expand Up @@ -225,14 +229,14 @@ func (m *MachinePoolScope) GetLongRunningOperationState() *infrav1.Future {
}

// SetProvisioningState sets the AzureMachinePool provisioning state.
func (m *MachinePoolScope) SetProvisioningState(v infrav1.VMState) {
func (m *MachinePoolScope) SetProvisioningState(v infrav1.ProvisioningState) {
switch {
case v == infrav1.VMStateSucceeded && *m.MachinePool.Spec.Replicas == m.AzureMachinePool.Status.Replicas:
case v == infrav1.Succeeded && *m.MachinePool.Spec.Replicas == m.AzureMachinePool.Status.Replicas:
// vmss is provisioned with enough ready replicas
m.AzureMachinePool.Status.ProvisioningState = &v
case v == infrav1.VMStateSucceeded && *m.MachinePool.Spec.Replicas != m.AzureMachinePool.Status.Replicas:
case v == infrav1.Succeeded && *m.MachinePool.Spec.Replicas != m.AzureMachinePool.Status.Replicas:
// not enough ready or too many ready replicas we must still be scaling up or down
updatingState := infrav1.VMStateUpdating
updatingState := infrav1.Updating
m.AzureMachinePool.Status.ProvisioningState = &updatingState
default:
m.AzureMachinePool.Status.ProvisioningState = &v
Expand All @@ -259,6 +263,26 @@ func (m *MachinePoolScope) SetFailureReason(v capierrors.MachineStatusError) {
m.AzureMachinePool.Status.FailureReason = &v
}

// SetBootstrapConditions sets the AzureMachinePool BootstrapSucceeded condition based on the extension provisioning states.
func (m *MachinePoolScope) SetBootstrapConditions(provisioningState string, extensionName string) error {
switch infrav1.ProvisioningState(provisioningState) {
case infrav1.Succeeded:
m.V(4).Info("extension provisioning state is succeeded", "vm extension", extensionName, "scale set", m.Name())
conditions.MarkTrue(m.AzureMachinePool, infrav1.BootstrapSucceededCondition)
Copy link
Copy Markdown
Contributor Author

@CecileRobertMichon CecileRobertMichon Mar 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it'd be nice to set conditions per VMSS VM, hopefully that gets easier with #819

Copy link
Copy Markdown
Contributor Author

@CecileRobertMichon CecileRobertMichon Mar 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now it's at the AzureMachinePool.Status level:

status:
    conditions:
    - lastTransitionTime: "2021-03-16T23:53:39Z"
      status: "True"
      type: BoostrapSucceeded
    - lastTransitionTime: "2021-03-16T23:49:37Z"
      status: "True"
      type: ScaleSetRunning

return nil
case infrav1.Creating:
m.V(4).Info("extension provisioning state is creating", "vm extension", extensionName, "scale set", m.Name())
conditions.MarkFalse(m.AzureMachinePool, infrav1.BootstrapSucceededCondition, infrav1.BootstrapInProgressReason, clusterv1.ConditionSeverityInfo, "")
return azure.WithTransientError(errors.New("extension still provisioning"), 30*time.Second)
case infrav1.Failed:
m.V(4).Info("extension provisioning state is failed", "vm extension", extensionName, "scale set", m.Name())
conditions.MarkFalse(m.AzureMachinePool, infrav1.BootstrapSucceededCondition, infrav1.BootstrapFailedReason, clusterv1.ConditionSeverityError, "")
return azure.WithTerminalError(errors.New("extension state failed"))
default:
return nil
}
}

// AdditionalTags merges AdditionalTags from the scope's AzureCluster and AzureMachinePool. If the same key is present in both,
// the value from AzureMachinePool takes precedence.
func (m *MachinePoolScope) AdditionalTags() infrav1.Tags {
Expand Down Expand Up @@ -368,6 +392,9 @@ func (m *MachinePoolScope) VMSSExtensionSpecs() []azure.VMSSExtensionSpec {
ScaleSetName: m.Name(),
Publisher: publisher,
Version: version,
ProtectedSettings: map[string]string{
"commandToExecute": azure.BootstrapExtensionCommand(),
},
},
}
}
Expand Down
2 changes: 1 addition & 1 deletion azure/services/scalesets/mock_scalesets/scalesets_mock.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading