-
Notifications
You must be signed in to change notification settings - Fork 117
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal of adding metal3host operator. #268
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,314 @@ | ||
# Baremetal host single instance and multi-instance operator | ||
|
||
## Introduction | ||
|
||
Currently, the metal3 system is to set up a Kubernetes cluster on bare-metal hosts which is tightly bound with cluster-api and Kubernetes. We target to allocate bare-metal hosts that just pre-installing the OS, or pre-installing the applications as required. | ||
|
||
## Motivation | ||
|
||
To meet the scenery that only requires a pure bare-metal host or with a simple application installed. | ||
|
||
## Goals | ||
|
||
- Allocate one pure bare-metal host | ||
- Allocate multiple bare-metal hosts | ||
- Allocate bare-metal host with pre-installed applications | ||
- Specify the different configurations for different bare-metal host | ||
|
||
## Non-goals | ||
|
||
- Allocate different types of bare-metal hosts at one time | ||
|
||
## Proposal | ||
|
||
### Architecture | ||
|
||
The operators include the metal3Host operator, metal3HostDeployment operator, and data operator. | ||
![modules](images/metal3hostModules.png) | ||
|
||
The bare-metal host controller and the ironic are not newly developed. | ||
|
||
Metal3Host is a virtual instance. Users could define it to require a bare-metal host. In the definition, the user could specify the image to install, could set the selector to choose a specific bare-metal host, and could define the boot data to automatically start their own applications. | ||
|
||
Metal3HostDeployment will define a group of metal3Hosts with the same configuration. It could define the number of replicas. It could scale up/down. We even could auto-scale by some rules after the monitor function is leveraged. | ||
|
||
The data operator will set up the boot time configuration & routine. It includes three parts: the userData, the metaData, and the networkData. The userData includes some user-specified data such as installing a package, or running a command like "mkdir /test". The metaData defines some host-specific data such as hostname. The networkData includes the network configuration. The data is bound with the baremetal host. For example, the networkData defines a NIC with an IP, the NIC name is host related. So a dataTemplate should be pre-defined, it will be bound to the bare-metal host when a metal3host is defined and associated with a bare-metal host. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is not clear to me why do you need a new pair of data and dataTemplate CR, wouldn't it be easier to just re-use the existing metal3Data and metal3DataTemplates? You would also use the new data and the dataTemplate for the same config process as the existing metal3Data and metal3DataTemplate is used for. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, almost the same with metal3Data. One reason I want to create a new CR is, the metal3Data is bound to the metal3Machine, so it's not easy to modify if any change is required. Another is that the metal3Host and data could be an independent application(new a repo). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What's the mechanism of linking machines to metal3Data? If it's ownerReference, we may be able to link different CR's there. |
||
|
||
### Routine | ||
|
||
#### Pre-define a dataTemplate | ||
|
||
![createDataTemplate](images/createDataTemplate.png) | ||
|
||
#### Require a metal3host | ||
|
||
![createMetal3Host](images/createMetal3Host.png) | ||
|
||
#### Require a metal3HostDeployment | ||
|
||
![createMetal3HostDeployment](images/createMetal3HostDeployment.png) | ||
|
||
### Configuration | ||
|
||
#### metal3host | ||
|
||
'''golang | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Formatting is off here, should be
without extra indentation. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks. I'll refine it. |
||
// Metal3HostSpec defines the desired state of Metal3Host | ||
type Metal3HostSpec struct { | ||
// ProviderID will be the Metal3Host in ProviderID format | ||
// (://<bmh-uuid>) | ||
// +optional | ||
ProviderID *string `json:"providerID,omitempty"` | ||
|
||
ConsumerRef *corev1.ObjectReference `json:"consumerRef,omitempty"` | ||
|
||
// UserScript is specified by user and it will run on every boot. | ||
BootScript string `json:"bootScript,omitempty"` | ||
|
||
// User is the configuration used to initialize user in guest os. | ||
// Default user is `root`. | ||
User User `json:"user,omitempty"` | ||
|
||
// Image is the image to be provisioned. | ||
Image Image `json:"image,omitempty"` | ||
|
||
// HostSelector specifies matching criteria for labels on BareMetalHosts. | ||
// This is used to limit the set of BareMetalHost objects considered for | ||
// claiming for a Metal3Host. | ||
// +optional | ||
HostSelector HostSelector `json:"hostSelector,omitempty"` | ||
|
||
// When set to disabled, automated cleaning of host disks will be skipped | ||
// during provisioning and deprovisioning. | ||
// +kubebuilder:validation:Enum:=metadata;disabled | ||
// +optional | ||
AutomatedCleaningMode *string `json:"automatedCleaningMode,omitempty"` | ||
} | ||
''' | ||
|
||
#### metal3hostDeployment | ||
|
||
'''golang | ||
// Metal3HostDeploymentSpec defines the desired state of Metal3HostDeployment. | ||
type Metal3HostDeploymentSpec struct { | ||
// Number of desired machines. Defaults to 1. | ||
// This is a pointer to distinguish between explicit zero and not specified. | ||
// +optional | ||
// +kubebuilder:default=1 | ||
Replicas *int32 `json:"replicas,omitempty"` | ||
|
||
// HostSelector specifies matching criteria for labels on BareMetalHosts. | ||
// This is used to limit the set of BareMetalHost objects considered for | ||
// claiming for a machine. | ||
// +optional | ||
HostSelector HostSelector `json:"hostSelector,omitempty"` | ||
|
||
// Template describes the metal3 hosts that will be created. | ||
Template Metal3HostTemplateSpec `json:"template"` | ||
|
||
// Minimum number of seconds for which a newly created machine should | ||
// be ready. | ||
// Defaults to 0 (machine will be considered available as soon as it | ||
// is ready) | ||
// +optional | ||
MinReadySeconds *int32 `json:"minReadySeconds,omitempty"` | ||
|
||
// The number of old MachineSets to retain to allow rollback. | ||
// This is a pointer to distinguish between explicit zero and not specified. | ||
// Defaults to 1. | ||
// +optional | ||
RevisionHistoryLimit *int32 `json:"revisionHistoryLimit,omitempty"` | ||
|
||
// Indicates that the deployment is paused. | ||
// +optional | ||
Paused bool `json:"paused,omitempty"` | ||
|
||
// The maximum time in seconds for a deployment to make progress before it | ||
// is considered to be failed. The deployment controller will continue to | ||
// process failed deployments and a condition with a ProgressDeadlineExceeded | ||
// reason will be surfaced in the deployment status. Note that progress will | ||
// not be estimated during the time a deployment is paused. Defaults to 600s. | ||
// +optional | ||
ProgressDeadlineSeconds *int32 `json:"progressDeadlineSeconds,omitempty"` | ||
} | ||
|
||
// Metal3HostDeploymentStatus defines the observed state of Metal3HostDeployment. | ||
type Metal3HostDeploymentStatus struct { | ||
// The generation observed by the deployment controller. | ||
// +optional | ||
ObservedGeneration int64 `json:"observedGeneration,omitempty"` | ||
|
||
// Total number of non-terminated machines targeted by this deployment | ||
// (their labels match the selector). | ||
// +optional | ||
Replicas int32 `json:"replicas"` | ||
|
||
// Total number of non-terminated machines targeted by this deployment | ||
// that have the desired template spec. | ||
// +optional | ||
UpdatedReplicas int32 `json:"updatedReplicas"` | ||
|
||
// Total number of ready machines targeted by this deployment. | ||
// +optional | ||
ReadyReplicas int32 `json:"readyReplicas"` | ||
|
||
// Total number of available machines (ready for at least minReadySeconds) | ||
// targeted by this deployment. | ||
// +optional | ||
AvailableReplicas int32 `json:"availableReplicas"` | ||
|
||
// Total number of unavailable machines targeted by this deployment. | ||
// This is the total number of machines that are still required for | ||
// the deployment to have 100% available capacity. They may either | ||
// be machines that are running but not yet available or machines | ||
// that still have not been created. | ||
// +optional | ||
UnavailableReplicas int32 `json:"unavailableReplicas"` | ||
|
||
// Phase represents the current phase of a Metal3HostDeployment (ScalingUp, ScalingDown, Running, Failed, or Unknown). | ||
// +optional | ||
Phase string `json:"phase,omitempty"` | ||
|
||
// FailureReason indicates that there is a fatal problem reconciling the | ||
// state, and will be set to a descriptive error message. | ||
// +optional | ||
FailureReason *string `json:"failureReason,omitempty"` | ||
|
||
Conditions capi.Conditions `json:"conditions,omitempty"` | ||
} | ||
''' | ||
|
||
#### datatemplate | ||
|
||
'''golang | ||
// DataTemplateSpec defines the desired state of DataTemplate | ||
type DataTemplateSpec struct { | ||
//MetaData contains the information needed to generate the metadata secret | ||
// +optional | ||
MetaData *MetaData `json:"metaData,omitempty"` | ||
|
||
//NetworkData contains the information needed to generate the networkdata | ||
// secret | ||
// +optional | ||
NetworkData *NetworkData `json:"networkData,omitempty"` | ||
|
||
//UserData contains the information defined by user | ||
// secret | ||
// +optional | ||
UserData *UserData `json:"userData,omitempty"` | ||
} | ||
|
||
// NetworkData represents a networkData object | ||
type NetworkData struct { | ||
// Links is a structure containing lists of different types objects | ||
// +optional | ||
Links NetworkDataLink `json:"links,omitempty"` | ||
|
||
//Networks is a structure containing lists of different types objects | ||
// +optional | ||
Networks NetworkDataNetwork `json:"networks,omitempty"` | ||
|
||
//Services is a structure containing lists of different types objects | ||
// +optional | ||
Services NetworkDataService `json:"services,omitempty"` | ||
} | ||
|
||
// UserData contains the information defined by user | ||
type UserData struct { | ||
//Users define the user&password information | ||
// +optional | ||
Users []User `json:"users"` | ||
|
||
//WriteFile files to be created at the boot time | ||
// +optional | ||
WriteFiles []WriteFile `json:"write_files"` | ||
|
||
//BootCmd commands to be run at the boot time | ||
// +optional | ||
BootCmd []string `json:"bootcmd"` | ||
} | ||
|
||
// MetaData represents a keyand value of the metadata | ||
type MetaData struct { | ||
// Strings is the list of metadata items to be rendered from strings | ||
// +optional | ||
Strings []MetaDataString `json:"strings,omitempty"` | ||
|
||
// ObjectNames is the list of metadata items to be rendered from the name | ||
// of objects. | ||
// +optional | ||
ObjectNames []MetaDataObjectName `json:"objectNames,omitempty"` | ||
|
||
// Indexes is the list of metadata items to be rendered from the index of the | ||
// CUMetalData | ||
// +optional | ||
Indexes []MetaDataIndex `json:"indexes,omitempty"` | ||
|
||
// Namespaces is the list of metadata items to be rendered from the namespace | ||
// +optional | ||
Namespaces []MetaDataNamespace `json:"namespaces,omitempty"` | ||
|
||
// IPAddressesFromPool is the list of metadata items to be rendered as ip addresses. | ||
// +optional | ||
IPAddressesFromPool []FromPool `json:"ipAddressesFromIPPool,omitempty"` | ||
|
||
// PrefixesFromPool is the list of metadata items to be rendered as network prefixes. | ||
// +optional | ||
PrefixesFromPool []FromPool `json:"prefixesFromIPPool,omitempty"` | ||
|
||
// GatewaysFromPool is the list of metadata items to be rendered as gateway addresses. | ||
// +optional | ||
GatewaysFromPool []FromPool `json:"gatewaysFromIPPool,omitempty"` | ||
|
||
// DNSServersFromPool is the list of metadata items to be rendered as dns servers. | ||
// +optional | ||
DNSServersFromPool []FromPool `json:"dnsServersFromIPPool,omitempty"` | ||
|
||
// FromHostInterfaces is the list of metadata items to be rendered as MAC | ||
// addresses of the host interfaces. | ||
// +optional | ||
FromHostInterfaces []MetaDataHostInterface `json:"fromHostInterfaces,omitempty"` | ||
|
||
// FromLabels is the list of metadata items to be fetched from object labels | ||
// +optional | ||
FromLabels []MetaDataFromLabel `json:"fromLabels,omitempty"` | ||
|
||
// FromAnnotations is the list of metadata items to be fetched from object | ||
// Annotations | ||
// +optional | ||
FromAnnotations []MetaDataFromAnnotation `json:"fromAnnotations,omitempty"` | ||
} | ||
''' | ||
|
||
#### data | ||
|
||
'''golang | ||
// DataSpec defines the desired state of Data | ||
type DataSpec struct { | ||
// DataTemplate is the CUMetalDataTemplate this was generated from. | ||
Template corev1.ObjectReference `json:"template"` | ||
|
||
// MetaData points to the rendered MetaData secret. | ||
// +optional | ||
MetaData *corev1.SecretReference `json:"metaData,omitempty"` | ||
|
||
// NetworkData points to the rendered NetworkData secret. | ||
// +optional | ||
NetworkData *corev1.SecretReference `json:"networkData,omitempty"` | ||
|
||
// NetworkData points to the rendered UserData secret. | ||
// +optional | ||
UserData *corev1.SecretReference `json:"userData,omitempty"` | ||
} | ||
|
||
// DataStatus defines the observed state of Data | ||
type DataStatus struct { | ||
// Ready is a flag set to True if the secrets were rendered properly | ||
// +optional | ||
Ready bool `json:"ready"` | ||
|
||
// ErrorMessage contains the error message | ||
// +optional | ||
ErrorMessage *string `json:"errorMessage,omitempty"` | ||
} | ||
''' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we actually want to have Metal3 in the names. The difference between Metal3Host and BareMetalHost is not going to be obvious. It's pity that the word "Deployment" has a different meaning...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I even wonder if we need an object at all. Cannot we only use BareMetalHost's and a new BareMetalDeployment (renamed from Metal3HostDeployment)? BareMetalHosts already have all the deployment information inside of them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your review.
The difference between metal3Host and baremetalHost is that metal3Host is a virtual host. When the user requires a host, a metal3Host instance will be created. The metal3Host instance has some features the user defines, such as the operating system, the user name/password, the bootup application, etc. After the instance is created, it will choose one suitable baremetalHost.
About the name, I'm not clear which is best.
Thanks & Regards,
Wei
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another side, if one baremetalHost is down, the metal3Host could choose another baremetalHost. We also could label the baremetalHost, so users could define their requirements such as they need a strong CPU host. Then the metal3Host could choose a host by the label. The metal3Host represents the users' requirement, the baremetal3Host represents the physical machine. I think it's more flexible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see so the host is handled like a workload, like a pod.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, I can agree with a new object, but I cannot agree with calling it Metal3Host. BareMetalHost is also a Metal3 host technically. Could be BareMetalInstance, ProvisionedBareMetal, BareMetalProvisioning... Having a Metal3 prefix is not terribly helpful, definitely does not solve the potential confusion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://docs.google.com/document/d/1FL8oA0_WNcPdiC-0zrToinl8aol-NPQUjWe-OkcvTqA/edit has some prior art with regards to naming.