Add bare metal fulfillment enhancement proposal

[tzumainn]

No description provided.

[okrieg]

Few tweaks;

One larger question, I discussed what we are doing with Rashid Khan, and Sean Cohen, and they wanted us to engage with them to see if they could enhance ESI to address challenges. So, not clear that we want to replace ESI after all, which is what this enhancement request keeps saying.

@mhrivnak @tzumainn @larsks should we create a seperate enhancement request to fix ESI that we can expose to them?

From the original design document:
ESI is primarily just a particular configuration of OpenStack services, and there are a number of reasons why we may eventually replace it:

• ESI is hard to deploy and configure. Red Hat’s OpenStack installers as of RHOS 17.1 have been difficult to configure and maintain, and ESI has added additional capabilities that complicate the deployment.
• Network management is provided by Neutron, leading to challenges when managing bare metal network interfaces (since we need to make API calls to two separate services and reconcile the information retrieved from both).
• Due to a variety of design decisions, OpenStack services are noticeably slow for this use, leading to a frustrating user experience.

In addition, I think we want to say that since ESI was designed for single VM at a time; it doesn't support parallel configuration; e.g., we would want to go to a switch and change all the required ports in one shot for switches that support.

[okrieg]

Please don't use word tag here, confusing for people thinking about vlan tags

[tzumainn]

Changed to property

[okrieg]

not sure if we want to integrate higher level use cases:

• Some tenants will want to deploy alternative clusters, e.g., SLURM
• We want to support broad use by Red Hat that involves developers installing their own cluster software, own OSes...

[tzumainn]

Added these use cases as examples of what this proposal is aiming to enable.

[tzumainn]

Thanks for the feedback! I've updated the document to address the comments (with one question where I didn't quite get what you were saying).

So, not clear that we want to replace ESI after all, which is what this enhancement request keeps saying.

I don't know that we actually keep saying this at all; perhaps the language can be softened, but most of the discussion around replacement is in the context of saying that we want to design bare metal fulfillment to allow easy replacement of the bare metal management implementation. Perhaps it would make sense to make that even more explicit?

@mhrivnak @tzumainn @larsks should we create a seperate enhancement request to fix ESI that we can expose to them?

I think it's worth having a discussion around this, but a formal request may be a little premature.

[hakasapl]

Maybe too implementation specific, but the user also probably wants to customize the L1 part of the connection, specifically whether to aggregate ports and apply the vlans on the port channel, or on the individual ports.

[tzumainn]

The intent is to create a separate networking service that can be used for both bare metal and VMs; I think this requirement would fall there.

[knikolla]

My understanding is that this would also include members of the specific project, even if they're not PIs or Managers. In the same way that today all members of an ESI project can manage the bare metal resources within the project.

[tzumainn]

Good point - updated to specify that the "tenant" is an organization which is represented by a ColdFront PI/manager.

[knikolla]

So a tenant would be requesting a host from a resource class and the fulfillment service would select one for the user? In other words, the user doesn't select a specific host? Will the fulfillment service take into account any characteristics, such as rack locality, etc?

[tzumainn]

Yep - tenants would not specify specific hosts. After discussion with Danni and Lars, I've added the "filter" concept to the document; essentially, tenants can specify filters (such as a specific rack) that constrains host selection.

[knikolla]

How is this different from the bullet point immediately above?

[tzumainn]

It's not super different I guess! The first is intended to be "network configuration on creation" while the second is "update network configuration".

[larsks]

Maybe rephrase them as such?

• As a tenant, I want to be able to specify network attachments for a host when I first acquire it.
• As a tenant, I want to be able to update the network attachments of a host that I have already acquired.

[tzumainn]

Makes sense! Updated

[knikolla]

Are certain parameters, such as resourceClass immutable once a HostPool has been created? Or would O-SAC in such a situation release nodes and acquire nodes of the appropriate class.

[tzumainn]

If a tenant updates their HostPool to remove one resource class and add another, O-SAC would do what you say and release nodes of the first resource class while acquiring nodes of the second.

[knikolla]

I think it's a good decision to have network configuration only applicable at the entire HostPool level.

• It matches the interface and API described above where operations are performed on HostPool. Eg. If you reduce the number of Hosts in a HostPool whose Hosts have been manually applied networking, which one do you release?
• It forces all hosts in a HostPool to have the same configuration, and more importantly trains users to work at the HostPool/Fleet/Herd/etc level, applying consistent policy and networking.

[tzumainn]

Makes sense! That seems to be the overwhelming consensus, so I've removed this bullet point.

[knikolla]

How does it choose which one to remove?

[tzumainn]

I've added an example in the HostPool section below, and mentioned that the HostPool filters can include the idea of excluding specific nodes.

[tzumainn]

@knikolla thanks for your comments! I think I've addressed them all; the bulk are answered by adding the idea of allowing host filtering ("hosts in rack X", "not host named Y"). The filtering concept is sprinkled throughout the document at the appropriate places, with examples given in the HostPool specification section.

[adriengentil]

few questions/nits, looks good overall!

[adriengentil]

what about several nodes being removed? does exclude takes a list? Does this transient information stays in the API until the user removes it?

An idea from hypershift: they allow to annotate the nodes that will be cleanup first when the nodepool is down-scaled. Maybe we can implement something similar by adding a tag on the hosts? That would be cleaner to me as the information about the exclusion goes away with the host and doesn't stay within the HostPool.

[tzumainn]

yep, exclude would take a list! I kinda like having the list associated with the hostpool, because it prevents O-SAC from simply trying to put the same host back into the hostpool

[adriengentil]

oh good point! I think it should be mentioned.

[tzumainn]

fair - added some text!

[tzumainn]

@adriengentil thanks for the review! I've answered your questions and made changes based on your feedback; let me know if there's anything I missed!

[larsks]

This is mis-indented. I think there are some tabs-vs-spaces issues happening in this document; you should detab it.

[tzumainn]

Whoops - I think I've fixed the issues.

[larsks]

Should we remove this?

[tzumainn]

Let me change remove to manage; it's something that was specifically mentioned before, so I think mentioning it at the very least makes sense.

[larsks]

What about attaching the same network to multiple interfaces? Supported? Not supported?

[tzumainn]

Added as a question. My gut is yes (since we do that at the MOC already); implementation-wise, I'd imagine that we can count requested network attachments against the current network state.

[tzumainn]

@larsks I've addressed your comments (changes in some places, responses in others). Let me know what you think!

[larsks]

We probably want to support standard Kubernetes matchLabel/matchExpression syntax in these selectors. Maybe something like this:

Suggested change

	hostSelectors:
	- rack: R2
	hostSelectors:
	matchLabels:
	row: R2
	matchExpressions:
	- op: NotIn
	key: cabinet
	values: ["C2"]

[tzumainn]

makes sense! updated (and added text before the example)

[larsks]

Per my earlier comment:

Suggested change

	hostSelectors:
	- rack: R2
	exclude:
	- HostX
	hostSelectors:
	matchLabels:
	row: R2
	matchExpressions:
	- op: NotIn
	key: nodeName
	values: ["HostX"]

[tzumainn]

updated as well!

[tzumainn]

@mhrivnak thanks for the review! I've updated the proposal to address most of your comments, and left questions/comments on a few.

[tzumainn]

@mhrivnak The latest commit adds text expanding details regarding the workflow from the fulfillment CLI to the fulfillment service to the O-SAC operator, with limited examples appearing in the API section. I used the VM proposal as a baseline; there isn't a full API specification, but I feel like there's enough to make the distinction between what the tenant requests, and what CRs are ultimately created. Let me know what you think!

[mhrivnak]

As long as we're ok with some of the network parts of the API changing when we have a more complete set of networking capabilities, LGTM.

[tzumainn]

Thanks all for the comments!