Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs on scheduler MultiPoint config #30442

Merged
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 182 additions & 0 deletions content/en/docs/reference/scheduling/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,8 @@ extension points:
least one bind plugin is required.
1. `postBind`: This is an informational extension point that is called after
a Pod has been bound.
1. `multiPoint`: This is a config-only field that allows plugins to be enabled
or disabled for all of their applicable extension points simultaneously.

For each extension point, you could disable specific [default plugins](#scheduling-plugins)
or enable your own. For example:
Expand Down Expand Up @@ -251,6 +253,186 @@ the same configuration parameters (if applicable). This is because the scheduler
only has one pending pods queue.
{{< /note >}}

### Plugins that apply to multiple extension points {#multipoint}
damemi marked this conversation as resolved.
Show resolved Hide resolved

Starting from `kubescheduler.config.k8s.io/v1beta3`, there is an additional field in the
profile config, `multiPoint`, which allows for easily enabling or disabling a plugin
across several extension points. The intent of `multiPoint` config is to simplify the
configuration needed for users and administrators when using custom profiles.

Consider a plugin, `MyPlugin`, which implements the `preScore`, `score`, `preFilter`,
and `filter` extension points. To enable `MyPlugin` for all its available extension
points, the profile config looks like:

```yaml
apiVersion: kubescheduler.config.k8s.io/v1beta3
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: multipoint-scheduler
plugins:
multiPoint:
enabled:
- name: MyPlugin
```

This would equate to manually enabling `MyPlugin` for all of its extension
points, like so:

```yaml
apiVersion: kubescheduler.config.k8s.io/v1beta3
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: non-multipoint-scheduler
plugins:
preScore:
enabled:
- name: MyPlugin
score:
enabled:
- name: MyPlugin
preFilter:
enabled:
- name: MyPlugin
filter:
enabled:
- name: MyPlugin
```
damemi marked this conversation as resolved.
Show resolved Hide resolved

One benefit of using `multiPoint` here is that if `MyPlugin` implements another
extension point in the future, the `multiPoint` config will automatically enable it
for the new extension.

Specific extension points can be excluded from `MultiPoint` expansion using
the `disabled` field for that extension point. This works with disabling default
plugins, non-default plugins, or with the wildcard (`'*'`) to disable all plugins.
An example of this, disabling `Score` and `PreScore`, would be:

```yaml
apiVersion: kubescheduler.config.k8s.io/v1beta3
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: non-multipoint-scheduler
plugins:
multiPoint:
enabled:
- name: 'MyPlugin'
preScore:
disabled:
- name: '*'
score:
disabled:
- name: '*'
```

In `v1beta3`, all [default plugins](#scheduling-plugins) are enabled internally through `MultiPoint`.
However, individual extension points are still available to allow flexible
reconfiguration of the default values (such as ordering and Score weights). For
example, consider two Score plugins `DefaultScore1` and `DefaultScore2`, each with
a weight of `1`. They can be reordered with different weights like so:

```yaml
apiVersion: kubescheduler.config.k8s.io/v1beta3
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: multipoint-scheduler
plugins:
score:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

multiPoint? Order only really matters for Filter

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was trying to show a couple things here, that we can reorder and re-weight the plugins. Order technically only matters for Filter, true, but I think this still gets both of those points across

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good

enabled:
- name: 'DefaultScore2'
weight: 5
```

In this example, it's unnecessary to specify the plugins in `MultiPoint` explicitly
because they are default plugins. And the only plugin specified in `Score` is `DefaultScore2`.
This is because plugins set through specific extension points will always take precedence
over `MultiPoint` plugins. So, this snippet essentially re-orders the two plugins
without needing to specify both of them.

The general hierarchy for precedence when configuring `MultiPoint` plugins is as follows:
1. Specific extension points run first, and their settings override whatever is set elsewhere
2. Plugins manually configured through `MultiPoint` and their settings
3. Default plugins and their default settings

To demonstrate the above hierarchy, the following example is based on these plugins:
|Plugin|Extension Points|
|---|---|
|`DefaultQueueSort`|`QueueSort`|
|`CustomQueueSort`|`QueueSort`|
|`DefaultPlugin1`|`Score`, `Filter`|
|`DefaultPlugin2`|`Score`|
|`CustomPlugin1`|`Score`, `Filter`|
|`CustomPlugin2`|`Score`, `Filter`|

A valid sample configuration for these plugins would be:

```yaml
apiVersion: kubescheduler.config.k8s.io/v1beta3
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: multipoint-scheduler
plugins:
multiPoint:
enabled:
- name: 'CustomQueueSort'
- name: 'CustomPlugin1'
weight: 3
- name: 'CustomPlugin2'
disabled:
- name: 'DefaultQueueSort'
filter:
disabled:
- name: 'DefaultPlugin1'
score:
enabled:
- name: 'DefaultPlugin2'
```

Note that there is no error for re-declaring a `MultiPoint` plugin in a specific
extension point. The re-declaration is ignored (and logged), as specific extension points
take precedence.

Besides keeping most of the config in one spot, this sample does a few things:
* Enables the custom `queueSort` plugin and disables the default one
* Enables `CustomPlugin1` and `CustomPlugin2`, which will run first for all of their extension points
* Disables `DefaultPlugin1`, but only for `filter`
* Reorders `DefaultPlugin2` to run first in `score` (even before the custom plugins)

In versions of the config before `v1beta3`, without `multiPoint`, the above snippet would equate to this:
```yaml
apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: multipoint-scheduler
plugins:

# Disable the default QueueSort plugin
queueSort:
enabled:
- name: 'CustomQueueSort'
disabled:
- name: 'DefaultQueueSort'

# Enable custom Filter plugins
filter:
enabled:
- name: 'CustomPlugin1'
- name: 'CustomPlugin2'
- name: 'DefaultPlugin2'
disabled:
- name: 'DefaultPlugin1'

# Enable and reorder custom score plugins
score:
enabled:
- name: 'DefaultPlugin2'
weight: 1
- name: 'DefaultPlugin1'
weight: 3
```

While this is a complicated example, it demonstrates the flexibility of `MultiPoint` config
as well as its seamless integration with the existing methods for configuring extension points.

## Scheduler configuration migrations

{{< tabs name="tab_with_md" >}}
Expand Down