Skip to content

Add inftyai-scheduler support and config updates #447

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 10, 2025
Merged

Add inftyai-scheduler support and config updates #447

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c6f907a
Add inftyai-scheduler support and config updates
carlory Jun 9, 2025
38cd59c
disable scheduler-plugins by default
carlory Jun 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,7 +317,9 @@ $(HELMIFY): $(LOCALBIN)

.PHONY: helm
helm: manifests kustomize helmify
$(KUSTOMIZE) build config/default | $(HELMIFY) -crd-dir
$(KUSTOMIZE) build config/default \
| yq 'select(.kind and (.kind != "ConfigMap" or .metadata.name != "llmaz-global-config"))' \
Copy link
Member

@kerthcet kerthcet Jun 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better not to add too much logic to the makefile command, what's the advantages over the current one?

Copy link
Member Author

@carlory carlory Jun 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the default value is default-scheduler in config/default/configmap.yaml and it can not be easily changed by user via helm install --set command.

I can not edit chart/template/global-config.yaml because it is generated by helmify. I want users to be able to easily use the chart without extra configuration.

Copy link
Member

@kerthcet kerthcet Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it's not a general solution, we have a lot of configurations.

Copy link
Member

@kerthcet kerthcet Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ok to let user configure the configmaps via global.values.yaml. It's also not difficult.

Copy link
Member Author

@carlory carlory Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we switch the default scheduler to inftyai-scheduler in config/default/configmap.yaml? And add make install-inftyai-scheduler and make uninstall-inftyai-scheduler to the Makefile.

install-inftyai-scheduler:
	kubectl apply --server-side -k config/inftyai-scheduler

.PHONY: uninstall-prometheus
uninstall-inftyai-scheduler:
	kubectl delete -k config/inftyai-scheduler

Copy link
Member Author

@carlory carlory Jun 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If not, we can not enable the inftyai-scheduler in chart by default due to the immutability of the configmap which is generated by helmify.

Copy link
Member Author

@carlory carlory Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ok to let user configure the configmaps via global.values.yaml. It's also not difficult.

This pr has supported it.

| $(HELMIFY) -crd-dir

.PHONY: helm-install
helm-install: helm
Expand Down
7 changes: 5 additions & 2 deletions chart/Chart.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@ dependencies:
- name: lws
repository: oci://registry.k8s.io/lws/charts
version: 0.6.1
- name: kube-scheduler
repository: oci://registry-1.docker.io/inftyai
version: 0.0.1
- name: open-webui
repository: https://helm.openwebui.com
version: 6.4.0
Expand All @@ -11,5 +14,5 @@ dependencies:
- name: ai-gateway-helm
repository: oci://registry-1.docker.io/envoyproxy
version: v0.1.5
digest: sha256:0ec1b490fc7cd028186811c1776f786b16b5f55faf93cfdf2a04c24b1ba08bdb
generated: "2025-05-20T21:08:50.138833832+08:00"
digest: sha256:b2b856b107e9e03d175f381a93b83b001211df02f1c1ef1ee13b23147aed50e8
generated: "2025-06-09T10:04:15.540138+08:00"
4 changes: 4 additions & 0 deletions chart/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ dependencies:
version: 0.6.1
repository: "oci://registry.k8s.io/lws/charts"
condition: leaderWorkerSet.enabled
- name: kube-scheduler
version: 0.0.1
repository: "oci://registry-1.docker.io/inftyai"
condition: kube-scheduler.enabled
- name: open-webui
version: "6.4.0"
repository: "https://helm.openwebui.com"
Expand Down
12 changes: 11 additions & 1 deletion chart/templates/global-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,14 @@ metadata:
labels:
{{- include "chart.labels" . | nindent 4 }}
data:
config.data: {{ .Values.globalConfig.configData | toYaml | indent 1 }}
config.data: |-
{{- $base := deepCopy .Values.globalConfig.configData }}
Copy link
Member

@kerthcet kerthcet Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean is we may have several configurations in globalconfig, so this is not a general solution, I would suggest revert to the original configurations, and people can modify the chart/values.global.yaml before installation/upgrade, does that workaround?

Copy link
Member Author

@carlory carlory Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works. Howerver, if an user want to use the scheduler, he have to do the following:

  1. Get values from chart via helm show values oci://registry-1.docker.io/inftyai/llmaz > /t mp/values.yaml

  2. Modify the values.yaml to enable the scheduler like this:

kube-scheduler:
  enabled: true

globalConfig:
  configData: |-
    scheduler-name: inftyai-scheduler
    # init-container-image: inftyai/model-loader:v0.0.10
  1. Install the chart with the modified values.yaml.

Copy link
Member Author

@carlory carlory Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can not add this config to values.global.yaml in codebase because make helm-package appends (not merge) the file to values.yaml.

Copy link
Member Author

@carlory carlory Jun 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another drawback is that users need to provide the whole content of the configData when they want to change only one field because the globalConfig.configData is a string, not a map.

{{- $ks := get .Values "kube-scheduler" }}
{{- if and $ks (get $ks "enabled") (hasKey $ks "scheduler") }}
{{- $scheduler := get $ks "scheduler" }}
{{- $name := get $scheduler "name" }}
{{- if $name }}
{{- $_ := set $base "scheduler-name" $name }}
{{- end }}
{{- end }}
{{- toYaml $base | nindent 4 }}
13 changes: 13 additions & 0 deletions chart/values.global.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@ backendRuntime:
leaderWorkerSet:
enabled: true

kube-scheduler:
enabled: true
# scheduler:
# name: inftyai-scheduler

prometheus:
# Prometheus is required to enable smart routing.
enabled: false
Expand Down Expand Up @@ -58,3 +63,11 @@ envoy-gateway:
enabled: true
envoy-ai-gateway:
enabled: true

globalConfig:
configData:
# If kube-scheduler is enabled, the scheduler-name will
# be automatically overwritten by the kube-scheduler.scheduler.name,
# when generating the global-config ConfigMap.
scheduler-name: default-scheduler
# init-container-image: inftyai/model-loader:v0.0.10
4 changes: 0 additions & 4 deletions chart/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,6 @@ controllerManager:
replicas: 1
serviceAccount:
annotations: {}
globalConfig:
configData: |-
scheduler-name: default-scheduler
# init-container-image: inftyai/model-loader:v0.0.10
kubernetesClusterDomain: cluster.local
metricsService:
ports:
Expand Down
Loading