feat(helm): Surface the scheduler's em_registry and the storage's task_instance_pool and job_cache_gc config in the values file.

[LinZhihao-723]

Description

This PR depends on #433.

Several runtime config sections are absent from values.yaml, so a Helm user has no way to see or override them: they silently fall back to the Rust Default impls. This PR surfaces the remaining ones, with values that mirror those defaults exactly, so deployments behave identically to before:

Service	Section	Field	Value
scheduler	runtime.em_registry	dead_em_cutoff_sec	30
		liveness_tracking_interval_ms	1000
storage	runtime.task_instance_pool	execution_manager_stale_cutoff_sec	60
		gc_interval_sec	30
		message_channel_capacity	128
storage	runtime.job_cache_gc	terminated_job_retention_sec	300
		gc_interval_sec	30

The storage entries need no template change: configmap.yaml already renders storage.yaml's runtime by emitting the computed db block and passing the rest of spiderConfig.storage.runtime through with toYaml, so new keys flow through automatically.

The scheduler's runtime block instead enumerated each field explicitly, which is why em_registry was missing in the first place. This PR switches it to the same passthrough shape used by storage, since advertised_endpoint is the only field the template needs to compute:

    runtime:
      advertised_endpoint:
        host: {{ include "spider.componentFullname" (dict "root" . "component" "scheduler") | quote }}
        port: {{ .Values.spiderConfig.scheduler.port }}
      {{- with .Values.spiderConfig.scheduler.runtime }}
      {{- toYaml . | nindent 6 }}
      {{- end }}

Besides removing the per-field plumbing, this makes every current and future RuntimeConfig field reachable from values.yaml without a template edit. stop_timeout_sec is one that was already unreachable and now works.

Two consequences of the passthrough worth noting for reviewers:

• toYaml sorts keys, so the rendered scheduler policy block now emits config before policy, and policy's value is no longer explicitly quoted. SchedulerConfig is an adjacently tagged enum, and Serde accepts the content field ahead of the tag, so both forms deserialize identically. This was verified by deserializing the rendered config through ServerConfig rather than by inspection alone (see below).
• Values that mirror a Rust default are no longer merely documentation: the chart now always sends them explicitly, so a future change to a Default impl will not reach Helm deployments unless values.yaml is updated in step.

The chart version is bumped to satisfy the chart linting workflow.

Checklist

• The PR satisfies the contribution guidelines.
• This isn't a breaking change: the added values match the Rust defaults that previously applied, so the effective config is unchanged.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• Ensure all workflows pass.

Summary by CodeRabbit

• New Features

  • Added configurable scheduler monitoring, including liveness timeouts and tracking intervals.
  • Added storage settings for automatic job-cache cleanup and task-instance capacity limits.
  • Expanded runtime configuration options while preserving existing ready-queue settings.
  • Improved deployment flexibility for scheduler and storage operations.

• Chores

  • Updated the deployment chart version to 0.1.11.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c64dd221-4dd8-4205-8e26-d0d685ef22f7

📥 Commits

Reviewing files that changed from the base of the PR and between 92de4c7 and a1661d6.

📒 Files selected for processing (1)

• tools/deployment/spider-helm/templates/configmap.yaml

🚧 Files skipped from review as they are similar to previous changes (1)

• tools/deployment/spider-helm/templates/configmap.yaml

---

Walkthrough

The Helm chart adds scheduler and storage runtime settings, renders scheduler runtime fields dynamically, and updates the chart version from 0.1.10 to 0.1.11.

Changes

Helm runtime updates

Layer / File(s)	Summary
Runtime values and dynamic rendering tools/deployment/spider-helm/values.yaml, tools/deployment/spider-helm/templates/configmap.yaml, tools/deployment/spider-helm/Chart.yaml	The chart adds execution-manager liveness, job-cache garbage-collection, and task-instance pool settings. The ConfigMap dynamically renders scheduler runtime fields while excluding advertised_endpoint. The chart version changes to 0.1.11.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• y-scope/spider#384: Modifies the same Helm chart runtime configuration files.
• y-scope/spider#403: Extends related scheduler configuration in the Helm chart.
• y-scope/spider#411: Updates related execution-manager and scheduler runtime settings.

Suggested reviewers: sitaowang1998

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: exposing scheduler and storage runtime settings in the Helm values file.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

components/spider-storage/tests/runtime_recovery_test.rs (1)

202-207: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Verify the serialised configuration contract in a test.

This helper constructs RuntimeConfig directly. It does not exercise deserialisation of db, ready_queue, task_instance_pool, or job_cache_gc. It cannot catch a Helm key mismatch or a legacy-key fallback.

Add a fixture with non-default values and deserialise the rendered storage configuration, or add an equivalent integration test.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/spider-storage/tests/runtime_recovery_test.rs` around lines 202 -
207, Extend the runtime configuration tests around create_runtime_config to add
a non-default serialized configuration fixture and deserialize the rendered
storage configuration into RuntimeConfig. Assert the db, ready_queue,
task_instance_pool, and job_cache_gc values so Helm key names and legacy-key
fallback behavior are exercised instead of only testing direct struct
construction.

tools/deployment/spider-helm/values.yaml (1)

44-45: 🚀 Performance & Scalability | 🔵 Trivial

Validate the one-second heartbeat defaults under load.

Both heartbeat intervals change from 10 seconds to 1 second. This increases heartbeat traffic and liveness checks by 10x. Confirm CPU, network, and transient-latency behaviour with a representative deployment before keeping these as chart defaults.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/deployment/spider-helm/values.yaml` around lines 44 - 45, Validate the
one-second defaults for scheduler_heartbeat_interval_sec and
storage_heartbeat_interval_sec under a representative deployment, measuring CPU,
network traffic, and transient latency; retain the 1-second chart defaults only
if the increased heartbeat load is acceptable, otherwise restore the prior
defaults.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/deployment/spider-helm/values.yaml`:
- Around line 62-69: Add scheduler fields em_registry and stop_timeout_sec, and
storage fields task_instance_pool and job_cache_gc, using their existing
Rust-default values. Update the scheduler and storage Helm templates to render
the complete runtime configuration, while preserving the computed
advertised_endpoint in the scheduler output.

---

Nitpick comments:
In `@components/spider-storage/tests/runtime_recovery_test.rs`:
- Around line 202-207: Extend the runtime configuration tests around
create_runtime_config to add a non-default serialized configuration fixture and
deserialize the rendered storage configuration into RuntimeConfig. Assert the
db, ready_queue, task_instance_pool, and job_cache_gc values so Helm key names
and legacy-key fallback behavior are exercised instead of only testing direct
struct construction.

In `@tools/deployment/spider-helm/values.yaml`:
- Around line 44-45: Validate the one-second defaults for
scheduler_heartbeat_interval_sec and storage_heartbeat_interval_sec under a
representative deployment, measuring CPU, network traffic, and transient
latency; retain the 1-second chart defaults only if the increased heartbeat load
is acceptable, otherwise restore the prior defaults.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9e6ef66c-63ba-427c-aeb8-e5b74724d902

📥 Commits

Reviewing files that changed from the base of the PR and between 00037af and d6ffc05.

📒 Files selected for processing (5)

• components/spider-storage/src/state/runtime.rs
• components/spider-storage/tests/runtime_recovery_test.rs
• tools/deployment/spider-helm/Chart.yaml
• tools/deployment/spider-helm/templates/configmap.yaml
• tools/deployment/spider-helm/values.yaml

[20001020ycx]

Verified end-to-end: rendered the chart with the new runtime fields for each component and confirmed the generated scheduler.yaml/storage.yaml deserialize correctly.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/deployment/spider-helm/templates/configmap.yaml`:
- Around line 29-31: Filter the reserved advertised_endpoint key from
.Values.spiderConfig.scheduler.runtime before rendering it with toYaml in the
configmap template. Preserve all other runtime values and keep the existing
nindent output behavior, ensuring advertised_endpoint is emitted only by its
dedicated configuration path.

In `@tools/deployment/spider-helm/values.yaml`:
- Around line 59-61: Add the missing stop_timeout_sec entry under the
spiderConfig.scheduler.runtime configuration in values.yaml, placing it at the
level specified by the Rust schema and assigning the schema’s default value.
Preserve the existing em_registry settings.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 355b02be-4d86-4e43-b70c-5946892d03bf

📥 Commits

Reviewing files that changed from the base of the PR and between d6ffc05 and 0436b7d.

📒 Files selected for processing (3)

• tools/deployment/spider-helm/Chart.yaml
• tools/deployment/spider-helm/templates/configmap.yaml
• tools/deployment/spider-helm/values.yaml

🚧 Files skipped from review as they are similar to previous changes (1)

• tools/deployment/spider-helm/Chart.yaml

[20001020ycx]

Pushed with a fix in incorporating key omission with toYaml copy