feat(huntsman): Update the default config in the values file for Helm deployment. - #432
Conversation
WalkthroughThe Helm values reduce scheduler and storage heartbeat intervals to 1 second. They also reduce active-job and dispatch queue capacities and increase ready-task capacity. ChangesScheduler Runtime Tuning
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tools/deployment/spider-helm/values.yaml (1)
44-45: 🩺 Stability & Availability | 🔵 TrivialKeep the one-second heartbeat interval aligned with liveness timeouts.
scheduler_heartbeat_interval_secandstorage_heartbeat_interval_secare applied verbatim to the EM liveness actors. With a one-second tick, ensure the schedulerdead_em_cutoff_sec/storage stale threshold and Kubernetes liveness probe timeout stay large enough for normal network and scheduling delay, otherwise the worker can be marked dead or replaced because a heartbeat RPC is slower than one interval.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tools/deployment/spider-helm/values.yaml` around lines 44 - 45, Update the liveness configuration associated with scheduler_heartbeat_interval_sec and storage_heartbeat_interval_sec so scheduler dead_em_cutoff_sec, the storage stale threshold, and the Kubernetes liveness probe timeout provide sufficient margin above the one-second heartbeat interval for normal RPC and scheduling delays. Keep both heartbeat intervals at one second while aligning all related timeout values.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tools/deployment/spider-helm/values.yaml`:
- Around line 62-67: Before shipping the one-million-task buffer configured by
ready_task_capacity in the scheduler values, add an explicit Kubernetes memory
resource limit to this chart’s scheduler deployment and create a load test that
exercises the million-entry ready queue, including OOM and queue-saturation
behavior. Ensure the test validates the configured capacity under the
deployment’s memory limit.
---
Nitpick comments:
In `@tools/deployment/spider-helm/values.yaml`:
- Around line 44-45: Update the liveness configuration associated with
scheduler_heartbeat_interval_sec and storage_heartbeat_interval_sec so scheduler
dead_em_cutoff_sec, the storage stale threshold, and the Kubernetes liveness
probe timeout provide sufficient margin above the one-second heartbeat interval
for normal RPC and scheduling delays. Keep both heartbeat intervals at one
second while aligning all related timeout values.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c2754654-1f95-45cc-8861-4d9b34ec44c7
📒 Files selected for processing (1)
tools/deployment/spider-helm/values.yaml
| active_job_queue_capacity: 16 | ||
| cleanup_ready_task_capacity: 256 | ||
| commit_ready_task_capacity: 256 | ||
| dispatch_queue_capacity: 64 | ||
| dispatch_queue_capacity: 16 | ||
| finalizing_job_expiration_timeout_sec: 300 | ||
| ready_task_capacity: 65536 | ||
| ready_task_capacity: 1048576 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
rg -n -C 8 \
'ready_task_capacity|task_capacity|resources:|limits:|memory:' .Repository: y-scope/spider
Length of output: 27472
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== helm chart templates for scheduler pod =="
for f in $(git ls-files 'tools/deployment/spider-helm/templates/*schedule*' 'tools/deployment/spider-helm/templates/*scheduler*' 'tools/deployment/spider-helm/templates/*' | egrep -i 'scheduler|schedule|deployment|statefulset|daemonset' || true); do
echo "--- $f"
wc -l "$f"
sed -n '1,220p' "$f"
done
echo "== values relevant to scheduler/storage resources =="
sed -n '1,140p' tools/deployment/spider-helm/values.yaml | nl -ba
echo "== search for scheduler image/env/resource references =="
rg -n -C 4 '\${?*scheduler|scheduler.*image|image:|resources:|limits:|memory:|ready_task_capacity|task_capacity|RoundRobinConfig' tools/deployment/spider-helm tools/deployment || trueRepository: y-scope/spider
Length of output: 18540
Confirm scheduler memory before shipping the one-million-task buffer.
ready_task_capacity is the total pending ready tasks buffered in the scheduler, and this value now matches storage.runtime.ready_queue_config.task_capacity. The scheduler deployment does not set Kubernetes resource limits in this chart, and the repository does not contain an OOM/queue-saturation test for the million-entry schedule. Add a memory limit and a load test before release.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tools/deployment/spider-helm/values.yaml` around lines 62 - 67, Before
shipping the one-million-task buffer configured by ready_task_capacity in the
scheduler values, add an explicit Kubernetes memory resource limit to this
chart’s scheduler deployment and create a load test that exercises the
million-entry ready queue, including OOM and queue-saturation behavior. Ensure
the test validates the configured capacity under the deployment’s memory limit.
2 participants
Description
This PR updates the following default values for Helm deployment, including:
active_job_queue_capacity: Change it from 64 to 16 since by default there're only 4 workers.dispatch_queue_capacity: Change it from 64 to 16 since by default there're only 4 workers.ready_task_capacity: Change it from 65536 to 1048576 to match storage's inbound ready queue's capacity.Checklist
breaking change.
Validation performed
Summary by CodeRabbit
Performance
Reliability