feat(spider-scheduler): Add channel-based dispatch queue implementation.

[LinZhihao-723]

Description

This PR depends on #330 and #331.

This PR implements an async-channel-backed dispatch queue. It also adds unit tests to cover its basic behaviors.

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• Ensure all workflows pass.
• Added unit tests to cover the enqueue/dequeue operations, as well as session bumping.

Summary by CodeRabbit

Release Notes

• Chores
  • Established new scheduler component crate with configured dependencies
  • Implemented dispatch queue system foundation with comprehensive test coverage
  • Added tests for concurrent operations, load balancing, and session management scenarios
  • Configured asynchronous runtime support for system operations

[coderabbitai]

Walkthrough

This PR introduces the dispatch queue layer of the spider-scheduler crate. It defines cloneable writer and reader handles backed by an async channel and shared session-ID state, implements monotonic session bumping with queued assignment invalidation, and provides a factory constructor. The module includes comprehensive async tests validating round-trip delivery, multi-consumer load balancing, session invalidation semantics, and concurrent pair-consistency.

Changes

Dispatch Queue Implementation

Layer / File(s)	Summary
Crate configuration and dependencies components/spider-scheduler/Cargo.toml	Defines the spider-scheduler crate metadata and runtime/development dependencies including async-channel, tokio with sync/time features, tokio-util, thiserror, async-trait, and the local spider-core path dependency.
Dispatch queue writer, reader, and factory with test suite components/spider-scheduler/src/dispatch_queue.rs	Implements DispatchQueueWriter with bounded enqueue and monotonic session-ID bumping that drains pre-bump assignments; DispatchQueueReader with bounded timeout-guarded dequeue that pairs assignments with the session ID at dequeue time; create_dispatch_queue factory; and an extensive async test suite validating round-trip delivery, load balancing across multiple readers, session bump invalidation, monotonicity enforcement, queue-size reset, and concurrent delivery consistency.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• y-scope/spider#330: This PR's concrete DispatchQueueWriter/DispatchQueueReader implementation and session-bump dequeue semantics directly fulfil the DispatchQueueSink/DispatchQueueSource trait abstractions introduced in the referenced PR.

Suggested reviewers

• sitaowang1998

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: implementing a channel-based dispatch queue in the spider-scheduler component.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/spider-scheduler/src/dispatch_queue.rs`:
- Around line 82-84: The queue must stamp session IDs at enqueue time instead of
inferring them at dequeue: modify the internal queue element type to store
(SessionId, TaskAssignment) and change DispatchQueueWriter::enqueue to read the
current session (from wherever bump_session_id updates it) and push the stamped
pair atomically; update DispatchQueue::dequeue to return the stamped pair (or
convert it to the existing return shape) rather than attaching the session on
receive. Alternatively, if you prefer serialization instead of changing the item
type, guard both DispatchQueueWriter::enqueue and
DispatchQueueWriter::bump_session_id with the same writer-side lock/shared mutex
so bump and enqueue cannot interleave; also ensure the cloneable
DispatchQueueWriter shares that lock (or remove Clone) so concurrent clones
cannot bypass the synchronization. Ensure corresponding consumer code (dequeue
handling) and any helpers that construct or consume TaskAssignment are updated
to the new (SessionId, TaskAssignment) shape.
- Around line 93-97: The writer currently holds a hidden clone of
assignment_receiver (DispatchQueueWriterInner / create_dispatch_queue),
preventing the channel from closing when all public DispatchQueueReader
instances are dropped so DispatchQueueWriter::enqueue never returns
DispatchQueueClosed; remove the stored clone from DispatchQueueWriterInner (do
not clone assignment_receiver into the writer in create_dispatch_queue) so only
real readers hold receiver handles, allowing async_channel::Sender::send to fail
and map to SchedulerError::DispatchQueueClosed when readers are gone.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 8c98dfdc-5b9c-4dfb-8017-d96d0b1b0205

📥 Commits

Reviewing files that changed from the base of the PR and between 27091f0 and 10cb6ad.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (19)

• Cargo.toml
• components/spider-core/src/types/id.rs
• components/spider-scheduler/Cargo.toml
• components/spider-scheduler/src/core.rs
• components/spider-scheduler/src/dispatch_queue.rs
• components/spider-scheduler/src/error.rs
• components/spider-scheduler/src/lib.rs
• components/spider-scheduler/src/storage_client.rs
• components/spider-scheduler/src/types.rs
• components/spider-storage/src/cache.rs
• components/spider-storage/src/cache/job.rs
• components/spider-storage/src/task_instance_pool.rs
• components/spider-storage/tests/scheduling_infra.rs
• components/spider-tdl/src/task.rs
• components/spider-tdl/src/task_context.rs
• components/spider-tdl/tests/test_task_macro.rs
• tests/huntsman/task-executor/src/lib.rs
• tests/huntsman/task-executor/tests/test_process_pool.rs
• tests/huntsman/tdl-integration/tests/complex.rs

💤 Files with no reviewable changes (1)

• components/spider-storage/src/cache.rs

[sitaowang1998]

Is there a better way to drain the queue than running a while loop?

[LinZhihao-723]

Unfortunately no...