feat(spider-scheduler): Add scheduler crate skeleton with trait and type abstractions.

[LinZhihao-723]

Description

This PR introduces a new crate spider-scheduler and lands the trait and type abstractions that the scheduler will be built on top of. No concrete implementations are included — those follow in subsequent PRs.

Architecture

The scheduler is the serial decision maker that turns ready tasks discovered by the storage layer into assignments for execution managers. It owns placement and ordering policy, not dependency resolution: storage decides what is ready, and the scheduler decides in what order and with what throttling ready tasks are offered to the fleet.

The pipeline:

  storage  ── authoritative ready queue (owned by the storage layer, not this crate)
        │
        │  poll_ready / poll_commit_ready / poll_cleanup_ready  (SchedulerStorageClient)
        ▼
  ┌───────────────────┐
  │   SchedulerCore   │  serial loop: poll → decide → enqueue
  └───────────────────┘
        │
        │  enqueue             (DispatchQueueSink — writer side)
        ▼
  ┌───────────────────┐
  │  dispatch queue   │  bounded SPMC; a full queue back-pressures the core
  └───────────────────┘
        │
        │  dequeue             (DispatchQueueSource — reader side)
        ▼
  ┌───────────────────┐
  │ scheduler service │ ──▶ execution managers (concurrent fan-out)
  └───────────────────┘

Trait seams

• SchedulerStorageClient — the scheduler's view of storage. Three lane-specific polls (poll_ready, poll_commit_ready, poll_cleanup_ready) mirror storage's ReadyQueueReceiverHandle lanes; each returns (SessionId, Vec<InboundEntry>) so a stale-session batch can be detected downstream. job_state(JobId) -> JobState exposes a read-only lookup for placement policies that gate on job lifecycle.

• SchedulerCore — the algorithm seam. Owns its decision loop: poll the inbound queue through its associated StorageClient, apply the scheduling algorithm, and write assignments to its associated Sink. Generic over both, so a real algorithm and a mock can share the same runtime. The loop terminates when its tokio_util::sync::CancellationToken is cancelled.

• DispatchQueueSink — the writer side of the dispatching queue. enqueue(TaskAssignment) awaits when the bounded queue is full, providing the back-pressure that throttles the core to fleet drain rate. bump_session_id(SessionId) advances the queue's current session and invalidates everything currently queued; the core calls it when it observes a strictly-higher session from a poll.

• DispatchQueueSource — the reader side, drained by the EM-facing service. dequeue() -> (SessionId, TaskAssignment) returns the next assignment paired with the session it was enqueued under, so the EM can compare against storage's current session at registration time and discard stale assignments.

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• Ensure all workflows pass.

Summary by CodeRabbit

• New Features
  • Introduced a new spider-scheduler component to provide scheduling and task-placement capabilities.
  • Adds standardized storage polling interfaces, a dispatch queue abstraction for enqueue/dequeue and session handling, well-defined task/entry types, and crate-level error types for clearer runtime reporting and handling.

[coderabbitai]

Walkthrough

Adds a new spider-scheduler crate that defines scheduler contracts: storage polling trait, dispatch queue sink/source traits, scheduler core orchestration trait, error types, and simple data types, and registers the crate in the workspace.

Changes

Spider Scheduler Core

Layer / File(s)	Summary
Data types and error contracts components/spider-scheduler/src/types.rs, components/spider-scheduler/src/error.rs	InboundEntry and TaskAssignment value types; StorageClientError and SchedulerError enums for storage and scheduler errors.
Storage and dispatch queue abstractions components/spider-scheduler/src/storage_client.rs, components/spider-scheduler/src/dispatch_queue.rs	SchedulerStorageClient trait with three inbound polling APIs and job_state; DispatchQueueSink/DispatchQueueSource traits for enqueue/dequeue, session bumping, and queue sizing.
Scheduler core orchestration components/spider-scheduler/src/core.rs	SchedulerCore async trait with associated StorageClient and Sink types and an async run method that drives polling and dispatch with cancellation.
Crate configuration and public surface components/spider-scheduler/Cargo.toml, components/spider-scheduler/src/lib.rs, Cargo.toml	New crate manifest and workspace registration; crate root exposes modules and re-exports core traits, dispatch interfaces, errors, storage client trait, and task/entry types.

Sequence Diagram

sequenceDiagram
  participant SchedulerCore
  participant SchedulerStorageClient
  participant DispatchQueueSink
  participant DispatchQueueSource
  SchedulerCore->>SchedulerStorageClient: poll_ready(max_items, wait)
  SchedulerStorageClient-->>SchedulerCore: (SessionId, Vec<InboundEntry>)
  SchedulerCore->>SchedulerCore: map InboundEntry -> TaskAssignment
  SchedulerCore->>DispatchQueueSink: enqueue(TaskAssignment)
  DispatchQueueSink-->>SchedulerCore: Result<(), SchedulerError>
  DispatchQueueSource->>SchedulerCore: dequeue(wait_time) (consumer side)

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested reviewers

• sitaowang1998

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: introducing a new spider-scheduler crate with foundational trait and type abstractions for the scheduler component.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sitaowang1998]

We should probably have a batched enqueue method for better performance.

[LinZhihao-723]

The current planned implementation won't benefit from a batch operation:

• The dispatch queue is implemented using async channel, meaning that all enqueue operations will be serialized.
• The scheduler decision maker pops assignments from the queue one by one; a batch operation means we need to construct/destruct vector on top of the popped results, which introduces unnecessary overhead.