feat(huntsman): Complete storage gRPC protocol and its clients:

[sitaowang1998]

• Add protobuf definitions for the storage gRPC protocol.
• Implement the storage client for spider-scheduler.
• Implement the liveness client for spider-execution-manager.

Description

This PR:

• Adds remaining gRPC protocol for storage interface.
• Adds task id and job state conversion between gRPC and spider-core.
• Adds gRPC implementation of execution manager's liveness client.
• Adds gRPC implementation of scheduler's storage client.

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• GitHub workflows pass.

Summary by CodeRabbit

• New Features

  • Added gRPC services for job orchestration, inbound task polling, resource-group management, execution-manager liveness (register/heartbeat) and session retrieval.
  • Introduced gRPC-backed clients for liveness and scheduler storage to enable remote polling and job-state operations.

• Error Handling

  • Replaced generic storage errors with domain-specific error types and richer error mappings for clearer failure categories.

• Tests

  • Added unit tests covering protocol conversions, response/error mapping, registration and heartbeat flows.

[coderabbitai]

Walkthrough

Adds domain-scoped storage proto services/enums, bidirectional protobuf↔core conversions, a GrpcLivenessClient, updates execution-manager storage adapter to task-instance types, and introduces a GrpcSchedulerStorageClient with scheduler wiring and public re-exports.

Changes

Proto, gRPC clients, and conversions

Layer / File(s)	Summary
Proto API definitions and enums components/spider-proto/storage/storage.proto	Adds JobOrchestrationService, InboundQueueService, ResourceGroupManagementService, ExecutionManagerLivenessService, SessionManagementService; replaces generic StorageOperationResponse/StorageError with domain-scoped operation responses and errors; adds JobState enum.
Proto-to-Rust conversions and error types components/spider-proto-rust/Cargo.toml, components/spider-proto-rust/src/error.rs, components/spider-proto-rust/src/id.rs, components/spider-proto-rust/src/job.rs, components/spider-proto-rust/src/lib.rs	Adds Error enum for conversion failures; implements TryFromstorage::TaskId → TaskId and From/TryFrom for JobState; exposes error and job modules; adds thiserror dependency.
GrpcLivenessClient implementation components/spider-execution-manager/src/client/grpc/liveness.rs	Implements GrpcLivenessClient with connect(), register, heartbeat; converts ExecutionManagerLivenessError codes to domain errors; includes response conversion helpers and tests.
Execution-manager storage adapter update components/spider-execution-manager/src/client/grpc/storage.rs	Switches adapter to TaskInstanceOperationResponse and TaskInstanceManagementError types; updates StorageResponseError From conversion and storage_operation_response_to_result; updates tests.
GrpcSchedulerStorageClient implementation components/spider-scheduler/src/storage_client/grpc.rs	Adds GrpcSchedulerStorageClient with connect() and SchedulerStorageClient method implementations (polls and job_state); converts InboundQueueResponseError and JobOrchestrationError to StorageClientError; adds payload conversion helpers and tests.
Scheduler error types and dependencies components/spider-scheduler/src/error.rs, components/spider-scheduler/Cargo.toml	Extends StorageClientError with StaleSession and categorized error variants; adds spider-proto-rust path dependency and tonic dependency.
Module structure and public API exports components/spider-execution-manager/src/client.rs, components/spider-execution-manager/src/client/grpc/mod.rs, components/spider-scheduler/src/lib.rs, components/spider-scheduler/src/storage_client/mod.rs	Adds grpc submodules and re-exports for GrpcLivenessClient and GrpcSchedulerStorageClient; updates crate re-exports to expose the new clients.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• y-scope/spider#323: Related liveness service methods and gRPC liveness wiring.
• y-scope/spider#328: Liveness trait/actor work that uses the concrete GrpcLivenessClient.
• y-scope/spider#329: Runtime/bootstrap changes that interact with liveness registration/heartbeat.

Suggested reviewers

• LinZhihao-723

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: adding gRPC protocol for storage and implementing gRPC clients in both the execution manager and scheduler components.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

components/spider-proto/storage/storage.proto (1)

1-352: 🛠️ Refactor suggestion | 🟠 Major | 🏗️ Heavy lift

Add comprehensive documentation to the proto API.

The proto file defines multiple public gRPC services and complex message structures but contains zero documentation comments. For a cross-component API contract, comprehensive documentation is essential to ensure maintainability, enable safe evolution, and support developer onboarding.

Please add proto comments documenting:

• Each service's purpose and usage context
• Each RPC's behaviour, preconditions, and error scenarios
• Message field semantics, constraints, and valid ranges
• Error code meanings and client handling expectations
• The JobState enum transitions and their lifecycle implications

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/spider-proto/storage/storage.proto` around lines 1 - 352, The
proto file lacks any documentation; add clear proto comments for all public
symbols: each service (JobManagementService, TaskInstanceManagementService,
SchedulerStorageService, ResourceGroupManagementService,
ExecutionManagerLivenessService, SessionManagementService) describing purpose
and usage; each RPC (e.g., SubmitJob, StartJob, PollReadyTasks,
RegisterTaskInstance, ReportTaskSuccess, AddResourceGroup,
RegisterExecutionManager, GetSession) explaining input preconditions, side
effects, success/failure semantics and typical error scenarios; each message and
important fields (SubmitJobRequest.serialized_task_graph, ReadyTask.task_id,
RegisterTaskInstanceRequest.execution_manager_id, etc.) documenting field
semantics, valid ranges and constraints; all error messages
(JobManagementError.ErrCode, TaskInstanceError.ErrCode,
SchedulerStorageError.ErrCode, ResourceGroupError.ErrCode,
ExecutionManagerLivenessError.ErrCode) describing when each code is returned and
recommended client handling; and the JobState enum documenting legal state
transitions (e.g., READY -> RUNNING -> COMMIT_READY/CLEANUP_READY ->
SUCCEEDED/FAILED/CANCELLED) and lifecycle implications. Keep comments concise,
use proto // or /** */ comments above definitions so they are included in
generated docs and ensure examples or notes for session_id and storage_session
consistency where relevant.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@components/spider-proto/storage/storage.proto`:
- Around line 1-352: The proto file lacks any documentation; add clear proto
comments for all public symbols: each service (JobManagementService,
TaskInstanceManagementService, SchedulerStorageService,
ResourceGroupManagementService, ExecutionManagerLivenessService,
SessionManagementService) describing purpose and usage; each RPC (e.g.,
SubmitJob, StartJob, PollReadyTasks, RegisterTaskInstance, ReportTaskSuccess,
AddResourceGroup, RegisterExecutionManager, GetSession) explaining input
preconditions, side effects, success/failure semantics and typical error
scenarios; each message and important fields
(SubmitJobRequest.serialized_task_graph, ReadyTask.task_id,
RegisterTaskInstanceRequest.execution_manager_id, etc.) documenting field
semantics, valid ranges and constraints; all error messages
(JobManagementError.ErrCode, TaskInstanceError.ErrCode,
SchedulerStorageError.ErrCode, ResourceGroupError.ErrCode,
ExecutionManagerLivenessError.ErrCode) describing when each code is returned and
recommended client handling; and the JobState enum documenting legal state
transitions (e.g., READY -> RUNNING -> COMMIT_READY/CLEANUP_READY ->
SUCCEEDED/FAILED/CANCELLED) and lifecycle implications. Keep comments concise,
use proto // or /** */ comments above definitions so they are included in
generated docs and ensure examples or notes for session_id and storage_session
consistency where relevant.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: a9894bac-13ee-4fdd-8177-6fede9a84856

📥 Commits

Reviewing files that changed from the base of the PR and between abd22d4 and a3a3f09.

⛔ Files ignored due to path filters (2)

• Cargo.lock is excluded by !**/*.lock
• components/spider-proto-rust/src/generated/storage.rs is excluded by !**/generated/**

📒 Files selected for processing (8)

• components/spider-execution-manager/src/client/grpc/liveness.rs
• components/spider-execution-manager/src/client/grpc/storage.rs
• components/spider-proto/storage/storage.proto
• components/spider-scheduler/Cargo.toml
• components/spider-scheduler/src/error.rs
• components/spider-scheduler/src/grpc/mod.rs
• components/spider-scheduler/src/grpc/storage_client.rs
• components/spider-scheduler/src/lib.rs

✅ Files skipped from review due to trivial changes (2)

• components/spider-scheduler/Cargo.toml
• components/spider-scheduler/src/grpc/mod.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• components/spider-execution-manager/src/client/grpc/liveness.rs

[LinZhihao-723]

Reviewed all changes and applied the following code-level changes:

proto

• JobManagementService -> JobOrchestrationService to match the protocol trait's name
  • JobManagementError -> JobOrchestrationError
• TaskInstanceError -> TaskInstanceManagementError. TaskInstanceError sounds like the error inside the task instance itself.
• ResourceGroupError -> ResourceGroupManagementError.
• serialized_inputs should be bytes, not repeated bytes. The inputs should be submitted in a serialized form, which allows the client and server to determine how the bytes should be decoded.
• SchedulerStorageService -> InboundQueueService to match the term we use in the scheduler.
  • SchedulerStorageError -> InboundQueueResponseError
• Remove IsExecutionManagerAlive and GetDeadExecutionManagers from ExecutionManagerLivenessService: these operations are internal to the storage service, we don't need gRPC methods for them.
• Change PollReadyTaskRequest's wait_ns to wait_ms: None of our clients use nanoseconds.

spider-proto-rust

• Create a dedicated error type under error.rs mod, export it through the library layer. Use this error type for id.rs and job.rs to replace the use of string errors.
  • Client implementation should be changed accordingly.

spider-execution-manager

• storage_error_to_liveness_error should be renamed to reflect the error type renaming.
• Use impl From<storage::ExecutionManagerLivenessError> for LivenessResponseError to replace the use of storage_error_to_liveness_error. This was done in the storage client in the previous PR.
• Use constants to replace magic numbers/strings in the test cases.

spider-scheduler

• Restructure the client mod in the following structure (to match the execution manager trait):
  • storage_client
    • mod.rs (containing the client trait)
    • grpc.rs (containing the gPRC client implementation)
• Drop inbound_queue_response_error_to_client_error.
• Doesn't make sense to return Transport error for request out-of-range in poll_ready_task_request. Use InvalidInput instead.
• Docstrings are not inconsistent with other _to_result methods in spider-execution-manager.
• The session tracker doesn't make any sense: no methods will update it, and used for all requests. If we look at the storage service in the dev branch, retrieving the job state doesn't need the session ID. Thus, the session ID is dropped from the proto source; the session tracker is removed from the client.
• Use constants to replace magic numbers/strings in the test cases.

---

Besides some naming issues, I need to point out that some changes above are not supposed to be my job for reviewing this PR:

• Considering we already have a gRPC PR reviewed and merged, I'd assume you should read my changes and make sure these changes are propagated to the consequent PRs. But part of my editing of this review is still repeating what I've done in the last PR.
• Some of the code generated by the coding agent doesn't make sense. You probably need to spend more time reviewing them yourself.

Please review my changes and be careful with the pending gRPC PR for the scheduler-em communication.

[LinZhihao-723]

For the PR title, how about:

 feat(huntsman): Complete storage gRPC protocol and its clients

* Add protobuf definitions for the storage gRPC protocol.
* Implement the storage client for `spider-scheduler`.
* Implement the liveness client for `spider-execution-manager`.