feat(spider-utils): Add a gRPC connection pool and route the execution-manager and scheduler gRPC clients through it.

[LinZhihao-723]

Description

Summary

A tonic [Channel] multiplexes every request over a single HTTP/2 connection, and cloning the channel does not change that — all clones share one connection and one dispatch buffer. Under concurrency this serializes request dispatch and caps throughput (see hyperium/h2#531). This PR adds a small ConnectionPool to spider-utils that holds several independent connections to one endpoint and hands them out round-robin, then migrates the execution-manager and scheduler gRPC clients to build on it instead of a single cloned channel.

New pool (spider-utils/src/grpc)

• Added grpc::client::ConnectionPool<GrpcServiceClientType>, a cheaply-cloneable handle (Arc-backed) over a fixed set of pre-connected service clients.
• ConnectionPool::connect(endpoint, pool_size, client_factory) eagerly opens pool_size independent connections and wraps each in a service client via the client_factory closure. pool_size is a NonZeroUsize so an empty pool is unrepresentable.
• ConnectionPool::get_client() returns the next client by a relaxed atomic round-robin counter; the returned client is a cheap clone that callers issue a single RPC on.
• Added the grpc::Error enum (InvalidEndpoint, TonicTransport) that the factory returns.

Execution-manager clients (spider-execution-manager)

GrpcStorageClient, GrpcLivenessClient, and GrpcSchedulerClient each replace their single …Client<Channel> field with a ConnectionPool<…Client<Channel>>. Their connect constructors gain a pool_size: NonZeroUsize parameter and build the pool, mapping the pool's grpc::Error through each module's existing transport-error conversion. Every per-RPC self.client.clone() becomes self.connection_pool.get_client(); the request/response and error-mapping logic is otherwise unchanged.

Scheduler client (spider-scheduler)

GrpcSchedulerStorageClient wraps two distinct storage services (InboundQueueService and JobOrchestrationService), so it now holds one pool per service (scheduler_connection_pool and job_connection_pool), each with pool_size connections. Its connect constructor gains the same pool_size parameter. Added spider-utils as a dependency of spider-scheduler.

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• Ensure all workflows pass.
• There's no e2e testing in the system yet since it doesn't wire to any binary yet. However, this idea has been tested in metalog with a similar implementation here.

Summary by CodeRabbit

• New Features

  • Added pooled gRPC connections for several scheduler, storage, and execution manager services.
  • Requests now rotate across multiple active connections, which can improve reliability and throughput.
  • Added shared gRPC utilities for connection pooling and error handling.

• Bug Fixes

  • Reduced dependence on a single connection for service calls, helping avoid disruptions when one connection becomes unavailable.

• Chores

  • Updated project dependencies to support the new gRPC connection pooling features.

[coderabbitai]

Warning

Review limit reached

@LinZhihao-723, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 51 minutes and 25 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: f20fee36-0d9a-4a28-84d9-56e5ba4a5aeb

📥 Commits

Reviewing files that changed from the base of the PR and between 2626892 and cbeb96a.

📒 Files selected for processing (6)

• components/spider-execution-manager/src/client/grpc/liveness.rs
• components/spider-execution-manager/src/client/grpc/scheduler.rs
• components/spider-execution-manager/src/client/grpc/storage.rs
• components/spider-scheduler/src/storage_client/grpc.rs
• components/spider-utils/src/grpc/client.rs
• components/spider-utils/src/grpc/mod.rs

Walkthrough

The PR adds a reusable gRPC ConnectionPool utility and updates execution-manager and scheduler gRPC clients to create pooled connections and fetch a client for each RPC. It also exposes the new gRPC module surface and adds required crate dependencies.

Changes

Pooled gRPC client rollout

Layer / File(s)	Summary
gRPC utility surface and pool components/spider-utils/Cargo.toml, components/spider-utils/src/lib.rs, components/spider-utils/src/grpc/mod.rs, components/spider-utils/src/grpc/client.rs	spider-utils exports grpc, adds the gRPC error type and tonic dependency, and defines ConnectionPool with round-robin client selection.
Execution-manager liveness pool components/spider-execution-manager/src/client/grpc/liveness.rs	GrpcLivenessClient stores a ConnectionPool, accepts pool_size in connect, and acquires a pooled client for register and heartbeat.
Execution-manager scheduler pool components/spider-execution-manager/src/client/grpc/scheduler.rs	GrpcSchedulerClient stores a ConnectionPool, accepts pool_size in connect, and acquires a pooled client for next_task, heartbeat, and shutdown.
Execution-manager storage pool components/spider-execution-manager/src/client/grpc/storage.rs	GrpcStorageClient stores a ConnectionPool, accepts pool_size in connect, and acquires a pooled client for register_task_instance, report_task_success, and report_task_failure.
Scheduler storage pool components/spider-scheduler/Cargo.toml, components/spider-scheduler/src/storage_client/grpc.rs	spider-scheduler adds spider-utils, and GrpcSchedulerStorageClient now uses separate inbound queue and job orchestration pools for its RPCs.

Sequence Diagram(s)

sequenceDiagram
  participant GrpcSchedulerStorageClient
  participant ConnectionPool
  participant InboundQueueServiceClient
  participant JobOrchestrationServiceClient
  GrpcSchedulerStorageClient->>ConnectionPool: connect(endpoint, pool_size)
  ConnectionPool-->>GrpcSchedulerStorageClient: inbound_queue_connection_pool and job_orchestration_connection_pool
  GrpcSchedulerStorageClient->>ConnectionPool: get_client()
  ConnectionPool-->>GrpcSchedulerStorageClient: InboundQueueServiceClient
  GrpcSchedulerStorageClient->>InboundQueueServiceClient: poll_ready_tasks / poll_ready_commit_tasks / poll_ready_cleanup_tasks
  GrpcSchedulerStorageClient->>ConnectionPool: get_client()
  ConnectionPool-->>GrpcSchedulerStorageClient: JobOrchestrationServiceClient
  GrpcSchedulerStorageClient->>JobOrchestrationServiceClient: get_job_state

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• y-scope/spider#333 — Introduced GrpcStorageClient; this PR changes that client to use ConnectionPool.
• y-scope/spider#340 — Introduced GrpcLivenessClient; this PR updates its constructor and RPC calls to pooled connections.
• y-scope/spider#342 — Introduced GrpcSchedulerClient; this PR applies the same pooling refactor in that file.

Suggested reviewers

• sitaowang1998

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adding a gRPC connection pool and wiring clients to use it.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/spider-execution-manager/src/client/grpc/liveness.rs`:
- Line 26: The rustdoc on the liveness client currently mentions the wrong
endpoint, which is misleading in public documentation. Update the doc comment on
the liveness client connection helper in liveness.rs so it refers to the
liveness gRPC endpoint instead of the storage gRPC endpoint, keeping the wording
aligned with the actual service used by the liveness client and its connection
logic.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 2ae0c52a-03a3-48e8-a334-0f2a54dfc7e2

📥 Commits

Reviewing files that changed from the base of the PR and between 7af2859 and 2626892.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (9)

• components/spider-execution-manager/src/client/grpc/liveness.rs
• components/spider-execution-manager/src/client/grpc/scheduler.rs
• components/spider-execution-manager/src/client/grpc/storage.rs
• components/spider-scheduler/Cargo.toml
• components/spider-scheduler/src/storage_client/grpc.rs
• components/spider-utils/Cargo.toml
• components/spider-utils/src/grpc/client.rs
• components/spider-utils/src/grpc/mod.rs
• components/spider-utils/src/lib.rs