feat(spider-client)!: Retry gRPC connection failures with a configurable exponential-backoff policy.

[LinZhihao-723]

Description

This PR makes SpiderClient retry transient gRPC failures instead of surfacing the first lost-connection error to the caller. The retry logic is a reusable async helper added to spider-utils, the tonic-specific "which failures are retriable" policy lives alongside it, and SpiderClient gains a builder so callers can tune the retry behavior (and the existing connection-pool size) with named, optional settings. Every gRPC round-trip the client issues is now wrapped in this retry policy.

Because retry needs a call the helper can invoke repeatedly, the wrapped closure re-pulls a client from the connection pool on each attempt, so a retry is re-sent over the pool's next round-robin connection rather than the one that just failed.

Generic retry helper (spider-utils)

• Adds spider_utils::grpc::retry::execute_with_retry(max_retries, max_backoff, grpc_call, is_retriable): an async helper generic over the success and error types, the async call closure (AsyncFnMut() -> Result<ResponseType, ErrorType>), and an is_retriable: Fn(&ErrorType) -> bool classifier. It returns as soon as the call succeeds, returns immediately when the error is classified non-retriable, and otherwise retries until the retry budget is exhausted and returns the last error. max_retries counts attempts after the initial one, so the call is invoked at most max_retries + 1 times.
• Between attempts it sleeps for an exponentially increasing backoff that doubles each retry (starting at 100ms), capped at max_backoff, plus a small uniform random jitter (up to 20ms) so concurrent clients don't retry in lockstep. The doubling is overflow-safe and the jitter is additive on top of the cap, so an actual wait may slightly exceed max_backoff.
• Adds RetryConfig { max_retries: usize, max_backoff: Duration } (Clone, Copy, Debug), a small bundle of the two knobs with a Default of 10 retries and a 3s backoff cap.
• Adds call_with_retry(retry_config, grpc_call), a thin tonic-facing wrapper over execute_with_retry that supplies the retriable policy: a gRPC Status is retried only when its code is UNAVAILABLE (a lost or unestablished connection). All other status codes are treated as deterministic and returned immediately.

Configurable client via a builder (spider-client)

• Adds SpiderClientBuilder and SpiderClient::builder(endpoint). The builder exposes consuming, #[must_use] setters — pool_size, max_retries, max_backoff — and an async connect() that establishes the pools and returns a SpiderClient. Unset knobs fall back to a default pool size of 8 and RetryConfig::default().
• Breaking: removes the previous SpiderClient::connect(endpoint, pool_size) constructor; the builder is now the sole entry point. Migration is SpiderClient::builder(endpoint).pool_size(pool_size).connect().await.
• Re-exports SpiderClientBuilder and RetryConfig from the crate root so callers configuring retries don't need to reach into spider-utils.

Retry wiring in the gRPC clients (spider-client)

• Both service clients (JobOrchestrationClient, ResourceGroupManagementClient) now carry a RetryConfig, threaded from the builder through their connect constructors.
• All eight service calls — the six job-orchestration calls (submit_job, start_job, cancel_job, get_job_state, get_job_outputs, get_job_error) and the two resource-group calls (add_resource_group, verify_resource_group) — now issue their gRPC round-trip through call_with_retry. Only the network call is inside the retried closure; request serialization/compression and response deserialization stay outside it and run exactly once. The per-attempt request is cloned so each retry re-sends over a freshly selected pooled connection.

Caller update (huntsman e2e)

• Updates the end-to-end test driver's client construction from the removed SpiderClient::connect(endpoint, concurrency) to SpiderClient::builder(endpoint).pool_size(concurrency).connect().await, and fixes the corresponding docstring reference.

Tests

• Adds unit tests for the generic helper in spider-utils: success on the first attempt, success after a run of retriable failures, immediate return on a non-retriable error, retry-budget exhaustion (asserting exactly max_retries + 1 invocations), and a bounds test asserting the jittered backoff always lands within [capped, capped + jitter] across the exponential and clamped regions.
• Adds tonic-path tests for call_with_retry: an UNAVAILABLE status is retried and then succeeds, and a non-retriable status (NOT_FOUND) returns immediately without retrying.

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• Ensure all workflows pass.

Summary by CodeRabbit

• New Features

  • Introduced a builder-based client connection flow to configure pool size and retry parameters.
  • Added retry support for transient gRPC failures, including exponential backoff with jitter.
  • Exposed retry configuration and builder APIs for wider reuse.

• Bug Fixes

  • Improved resilience by automatically retrying eligible gRPC operations when the service is temporarily unavailable, reducing transient connection failures.

• Tests

  • Updated end-to-end test driver to use the new builder-based client connection path.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 22d87144-121b-4372-9371-aeca6eeb4a71

📥 Commits

Reviewing files that changed from the base of the PR and between 7f8c4b3 and fe1d815.

📒 Files selected for processing (1)

• components/spider-utils/src/grpc/retry.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• components/spider-utils/src/grpc/retry.rs

---

Walkthrough

Changes

Spider client construction now uses a configurable builder with pool and retry settings. Shared retry utilities handle transient gRPC failures with bounded backoff, and job/resource-group RPCs use them. The e2e test driver adopts the builder API.

Spider client retry integration

Layer / File(s)	Summary
Retry policy and execution components/spider-utils/src/grpc/retry.rs, components/spider-utils/Cargo.toml, components/spider-utils/src/grpc/mod.rs	Adds RetryConfig, retry execution, unavailable-status handling, jittered backoff, dependencies, and unit tests.
Builder configuration and connection pools components/spider-client/src/client.rs, components/spider-client/src/lib.rs	Adds SpiderClientBuilder, configurable pool/retry settings, default values, and public re-exports.
Retry-enabled gRPC operations components/spider-client/src/grpc/job.rs, components/spider-client/src/grpc/resource_group.rs	Stores retry configuration and routes job and resource-group RPCs through call_with_retry.
Test-driver initialization tests/huntsman/e2e/src/test_driver.rs	Updates client construction to use the builder-based API.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant SpiderClientBuilder
  participant JobOrchestrationClient
  participant ResourceGroupManagementClient
  participant call_with_retry
  participant gRPCService
  Caller->>SpiderClientBuilder: configure and connect
  SpiderClientBuilder->>JobOrchestrationClient: create pool with RetryConfig
  SpiderClientBuilder->>ResourceGroupManagementClient: create pool with RetryConfig
  Caller->>JobOrchestrationClient: invoke job RPC
  JobOrchestrationClient->>call_with_retry: execute RPC
  call_with_retry->>gRPCService: retry unavailable calls
  Caller->>ResourceGroupManagementClient: invoke resource-group RPC
  ResourceGroupManagementClient->>call_with_retry: execute RPC

Loading

Possibly related PRs

• y-scope/spider#363: Introduced the user-facing spider-client API that this change refactors.
• y-scope/spider#379: Updated e2e initialization to use the builder-based client API.

Suggested reviewers: sitaowang1998

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding configurable exponential-backoff retries for Spider gRPC failures.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/spider-client/src/grpc/job.rs`:
- Around line 93-101: Remove the call_with_retry wrapper around register_job in
the job submission flow, invoking the RPC once while preserving the existing
status mapping and into_inner handling. Keep retry behavior available only for
idempotent RPCs, and do not add retries without an idempotency key.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 2c69add3-d223-4cfd-a082-4571585aa8e2

📥 Commits

Reviewing files that changed from the base of the PR and between 1fe99d6 and 7f8c4b3.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (8)

• components/spider-client/src/client.rs
• components/spider-client/src/grpc/job.rs
• components/spider-client/src/grpc/resource_group.rs
• components/spider-client/src/lib.rs
• components/spider-utils/Cargo.toml
• components/spider-utils/src/grpc/mod.rs
• components/spider-utils/src/grpc/retry.rs
• tests/huntsman/e2e/src/test_driver.rs

[sitaowang1998]

tonic sometimes map tonic::transport::Error to Status::Unknown instead of Status::Unavailable. This is a known issue and has been reported and closed without fix. Our e2e test also shows that Status::Unknown is thrown when a server goes down. Thus, we need to handle this as well.

[LinZhihao-723]

tonic sometimes map tonic::transport::Error to Status::Unknown instead of Status::Unavailable. This is a known issue and has been reported and closed without fix. Our e2e test also shows that Status::Unknown is thrown when a server goes down. Thus, we need to handle this as well.

I will do it. In the meanwhile, can you check what exactly the error is when the status is Unknown (e.g., provide the details like the man reported in the issue)?

[sitaowang1998]

tonic sometimes map tonic::transport::Error to Status::Unknown instead of Status::Unavailable. This is a known issue and has been reported and closed without fix. Our e2e test also shows that Status::Unknown is thrown when a server goes down. Thus, we need to handle this as well.

I will do it. In the meanwhile, can you check what exactly the error is when the status is Unknown (e.g., provide the details like the man reported in the issue)?

[DIAG] job_status_to_error: code=Unknown message="transport error" source_chain=
  [0] tonic::transport::Error(Transport, hyper::Error(Io, Kind(ConnectionReset)))
  [1] hyper::Error(Io, Kind(ConnectionReset))
  [2] Kind(ConnectionReset)

[sitaowang1998]

Verified that the e2e test pass with a temporary fix for storage enqueue discussed offline.