feat(spider-storage): Add storage service runtime; Replace `ExecutionManagerLivenessStore` with `ExecutionManagerLivenessManagement` DB protocol in `TaskInstancePool`. by sitaowang1998 · Pull Request #324 · y-scope/spider

sitaowang1998 · 2026-05-14T19:58:51Z

Description

This PR:

Adds a new ServerRuntime type that keeps a CancellationToken and JoinHandle to gracefully shutdown the service.
Merges ExecutionManagerLivenessStore in task instance pool into ExecutionManagerLivenessManagement in db.
Adds graceful shutdown in task instance pool that drains all messages.
Refactors task instance pool handle creation into a factory method.

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

New tests added for the graceful shutdown and task instance pool factory method.
GitHub workflows pass.

Summary by CodeRabbit

Release Notes

Chores
- Updated Tokio dependencies to latest compatible versions for improved stability
Refactor
- Restructured task instance pool API with new configuration options
- Introduced server runtime abstraction for better background task coordination
- Enhanced error handling for database operations in cache layer

coderabbitai · 2026-05-14T19:58:59Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 1a22ebd1-628b-48d7-bfe7-9098ebb69c31

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title accurately describes the two main changes: adding ServerRuntime for storage service and replacing ExecutionManagerLivenessStore with ExecutionManagerLivenessManagement in TaskInstancePool.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/spider-storage/src/state/server.rs`:
- Around line 102-109: The timeout currently moves
self.task_instance_pool_join_handle into tokio::time::timeout so if the timeout
fires the handle is dropped and the background task keeps running; change the
logic to detect timeout without consuming the handle (e.g., use tokio::select!
between the join handle future and tokio::time::sleep) so you can call abort()
on the JoinHandle (self.task_instance_pool_join_handle.abort()) when the timeout
branch wins, then return the StorageServerError::Stopping error; reference the
existing stop_timeout_sec and task_instance_pool_join_handle symbols and ensure
you await or ignore the aborted handle appropriately after aborting.

In `@components/spider-storage/src/task_instance_pool.rs`:
- Around line 169-170: Clamp the public config fields channel_size and
gc_interval to at least 1 before creating Tokio primitives to avoid panics: when
calling mpsc::channel(...) use config.channel_size.max(1) and when creating the
interval with tokio::time::interval(...) use config.gc_interval.max(1) (or
assign clamped values to locals first). Update the TaskInstancePool
construction/initialization sites (references: TaskInstancePoolConfig,
mpsc::channel, tokio::time::interval, variables sender/receiver and the GC
interval setup) to pass the clamped values so zero cannot be forwarded into
Tokio APIs.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 71fa0b19-7bcb-4d56-96c9-4681b1e1b273

📥 Commits

Reviewing files that changed from the base of the PR and between dfb2170 and 5b8eccf.

📒 Files selected for processing (5)

components/spider-storage/Cargo.toml
components/spider-storage/src/cache/error.rs
components/spider-storage/src/state.rs
components/spider-storage/src/state/server.rs
components/spider-storage/src/task_instance_pool.rs

LinZhihao-723 · 2026-06-07T21:27:07Z

Reviewed and made the following changes:

server.rs -> runtime.rs. This is a runtime, not an actual server.

stop_timeout stored as a Duration instance.

stop_background_tasks_cancels_and_joins_task -> stop for simplicity.

Change create_server_runtime to accept all configs needed, and make sure it returns the cancellation token as well so that the main loop can monitor its changes.

LinZhihao-723 · 2026-06-07T21:27:39Z

+        task_instance_pool_config,
+    );
+
+    // TODO: Recover jobs from the database.


This might be a high-priority task: we need to complete this stage to make the storage fully serviceable.

LinZhihao-723 · 2026-06-07T22:23:06Z

Reviewed and polished with the following changes:

Fix symbol ordering.

Field renames:

execution_manager_stale_after_sec → execution_manager_stale_cutoff_sec

gc_interval → gc_interval_sec

channel_size → message_channel_capacity

Validation moved out of new, and enforced inside create_task_instance_pool.

Deleted drain_received_messages: there's no need to drain the message queue. If there are any blocked service tasks, they will be aborted anyway by tokio runtime. We only need to make sure there can't be new requests being served.

Should we have a dedicated type for validated config?

I think the current pattern should be fine: in the final deliverable, task instance pool creation must go through creat_task_instance_pool, which enforces the config validation. We probably need a dedicated type if it's been used in multiple places.

LinZhihao-723

Directly modified the PR title.

…343) Related PRs: #319, #321, #322, #323, #324, #338, #339, #341. Co-authored-by: sitaowang1998 <sitaowang1998@outlook.com>

sitaowang1998 added 7 commits May 13, 2026 23:57

Add graceful shutdown & server runtime

6d421d5

Merge two em liveness trait into one

fc6e11e

Use u64 instead of duration

6a3be1e

Fix test

6bba976

Rename variable

5ccfa8d

Refactor server runtime

e772967

Fix import

5b8eccf

sitaowang1998 requested a review from a team as a code owner May 14, 2026 19:58

coderabbitai Bot reviewed May 14, 2026

View reviewed changes

Comment thread components/spider-storage/src/state/server.rs Outdated

Comment thread components/spider-storage/src/task_instance_pool.rs Outdated

Address comment

76dafa0

sitaowang1998 requested a review from LinZhihao-723 May 14, 2026 20:46

Review .

cda4e6b

LinZhihao-723 requested changes Jun 7, 2026

View reviewed changes

Review task instance pool changes.

cf7235a

LinZhihao-723 requested changes Jun 7, 2026

View reviewed changes

Rename factory API.

4f721c7

LinZhihao-723 changed the title ~~feat(spider-storage): Add ServerRuntime to gracefully shutdown storage service.~~ feat(spider-storage): Add storage service runtime; Replace ExecutionManagerLivenessStore with ExecutionManagerLivenessManagement DB protocol in TaskInstancePool. Jun 7, 2026

LinZhihao-723 approved these changes Jun 7, 2026

View reviewed changes

Toml fix.

b7c8c70

sitaowang1998 merged commit 2cbdff6 into y-scope:storage-service-dev Jun 8, 2026
11 checks passed

sitaowang1998 deleted the graceful-shutdown branch June 8, 2026 00:22

LinZhihao-723 mentioned this pull request Jun 15, 2026

feat(spider-storage): Merge storage-service-dev branch into main. #343

Merged

4 tasks

LinZhihao-723 added a commit that referenced this pull request Jun 15, 2026

feat(spider-storage): Merge storage-service-dev branch into main. (#…

1d1782b

…343) Related PRs: #319, #321, #322, #323, #324, #338, #339, #341. Co-authored-by: sitaowang1998 <sitaowang1998@outlook.com>

This was referenced Jul 10, 2026

fix(spider-storage): Cancel the service when a background tokio task exits on error. #387

Merged

fix(spider-storage): Refill the inbound queue when the server restarts. #398

Merged

Navigation Menu

Uh oh!

feat(spider-storage): Add storage service runtime; Replace `ExecutionManagerLivenessStore` with `ExecutionManagerLivenessManagement` DB protocol in `TaskInstancePool`. - #324

feat(spider-storage): Add storage service runtime; Replace `ExecutionManagerLivenessStore` with `ExecutionManagerLivenessManagement` DB protocol in `TaskInstancePool`.#324
sitaowang1998 merged 12 commits into
y-scope:storage-service-devfrom
sitaowang1998:graceful-shutdown

sitaowang1998 commented May 14, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented May 14, 2026 •

edited

Loading

Review skipped

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

LinZhihao-723 Jun 7, 2026

Uh oh!

LinZhihao-723 Jun 7, 2026

Uh oh!

LinZhihao-723 Jun 7, 2026

Uh oh!

sitaowang1998 Jun 7, 2026

Uh oh!

LinZhihao-723 Jun 7, 2026

Uh oh!

LinZhihao-723 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sitaowang1998 commented May 14, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Validation performed

Summary by CodeRabbit

Release Notes

Uh oh!

coderabbitai Bot commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

LinZhihao-723 Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

LinZhihao-723 Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

LinZhihao-723 Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

sitaowang1998 Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

LinZhihao-723 Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

LinZhihao-723 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sitaowang1998 commented May 14, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 14, 2026 •

edited

Loading