Skip to content

feat(task): stream remote cache artifacts - #11623

Merged
jdx merged 4 commits into
agent/task-cache-compositefrom
agent/task-cache-streaming
Aug 3, 2026
Merged

jdx merged 4 commits into
agent/task-cache-compositefrom
agent/task-cache-streaming

Conversation

@jdx

@jdx jdx commented Aug 2, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • make the task cache store interface asynchronous for network-backed implementations
  • stream remote artifact uploads from disk and downloads into temporary files
  • publish artifacts before manifests and validate protocol keys, headers, and media types

Tests

  • cargo test task_cache_store::tests
  • cargo clippy --workspace --all-features --all-targets -- -D warnings
  • mise run test:e2e e2e/tasks/test_task_artifact_cache_concurrency e2e/tasks/test_task_artifact_cache_no_outputs e2e/tasks/test_task_artifact_cache_resilience

AI-assisted — Tool: Codex; model: unavailable/unavailable; version: unavailable.


Note

Medium Risk
Touches core task cache restore/store paths and adds a large network/CAS layer; failures are mostly softened with warnings, but bad remote data or partial failures could still affect cache hits and artifact integrity checks.

Overview
Adds an async TaskCacheStore and wires task execution to await cache get, restore, store, and current-output paths.

Remote HTTP backend implements the v1 cache protocol: action results and content-addressed blobs (action bytes, canonical client metadata, directory trees, file chunks). Artifact uploads stream from disk; downloads stream into temp files before reassembly into a local tar.zst for restore. The composite store still promotes remote hits locally and mirrors writes, with lookup failures degrading to local-only.

Cache keys now hash canonical JSON key material; commits pass the serialized action bytes so remote lookups are keyed by (blake3 key, action size). Manifest/types are shared with the store layer; artifact paths use a small wrapper for on-disk vs temporary files.

Reviewed by Cursor Bugbot for commit 16390f1. Bugbot is set up for automated code reviews on this repo. Configure here.

@coderabbitai

coderabbitai Bot commented Aug 2, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 10950524-4b1f-4a94-be5c-450066ed6a20

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Comment thread src/task/task_cache_store.rs Outdated
@greptile-apps

greptile-apps Bot commented Aug 2, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR makes task-cache stores asynchronous and adds streamed, content-addressed HTTP artifact transport.

  • Canonicalizes and persists action-key material for remote protocol validation.
  • Splits output archives into directory, file, and symlink CAS objects and reconstructs them on download.
  • Keeps local-first composite behavior while treating remote failures as cache misses.

Confidence Score: 5/5

The PR appears safe to merge within the scope of this follow-up review.

No blocking failure remains from an eligible prior finding.

Important Files Changed

Filename Overview
src/task/task_cache.rs Makes cache reads, restores, and writes asynchronous while carrying canonical action bytes through the store interface.
src/task/task_cache_store.rs Adds the HTTP CAS protocol, streamed artifact transfer, temporary artifact ownership, and asynchronous local/composite store implementations.
src/task/task_executor.rs Awaits asynchronous cache lookup, restore, and persistence operations in the task execution flow.
TASK_CACHE_PARITY.md Marks streamed artifact uploads and downloads as implemented.

Reviews (12): Last reviewed commit: "fix(task): verify and canonicalize remot..." | Re-trigger Greptile

@jdx
jdx force-pushed the agent/task-cache-streaming branch from e6bad52 to cb45fdc Compare August 2, 2026 03:15
@jdx
jdx force-pushed the agent/task-cache-streaming branch from cb45fdc to 6714f71 Compare August 2, 2026 03:17
Comment thread src/task/task_cache.rs Outdated
Comment thread src/task/task_cache_store.rs Outdated
@jdx
jdx force-pushed the agent/task-cache-streaming branch from 6714f71 to 183eea9 Compare August 2, 2026 03:36
@github-actions

github-actions Bot commented Aug 2, 2026 •

Copy link
Copy Markdown
Contributor

Instruction counts

benchmark trend instructions Δ wall (min) Δ
env ▄█▃▁▁▂█▆▇▄ 80,074,952 → 80,045,402 -0.04% 17.64 → 17.44ms -1.13%
hook-env █▇▁▄▃▂▅▄▆▇ 82,190,914 → 82,215,898 +0.03% 18.34 → 18.44ms +0.56%
ls ▆▅▁▁▂███▇▅ 73,127,800 → 73,084,830 -0.06% 15.71 → 16.91ms +7.64%
registry ▁▁▂▂▂▄▄▅▂█ 49,102,713 → 49,255,363 +0.31% 11.64 → 13.06ms +12.11%
startup ▄▄▁▃▃▂▂▃▃█ 15,784,751 → 15,811,232 +0.17% 9.04 → 9.46ms +4.65%

No instruction-count regression above 1%.

Only instruction counts gate. Wall clock is shown for context — on identical hardware it moves 4-20% run to run.

Measured by tak — instruction-counted CLI benchmarks, stored in this repository's git notes.

16390f17a220 vs e5230aa14775 · measured on the runner, not pushed to the history.

@jdx
jdx force-pushed the agent/task-cache-streaming branch from 183eea9 to da1f8a7 Compare August 2, 2026 13:10
@jdx
jdx force-pushed the agent/task-cache-streaming branch from da1f8a7 to 225b0d6 Compare August 2, 2026 15:18
@jdx
jdx force-pushed the agent/task-cache-streaming branch 3 times, most recently from 26bac7f to 5c12871 Compare August 2, 2026 17:33
Comment thread src/task/task_cache_store.rs
Comment thread src/task/task_cache_store.rs
Comment thread src/task/task_cache_store.rs
@jdx
jdx force-pushed the agent/task-cache-streaming branch from 5c12871 to d4e8afd Compare August 2, 2026 18:18
Comment thread src/task/task_cache_store.rs Outdated
@jdx
jdx force-pushed the agent/task-cache-streaming branch from d4e8afd to 7bd9cf2 Compare August 2, 2026 18:58
Comment thread src/task/task_cache_store.rs
@jdx
jdx force-pushed the agent/task-cache-streaming branch from 7bd9cf2 to 96e6d03 Compare August 2, 2026 19:10

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 96e6d03. Configure here.

Comment thread src/task/task_cache_store.rs
@jdx
jdx force-pushed the agent/task-cache-streaming branch from 96e6d03 to 16390f1 Compare August 2, 2026 21:25
@jdx
jdx merged commit 450bd5f into main Aug 3, 2026
30 checks passed
@jdx
jdx deleted the agent/task-cache-streaming branch August 3, 2026 03:04
donbeave pushed a commit to donbeave/mise that referenced this pull request Aug 3, 2026
## Summary
- make the task cache store interface asynchronous for network-backed
implementations
- stream remote artifact uploads from disk and downloads into temporary
files
- publish artifacts before manifests and validate protocol keys,
headers, and media types

## Tests
- `cargo test task_cache_store::tests`
- `cargo clippy --workspace --all-features --all-targets -- -D warnings`
- `mise run test:e2e e2e/tasks/test_task_artifact_cache_concurrency
e2e/tasks/test_task_artifact_cache_no_outputs
e2e/tasks/test_task_artifact_cache_resilience`

*AI-assisted — Tool: Codex; model: unavailable/unavailable; version:
unavailable.*

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Touches core task cache restore/store paths and adds a large
network/CAS layer; failures are mostly softened with warnings, but bad
remote data or partial failures could still affect cache hits and
artifact integrity checks.
> 
> **Overview**
> Adds an **async** `TaskCacheStore` and wires task execution to
**await** cache get, restore, store, and current-output paths.
> 
> **Remote HTTP backend** implements the v1 cache protocol: action
results and content-addressed blobs (action bytes, canonical client
metadata, directory trees, file chunks). Artifact **uploads stream from
disk**; **downloads stream into temp files** before reassembly into a
local `tar.zst` for restore. The composite store still promotes remote
hits locally and mirrors writes, with lookup failures degrading to
local-only.
> 
> Cache keys now hash **canonical JSON** key material; commits pass the
serialized **action** bytes so remote lookups are keyed by `(blake3 key,
action size)`. Manifest/types are shared with the store layer; artifact
paths use a small wrapper for on-disk vs temporary files.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
16390f1. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant