Skip to content

perf(task): benchmark artifact cache phases - #11614

Merged
jdx merged 1 commit into
agent/task-cache-portabilityfrom
agent/task-cache-benchmarks
Aug 3, 2026
Merged

jdx merged 1 commit into
agent/task-cache-portabilityfrom
agent/task-cache-benchmarks

Conversation

@jdx

@jdx jdx commented Aug 2, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • add an isolated task artifact cache benchmark harness
  • measure source hashing, archive publication, restoration, and cached graph traversal separately
  • generate configurable large source, artifact, and dependency-graph fixtures
  • document usage and complete the benchmark tracker item

Validation

  • TASK_CACHE_BENCH_SOURCES=10 TASK_CACHE_BENCH_ARTIFACT_BYTES=65536 TASK_CACHE_BENCH_TASKS=5 TASK_CACHE_BENCH_RUNS=2 MISE_BIN="$PWD/target/debug/mise" bash benchmarks/task-cache/benchmark.sh
  • mise run lint-fix

AI-assisted — Tool: Codex; model: unavailable/unavailable; version: unavailable.


Note

Low Risk
Benchmark and documentation only; no changes to cache runtime behavior or production code paths.

Overview
Adds an end-to-end benchmark harness for the experimental local task artifact cache, wired as mise run perf:task-cache (release build + benchmarks/task-cache/benchmark.sh).

The script spins up a temporary mise project with configurable scale (defaults: 2k sources, 32 MiB artifact, 500-node graph) and reports median wall-clock times for four phases: source hashing / result-only publish, task run + archive creation, cache restore, and fully cached graph traversal. Fixture prep and cache warm-up sit outside the timed windows; MISE_BIN and TASK_CACHE_BENCH_* env vars tune binary and size.

Docs and tracking: README describes usage and env overrides; TASK_CACHE_PARITY.md marks the hashing/archive/restore benchmark tracker item done; tasks.md documents the new task.

Reviewed by Cursor Bugbot for commit 03aa160. Bugbot is set up for automated code reviews on this repo. Configure here.

@coderabbitai

coderabbitai Bot commented Aug 2, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: b65b4c9c-78cd-42d1-9b8e-58e17b233c18

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Comment thread benchmarks/task-cache/benchmark.sh Resolved
@greptile-apps

greptile-apps Bot commented Aug 2, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds an isolated benchmark harness for task artifact-cache hashing, publication, restoration, and graph traversal, plus task-runner wiring and usage documentation.

  • Generates configurable source, artifact, and dependency-graph fixtures in a temporary project.
  • Reports median wall-clock measurements for four cache phases.
  • Adds perf:task-cache and marks the corresponding parity-tracker item complete.

Confidence Score: 4/5

The PR is not ready to merge because the restore and cached-graph phases still measure the wrong behavior.

The restore setup deletes the output but the current-output fast path skips before extraction, while source-less graph tasks do not receive cache configuration and therefore never create the entries the cached traversal phase claims to measure.

Files Needing Attention: benchmarks/task-cache/benchmark.sh, TASK_CACHE_PARITY.md

Important Files Changed

Filename Overview
benchmarks/task-cache/benchmark.sh Adds the benchmark fixture and timing harness, but the previously reported restore and cached-graph measurement defects remain.
benchmarks/task-cache/README.md Documents benchmark phases, defaults, configuration variables, and comparison guidance.
mise.toml Adds a task that builds the release binary and launches the benchmark harness.
TASK_CACHE_PARITY.md Marks the benchmark tracker item complete, although two benchmark phases still do not measure their stated workloads.
tasks.md Adds generated usage documentation for the new performance task.

Reviews (10): Last reviewed commit: "perf(task): benchmark artifact cache pha..." | Re-trigger Greptile

Comment thread benchmarks/task-cache/benchmark.sh Resolved
Comment thread benchmarks/task-cache/benchmark.sh Resolved
Comment thread benchmarks/task-cache/benchmark.sh Resolved
@jdx
jdx force-pushed the agent/task-cache-benchmarks branch from 0183658 to db16a64 Compare August 2, 2026 01:33
Comment thread benchmarks/task-cache/benchmark.sh Resolved
@github-actions

github-actions Bot commented Aug 2, 2026 •

Copy link
Copy Markdown
Contributor

Instruction counts

benchmark trend instructions Δ wall (min) Δ
env ▂▃▂▁▁▁▃▃▃█ 80,074,952 → 80,255,975 +0.23% 17.64 → 18.30ms +3.74%
hook-env ▅▅▁▃▂▂▃▃▄█ 82,190,914 → 82,339,219 +0.18% 18.34 → 19.41ms +5.85%
ls ▄▃▁▁▁▅▄▅▄█ 73,127,800 → 73,275,815 +0.20% 15.71 → 16.44ms +4.62%
registry ▁▁▂▁▂▃▃▄▁█ 49,102,713 → 49,323,289 +0.45% 11.64 → 12.33ms +5.91%
startup ▃▃▁▂▂▁▁▂▂█ 15,784,751 → 15,840,006 +0.35% 9.04 → 9.28ms +2.65%

No instruction-count regression above 1%.

Only instruction counts gate. Wall clock is shown for context — on identical hardware it moves 4-20% run to run.

Measured by tak — instruction-counted CLI benchmarks, stored in this repository's git notes.

03aa16066835 vs e5230aa14775 · measured on the runner, not pushed to the history.

@jdx
jdx force-pushed the agent/task-cache-benchmarks branch from db16a64 to c67b7b1 Compare August 2, 2026 01:52

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c67b7b1. Configure here.

Comment thread benchmarks/task-cache/benchmark.sh Resolved
@jdx
jdx force-pushed the agent/task-cache-benchmarks branch 2 times, most recently from 04a9c68 to f338896 Compare August 2, 2026 02:08
@jdx
jdx force-pushed the agent/task-cache-benchmarks branch from f338896 to 4711996 Compare August 2, 2026 02:26
@jdx
jdx force-pushed the agent/task-cache-benchmarks branch from 4711996 to 5f69079 Compare August 2, 2026 02:36
@jdx
jdx force-pushed the agent/task-cache-benchmarks branch from 5f69079 to 4711996 Compare August 2, 2026 02:39
@jdx
jdx force-pushed the agent/task-cache-benchmarks branch from 4711996 to 8cf46ff Compare August 2, 2026 13:10
greptile-apps[bot]
greptile-apps Bot previously approved these changes Aug 2, 2026
@jdx
jdx force-pushed the agent/task-cache-benchmarks branch from 8cf46ff to 03aa160 Compare August 2, 2026 15:18
@greptile-apps
greptile-apps Bot dismissed their stale review August 2, 2026 15:18

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

@jdx
jdx merged commit 34144d7 into main Aug 3, 2026
45 of 57 checks passed
@jdx
jdx deleted the agent/task-cache-benchmarks branch August 3, 2026 03:04
donbeave pushed a commit to donbeave/mise that referenced this pull request Aug 3, 2026
## Summary
- add an isolated task artifact cache benchmark harness
- measure source hashing, archive publication, restoration, and cached
graph traversal separately
- generate configurable large source, artifact, and dependency-graph
fixtures
- document usage and complete the benchmark tracker item

## Validation
- `TASK_CACHE_BENCH_SOURCES=10 TASK_CACHE_BENCH_ARTIFACT_BYTES=65536
TASK_CACHE_BENCH_TASKS=5 TASK_CACHE_BENCH_RUNS=2
MISE_BIN="$PWD/target/debug/mise" bash
benchmarks/task-cache/benchmark.sh`
- `mise run lint-fix`

*AI-assisted — Tool: Codex; model: unavailable/unavailable; version:
unavailable.*

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Benchmark and documentation only; no changes to cache runtime behavior
or production code paths.
> 
> **Overview**
> Adds an **end-to-end benchmark harness** for the experimental local
task artifact cache, wired as `mise run perf:task-cache` (release build
+ `benchmarks/task-cache/benchmark.sh`).
> 
> The script spins up a **temporary mise project** with configurable
scale (defaults: 2k sources, 32 MiB artifact, 500-node graph) and
reports **median wall-clock** times for four phases: source hashing /
result-only publish, task run + archive creation, cache restore, and
fully cached graph traversal. Fixture prep and cache warm-up sit outside
the timed windows; `MISE_BIN` and `TASK_CACHE_BENCH_*` env vars tune
binary and size.
> 
> **Docs and tracking:** README describes usage and env overrides;
`TASK_CACHE_PARITY.md` marks the hashing/archive/restore benchmark
tracker item done; `tasks.md` documents the new task.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
03aa160. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant