Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions TASK_CACHE_PARITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,8 @@ outside this tracker.

## Local cache parity

- [ ] Include dependency artifact keys in downstream task keys
- [ ] Allow dependents to restore after dependencies execute or restore
- [x] Include dependency artifact keys in downstream task keys
- [x] Allow dependents to restore after dependencies execute or restore
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
- [ ] Capture and replay stdout and stderr while respecting mise output modes
- [ ] Cache useful results for tasks with no filesystem outputs
- [ ] Support reusable and global input groups
Expand Down
13 changes: 7 additions & 6 deletions docs/tasks/task-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -491,18 +491,19 @@ cache = { enabled = false }
```

The cache key includes source contents, the task definition and arguments, resolved task environment,
the values (or absence) of variables named in `cache.env`, resolved tool versions, and the operating
system and architecture. Variables inherited from the ambient process are ignored unless listed in
`cache.env`.
the values (or absence) of variables named in `cache.env`, resolved tool versions, dependency
artifact keys, and the operating system and architecture. Variables inherited from the ambient
process are ignored unless listed in `cache.env`.

Artifacts are stored under `MISE_CACHE_DIR/task-artifacts/v1` as zstd-compressed tar archives. They
are included in the existing `mise cache clear` and `mise cache prune` behavior. Only successful task
runs are cached. Cache read/write failures are treated as misses and never turn a successful task run
into a failure.

This initial implementation is local-only and does not cache or replay task logs. If a dependency
executes or restores outputs during the current run, dependent tasks conservatively execute rather
than restoring an artifact.
This initial implementation is local-only and does not cache or replay task logs. Cacheable
dependencies contribute their artifact keys to dependent task keys, so a dependent can restore the
matching artifact after its dependencies execute, skip, or restore. If a dependency executes without
a stable artifact key, its dependents conservatively execute.

### `shell`

Expand Down
101 changes: 101 additions & 0 deletions e2e/tasks/test_task_artifact_cache_dependencies
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
#!/usr/bin/env bash

cat <<'EOF' >mise.toml
[settings]
experimental = true

[task_config.cache]
enabled = true

[tasks.lib]
run = '''
mkdir -p lib-dist
cat lib.txt > lib-dist/result.txt
printf 'ran\n' >> lib-runs.txt
'''
sources = ["lib.txt"]
outputs = ["lib-dist"]

[tasks.app]
depends = ["lib"]
run = '''
mkdir -p app-dist
cat lib-dist/result.txt app.txt > app-dist/result.txt
printf 'ran\n' >> app-runs.txt
'''
sources = ["app.txt"]
outputs = ["app-dist"]

[tasks.prepare]
run = '''
mkdir -p prepare-dist
cat prepare.txt > prepare-dist/result.txt
printf 'ran\n' >> prepare-runs.txt
'''
sources = ["prepare.txt"]
outputs = ["prepare-dist"]
cache = { enabled = false }

[tasks.consume]
depends = ["prepare"]
run = '''
mkdir -p consume-dist
cat prepare-dist/result.txt consume.txt > consume-dist/result.txt
printf 'ran\n' >> consume-runs.txt
'''
sources = ["consume.txt"]
outputs = ["consume-dist"]
EOF

printf 'lib-one\n' >lib.txt
printf 'app\n' >app.txt
printf 'prepare-one\n' >prepare.txt
printf 'consume\n' >consume.txt

mise run app
assert "wc -l < lib-runs.txt | tr -d ' '" "1"
assert "wc -l < app-runs.txt | tr -d ' '" "1"
assert "cat app-dist/result.txt" $'lib-one\napp'

# A fresh dependency still publishes its identity when it skips, allowing a
# missing downstream output to restore.
rm -rf app-dist
assert_contains "mise run app 2>&1" "restored outputs from cache"
assert "wc -l < lib-runs.txt | tr -d ' '" "1"
assert "wc -l < app-runs.txt | tr -d ' '" "1"

# A restored dependency contributes the same identity to the downstream key,
# so both tasks can restore without executing.
rm -rf lib-dist app-dist
assert_contains "mise run app 2>&1" "restored outputs from cache"
assert "wc -l < lib-runs.txt | tr -d ' '" "1"
assert "wc -l < app-runs.txt | tr -d ' '" "1"
assert "cat app-dist/result.txt" $'lib-one\napp'

# Changing the dependency changes its artifact identity and invalidates the
# downstream artifact even though the downstream task's own sources are unchanged.
printf 'lib-two\n' >lib.txt
mise run app
assert "wc -l < lib-runs.txt | tr -d ' '" "2"
assert "wc -l < app-runs.txt | tr -d ' '" "2"
assert "cat app-dist/result.txt" $'lib-two\napp'

# Returning to a previously cached dependency identity restores both the
# dependency and the matching downstream artifact.
printf 'lib-one\n' >lib.txt
rm -rf lib-dist app-dist
assert_contains "mise run app 2>&1" "restored outputs from cache"
assert "wc -l < lib-runs.txt | tr -d ' '" "2"
assert "wc -l < app-runs.txt | tr -d ' '" "2"
assert "cat app-dist/result.txt" $'lib-one\napp'

# A dependency that executes without a stable cache identity still forces
# conservative downstream execution.
mise run consume
assert "wc -l < prepare-runs.txt | tr -d ' '" "1"
assert "wc -l < consume-runs.txt | tr -d ' '" "1"
printf 'prepare-two\n' >prepare.txt
mise run consume
assert "wc -l < prepare-runs.txt | tr -d ' '" "2"
assert "wc -l < consume-runs.txt | tr -d ' '" "2"
assert "cat consume-dist/result.txt" $'prepare-two\nconsume'
32 changes: 18 additions & 14 deletions src/cli/run.rs
Original file line number Diff line number Diff line change
Expand Up @@ -636,16 +636,16 @@ impl Run {
let semaphore = ctx.semaphore.clone();
ctx.jset.lock().await.spawn(async move {
let mut permit = permit_opt;
let (completed, dep_ran) = {
let (completion_state, dependency_state) = {
let deps = deps_for_remove.lock().await;
(deps.handled_task_keys(), deps.any_dep_ran(&task))
(deps.completion_state(), deps.dependency_state(&task))
};
let (result, panicked) = match AssertUnwindSafe(this.run_task_sched(
&task,
&ctx.config,
ctx.sched_tx.clone(),
completed,
dep_ran,
completion_state,
dependency_state,
semaphore,
&mut permit,
))
Expand All @@ -658,13 +658,17 @@ impl Run {
true,
),
};
// If the task actually ran (not skipped) and has sources defined,
// If the task executed or restored outputs and has sources defined,
// mark it so dependents' source freshness checks are invalidated.
// Tasks without sources always run and should not trigger invalidation.
if let Ok(true) = &result
&& !task.sources.is_empty()
{
deps_for_remove.lock().await.mark_ran(&task);
if let Ok(outcome) = &result {
let mut deps = deps_for_remove.lock().await;
if outcome.did_work && !task.sources.is_empty() {
deps.mark_did_work(&task);
}
if let Some(cache_key) = &outcome.cache_key {
deps.mark_cache_key(&task, cache_key.clone());
}
}
if let Err(err) = &result {
let status = if panicked {
Expand Down Expand Up @@ -862,20 +866,20 @@ impl Run {
task: &Task,
config: &Arc<Config>,
sched_tx: Arc<tokio::sync::mpsc::UnboundedSender<(Task, Arc<Mutex<Deps>>)>>,
completed_tasks: std::collections::HashSet<crate::task::TaskKey>,
dep_ran: bool,
completion_state: crate::task::TaskCompletionState,
dependency_state: crate::task::TaskDependencyState,
semaphore: Arc<tokio::sync::Semaphore>,
permit: &mut Option<tokio::sync::OwnedSemaphorePermit>,
) -> Result<bool> {
) -> Result<crate::task::task_executor::TaskRunOutcome> {
self.executor
.as_ref()
.expect("executor must be initialized before running tasks")
.run_task_sched(
task,
config,
sched_tx,
completed_tasks,
dep_ran,
completion_state,
dependency_state,
semaphore,
permit,
)
Expand Down
76 changes: 57 additions & 19 deletions src/task/deps.rs
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,28 @@ impl fmt::Display for TaskCycleError {

impl std::error::Error for TaskCycleError {}

#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct TaskDependencyState {
pub cache_keys: Vec<String>,
pub any_did_work: bool,
pub any_unkeyed_did_work: bool,
}

#[derive(Debug, Clone, Default)]
pub struct TaskCompletionState {
completed: HashSet<TaskKey>,
did_work: HashSet<TaskKey>,
cache_keys: HashMap<TaskKey, String>,
}

#[derive(Debug)]
pub struct Deps {
pub graph: DiGraph<Task, ()>,
sent: HashSet<TaskKey>, // tasks that have already started so should not run again
removed: HashSet<TaskKey>, // tasks that have already finished to track if we are in an infinite loop
executed: HashSet<TaskKey>, // tasks that actually began executing (not just scheduled)
ran: HashSet<TaskKey>, // tasks that actually ran their commands (not skipped due to fresh sources)
did_work: HashSet<TaskKey>, // tasks that executed or restored outputs (not freshness-skipped)
cache_keys: HashMap<TaskKey, String>, // stable artifact identities published by completed tasks
dep_edges: HashMap<TaskKey, HashSet<TaskKey>>, // maps each task to its direct dependency task keys
post_dep_parents: HashMap<TaskKey, HashSet<TaskKey>>, // maps each post-dep to its parent tasks
tx: mpsc::UnboundedSender<Option<Task>>,
Expand Down Expand Up @@ -202,14 +217,16 @@ impl Deps {
let sent = HashSet::new();
let removed = HashSet::new();
let executed = HashSet::new();
let ran = HashSet::new();
let did_work = HashSet::new();
let cache_keys = HashMap::new();
Ok(Self {
graph,
tx,
sent,
removed,
executed,
ran,
did_work,
cache_keys,
dep_edges,
post_dep_parents,
})
Expand All @@ -221,13 +238,15 @@ impl Deps {
pub async fn new_pruned(
config: &Arc<Config>,
tasks: Vec<Task>,
completed: &HashSet<TaskKey>,
completed: &TaskCompletionState,
) -> eyre::Result<Self> {
let mut deps = Self::new(config, tasks).await?;
deps.did_work.extend(completed.did_work.iter().cloned());
deps.cache_keys.extend(completed.cache_keys.clone());
let mut to_remove = vec![];
for idx in deps.graph.node_indices() {
let key = task_key(&deps.graph[idx]);
if completed.contains(&key) {
if completed.completed.contains(&key) {
to_remove.push(idx);
}
}
Expand Down Expand Up @@ -284,12 +303,13 @@ impl Deps {
self.graph.node_count() == 0
}

/// Snapshot of task keys that have completed (removed from the graph).
/// Used by `new_pruned` so sub-graphs skip tasks the parent already ran.
/// Only includes confirmed-complete tasks, not in-flight ones, to
/// preserve dependency ordering in the sub-graph.
pub fn handled_task_keys(&self) -> HashSet<TaskKey> {
self.removed.clone()
/// Snapshot completed task state for nested task sub-graphs.
pub fn completion_state(&self) -> TaskCompletionState {
TaskCompletionState {
completed: self.removed.clone(),
did_work: self.did_work.clone(),
cache_keys: self.cache_keys.clone(),
}
}

/// Check if a post-dep task should actually run: it must be a post-dependency
Expand All @@ -310,18 +330,36 @@ impl Deps {
self.executed.insert(task_key(task));
}

/// Mark a task as having actually run its commands (not skipped due to fresh sources).
/// Mark a task as having executed or restored outputs.
/// Used to invalidate dependent tasks' source freshness checks.
pub fn mark_ran(&mut self, task: &Task) {
self.ran.insert(task_key(task));
pub fn mark_did_work(&mut self, task: &Task) {
self.did_work.insert(task_key(task));
}

/// Check if any direct dependency of the given task actually ran (not skipped).
pub fn any_dep_ran(&self, task: &Task) -> bool {
/// Record a stable artifact identity produced or reused by a completed task.
pub fn mark_cache_key(&mut self, task: &Task, cache_key: String) {
self.cache_keys.insert(task_key(task), cache_key);
}

/// Return the completed direct-dependency state needed for freshness and artifact caching.
pub fn dependency_state(&self, task: &Task) -> TaskDependencyState {
let key = task_key(task);
self.dep_edges
.get(&key)
.is_some_and(|deps| deps.iter().any(|dep_key| self.ran.contains(dep_key)))
let Some(deps) = self.dep_edges.get(&key) else {
return TaskDependencyState::default();
};
let mut cache_keys = deps
.iter()
.filter_map(|dep_key| self.cache_keys.get(dep_key).cloned())
.collect::<Vec<_>>();
cache_keys.sort();
cache_keys.dedup();
TaskDependencyState {
cache_keys,
any_did_work: deps.iter().any(|dep_key| self.did_work.contains(dep_key)),
any_unkeyed_did_work: deps.iter().any(|dep_key| {
self.did_work.contains(dep_key) && !self.cache_keys.contains_key(dep_key)
}),
}
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

/// Remove multiple tasks from the graph in a batch, emitting leaves only once at the end.
Expand Down
2 changes: 1 addition & 1 deletion src/task/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ use crate::file::display_path;
use crate::fuzzy::{FuzzyMatcher, FuzzyPattern};
use crate::toolset::{ToolRequest, ToolSource, ToolVersionOptions, Toolset};
use crate::ui::style;
pub use deps::{Deps, TaskCycleError, TaskKey};
pub use deps::{Deps, TaskCompletionState, TaskCycleError, TaskDependencyState, TaskKey};
use task_dep::TaskDep;
use task_sources::{RawOutputTemplates, TaskOutputs};

Expand Down
4 changes: 4 additions & 0 deletions src/task/task_cache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ struct CacheKeyMaterial<'a> {
outputs: Vec<String>,
root: PathBuf,
source_hash: String,
#[serde(skip_serializing_if = "Vec::is_empty")]
dependency_keys: Vec<String>,
environment: BTreeMap<String, Option<String>>,
vars: BTreeMap<String, String>,
tools: Vec<String>,
Expand Down Expand Up @@ -63,6 +65,7 @@ impl TaskArtifactCache {
toolset: &Toolset,
resolved_env: &BTreeMap<String, String>,
declared_env: &[(String, String)],
dependency_keys: &[String],
persist_content_hash_cache: bool,
) -> Result<Option<Self>> {
Settings::get().ensure_experimental("task artifact caching")?;
Expand Down Expand Up @@ -124,6 +127,7 @@ impl TaskArtifactCache {
outputs: task.outputs.patterns(),
root: inputs.root_identity,
source_hash: inputs.source_hash,
dependency_keys: dependency_keys.to_vec(),
environment,
vars,
tools,
Expand Down
Loading
Loading