Skip to content

fix(tasks): restore parallel starts with poetry via list_paths cache and stable exec-env cache#6237

Merged
jdx merged 1 commit intomainfrom
poetry-tasks
Sep 9, 2025
Merged

fix(tasks): restore parallel starts with poetry via list_paths cache and stable exec-env cache#6237
jdx merged 1 commit intomainfrom
poetry-tasks

Conversation

@jdx
Copy link
Owner

@jdx jdx commented Sep 8, 2025

Summary

Minimal fix for parallel task execution with poetry. Tasks now start concurrently again (~4ms spread vs ~6s).

Problem

With poetry installed, parallel tasks were serializing due to:

  1. Repeated list_paths computations for each task
  2. Cache invalidations from dirs::DATA freshness checks
  3. Lock contention in external_plugin_cache

Solution

  1. Added LIST_PATHS_CACHE to cache toolset list_paths results per project/toolset
  2. Use Arc in ExternalPluginCache to release locks earlier
  3. Removed dirs::DATA fresh check that caused spurious cache invalidations
  4. Added trace logs for debugging task startup timing

Testing

Updated test_task_parallel_execution to demonstrate the bug with poetry installed. Test now passes reliably with all tasks starting within milliseconds.

Verification

  • Before: Tasks start spread ~6000ms (sequential)
  • After: Tasks start spread ~4ms (parallel)
  • Total execution time: ~3s (vs ~8s+ when sequential)

@github-actions
Copy link

github-actions bot commented Sep 8, 2025

Hyperfine Performance

mise x -- echo

Command Mean [ms] Min [ms] Max [ms] Relative
mise-2025.9.6 x -- echo 18.3 ± 0.3 17.9 20.9 1.00
mise x -- echo 18.9 ± 0.2 18.5 19.7 1.03 ± 0.02

mise env

Command Mean [ms] Min [ms] Max [ms] Relative
mise-2025.9.6 env 17.8 ± 0.3 17.3 20.0 1.00
mise env 18.4 ± 0.4 17.8 21.8 1.03 ± 0.03

mise hook-env

Command Mean [ms] Min [ms] Max [ms] Relative
mise-2025.9.6 hook-env 17.4 ± 0.2 17.0 18.3 1.00
mise hook-env 18.0 ± 0.3 17.6 20.4 1.03 ± 0.02

mise ls

Command Mean [ms] Min [ms] Max [ms] Relative
mise-2025.9.6 ls 15.9 ± 0.2 15.5 16.9 1.00
mise ls 16.0 ± 0.2 15.5 17.5 1.01 ± 0.02

xtasks/test/perf

Command mise-2025.9.6 mise Variance
install (cached) 163ms ✅ 102ms +59%
ls (cached) 62ms 62ms +0%
bin-paths (cached) 65ms 68ms -4%
task-ls (cached) 465ms 463ms +0%

✅ Performance improvement: install cached is 59%

@jdx jdx changed the title fix: perf issue with poetry and tasks fix(tasks): restore parallel starts with poetry via list_paths cache and stable exec-env cache Sep 8, 2025
@jdx jdx marked this pull request as ready for review September 9, 2025 13:35
Copilot AI review requested due to automatic review settings September 9, 2025 13:35
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes parallel task execution with poetry by implementing a comprehensive caching strategy to prevent serialization of concurrent tasks. The issue was that tasks were recomputing environment and path information repeatedly, causing them to run sequentially instead of in parallel.

  • Introduces a global LIST_PATHS_CACHE to cache toolset path computations across identical configurations
  • Removes spurious cache invalidations from poetry exec-env cache by eliminating the dirs::DATA freshness check
  • Adds detailed tracing to identify performance bottlenecks in task execution

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/toolset/mod.rs Implements global LIST_PATHS_CACHE with stable cache keys and adds tracing for list_bin_paths operations
src/cli/run.rs Adds comprehensive timing traces for task execution phases including semaphore acquisition and environment rendering
src/backend/external_plugin_cache.rs Refactors cache managers to use Arc for thread safety and adds timing traces for cache operations
src/backend/asdf.rs Removes dirs::DATA freshness check and adds exec_env timing traces
e2e/tasks/test_task_parallel_execution Enhances test with poetry tooling and multiple runs for reliability verification

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.


// Cache Toolset::list_paths results across identical toolsets within a process.
// Keyed by project_root plus sorted list of backend@version pairs currently installed.
static LIST_PATHS_CACHE: Lazy<DashMap<String, Vec<PathBuf>>> = Lazy::new(DashMap::new);
Copy link

Copilot AI Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cache key construction could lead to collisions if project paths contain the delimiter '|'. Consider using a more robust serialization approach like JSON or a structured hash.

Copilot uses AI. Check for mistakes.
Comment on lines +15 to +16
list_bin_paths: RwLock<HashMap<ToolRequest, Arc<CacheManager<Vec<String>>>>>,
exec_env: RwLock<HashMap<ToolRequest, Arc<CacheManager<EnvMap>>>>,
Copy link

Copilot AI Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nested Arc<CacheManager> pattern creates unnecessary indirection. Consider using DashMap directly for better performance and simpler code structure.

Copilot uses AI. Check for mistakes.
cursor[bot]

This comment was marked as outdated.

- Add LIST_PATHS_CACHE to cache toolset list_paths results
- Use Arc<CacheManager> in ExternalPluginCache to release locks earlier
- Remove dirs::DATA fresh check that caused spurious cache invalidations
- Add trace logs for debugging task startup timing
- Update test to demonstrate the bug with poetry installed
@jdx jdx enabled auto-merge (squash) September 9, 2025 13:51
@jdx jdx merged commit 384e12c into main Sep 9, 2025
18 checks passed
@jdx jdx deleted the poetry-tasks branch September 9, 2025 14:03
@jdx jdx mentioned this pull request Sep 9, 2025
jdx added a commit that referenced this pull request Sep 9, 2025
### 🐛 Bug Fixes

- **(env)** allow mixed map for env._.file by @risu729 in
[#6148](#6148)
- **(tasks)** restore parallel starts with poetry via list_paths cache
and stable exec-env cache by @jdx in
[#6237](#6237)
- add 'unknown' to the list of OS patterns by @efussi in
[#6235](#6235)
- propagate errors from backend installs by @jdx in
[#6236](#6236)

### 📦️ Dependency Updates

- update taiki-e/install-action digest to 0c5db7f by @renovate[bot] in
[#6244](#6244)
- update golang docker tag to v1.25.1 by @renovate[bot] in
[#6247](#6247)
- update dependency vitepress to v1.6.4 by @renovate[bot] in
[#6246](#6246)

### New Contributors

- @efussi made their first contribution in
[#6235](#6235)

Co-authored-by: mise-en-dev <release@mise.jdx.dev>
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Sep 19, 2025
## [2025.9.12](https://github.com/jdx/mise/compare/v2025.9.11..v2025.9.12) - 2025-09-16

### 🐛 Bug Fixes

- **(ci)** properly exclude aqua-registry files from hk linting by @jdx in [42d7758](jdx/mise@42d7758)

### Chore

- **(release)** embed aqua-registry under crate and publish like vfox by @jdx in [#6306](jdx/mise#6306)
- ignore aqua-registry assets from prettier by @jdx in [047d77b](jdx/mise@047d77b)
- disable "useless cat" shellcheck by @jdx in [a6def59](jdx/mise@a6def59)

## [2025.9.11](https://github.com/jdx/mise/compare/v2025.9.10..v2025.9.11) - 2025-09-16

### 📦 Registry

- indicate aws-cli is v2 by @jayvdb in [#6300](jdx/mise#6300)

### 🚀 Features

- **(ci)** run all registry tools when workflow_dispatch is triggered by @jdx in [#6295](jdx/mise#6295)
- **(cli)** handle non-existent working directories gracefully by @jdx in [#6304](jdx/mise#6304)
- **(set)** add -E/--env flag to mise set command by @jdx in [#6291](jdx/mise#6291)
- **(tasks)** make auto outputs default by @risu729 in [#6137](jdx/mise#6137)

### 🐛 Bug Fixes

- align crate versions to fix release-plz script by @jdx in [5a464e9](jdx/mise@5a464e9)

### 🚜 Refactor

- **(aqua)** extract aqua registry into internal subcrate by @jdx in [#6293](jdx/mise#6293)

### 📚 Documentation

- fix mkdir paths by @risu729 in [#6298](jdx/mise#6298)
- fix rust profile default by @risu729 in [#6305](jdx/mise#6305)

### Chore

- **(aqua-registry)** remove unused aqua-registry files by @jdx in [#6294](jdx/mise#6294)
- **(release)** make release-plz idempotent for existing crate versions by @jdx in [dbdfd6a](jdx/mise@dbdfd6a)
- **(release)** skip publishing mise when aqua-registry is a path dep by @jdx in [47efffd](jdx/mise@47efffd)
- keep aqua-registry LICENSE file by @risu729 in [#6297](jdx/mise#6297)

### New Contributors

- @jayvdb made their first contribution in [#6300](jdx/mise#6300)

## [2025.9.10](https://github.com/jdx/mise/compare/v2025.9.9..v2025.9.10) - 2025-09-13

### 📦 Registry

- use asdf to install semver-tool by @jylenhof in [#6233](jdx/mise#6233)

### 🐛 Bug Fixes

- **(tool-stub)** detect binary names from single-file downloads by @jdx in [#6281](jdx/mise#6281)

### 🚜 Refactor

- allow any collection types in deserialize_arr by @risu729 in [#6282](jdx/mise#6282)
- use deserialize_arr for task runs by @risu729 in [#6253](jdx/mise#6253)

### 📚 Documentation

- **(getting-started)** add backends step with diagram, github example, env vars and simple tasks by @jdx in [#6288](jdx/mise#6288)
- simplify OS detection in tool plugin development by @jdx in [#6287](jdx/mise#6287)
- update backend plugin template references by @jdx in [942f5eb](jdx/mise@942f5eb)

### 📦️ Dependency Updates

- update rust crate chrono to v0.4.42 by @renovate[bot] in [#6274](jdx/mise#6274)
- update taiki-e/install-action digest to 0154864 by @renovate[bot] in [#6273](jdx/mise#6273)

### Chore

- **(schema)** fix schema for subtasks by @risu729 in [#6289](jdx/mise#6289)
- update render:schema task by @risu729 in [#6275](jdx/mise#6275)

## [2025.9.9](https://github.com/jdx/mise/compare/v2025.9.8..v2025.9.9) - 2025-09-11

### 🐛 Bug Fixes

- **(backend)** make HTTP installs atomic and serialize with cache lock by @jdx in [#6259](jdx/mise#6259)
- **(env)** allow nested env._.path directives by @risu729 in [#6160](jdx/mise#6160)
- **(env)** disallow nested env objects by @risu729 in [#6268](jdx/mise#6268)
- **(schema)** allow nested arrays in task.depends by @risu729 in [#6265](jdx/mise#6265)
- **(task)** resolve jobs=1 hang and keep-order panic; improve Ctrl-C handling by @jdx in [#6264](jdx/mise#6264)
- **(tasks)** stop CLI group after first failure without --continue-on-error by @jdx in [#6270](jdx/mise#6270)

### 📚 Documentation

- fixed toml issues in URL replacements settings documentation by @ThomasSteinbach in [#6269](jdx/mise#6269)

### Chore

- **(schema)** strict oneOf branches and DRY env_directive in schemas by @jdx in [#6271](jdx/mise#6271)
- add schema linter by @risu729 in [#6267](jdx/mise#6267)

## [2025.9.8](https://github.com/jdx/mise/compare/v2025.9.7..v2025.9.8) - 2025-09-10

### 🐛 Bug Fixes

- **(tasks)** prevent hang when task fails in sequence by @jdx in [#6260](jdx/mise#6260)
- **(version)** fetch mise version if cached version is older than the current by @risu729 in [#6252](jdx/mise#6252)

### 📦️ Dependency Updates

- update rhysd/action-setup-vim action to v1.4.3 by @renovate[bot] in [#6249](jdx/mise#6249)

## [2025.9.7](https://github.com/jdx/mise/compare/v2025.9.6..v2025.9.7) - 2025-09-09

### 🐛 Bug Fixes

- **(env)** allow mixed map for env._.file by @risu729 in [#6148](jdx/mise#6148)
- **(tasks)** restore parallel starts with poetry via list_paths cache and stable exec-env cache by @jdx in [#6237](jdx/mise#6237)
- add 'unknown' to the list of OS patterns by @efussi in [#6235](jdx/mise#6235)
- propagate errors from backend installs by @jdx in [#6236](jdx/mise#6236)

### 📦️ Dependency Updates

- update taiki-e/install-action digest to 0c5db7f by @renovate[bot] in [#6244](jdx/mise#6244)
- update golang docker tag to v1.25.1 by @renovate[bot] in [#6247](jdx/mise#6247)
- update dependency vitepress to v1.6.4 by @renovate[bot] in [#6246](jdx/mise#6246)

### New Contributors

- @efussi made their first contribution in [#6235](jdx/mise#6235)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants