Skip to content

fix: rayon -> tokio#5172

Merged
jdx merged 1 commit into
mainfrom
tokio
May 26, 2025
Merged

fix: rayon -> tokio#5172
jdx merged 1 commit into
mainfrom
tokio

Conversation

@jdx

@jdx jdx commented May 24, 2025

Copy link
Copy Markdown
Owner

The deadlock issue reported by many people but the cause seems related to using rayon for parallelism. This PR switches to using async rust with tokio which should resolve it.

This is obviously a massive PR so don't be surprised if there is a bug in some untested code path.

Fixes #5149
Fixes #5144
Fixes #4302
Fixes #4127

@github-actions

Copy link
Copy Markdown

If you are adding a new tool to mise, note the following:

  • New asdf plugins are not accepted
  • Tools may be rejected if they are not notable
  • A test is required in registry.toml

@jdx jdx force-pushed the tokio branch 2 times, most recently from 7412bc4 to c48a2d1 Compare May 25, 2025 00:57
@jdx jdx requested a review from Copilot May 25, 2025 01:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fully converts the CLI and backend implementations from synchronous to asynchronous execution under Tokio, replacing blocking patterns and Rayon. Key updates include:

  • Changing all run, cache, and backend methods to async fn and using async_trait for the Backend trait.
  • Introducing cache_async and a new get_or_try_init_async in CacheManager for asynchronous caching.
  • Refactoring Ubi installation logic into an async free function and updating many backends to use .await.

Reviewed Changes

Copilot reviewed 156 out of 156 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/cli/alias/*.rs Converted alias commands’ run to async fn and awaited config fetches
src/cache.rs Added cache_async: OnceCell and get_or_try_init_async
src/backend/mod.rs Annotated Backend trait with async_trait, made core methods async
src/backend/vfox.rs Converted VfoxBackend methods to async/await
src/backend/ubi.rs Extracted Ubi install into async helper (install), removed blocking runtime use
src/backend/external_plugin_cache.rs Made external plugin cache methods async
Cargo.toml Added async-trait and expanded Tokio features
Comments suppressed due to low confidence (3)

src/backend/vfox.rs:48

  • VfoxBackend does not implement Clone, so calling self.clone() will not compile. Consider capturing an Arc<Self> in the async context or deriving Clone for VfoxBackend and ensuring correct ownership.
let this = self.clone();

src/backend/ubi.rs:340

  • Calling block_on inside an async function can deadlock the Tokio runtime. Instead, await ubi.install_binary().await directly without using block_on.
rt.block_on(async {

src/backend/external_plugin_cache.rs:21

  • The signature references Future but std::future::Future is not imported. Add use std::future::Future; or fully qualify the type to avoid compilation errors.
pub async fn list_bin_paths<F, Fut>(

@jdx jdx force-pushed the tokio branch 13 times, most recently from 5cedad3 to 1de6385 Compare May 25, 2025 19:43
@github-actions

github-actions Bot commented May 25, 2025

Copy link
Copy Markdown

e2e/perf/test_many_tasks

  • NUM_TASKS: 2000
  • NUM_TOOLS: 200
Command Uncached mise Cached mise Uncached mise-2025.5.12 Cached mise-2025.5.12
install 2171ms 198ms 510ms 182ms
ls 700ms 80ms 374ms 56ms
bin-paths 841ms 76ms 384ms 59ms
task-ls 6090ms 320ms 10006ms 10006ms

@jdx jdx force-pushed the tokio branch 9 times, most recently from b46c28f to 89c6f40 Compare May 25, 2025 21:22
@jdx jdx marked this pull request as ready for review May 25, 2025 21:22
@jdx jdx force-pushed the tokio branch 11 times, most recently from 6730661 to e2d464b Compare May 26, 2025 00:07
@jdx jdx merged commit 9d0bb67 into main May 26, 2025
25 of 26 checks passed
@jdx jdx deleted the tokio branch May 26, 2025 01:15
jdx pushed a commit that referenced this pull request May 26, 2025
### 🐛 Bug Fixes

- output was silenced on task fail with keep-order by
[@artemisart](https://github.com/artemisart) in
[#5175](#5175)
- avoid mapfile to run e2e tests on macOS (bash 3.2) by
[@artemisart](https://github.com/artemisart) in
[#5170](#5170)
- flaky keep-order e2e test by
[@artemisart](https://github.com/artemisart) in
[#5178](#5178)
- watch mise.lock for changes by [@jdx](https://github.com/jdx) in
[#5184](#5184)
- remote task dependency does not work by
[@roele](https://github.com/roele) in
[#5183](#5183)
- rayon -> tokio by [@jdx](https://github.com/jdx) in
[#5172](#5172)
- cache results from version host by [@jdx](https://github.com/jdx) in
[#5187](#5187)
- cache results from version host for aqua packages by
[@jdx](https://github.com/jdx) in
[#5188](#5188)

### 📚 Documentation

- standardize subcommand format to 'u|use' for consistency by
[@LuckyWindsck](https://github.com/LuckyWindsck) in
[#5167](#5167)
- clarify how to enable ideomatic version file reading for ruby by
[@amkisko](https://github.com/amkisko) in
[#5163](#5163)

### 🧪 Testing

- added perf test by [@jdx](https://github.com/jdx) in
[#5179](#5179)
- skip benchmark errors for now by [@jdx](https://github.com/jdx) in
[#5186](#5186)

### Chore

- fix clippy issue in xonsh by [@jdx](https://github.com/jdx) in
[#5180](#5180)
- improve shfmt linter by [@jdx](https://github.com/jdx) in
[#5181](#5181)
- cargo up by [@jdx](https://github.com/jdx) in
[3ece604](3ece604)
- fix hyperfine step summary by [@jdx](https://github.com/jdx) in
[36ab4a1](36ab4a1)
- adjust perf thresholds by [@jdx](https://github.com/jdx) in
[4113a3b](4113a3b)

### New Contributors

- @amkisko made their first contribution in
[#5163](#5163)
- @LuckyWindsck made their first contribution in
[#5167](#5167)
stanhu added a commit to stanhu/mise that referenced this pull request Jun 4, 2025
With the switch from Rayon (parallel processing) to Tokio (async) in
jdx#5172, `mise exec` no longer correctly
resolves tool versions if `.tool-versions` contains an unknown entry.

This occurred because a single failure would cause a hard faliure in
the `resolve()` method in `src/toolset/mod.rs`. Fix this by separating
successful and unsuccessful results and only failing if nothing was
resolved.

Relates to jdx#5282
stanhu added a commit to stanhu/mise that referenced this pull request Jun 5, 2025
With the switch from Rayon (parallel processing) to Tokio (async) in
jdx#5172, `mise exec` no longer correctly
resolves tool versions if `.tool-versions` contains an unknown entry.

This occurred because a single failure would cause a hard faliure in
the `resolve()` method in `src/toolset/mod.rs`. Fix this by separating
successful and unsuccessful results and only failing if nothing was
resolved.

Relates to jdx#5282
stanhu added a commit to stanhu/mise that referenced this pull request Jun 5, 2025
With the switch from Rayon (parallel processing) to Tokio (async) in
jdx#5172, `mise exec` no longer correctly
resolves tool versions if `.tool-versions` contains an unknown entry.

This occurred because a single failure would cause a hard faliure in
the `resolve()` method in `src/toolset/mod.rs`. Fix this by separating
successful and unsuccessful results and only failing if nothing was
resolved.

Relates to jdx#5282
stanhu added a commit to stanhu/mise that referenced this pull request Jun 5, 2025
With the switch from Rayon (parallel processing) to Tokio (async) in
jdx#5172, `mise exec` no longer correctly
resolves tool versions if `.tool-versions` contains an unknown entry.

This occurred because a single failure would cause a hard faliure in
the `resolve()` method in `src/toolset/mod.rs`. Fix this by separating
successful and unsuccessful results and only failing if nothing was
resolved.

Relates to jdx#5282
stanhu added a commit to stanhu/mise that referenced this pull request Jun 5, 2025
With the switch from Rayon (parallel processing) to Tokio (async) in
jdx#5172, `mise exec` no longer correctly
resolves tool versions if `.tool-versions` contains an unknown entry.

This occurred because a single failure would cause a hard faliure in
the `resolve()` method in `src/toolset/mod.rs`. Fix this by separating
successful and unsuccessful results and only failing if nothing was
resolved.

Relates to jdx#5282
stanhu added a commit to stanhu/mise that referenced this pull request Jun 5, 2025
With the switch from Rayon (parallel processing) to Tokio (async) in
jdx#5172, `mise exec` no longer correctly
resolves tool versions if `.tool-versions` contains an unknown entry.

This occurred because a single failure would cause a hard faliure in
the `resolve()` method in `src/toolset/mod.rs`. Fix this by separating
successful and unsuccessful results and only failing if nothing was
resolved.

Relates to jdx#5282
stanhu added a commit to stanhu/mise that referenced this pull request Jun 5, 2025
With the switch from Rayon (parallel processing) to Tokio (async) in
jdx#5172, `mise exec` no longer correctly
resolves tool versions if `.tool-versions` contains an unknown entry.

This occurred because a single failure would cause a hard faliure in
the `resolve()` method in `src/toolset/mod.rs`. Fix this by separating
successful and unsuccessful results and only failing if nothing was
resolved.

Relates to jdx#5282
stanhu added a commit to stanhu/mise that referenced this pull request Jun 5, 2025
With the switch from Rayon (parallel processing) to Tokio (async) in
jdx#5172, `mise exec` no longer correctly
resolves tool versions if `.tool-versions` contains an unknown entry.

This occurred because a single failure would cause a hard faliure in
the `resolve()` method in `src/toolset/mod.rs`. Fix this by setting
successful tool resolutions while returning the errors, which
can be ignored by the caller.

Relates to jdx#5282
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Non-Reproducible Deadlock during mise run on GitHub Actions Sometimes mise just hangs when executing a task

2 participants