Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Task dumps can deadlock. #6035

Closed
jswrenn opened this issue Sep 26, 2023 · 1 comment · Fixed by #6036
Closed

Task dumps can deadlock. #6035

jswrenn opened this issue Sep 26, 2023 · 1 comment · Fixed by #6036
Labels
A-tokio Area: The main tokio crate C-bug Category: This is a bug. M-taskdump --cfg tokio_taskdump

Comments

@jswrenn
Copy link
Contributor

jswrenn commented Sep 26, 2023

Requesting a task dump can deadlock. When a task dump is requested, OwnedTasks is locked and iterated over, which each task being polled.

Tasks are polled in 'tracing mode', causing most tokio leaf futures to produce Poll::Pending, even if they are otherwise ready. However, if a non-tokio or non-instrumented leaf future is used and completes during tracing, the system will deadlock: completed futures are removed from OwnedTasks and doing so requires locking OwnedTasks (which can't happen, because it's already locked for tracing).
@jswrenn jswrenn added C-bug Category: This is a bug. A-tokio Area: The main tokio crate labels Sep 26, 2023
@Darksonn Darksonn added the M-taskdump --cfg tokio_taskdump label Sep 26, 2023
@Darksonn
Copy link
Contributor

Yeah, we definitely shouldn't hold the lock while polling stuff. Making a vector of tasks seems like a better solution.

jswrenn added a commit to jswrenn/tokio that referenced this issue Sep 27, 2023
@jswrenn
rt: do not trace tasks while locking OwnedTasks
3e25f2c 
If a polled tasks completes while `OwnedTasks` is locked, it will
not be able to remove itself from `OwnedTasks`, resulting in a
deadlock.

Fixes tokio-rs#6035.
jswrenn added a commit to jswrenn/tokio that referenced this issue Sep 27, 2023
@jswrenn
rt: do not trace tasks while locking OwnedTasks
a8577e8 
If a polled tasks completes while `OwnedTasks` is locked, it will
not be able to remove itself from `OwnedTasks`, resulting in a
deadlock.

Fixes tokio-rs#6035.
@jswrenn jswrenn mentioned this issue Sep 27, 2023
Merged
jswrenn added a commit to jswrenn/tokio that referenced this issue Oct 2, 2023
@jswrenn
rt: do not trace tasks while locking OwnedTasks
3912db9 
If a polled tasks completes while `OwnedTasks` is locked, it will
not be able to remove itself from `OwnedTasks`, resulting in a
deadlock.

Fixes tokio-rs#6035.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-tokio Area: The main tokio crate C-bug Category: This is a bug. M-taskdump --cfg tokio_taskdump
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

rt: do not trace tasks while locking OwnedTasks jswrenn/tokio
2 participants
@Darksonn @jswrenn