Console may show "ghost" tasks when the event buffer is at capacity

[asonix]

Version

tokio v1.15.0
├── console-subscriber v0.1.0
│   └── current v0.1.0 (/home/asonix/Development/rust/current)
├── current v0.1.0 (/home/asonix/Development/rust/current)
├── h2 v0.3.9
│   ├── hyper v0.14.16
│   │   ├── hyper-timeout v0.4.1
│   │   │   └── tonic v0.6.2
│   │   │       ├── console-api v0.1.0
│   │   │       │   └── console-subscriber v0.1.0 (*)
│   │   │       └── console-subscriber v0.1.0 (*)
│   │   └── tonic v0.6.2 (*)
│   └── tonic v0.6.2 (*)
├── hyper v0.14.16 (*)
├── hyper-timeout v0.4.1 (*)
├── tokio-io-timeout v1.1.1
│   └── hyper-timeout v0.4.1 (*)
├── tokio-stream v0.1.8
│   ├── console-subscriber v0.1.0 (*)
│   ├── tonic v0.6.2 (*)
│   └── tower v0.4.11
│       └── tonic v0.6.2 (*)
├── tokio-util v0.6.9
│   ├── h2 v0.3.9 (*)
│   ├── tonic v0.6.2 (*)
│   └── tower v0.4.11 (*)
├── tonic v0.6.2 (*)
└── tower v0.4.11 (*)

Platform
Linux graystripe 5.13.0-22-generic tokio-rs/tokio#22~20.04.1-Ubuntu SMP Tue Nov 9 15:07:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Description
When observing tokio with tokio-console, it seems sometimes tasks are never reaped. I noticed this first in an actix-web application, and I've created a minimal reproduction with a normal tokio multi-threaded environment

I tried this code:

[package]
name = "current"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
console-subscriber = "0.1"
tokio = { version = "1.15", features = ["full", "tracing"] }

#[tokio::main]
async fn main() -> std::io::Result<()> {
    console_subscriber::init();

    println!("Hello");
    tokio::signal::ctrl_c().await?;

    for _ in 0..10000 {
        tokio::task::spawn_blocking(move || {
            println!("Spawned");
        })
        .await?;
    }

    tokio::signal::ctrl_c().await
}

Here's a screenshot of my console session:

It seems like this blocking task is marked as "running" and it does not go away, no matter how long I wait. I wonder if this is an issue with tokio's reporting, or if a blocking task truly has not been reaped.

Here's another screenshot:

In this one, we have a number of blocking tasks that have not been reaped, two of which seem to never have been polled.

[hawkw]

It's possible that this is just a console issue. What happens if you dramatically increase the value of the console-subscriber layer's maximum event buffer capacity? Do you still see apparently-dangling tasks?

[asonix]

It does seem like increasing the event buffer capacity is the solution here, thanks!

[hawkw]

This should probably be a bug against the tokio-rs/console repository, rather than Tokio. I think what's happening here is that a bunch of tasks are being spawned, and they are eventually completing, but the event that records their completion is dropped because the console's event buffer is full. In that case, we never notice that the task has completed.

We might want to implement something where the console's aggregator task actively checks if task spans still exist, rather than just relying on a completion event. If we check for task existence and find that some task span is no longer active, but we missed the notification about its completion, we won't necessarily be able to record the correct time of completion, but we can at least remove it from the list of currently active tasks, so that we no longer show "ghost tasks" that should have completed.