time::timeout ignores panic and waits forever

[stevenroose]

Version
List the versions of all tokio crates you are using. The easiest way to get
this information is using cargo tree subcommand:

│   ├── tokio v1.39.2
│   │   └── tokio-macros v2.4.0 (proc-macro)
│       │   │   │   ├── tokio v1.39.2 (*)
│       │   │   │   ├── tokio-util v0.7.10
│       │   │   │   │   ├── tokio v1.39.2 (*)
│       │   │   ├── tokio v1.39.2 (*)
│       │   │   ├── tokio v1.39.2 (*)
│       │   │   ├── tokio-util v0.7.10 (*)
│       │   ├── tokio v1.39.2 (*)
│       │   └── tokio-io-timeout v1.2.0
│       │       └── tokio v1.39.2 (*)
│       ├── tokio v1.39.2 (*)
│       ├── tokio-rustls v0.24.1
│       │   └── tokio v1.39.2 (*)
│       ├── tokio-stream v0.1.14
│       │   ├── tokio v1.39.2 (*)
│       │   └── tokio-util v0.7.10 (*)
├── tokio v1.39.2 (*)
├── tokio v1.39.2 (*)
├── tokio-stream v0.1.14 (*)
│   │   │   │   ├── tokio v1.39.2 (*)
│   │   │   │   └── tokio-native-tls v0.3.1
│   │   │   │       └── tokio v1.39.2 (*)
│   │   │   ├── tokio v1.39.2 (*)
│   │   │   ├── tokio-native-tls v0.3.1 (*)
│   │   │   ├── tokio-socks v0.5.1
│   │   │   │   └── tokio v1.39.2 (*)
├── tokio v1.39.2 (*)
├── tokio-stream v0.1.14 (*)

But I also tried after calling cargo update, so anything latest of time of this posting is also affected.

Platform

Linux hostname 6.6.47 #1-NixOS SMP PREEMPT_DYNAMIC Mon Aug 19 04:04:32 UTC 2024 x86_64 GNU/Linux

Description

I tried to create a small project to reproduce this, but it's too involved, I think tonic and an unavailable gRPC endpoint might be involved.

This is a summary:

• I am running cargo test with a #[tokio::test] test
• I am calling a tokio::time::timeout on a method that has the following relevant contents:
  • tries to connect to a tonic gRPC endpoint (presumably failing to) and unwraps the result
    • before, we used to check for .is_err() on this result and retrying, and that worked, so it's this unwrap inside the tokio::timeout that is actually causing the hang
  • supposedly tokio::time::sleeps after failure

It seems to me that maybe the tonic connect attempt hangs for a long time and tokio's timeout is not cutting that short.

Reproduce

In my own project, you can reproduce as follows, sorry it's pretty annoying:

• install bitcoind (bitcoin-core), sorry it's a dep and it's a bit annoying to remove it (package managers have it nowadays)
• clone our repo https://codeberg.org/ark-bitcoin/bark.git
• checkout branch tokio-hang or commit b94a71be3a70e167bfd38a258322e29141666814
• then either run just test-integration fund_ or do the following:
  • run cargo build --workspace
  • set the ASPD_EXEC env var to the built aspd binary, depending on your cargo target dir, it's probably $(pwd)/target/debug/aspd
  • run cargo test -p ark-testing fund_

[Darksonn]

I can't help you without a smaller example. The timeout function definitely doesn't ignore or hang on panic. Something else must be going on. This issue doesn't sound like a Tokio bug from what you have described so far.