Skip to content

perf: Release tokio runtime at exit so background DNS lookups never stall the user - #13231

Merged
anthonyshew merged 3 commits into
mainfrom
shew/telemetry-exit-flush
Jul 4, 2026
Merged

anthonyshew merged 3 commits into
mainfrom
shew/telemetry-exit-flush

Conversation

@anthonyshew

@anthonyshew anthonyshew commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Why

On slow or lossy networks, turbo can hang for ~4-5 seconds after all work is done and the last output is written, on every invocation. We hit this repeatedly while benchmarking on a cloud sandbox: roughly half of all runs in a bad-network window took +4.4s, with -vv timestamps showing the process idle between the final log line and exit.

The cause is not a missing timeout in telemetry itself — the telemetry handle close is bounded to 1s and posts run under a 10s timeout. The problem is Runtime::drop: it joins blocking-pool threads with no deadline. Telemetry posts are detached best-effort tasks, and their DNS resolution (getaddrinfo, via reqwest's default resolver) runs on the blocking pool and cannot be cancelled. When block_on(run_main) returns with a lookup still in flight, the future-level timeouts are dropped with their tasks — but drop still sits on the uncancellable getaddrinfo until the resolver gives up.

What

cli::run now releases the runtime with shutdown_background() instead of inheriting drop's unbounded join — the process never waits on background work at exit, not even a grace period. Everything owed to the user is awaited inside run_main (telemetry and analytics handles get a bounded close there); anything still running at teardown is strictly best-effort by construction, and a grace period is not a correctness mechanism — the old teardown already cancelled pending async tasks, so waiting bought nothing but latency. Orphaned blocking threads die with the process.

How to verify

  • Mechanism isolated in a 20-line reproducer: a runtime holding one stuck spawn_blocking task takes 4.90s to drop, versus 123µs with shutdown_background().
  • Field validation under real bad-network conditions (interleaved pairs on the same cloud sandbox during degraded windows, telemetry enabled), reproduced independently for both revisions of this fix: main stalled at ~6.4s on 3 of 8 runs each time; this fix had 0 of 8 above normal run time, with normal runs unaffected.
  • 431 turborepo-lib tests pass; clippy and fmt clean.

@anthonyshew
anthonyshew requested a review from a team as a code owner July 3, 2026 19:32
@anthonyshew
anthonyshew requested review from tknickman and removed request for a team July 3, 2026 19:32
@vercel

vercel Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
examples-basic-web Ready Ready Preview, Comment, Open in v0 Jul 4, 2026 2:32am
examples-designsystem-docs Ready Ready Preview, Comment, Open in v0 Jul 4, 2026 2:32am
examples-gatsby-web Ready Ready Preview, Comment, Open in v0 Jul 4, 2026 2:32am
examples-kitchensink-blog Ready Ready Preview, Comment, Open in v0 Jul 4, 2026 2:32am
examples-nonmonorepo Ready Ready Preview, Comment, Open in v0 Jul 4, 2026 2:32am
examples-svelte-web Ready Ready Preview, Comment, Open in v0 Jul 4, 2026 2:32am
examples-tailwind-web Ready Ready Preview, Comment, Open in v0 Jul 4, 2026 2:32am
examples-vite-web Ready Ready Preview, Comment, Open in v0 Jul 4, 2026 2:32am
turbo-site Building Building Preview, Comment, Open in v0 Jul 4, 2026 2:32am
turborepo-eve-agent Ready Ready Preview, Comment, Open in v0 Jul 4, 2026 2:32am

@anthonyshew anthonyshew changed the title fix: Bound tokio runtime teardown so background DNS lookups cannot stall exit fix: Release tokio runtime at exit so background DNS lookups never stall the user Jul 3, 2026
@anthonyshew anthonyshew changed the title fix: Release tokio runtime at exit so background DNS lookups never stall the user perf: Release tokio runtime at exit so background DNS lookups never stall the user Jul 3, 2026
@anthonyshew
anthonyshew merged commit ae635bd into main Jul 4, 2026
42 checks passed
@anthonyshew
anthonyshew deleted the shew/telemetry-exit-flush branch July 4, 2026 20:20
anthonyshew pushed a commit that referenced this pull request Jul 4, 2026
## Release v2.10.4-canary.1

> [!CAUTION]
> Versioned docs aliasing FAILED. [View
logs](https://github.com/vercel/turborepo/actions/runs/28719728846)

### Changes

- release(turborepo): 2.10.3-canary.8 (#13200) (`5987ed1`)
- fix(watch): recover from slow initial hash instead of timing out
(#13159) (`05e26cb`)
- chore: Update vitest example to vitest 4 and use native test merging
(#13202) (`e2b61dc`)
- feat: Toggle between the TUI and streamed logs (#13203) (`1325455`)
- fix: Resolve Bun scoped package deps that share a name with a
dependency (#13207) (`e4773ab`)
- feat: Select tasks by clicking rows in the TUI task list (#13206)
(`c568f67`)
- perf: Avoid deep-cloning dependency maps in lockfile closure cache
(#13209) (`abd84cf`)
- perf: Cache root internal dependencies in package graph (#13211)
(`7ec8c6d`)
- feat: Automatically copy TUI selection to clipboard on mouse release
(#13208) (`61d6013`)
- perf: Cache env wildcard matches across tasks during hashing (#13210)
(`148b1dd`)
- perf: Replace per-package graph traversals in scope filtering (#13212)
(`28d1871`)
- perf: Derive dirty hash from repo index (#13213) (`b521d32`)
- fix: Include untracked symlinks in repo-index dirty hash (#13218)
(`c3c91ab`)
- refactor: Use upstream libghostty-vt crates instead of vendored
bindings (#13205) (`c2115dc`)
- fix: Remove devtools feature flag (#13219) (`65efe27`)
- fix: Harden TUI terminal restore during shutdown (#13220) (`dccab93`)
- fix: Correct gitignore precedence in untracked walk and memoize
matcher chains (#13221) (`bc32fcc`)
- release(turborepo): 2.10.3 (#13222) (`02f4ed0`)
- ci: Retry Windows nextest aborts from transient 0xc0000142 spawn
failures (#13240) (`222ffff`)
- perf: Use mimalloc as the global allocator (#13237) (`c42a8a9`)
- perf: Parse yarn v1 lockfiles in a single pass (#13241) (`6207099`)
- ci: Fix cache outputs for Eve app (#13243) (`d8b556d`)
- perf: Stop materializing spans for the disabled daemon log layer
(#13244) (`623aadc`)
- fix: Keep ancestor-scoped bun dependencies resolvable after prune
renames (#13236) (`8658b66`)
- perf: Overlap external dependency hashing with package file hashing
(#13234) (`445b0c7`)
- perf: Build tracked repo index concurrently with package graph
(#13232) (`f4699c6`)
- perf: Release tokio runtime at exit so background DNS lookups never
stall the user (#13231) (`ae635bd`)
- perf: Overhaul pnpm lockfile parsing and dependency closure
computation (#13228) (`669676d`)
- feat: Add futureFlags.experimentalCargoWorkspaces flag (no-op)
(#13227) (`f546930`)
- feat: Introduce toolchain provider abstraction (#13235) (`f03addb`)
- perf: Parse Berry lockfiles in a single pass (#13242) (`6d36125`)
- fix: Apply input exclusion globs to the filesystem walk (#13224)
(`d3f9dd3`)
- fix: Stop root-directory packages from claiming every file in change
mapping (#13225) (`f80203a`)
- chore: Rename turborepo-repository napi package to @turbo/repository
(#13226) (`c1972b3`)
- perf: Hash git blobs with hardware-accelerated SHA-1 (#13245)
(`17ffa0c`)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
anthonyshew pushed a commit that referenced this pull request Jul 6, 2026
## Release v2.10.4

> [!CAUTION]
> Versioned docs aliasing FAILED. [View
logs](https://github.com/vercel/turborepo/actions/runs/28819266360)

### Changes

- release(turborepo): 2.10.3 (#13222) (`02f4ed0`)
- ci: Retry Windows nextest aborts from transient 0xc0000142 spawn
failures (#13240) (`222ffff`)
- perf: Use mimalloc as the global allocator (#13237) (`c42a8a9`)
- perf: Parse yarn v1 lockfiles in a single pass (#13241) (`6207099`)
- ci: Fix cache outputs for Eve app (#13243) (`d8b556d`)
- perf: Stop materializing spans for the disabled daemon log layer
(#13244) (`623aadc`)
- fix: Keep ancestor-scoped bun dependencies resolvable after prune
renames (#13236) (`8658b66`)
- perf: Overlap external dependency hashing with package file hashing
(#13234) (`445b0c7`)
- perf: Build tracked repo index concurrently with package graph
(#13232) (`f4699c6`)
- perf: Release tokio runtime at exit so background DNS lookups never
stall the user (#13231) (`ae635bd`)
- perf: Overhaul pnpm lockfile parsing and dependency closure
computation (#13228) (`669676d`)
- feat: Add futureFlags.experimentalCargoWorkspaces flag (no-op)
(#13227) (`f546930`)
- feat: Introduce toolchain provider abstraction (#13235) (`f03addb`)
- perf: Parse Berry lockfiles in a single pass (#13242) (`6d36125`)
- fix: Apply input exclusion globs to the filesystem walk (#13224)
(`d3f9dd3`)
- fix: Stop root-directory packages from claiming every file in change
mapping (#13225) (`f80203a`)
- chore: Rename turborepo-repository napi package to @turbo/repository
(#13226) (`c1972b3`)
- perf: Hash git blobs with hardware-accelerated SHA-1 (#13245)
(`17ffa0c`)
- release(turborepo): 2.10.4-canary.1 (#13247) (`a31389e`)
- feat: Discover Cargo crates as packages (#13248) (`d813d12`)
- perf: Reuse per-package external dependency hashes in run summaries
(#13249) (`ef7ef06`)
- perf: Stat workspace turbo.json files concurrently during discovery
(#13251) (`57be8e5`)
- chore: Add tracing spans to workspace discovery and globwalk phases
(#13252) (`bd76946`)
- fix: Avoid non-reentrant libc calls in concurrent shutdown process
scans (#13256) (`bf2d827`)
- perf: Memoize resolved task definitions during engine construction
(#13257) (`9085289`)
- perf: Compute transitive closures and external dependency hashes
concurrently with run setup (#13250) (`b305fe2`)
- perf: Probe microfrontends configs in parallel (#13262) (`ea7c1f8`)
- feat: Execute Cargo crate tasks via cargo (#13261) (`ee81349`)
- perf: Pre-size engine task collections (#13265) (`17bc22e`)
- feat: Expose resolved experimentalCI task configuration in `turbo
query` (#13264) (`0f564e5`)
- feat: Derive input and output globs for Cargo tasks (#13263)
(`05b0e74`)
- perf: Parse large pnpm lockfile sections in parallel (#13266)
(`ac4155e`)
- feat: Hash Cargo external dependencies per-crate (#13267) (`71300aa`)
- ci: Install pnpm 10 in musl containers for Library Release (#13269)
(`5ba9f66`)
- ci: Force pnpm overwrite in Library Release musl containers (#13270)
(`560ee2f`)
- perf: Defer the untracked-scan barrier to first file-hash use (#13268)
(`96b0f5a`)
- ci: Fetch API-created commit before updating local ref (#13271)
(`4cbaf48`)
- release(library): 0.0.1-canary.22 (#13272) (`d2e0691`)
- perf: Remove lock and dispatch overhead from task hash precomputation
(#13273) (`dd6de07`)
- fix: Fall back to lockfile detection in @turbo/repository when package
manager is undeclared (#13275) (`8cf0e75`)
- test: Add end-to-end coverage for Cargo workspaces (#13274)
(`ce18f0a`)
- release(library): 0.0.1-canary.23 (#13276) (`8e3a59f`)
- release(turborepo): 2.10.4-canary.2 (#13278) (`7e02f94`)
- fix: Collapse nested package-manager fallback conditional (#13279)
(`af01fdf`)
- feat: Make turbo watch Cargo-aware (#13280) (`39d623e`)
- perf: Skip dependency-closure assembly for toolchains that derive
nothing (#13277) (`ff0d508`)
- feat: Make turbo prune Cargo-aware (#13281) (`ddc584d`)
- fix: Raise the open-file soft limit at startup (#13282) (`947b478`)
- fix: Stop flagging relative imports that resolve into node_modules in
boundaries (#13284) (`11a68c7`)
- perf: Evaluate simple include globs without wax compilation (#13285)
(`189897a`)
- chore: Dogfood native Cargo support in this repository (#13283)
(`42f067b`)

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

This branch was successfully deployed

1 active deployment
Preview – turborepo-eve-agent b2dd261b Deployed Jul 4, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants