Optimize dependency file search by Kobzol · Pull Request #153131 · rust-lang/rust

Kobzol · 2026-02-26T12:41:29Z

I tried to look into the slowdown reported in rust-lang/cargo#16665.

I created a Rust hello world program, and used this Python script to create a directory containing 200k files:

from pathlib import Path

dir = Path("deps")
dir.mkdir(parents=True, exist_ok=True)
for i in range(200000):
    path = dir / f"file{i:07}.o"
    with open(path, "w") as f:
        f.write("\n")

Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.

Here are the results on main vs the last commit with the stage1 compiler on Linux, using hyperfine "rustc +stage1 src/main.rs -L deps" -r 30 (there's IO involved, so it's good to let it run for a while):

Benchmark 1: rustc +stage1 src/main.rs -L deps
  Time (mean ± σ):     299.4 ms ±   2.7 ms    [User: 161.9 ms, System: 144.9 ms]
  Range (min … max):   294.8 ms … 307.1 ms    30 runs

Benchmark 1: rustc +stage1 src/main.rs -L deps
  Time (mean ± σ):     208.1 ms ±   4.5 ms    [User: 87.3 ms, System: 128.7 ms]
  Range (min … max):   202.4 ms … 219.6 ms    30 runs

Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).

I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with lib would get us down to ~150ms here. (The baseline without -L is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.

r? @nnethercote

Kobzol · 2026-02-26T12:45:37Z

Not expecting any big wins there, but let's check:

@bors try @rust-timer queue

Optimize dependency file search

compiler/rustc_session/src/search_paths.rs

rust-bors · 2026-02-26T14:56:18Z

☀️ Try build successful (CI)
Build commit: 71d4456 (71d445654a32204fbf86bfec11e657fb97adcd22, parent: 69b78537fac74de40f009b076bcbbf54b77683ad)

ehuss · 2026-02-26T15:06:30Z

Unfortunately I am unable to test because of #153077, I am unable to get rustc to build.

rust-timer · 2026-02-26T15:36:30Z

Finished benchmarking commit (71d4456): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.3%]	4
Regressions ❌ (secondary)	0.0%	[0.0%, 0.0%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.4%	[-0.6%, -0.1%]	7
All ❌✅ (primary)	0.3%	[0.2%, 0.3%]	4

Max RSS (memory usage)

Results (secondary 0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	6.8%	[6.8%, 6.8%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-6.7%	[-6.7%, -6.7%]	1
All ❌✅ (primary)	-	-	0

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 480.132s -> 480.258s (0.03%)
Artifact size: 395.80 MiB -> 395.77 MiB (-0.01%)

ehuss · 2026-02-26T19:49:56Z

OK, got some data:

With this PR:

Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
  Time (mean ± σ):     197.4 ms ±   3.9 ms    [User: 143.7 ms, System: 117.9 ms]
  Range (min … max):   192.4 ms … 207.7 ms    30 runs

Without:

Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
  Time (mean ± σ):     248.9 ms ±   5.1 ms    [User: 189.5 ms, System: 124.2 ms]
  Range (min … max):   241.1 ms … 262.6 ms    30 runs

The tests are super noisy, though (I get significantly different results between runs). But it seems like a pretty good win.

nnethercote · 2026-02-27T10:00:22Z

Nice wins.

@bors r+

rust-bors · 2026-02-27T10:00:24Z

📌 Commit f450289 has been approved by nnethercote

It is now in the queue for this repository.

bjorn3 · 2026-02-27T10:13:18Z

compiler/rustc_session/src/search_paths.rs

            Err(..) => Default::default(),
        };
-        files.sort_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
+        files.sort_unstable_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));


Is there no possibility of two files with the same file_name_str?

Although I guess that would result in either one being picked deterministically (if it is the only matching one), which one is picked not mattering (if both files having the same crate hash) or an error about multiple candidates getting emitted.

We sorted (and searched) by the UTF-8 name before anyway, so there shouldn't hopefully be any change in behavior.

@ehuss

Optimize dependency file search I tried to look into the slowdown reported in rust-lang/cargo#16665. I created a Rust hello world program, and used this Python script to create a directory containing 200k files: ```python from pathlib import Path dir = Path("deps") dir.mkdir(parents=True, exist_ok=True) for i in range(200000): path = dir / f"file{i:07}.o" with open(path, "w") as f: f.write("\n") ``` Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect. Here are the results on `main` vs the last commit with the stage1 compiler on Linux, using `hyperfine "rustc +stage1 src/main.rs -L deps" -r 30` (there's IO involved, so it's good to let it run for a while): ```bash Benchmark 1: rustc +stage1 src/main.rs -L deps Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms] Range (min … max): 294.8 ms … 307.1 ms 30 runs Benchmark 1: rustc +stage1 src/main.rs -L deps Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms] Range (min … max): 202.4 ms … 219.6 ms 30 runs ``` Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ). I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with `lib` would get us down to ~150ms here. (The baseline without `-L` is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory. r? @nnethercote

rust-bors · 2026-03-01T08:41:35Z

💔 Test for 07af97d failed: CI. Failed job:

auto - aarch64-gnu (web logs, enhanced plaintext logs)

rust-log-analyzer · 2026-03-01T08:44:38Z

The job aarch64-gnu failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

/dev/sda15       98M  6.4M   92M   7% /boot/efi
tmpfs           1.6G   16K  1.6G   1% /run/user/1001
================================================================================

Sufficient disk space available (120307208KB >= 52428800KB). Skipping cleanup.
##[group]Run src/ci/scripts/setup-environment.sh
src/ci/scripts/setup-environment.sh
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
[1773/3894] Building ARMGenMCCodeEmitter.inc...
##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
[1774/3894] Building ARMGenMCPseudoLowering.inc...

Session terminated, killing shell...::group::Clock drift check
  local time: Sun Mar  1 08:40:58 UTC 2026
  network time: [1775/3894] Building ARMGenInstrInfo.inc...
Sun, 01 Mar 2026 08:40:58 GMT
##[endgroup]
[1776/3894] Building ARMGenRegisterBank.inc...

Kobzol · 2026-03-01T08:45:14Z

@bors retry

Spurious worker restart.

rust-bors · 2026-03-02T02:40:54Z

☀️ Test successful - CI
Approved by: nnethercote
Duration: 3h 40m 3s
Pushing e7d90c6 to main...

github-actions · 2026-03-02T02:44:10Z

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 8038127 (parent) -> e7d90c6 (this PR)

Test differences

Show 2 test diffs

2 doctest diffs were found. These are ignored, as they are noisy.

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard e7d90c695a39426baf5ae705de2f9570a72229de --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

dist-aarch64-llvm-mingw: 1h 33m -> 1h 46m (+14.1%)
dist-aarch64-apple: 1h 56m -> 2h 12m (+13.1%)
aarch64-apple: 3h 8m -> 3h 33m (+13.0%)
dist-x86_64-apple: 2h 42m -> 2h 25m (-10.7%)
arm-android: 1h 37m -> 1h 47m (+10.4%)
dist-ohos-x86_64: 1h 14m -> 1h 21m (+9.7%)
x86_64-gnu-aux: 2h 7m -> 2h 18m (+8.5%)
x86_64-gnu-distcheck: 2h 7m -> 2h 18m (+8.4%)
x86_64-gnu-miri: 1h 23m -> 1h 30m (+7.7%)
dist-arm-linux-musl: 1h 38m -> 1h 30m (-7.3%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

rust-timer · 2026-03-02T03:21:47Z

Finished benchmarking commit (e7d90c6): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Our benchmarks found a performance regression caused by this PR.
This might be an actual regression, but it can also be just noise.

Next Steps:

If the regression was expected or you think it can be justified,
please write a comment with sufficient written justification, and add
@rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
If you think that you know of a way to resolve the regression, try to create
a new PR with a fix for the regression.
If you do not understand the regression or you think that it is just noise,
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
were already notified of this PR).

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.1%	[0.1%, 0.1%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.4%	[-0.6%, -0.2%]	7
All ❌✅ (primary)	0.1%	[0.1%, 0.1%]	1

Max RSS (memory usage)

Results (secondary -4.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.0%	[-5.4%, -2.6%]	3
All ❌✅ (primary)	-	-	0

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

Results (secondary 0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.1%	[0.1%, 0.1%]	3
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Bootstrap: 480.971s -> 480.149s (-0.17%)
Artifact size: 396.96 MiB -> 396.96 MiB (0.00%)

Kobzol · 2026-03-02T06:54:34Z

The regression seems like noise. Tiny wins on large-workspace, matching pre-merge run.

@rustbot label: +perf-regression-triaged

Kobzol added 4 commits February 26, 2026 13:18

Remove unnecessary Arc from FilesIndex

7dbdb29

Use unstable sort for sorting paths in a search index

516bd8c

Do not store copy of the full path for each SearchPathFile

1c3449d

Do not store directory at all in SearchPathFile

f450289

rustbot assigned nnethercote Feb 26, 2026

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Feb 26, 2026

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 26, 2026

rust-bors bot pushed a commit that referenced this pull request Feb 26, 2026

Auto merge of #153131 - Kobzol:filesearch-opt, r=<try>

71d4456

Optimize dependency file search

petrochenkov reviewed Feb 26, 2026

View reviewed changes

compiler/rustc_session/src/search_paths.rs Show resolved Hide resolved

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Feb 26, 2026

nnethercote approved these changes Feb 27, 2026

View reviewed changes

rust-bors bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 27, 2026

bjorn3 reviewed Feb 27, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

rust-bors bot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 1, 2026

rust-bors bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 1, 2026

This was referenced Mar 1, 2026

Show recent PRs when a user is banned rust-lang/triagebot#2313

Merged

Rename the comments command to user-info and extend it rust-lang/triagebot#2317

Merged

This comment has been minimized.

Sign in to view

rust-bors bot added merged-by-bors This PR was explicitly merged by bors. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 2, 2026

rust-bors bot merged commit e7d90c6 into rust-lang:main Mar 2, 2026
13 checks passed

rustbot added this to the 1.96.0 milestone Mar 2, 2026

rustbot added the perf-regression-triaged The performance regression has been triaged. label Mar 2, 2026

Uh oh!

Conversation

Kobzol commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kobzol commented Feb 26, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

rust-bors bot commented Feb 26, 2026

Uh oh!

This comment has been minimized.

ehuss commented Feb 26, 2026

Uh oh!

rust-timer commented Feb 26, 2026

Overall result: ❌✅ regressions and improvements - please read the text below

Uh oh!

ehuss commented Feb 26, 2026

Uh oh!

nnethercote commented Feb 27, 2026

Uh oh!

rust-bors bot commented Feb 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

rust-bors bot commented Mar 1, 2026

Uh oh!

rust-log-analyzer commented Mar 1, 2026

Uh oh!

Kobzol commented Mar 1, 2026

Uh oh!

This comment has been minimized.

rust-bors bot commented Mar 2, 2026

Uh oh!

Uh oh!

github-actions bot commented Mar 2, 2026

Test differences

Job duration changes

Uh oh!

rust-timer commented Mar 2, 2026

Overall result: ❌✅ regressions and improvements - please read the text below

Uh oh!

Kobzol commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Kobzol commented Feb 26, 2026 •

edited

Loading

Kobzol commented Mar 2, 2026 •

edited

Loading