Split Components iterator to prefixed and non-prefixed versions and optimize Components for non-prefix path based platforms - #156496
Open
asder8215 wants to merge 13 commits into
Open
Conversation
rustbot
added
S-waiting-on-review
Collaborator
|
rustbot has assigned @Mark-Simulacrum. Use Why was this reviewer chosen?The reviewer was selected based on:
|
This comment has been minimized.
This comment has been minimized.
asder8215
force-pushed
the
components_rewrite
branch
from
May 12, 2026 09:09
1627e2f to
33e69e1
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
asder8215
force-pushed
the
components_rewrite
branch
from
May 12, 2026 17:05
33e69e1 to
ed9d33d
Compare
This comment has been minimized.
This comment has been minimized.
asder8215
force-pushed
the
components_rewrite
branch
from
May 12, 2026 17:19
ed9d33d to
0b0f84c
Compare
This comment has been minimized.
This comment has been minimized.
asder8215
force-pushed
the
components_rewrite
branch
from
May 12, 2026 22:05
0b0f84c to
8ed33ea
Compare
This comment was marked as outdated.
This comment was marked as outdated.
asder8215
force-pushed
the
components_rewrite
branch
from
May 13, 2026 22:21
2151b8f to
83cdbed
Compare
This comment was marked as outdated.
This comment was marked as outdated.
asder8215
force-pushed
the
components_rewrite
branch
from
May 15, 2026 00:30
83cdbed to
3921fff
Compare
asder8215
marked this pull request as draft
May 16, 2026 12:22
rustbot
added
S-waiting-on-author
asder8215
force-pushed
the
components_rewrite
branch
from
May 17, 2026 16:09
0a25dda to
92e0132
Compare
Contributor
Author
|
New benchmarking results. You can see what the benchmark code looks like here and run it yourself to see if there are any difference in measurements on your end: This is the measurement of the current implementation of This is the measurement of the new implementation of Edit: Updated |
Contributor
Author
|
Here are the benchmark results with black box: From current From this Edit: Updated Edit 2: Took off Path ordering benchmark here since it was incorrect see below to see corrected path ordering benchmarks. |
asder8215
force-pushed
the
components_rewrite
branch
from
May 17, 2026 18:41
92e0132 to
574d7f2
Compare
asder8215
marked this pull request as ready for review
May 17, 2026 18:59
rustbot
added
S-waiting-on-review
Contributor
Author
|
I'm confident this code works (passed CI in previous run, the current amended commit change I made doesn't change logic, but makes the code written in a more idiomatic way). In my opinion, the logic in this code should look more readable than how |
Contributor
Author
|
@rustbot label +I-libs-nominated Since |
rustbot
added
the
I-libs-nominated
asder8215
commented
Jul 29, 2026
| #[derive(Clone)] | ||
| pub struct Components<'a> { | ||
| // The path left to parse components from | ||
| path: &'a [u8], |
Contributor
Author
There was a problem hiding this comment.
I decided to keep the field name as path to keep it consistent with the non-prefixed version of Components in case I want to refer to access the path field potentially outside of this file (e.g. for the PR I was working on in implementing DoubledEndedIterator on Ancestors iterator).
Contributor
Author
|
I made one more optimization for the non-prefixed implementation of I'm hoping by this weekend, I'll have a chart with the performance difference between stdlib's current |
This comment has been minimized.
This comment has been minimized.
asder8215
force-pushed
the
components_rewrite
branch
from
August 21, 2026 05:37
7ddc854 to
998374b
Compare
This comment has been minimized.
This comment has been minimized.
Contributor
|
So, generally, being slower in debug mode is just a case of needing |
asder8215
force-pushed
the
components_rewrite
branch
from
August 21, 2026 19:27
998374b to
f6e476f
Compare
Contributor
Author
I think the issue for why debug mode is slower for
I get where you're coming from. I'm unsure how to remove that debug perf slowdown though because I believe it boils down to a difference in the implementation and how the compiler optimizes/outputs code for the two. |
…rtialOrd implementation for non-prefixed Components via directly comparing mismatched byte if it's not a separator byte or cur dir byte; refactor while loop code to use iter().enumerate().find() instead
asder8215
force-pushed
the
components_rewrite
branch
from
August 22, 2026 06:27
f6e476f to
006de5e
Compare
…e conditional style code rather than match cases (degrades Components::next_back for some reason)
Contributor
Author
|
Disclaimer: I used Claude Sonnet 5 to parse through the raw Criterion measurement results I got from benchmarking Components iterator vs std Components iterator into well formatted md code tables (there's like 374 rows, so it would've taken me way longer to do this by hand or take some time to write a python script for this). I have cross-checked it with the raw Criterion results and it should be reflect the avg measurement results from what I got from Criterion. All benchmarking code is written myself, and I will explain the process of what I did to benchmark this below. You can find the benchmarking code in the Note, I have only done the release optimizations benchmarking for Right now my methodology is as follows:
Looking back, there's a lot more I could've benchmarked and experimented with. For example, I could've varied the number of path components we had in total instead of doing minimum number of component (short path, 1 comp) and maximum number of path components (long path, Here are the release mode benchmarks: Criterion Benchmark Results — Release ModeTimes shown are the Criterion point estimate (middle of the confidence interval). Rel Short Path with 1 byte comps
Abs Short Path with 1 byte comps
Rel Short Path with 3 byte comps
Abs Short Path with 3 byte comps
Rel Short Path with 7 byte comps
Abs Short Path with 7 byte comps
Rel Short Path with 15 byte comps
Abs Short Path with 15 byte comps
Rel Short Path with 31 byte comps
Abs Short Path with 31 byte comps
Rel Short Path with 63 byte comps
Abs Short Path with 63 byte comps
Rel Short Path with 127 byte comps
Abs Short Path with 127 byte comps
Rel Short Path with 255 byte comps
Abs Short Path with 255 byte comps
Rel Long Path with 1 byte comps
Abs Long Path with 1 byte comps
Rel Long Path with 3 byte comps
Abs Long Path with 3 byte comps
Rel Long Path with 7 byte comps
Abs Long Path with 7 byte comps
Rel Long Path with 15 byte comps
Abs Long Path with 15 byte comps
Rel Long Path with 31 byte comps
Abs Long Path with 31 byte comps
Rel Long Path with 63 byte comps
Abs Long Path with 63 byte comps
Rel Long Path with 127 byte comps
Abs Long Path with 127 byte comps
Rel Long Path with 255 byte comps
Abs Long Path with 255 byte comps
Rel Long Path Inconsistent Comp
Abs Long Path Inconsistent Comp
There are 12 rows where this Components iterator was slower than the std version.
Everything else is an improvement or the same on this Components iterator end with release mode optimizations. It's interesting to see that degradation occurs mostly when the components are at sizes of 63 bytes or further, but it still performs better/significantly better than Std Components with 31 bytes or less. On the other hand, it may be possible that std Components struggle with longer paths. Throwing in personal opinion, but I feel like that's a reasonable choice to choose this iterator as we don't often deal with paths with components with more than 20 bytes per component. |
Contributor
|
So, just to double-check right now: I've been mostly letting you continue to work on this with it still |
Contributor
Author
|
Yup, I'm ready (I actually forgot that it was labeled as waiting on author, I usually click the refresh icon thing next to your name for reviews). All I'll show next is debug performance results (likely tomorrow), but know that debug perf is significantly worse than the current std |
rustbot
added
S-waiting-on-review
Contributor
Author
|
Took longer than it should've to share this, but I got around to getting the debug perf output: Criterion Benchmark Results — Debug ModeTimes shown are the Criterion point estimate (middle of the confidence interval). Rel Short Path with 1 byte comps
Abs Short Path with 1 byte comps
Rel Short Path with 3 byte comps
Abs Short Path with 3 byte comps
Rel Short Path with 7 byte comps
Abs Short Path with 7 byte comps
Rel Short Path with 15 byte comps
Abs Short Path with 15 byte comps
Rel Short Path with 31 byte comps
Abs Short Path with 31 byte comps
Rel Short Path with 63 byte comps
Abs Short Path with 63 byte comps
Rel Short Path with 127 byte comps
Abs Short Path with 127 byte comps
Rel Short Path with 255 byte comps
Abs Short Path with 255 byte comps
Rel Long Path with 1 byte comps
Abs Long Path with 1 byte comps
Rel Long Path with 3 byte comps
Abs Long Path with 3 byte comps
Rel Long Path with 7 byte comps
Abs Long Path with 7 byte comps
Rel Long Path with 15 byte comps
Abs Long Path with 15 byte comps
Rel Long Path with 31 byte comps
Abs Long Path with 31 byte comps
Rel Long Path with 63 byte comps
Abs Long Path with 63 byte comps
Rel Long Path with 127 byte comps
Abs Long Path with 127 byte comps
Rel Long Path with 255 byte comps
Abs Long Path with 255 byte comps
Rel Long Path Inconsistent Comp
Abs Long Path Inconsistent Comp
Yeah, the debug performance is horrendous for this |
Contributor
|
FWIW, have noticed that this is ready for review and do plan to get to this relatively soonish. Since I'm going to be at rustconf in about a week I want to try and merge this before then unless there are major changes needed. |
Contributor
Author
I guess my concern here is: does this require a libs team discussion on the debug mode performance of this |
Contributor
|
Oh, wow, I hadn't actually looked at those numbers, but uh, 10x in some cases is pretty bad. I guess we could try a perf run: @bors try |
This comment has been minimized.
This comment has been minimized.
rustbot
added
the
S-waiting-on-perf
This comment has been minimized.
This comment has been minimized.
rust-bors Bot
pushed a commit
that referenced
this pull request
Aug 30, 2026
Split `Components` iterator to prefixed and non-prefixed versions and optimize `Components` for non-prefix path based platforms
Contributor
This comment has been minimized.
This comment has been minimized.
Collaborator
|
Finished benchmarking commit (6e44b7b): comparison URL. Overall result: ✅ improvements - no action neededBenchmarking means the PR may be perf-sensitive. Consider adding rollup=never if this change is not fit for rolling up. @rustbot label: -S-waiting-on-perf -perf-regression Instruction countOur most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
Max RSS (memory usage)Results (primary -0.2%, secondary 2.7%)A less reliable metric. May be of interest, but not used to determine the overall result above.
CyclesThis perf run didn't have relevant results for this metric. Binary sizeResults (primary 0.2%)A less reliable metric. May be of interest, but not used to determine the overall result above.
Bootstrap: 474.853s -> 473.232s (-0.34%) |
rustbot
removed
the
S-waiting-on-perf
Contributor
|
Yeah, that's expectedly unhelpful, probably because the standard library is compiled in release mode. I'm not actually sure you can easily run the debug versions without build-std or testing the repo? Will have to double-check, but pretty sure this should be fine. If you want to try running your benchmarks with debug mode combined with the release-compiled std, pretty sure the rustc dev guide has some notes on making rustup use a custom libstd build. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
View all comments
This PR entirely changes how
Components<'_>is implemented.Currently, theThisComponents<'_>iterator 'consumes' components through mutating its path field to a subslice that presents the left over unconsumed path components (this consumed path component is what's returned inComponents::nextorComponents::next_back). However, this PR keeps the path field alive/unmodified and uses front and back indexing strategy to extract consumed/unconsumed components.Components<'_>implementation still uses a subslicing approach similar to the original implementation, but it's highly optimized for non-Windows platform, given that there's noPrefix<'_>components to check with.Ideally, at least from a few benchmarking test I've done locally, this
Components<'_>implementation should perform similarly with the original one when it comes toComponents::next/Componentsordering comparisons (the original one was already highly optimized as is). Where this non-Windows implementation shine though is improvingComponents::as_path(as a result certain things benefit from this likeAncestorsiterator,std::fs::create_dir_all, etc.) andComponents::next_back(and as a resultPathequality).There are a lot of different things I've done to make sure that
Componentsequality is improved dramatically. For example, just like whatComponentscomparison does with checking if we see a mismatched byte in the forward direction and setting ourComponentsiterator to the nearest separator, we do the same thing withComponentsequality in the opposite direction (this actually is one of the significant reason why this implementation ofComponentsequality is really, really faster now). Additionally, there are other optimizations I do here, like doing less boolean checks or usize comparisons withinComponents::next_backand normalizing any separator/curr directory bytes upfront (the original implementation had a lot of conditional checks that could be way more simplified; the prefix checking also didn't benefit non-Windows platform). With the way I haveStateenum implemented in thisComponentsiterator, I can do other small optimizations like checking for empty paths, so that I can returntrueorfalseimmediately.Because the implementation I'm going for is specific to non-Windows/non-prefix-supporting platforms, I chose to split the
Componentsiterator implementation to two different version that is re-exported accordingly to individual platforms (this leads toHAS_PREFIXESconstant being redundant and unnecessary). I'm open to discussion on how to organize the code here better, naming stuff, etc. Some prefix-specific things have been refactored to be exclusively on platforms that utilize prefix components.So far, I've been benchmarking it locally. I formerly did it with Criterion, but now I've just been benchmarking manually through hyperfine. What I've been testing was how fast can this implementation of
Components::next/Components::next_backrun paths like "a1..aN/a1..aN" vs "b/a1..aN/a1..aN/" (where N is some number of a's I put for each component, and you can assume that there are some X path components in the path); I also played around with the mismatching component placed at the end of the path versus beginning, and benchmarked running the same path equality/comparisons 10000 times to see the result cumulatively. You can try hyperfining the code in this repo; I'll get around to consolidating those benchmarks in Criterion a bit later and then posting results in the comments below.