Use UnordMap and UnordSet for id collections (DefIdMap, LocalDefIdMap, etc) #106977

michaelwoerister · 2023-01-17T12:17:12Z

This PR changes the rustc_data_structures::define_id_collections! macro to use UnordMap and UnordSet instead of FxHashMap and FxHashSet. This should account for a large portion of hash-maps being used in places where they can cause trouble.

The changes required are moderate but non-zero:

In some places the collections are extracted into sorted vecs.
There are a few instances where for-loops have been changed to extends.

Let's see what the performance impact is. With a bit more refactoring, we might be able to get rid of some of the additional sorting -- but the change set is already big enough. Unless there's a performance impact, I'd like to do further changes in subsequent PRs.

Performance does not seem to be negatively affected (perf-run here).

Part of MCP 533.

r? @ghost

michaelwoerister · 2023-01-17T13:19:29Z

@bors try @rust-timer queue

bors · 2023-01-17T13:19:38Z

⌛ Trying commit 5185c3f658c566a8a55ccb27b335169cdedc7ad9 with merge de6228c8f8104f4bed1e7fb6b1874f3531c6aa31...

bors · 2023-01-17T15:45:44Z

☀️ Try build successful - checks-actions
Build commit: de6228c8f8104f4bed1e7fb6b1874f3531c6aa31 (de6228c8f8104f4bed1e7fb6b1874f3531c6aa31)

bors · 2023-01-17T15:45:45Z

☀️ Try build successful - checks-actions
Build commit: de6228c8f8104f4bed1e7fb6b1874f3531c6aa31 (de6228c8f8104f4bed1e7fb6b1874f3531c6aa31)

rust-timer · 2023-01-17T19:17:18Z

Finished benchmarking commit (de6228c8f8104f4bed1e7fb6b1874f3531c6aa31): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.6%	[0.2%, 1.5%]	4
Improvements ✅ (primary)	-0.2%	[-0.2%, -0.2%]	1
Improvements ✅ (secondary)	-0.4%	[-1.2%, -0.2%]	16
All ❌✅ (primary)	-0.2%	[-0.2%, -0.2%]	1

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.4%	[-5.6%, -3.1%]	5
All ❌✅ (primary)	-	-	0

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.4%	[-2.9%, -2.0%]	5
All ❌✅ (primary)	-	-	0

michaelwoerister · 2023-01-18T11:59:14Z

@bors try @rust-timer queue

bors · 2023-01-18T11:59:24Z

⌛ Trying commit 285e2c1be2918f18fcec3386f91ab8cebe3d19dc with merge 587656503f974defd16cf67c0f3894dcc9d7b565...

bors · 2023-01-18T14:21:33Z

☀️ Try build successful - checks-actions
Build commit: 587656503f974defd16cf67c0f3894dcc9d7b565 (587656503f974defd16cf67c0f3894dcc9d7b565)

michaelwoerister · 2023-01-19T09:29:30Z

@rust-timer build 587656503f974defd16cf67c0f3894dcc9d7b565

rust-timer · 2023-01-19T10:49:22Z

Finished benchmarking commit (587656503f974defd16cf67c0f3894dcc9d7b565): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.1%	[0.1%, 0.1%]	1
Regressions ❌ (secondary)	1.7%	[1.7%, 1.7%]	1
Improvements ✅ (primary)	-0.5%	[-0.5%, -0.5%]	3
Improvements ✅ (secondary)	-0.5%	[-1.2%, -0.2%]	20
All ❌✅ (primary)	-0.4%	[-0.5%, 0.1%]	4

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.7%	[3.3%, 4.2%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.5%	[-5.9%, -2.3%]	4
All ❌✅ (primary)	-	-	0

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	5.5%	[5.2%, 5.7%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.1%	[-2.1%, -2.1%]	1
All ❌✅ (primary)	-	-	0

rustbot · 2023-01-19T14:57:18Z

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

michaelwoerister · 2023-01-19T14:59:06Z

Performance looks OK now, I think.

r? compiler (see MCP 533 for background information)

oli-obk · 2023-01-19T15:39:38Z

compiler/rustc_data_structures/src/unord.rs

+        let mut items: Vec<(&K, &V)> = self.inner.iter().collect();
+        if cache_sort_key {
+            items.sort_by_cached_key(|(k, _)| k.to_stable_hash_key(hcx));
+        } else {
+            items.sort_unstable_by_key(|(k, _)| k.to_stable_hash_key(hcx));
+        }
+        items.into_iter().map(|(_, v)| v)


This is a recurring pattern, maybe give it a helper?

It should at least have documentation what cache_sort_key does

oli-obk · 2023-01-19T16:40:10Z

@bors r+

bors · 2023-01-19T16:40:12Z

📌 Commit f219771 has been approved by oli-obk

It is now in the queue for this repository.

michaelwoerister · 2023-01-20T09:43:40Z

Thanks for the review, @oli-obk!

bors · 2023-01-21T14:18:21Z

⌛ Testing commit f219771 with merge 005fc0f...

bors · 2023-01-21T17:31:46Z

☀️ Test successful - checks-actions
Approved by: oli-obk
Pushing 005fc0f to master...

rust-timer · 2023-01-21T20:41:51Z

Finished benchmarking commit (005fc0f): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.3%]	2
Regressions ❌ (secondary)	0.6%	[0.4%, 0.8%]	7
Improvements ✅ (primary)	-0.5%	[-0.5%, -0.5%]	3
Improvements ✅ (secondary)	-0.3%	[-0.7%, -0.2%]	20
All ❌✅ (primary)	-0.2%	[-0.5%, 0.3%]	5

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.5%	[1.2%, 5.3%]	4
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.9%	[-6.6%, -2.6%]	3
All ❌✅ (primary)	-	-	0

Cycles

This benchmark run did not return any relevant results for this metric.

rustdoc: Use `DefId(Map,Set)` instead of `FxHash(Map,Set)` Not all uses are converted, a few cases iterating through maps/sets and requiring nontrivial changes are kept. cc rust-lang#106977

rylev · 2023-01-25T14:49:39Z

@michaelwoerister perf got slightly worse from the original perf run, but it's still enough of a wash that I don't think it needs investigating.

@rustbot label +perf-regression-triaged

Use UnordMap and UnordSet for id collections (DefIdMap, LocalDefIdMap, etc) This PR changes the `rustc_data_structures::define_id_collections!` macro to use `UnordMap` and `UnordSet` instead of `FxHashMap` and `FxHashSet`. This should account for a large portion of hash-maps being used in places where they can cause trouble. The changes required are moderate but non-zero: - In some places the collections are extracted into sorted vecs. - There are a few instances where for-loops have been changed to extends. ~~Let's see what the performance impact is. With a bit more refactoring, we might be able to get rid of some of the additional sorting -- but the change set is already big enough. Unless there's a performance impact, I'd like to do further changes in subsequent PRs.~~ Performance does not seem to be negatively affected ([perf-run here](rust-lang/rust#106977 (comment))). Part of [MCP 533](rust-lang/compiler-team#533). r? `@ghost`

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jan 17, 2023

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jan 17, 2023

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Jan 17, 2023

michaelwoerister force-pushed the unord_id_collections branch from 1142d90 to 285e2c1 Compare January 18, 2023 10:19

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jan 18, 2023

michaelwoerister removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 18, 2023

Use UnordSet instead of FxHashSet in define_id_collections!().

8a1de57

michaelwoerister force-pushed the unord_id_collections branch from 285e2c1 to 57ee1bb Compare January 19, 2023 09:26

This comment has been minimized.

Sign in to view

michaelwoerister added 2 commits January 19, 2023 10:40

Use UnordMap instead of FxHashMap in define_id_collections!().

c3d2573

Allow for more efficient sorting when exporting Unord collections.

72ee14c

michaelwoerister force-pushed the unord_id_collections branch from 57ee1bb to 72ee14c Compare January 19, 2023 09:41

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jan 19, 2023

michaelwoerister marked this pull request as ready for review January 19, 2023 14:57

michaelwoerister changed the title ~~(WIP) Use UnordMap and UnordSet for id collections (DefIdMap, LocalDefIdMap, etc)~~ Use UnordMap and UnordSet for id collections (DefIdMap, LocalDefIdMap, etc) Jan 19, 2023

rustbot assigned oli-obk Jan 19, 2023

oli-obk reviewed Jan 19, 2023

View reviewed changes

Clean up and document unord collections a bit.

f219771

bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Jan 19, 2023

bors added the merged-by-bors This PR was explicitly merged by bors. label Jan 21, 2023

bors merged commit 005fc0f into rust-lang:master Jan 21, 2023

rustbot added this to the 1.68.0 milestone Jan 21, 2023

bors mentioned this pull request Jan 21, 2023

rustc_metadata: Encode doc(hidden) flag to metadata #107136

Merged

petrochenkov mentioned this pull request Jan 21, 2023

rustdoc: Use DefId(Map,Set) instead of FxHash(Map,Set) #107174

Merged

xFrednet mentioned this pull request Jan 22, 2023

Bump rust nightly -> 2023-01-26 rust-marker/marker#70

Closed

rustbot added the perf-regression-triaged The performance regression has been triaged. label Jan 25, 2023

michaelwoerister mentioned this pull request Apr 19, 2023

Create a deterministic FxHashMap wrapper #63713

Closed

jieyouxu mentioned this pull request Aug 14, 2024

Tracking Issue for Reproducible Build bugs and challenges #129080

Open

81 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use UnordMap and UnordSet for id collections (DefIdMap, LocalDefIdMap, etc) #106977

Use UnordMap and UnordSet for id collections (DefIdMap, LocalDefIdMap, etc) #106977

michaelwoerister commented Jan 17, 2023 •

edited

Loading

michaelwoerister commented Jan 17, 2023

This comment has been minimized.

bors commented Jan 17, 2023

bors commented Jan 17, 2023

bors commented Jan 17, 2023

This comment has been minimized.

rust-timer commented Jan 17, 2023

michaelwoerister commented Jan 18, 2023

This comment has been minimized.

bors commented Jan 18, 2023

bors commented Jan 18, 2023

michaelwoerister commented Jan 19, 2023

This comment has been minimized.

rust-timer commented Jan 19, 2023

rustbot commented Jan 19, 2023

michaelwoerister commented Jan 19, 2023

oli-obk Jan 19, 2023

oli-obk commented Jan 19, 2023

bors commented Jan 19, 2023

michaelwoerister commented Jan 20, 2023

bors commented Jan 21, 2023

bors commented Jan 21, 2023

rust-timer commented Jan 21, 2023

rylev commented Jan 25, 2023

Use UnordMap and UnordSet for id collections (DefIdMap, LocalDefIdMap, etc) #106977

Use UnordMap and UnordSet for id collections (DefIdMap, LocalDefIdMap, etc) #106977

Conversation

michaelwoerister commented Jan 17, 2023 • edited Loading

michaelwoerister commented Jan 17, 2023

This comment has been minimized.

bors commented Jan 17, 2023

bors commented Jan 17, 2023

bors commented Jan 17, 2023

This comment has been minimized.

rust-timer commented Jan 17, 2023

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

michaelwoerister commented Jan 18, 2023

This comment has been minimized.

bors commented Jan 18, 2023

bors commented Jan 18, 2023

michaelwoerister commented Jan 19, 2023

This comment has been minimized.

rust-timer commented Jan 19, 2023

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

rustbot commented Jan 19, 2023

michaelwoerister commented Jan 19, 2023

Choose a reason for hiding this comment

oli-obk commented Jan 19, 2023

bors commented Jan 19, 2023

michaelwoerister commented Jan 20, 2023

bors commented Jan 21, 2023

bors commented Jan 21, 2023

rust-timer commented Jan 21, 2023

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

rylev commented Jan 25, 2023

michaelwoerister commented Jan 17, 2023 •

edited

Loading