rewrite the predecessors code to create a reduced graph #39424

nikomatsakis · 2017-01-31T17:08:11Z

The old code created a flat listing of "HIR -> WorkProduct" edges.
While perfectly general, this could lead to a lot of repetition if the
same HIR nodes affect many work-products. This is set to be a problem
when we start to skip typeck, since we will be adding a lot more
"work-product"-like nodes.

The newer code uses an alternative strategy: it "reduces" the graph
instead. Basically we walk the dep-graph and convert it to a DAG, where
we only keep intermediate nodes if they are used by multiple
work-products.

This DAG does not contain the same set of nodes as the original graph,
but it is guaranteed that (a) every output node is included in the graph
and (b) the set of input nodes that can reach each output node is
unchanged.

(Input nodes are basically HIR nodes and foreign metadata; output nodes
are nodes that have assocaited state which we will persist to disk in
some way. These are assumed to be disjoint sets.)

r? @michaelwoerister

Fixes #39494

The old code created a flat listing of "HIR -> WorkProduct" edges. While perfectly general, this could lead to a lot of repetition if the same HIR nodes affect many work-products. This is set to be a problem when we start to skip typeck, since we will be adding a lot more "work-product"-like nodes. The newer code uses an alternative strategy: it "reduces" the graph instead. Basically we walk the dep-graph and convert it to a DAG, where we only keep intermediate nodes if they are used by multiple work-products. This DAG does not contain the same set of nodes as the original graph, but it is guaranteed that (a) every output node is included in the graph and (b) the set of input nodes that can reach each output node is unchanged. (Input nodes are basically HIR nodes and foreign metadata; output nodes are nodes that have assocaited state which we will persist to disk in some way. These are assumed to be disjoint sets.)

nikomatsakis · 2017-02-01T17:04:08Z

@michaelwoerister I did some measurements of syntex-syntax. The rust master takes 3.6 seconds to encode the dep-graph. This branch takes 2.6 seconds. So this seems like a clear win overall.

michaelwoerister

I did one pass over this and it looks very good! I want to take another, closer look at the implementation and tests for the graph reduction algorithm.

michaelwoerister · 2017-02-01T22:08:43Z

src/librustc_incremental/persist/load.rs

+    let mut len = 0;
+    while len != dirty_nodes.len() {
+        len = dirty_nodes.len();
+        for edge in edges {


This implementation looks a bit inefficient.

Yeah, I was lazy. :) I can rewrite it to use a work-list or some such thing. One thing is that we don't have the edges indexed by their target, which we would want here. Perhaps I'll change how things are serialized to be a Map<Target, Vec<Source>>. I think we even build one of those in the "reduction" algorithm, so perhaps I should just return that (i.e., don't return a Graph, but some kind of ReducedGraph). I have to look at how the code works there.

michaelwoerister · 2017-02-01T22:44:41Z

src/librustc_incremental/persist/preds/compress/dag_id.rs

+}
+
+impl DagId {
+    pub fn from_in_index(n: NodeIndex) -> Self {


where does the in in from_in_index come from? Wouldn't from_node_index be clearer?

Hmm, I meant in as in "the index in the INPUT graph". Perhaps input_index?

michaelwoerister · 2017-02-01T23:02:27Z

src/librustc_incremental/persist/preds/compress/test.rs

+    ]);
+}
+
+//#[test]


Unported test.

Ah yes I forgot about that one.

michaelwoerister · 2017-02-01T23:06:39Z

src/librustc_incremental/persist/save.rs

@@ -178,7 +179,9 @@ pub fn encode_dep_graph(preds: &Predecessors,
    // Create a flat list of (Input, WorkProduct) edges for


That's note quite true anymore...

The old algorithm was O(graph)

nikomatsakis · 2017-02-03T11:10:29Z

@michaelwoerister I addressed (I think) your feedback w/ exception of commented out test, jfyi.

nikomatsakis · 2017-02-03T17:42:47Z

@michaelwoerister ok, ported the test.

michaelwoerister · 2017-02-03T20:43:54Z

src/librustc_incremental/persist/preds/compress/classify/test.rs

+    let mut reduce = GraphReduce::new(&graph, |n| inputs.contains(n), |n| outputs.contains(n));
+    Classify::new(&mut reduce).walk();
+
+    assert!(reduce.in_cycle(nodes("B"), nodes("C")));


Can you also add assert!(reduce.in_cycle(nodes("A"), nodes("C"))); and assert!(reduce.in_cycle(nodes("A"), nodes("B")));?

michaelwoerister · 2017-02-03T21:10:17Z

OK, I did another pass. Looks good to me. The graph reduction algorithm is very nice!
Please add the assertions in the test case, if you don't mind. Also, make tidy complains. Otherwise, consider the PR approved.

nikomatsakis · 2017-02-04T13:23:43Z

@bors r=mw

bors · 2017-02-04T13:23:44Z

📌 Commit b3096e2 has been approved by mw

@michaelwoerister

…3, r=mw rewrite the predecessors code to create a reduced graph The old code created a flat listing of "HIR -> WorkProduct" edges. While perfectly general, this could lead to a lot of repetition if the same HIR nodes affect many work-products. This is set to be a problem when we start to skip typeck, since we will be adding a lot more "work-product"-like nodes. The newer code uses an alternative strategy: it "reduces" the graph instead. Basically we walk the dep-graph and convert it to a DAG, where we only keep intermediate nodes if they are used by multiple work-products. This DAG does not contain the same set of nodes as the original graph, but it is guaranteed that (a) every output node is included in the graph and (b) the set of input nodes that can reach each output node is unchanged. (Input nodes are basically HIR nodes and foreign metadata; output nodes are nodes that have assocaited state which we will persist to disk in some way. These are assumed to be disjoint sets.) r? @michaelwoerister Fixes rust-lang#39494

@michaelwoerister

…3, r=mw rewrite the predecessors code to create a reduced graph The old code created a flat listing of "HIR -> WorkProduct" edges. While perfectly general, this could lead to a lot of repetition if the same HIR nodes affect many work-products. This is set to be a problem when we start to skip typeck, since we will be adding a lot more "work-product"-like nodes. The newer code uses an alternative strategy: it "reduces" the graph instead. Basically we walk the dep-graph and convert it to a DAG, where we only keep intermediate nodes if they are used by multiple work-products. This DAG does not contain the same set of nodes as the original graph, but it is guaranteed that (a) every output node is included in the graph and (b) the set of input nodes that can reach each output node is unchanged. (Input nodes are basically HIR nodes and foreign metadata; output nodes are nodes that have assocaited state which we will persist to disk in some way. These are assumed to be disjoint sets.) r? @michaelwoerister Fixes rust-lang#39494

bors · 2017-02-04T18:39:00Z

⌛ Testing commit b3096e2 with merge eb5cb95...

@michaelwoerister

rewrite the predecessors code to create a reduced graph The old code created a flat listing of "HIR -> WorkProduct" edges. While perfectly general, this could lead to a lot of repetition if the same HIR nodes affect many work-products. This is set to be a problem when we start to skip typeck, since we will be adding a lot more "work-product"-like nodes. The newer code uses an alternative strategy: it "reduces" the graph instead. Basically we walk the dep-graph and convert it to a DAG, where we only keep intermediate nodes if they are used by multiple work-products. This DAG does not contain the same set of nodes as the original graph, but it is guaranteed that (a) every output node is included in the graph and (b) the set of input nodes that can reach each output node is unchanged. (Input nodes are basically HIR nodes and foreign metadata; output nodes are nodes that have assocaited state which we will persist to disk in some way. These are assumed to be disjoint sets.) r? @michaelwoerister Fixes #39494

bors · 2017-02-04T21:13:00Z

☀️ Test successful - status-appveyor, status-travis
Approved by: mw
Pushing eb5cb95 to master...

rust-highfive assigned michaelwoerister Jan 31, 2017

michaelwoerister reviewed Feb 2, 2017

View reviewed changes

nikomatsakis added 2 commits February 3, 2017 06:08

make dirty process O(dirty)

ef9ae85

The old algorithm was O(graph)

s/in_index/input_index/

afbf6c8

add a comment about optimality that somehow got removed

7abab8a

nikomatsakis mentioned this pull request Feb 3, 2017

Refactor dependency graph reduction to allow for more work-products #39494

Closed

michaelwoerister reviewed Feb 3, 2017

View reviewed changes

pacify the mercilous tidy, improve cycle unit test

b3096e2

frewsxcv mentioned this pull request Feb 4, 2017

Rollup of 11 pull requests #39536

Closed

frewsxcv mentioned this pull request Feb 4, 2017

Rollup of 10 pull requests #39537

Closed

bors merged commit b3096e2 into rust-lang:master Feb 4, 2017

SimonSapin mentioned this pull request Feb 5, 2017

Experiment with incremental builds servo/servo#14870

Closed

michaelwoerister mentioned this pull request Feb 6, 2017

Nightly rustc 1.17.0-nightly (ea7a6486a 2017-02-04) crashes #39569

Closed

nikomatsakis deleted the incr-comp-skip-typeck-3 branch April 14, 2017 10:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rewrite the predecessors code to create a reduced graph #39424

rewrite the predecessors code to create a reduced graph #39424

nikomatsakis commented Jan 31, 2017 •

edited

Loading

nikomatsakis commented Feb 1, 2017

michaelwoerister left a comment

michaelwoerister Feb 1, 2017

nikomatsakis Feb 2, 2017

michaelwoerister Feb 1, 2017

nikomatsakis Feb 2, 2017

michaelwoerister Feb 1, 2017

nikomatsakis Feb 2, 2017

michaelwoerister Feb 1, 2017

nikomatsakis commented Feb 3, 2017

nikomatsakis commented Feb 3, 2017

michaelwoerister Feb 3, 2017

michaelwoerister commented Feb 3, 2017

nikomatsakis commented Feb 4, 2017

bors commented Feb 4, 2017

bors commented Feb 4, 2017

bors commented Feb 4, 2017

		@@ -178,7 +179,9 @@ pub fn encode_dep_graph(preds: &Predecessors,
		// Create a flat list of (Input, WorkProduct) edges for

rewrite the predecessors code to create a reduced graph #39424

rewrite the predecessors code to create a reduced graph #39424

Conversation

nikomatsakis commented Jan 31, 2017 • edited Loading

nikomatsakis commented Feb 1, 2017

michaelwoerister left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikomatsakis commented Feb 3, 2017

nikomatsakis commented Feb 3, 2017

Choose a reason for hiding this comment

michaelwoerister commented Feb 3, 2017

nikomatsakis commented Feb 4, 2017

bors commented Feb 4, 2017

bors commented Feb 4, 2017

bors commented Feb 4, 2017

nikomatsakis commented Jan 31, 2017 •

edited

Loading