[SPARK-8725][PROJECT-INFRA] Test modules in topologically-sorted order in dev/run-tests

[JoshRosen]

This patch improves our dev/run-tests script to test modules in a topologically-sorted order based on modules' dependencies. This will help to ensure that bugs in upstream projects are not misattributed to downstream projects because those projects' tests were the first ones to exhibit the failure

Topological sorting is also useful for shortening the feedback loop when testing pull requests: if I make a change in SQL then the SQL tests should run before MLlib, not after.

In addition, this patch also updates our test module definitions to split sql into catalyst, sql, and hive in order to allow more tests to be skipped when changing only hive/ files.

[JoshRosen]

@srowen, I didn't want to write my own sort so I just copied this one from the toposort library. Do I need to update the NOTICE file in this case?

[JoshRosen]

Ping again. @srowen @pwendell, can you comment on the NOTICE file considerations here?

[srowen]

Oops, I missed this. It looks like it's Apache licensed and you have correctly preserved the header. https://bitbucket.org/ericvsmith/toposort/src/25b5894c4229cb888f77cf0c077c05e2464446ac/LICENSE.txt?fileviewer=file-view-default

You'll need to put the NOTICE contents in our NOTICE:
https://bitbucket.org/ericvsmith/toposort/src/25b5894c4229cb888f77cf0c077c05e2464446ac/NOTICE?fileviewer=file-view-default

That should be all.

[JoshRosen]

Do I just append the entire contents of the notice verbatim? Is there a shorthand way of doing this (e.g. just including just the copyright)? I got a bit confused looking at the existing NOTICE file and since it hasn't been updated in a while I didn't spot any smaller patches / diffs to model my change on.

[srowen]

Yes just append. I don't know if we've got a strong format going, except to try to precede the text from Foo with a line like "Foo" or "For Foo:". Anything sane should be OK.

Technically you reproduce all relevant parts of the NOTICE and only relevant parts. All is relevant here. Really, not sure the project needed this file but hey. Practically you could argue you can abbreviate it, but given it's short, I'd just copy it.

[JoshRosen]

Alright, updated the NOTICE to append it verbatim.

[SparkQA]

Test build #49943 has finished for PR 10885 at commit 76a446e.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49944 has finished for PR 10885 at commit c122c55.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Whoops:

========================================================================
Traceback (most recent call last):
  File "./python/run-tests.py", line 42, in <module>
    from sparktestsupport.modules import all_modules  # noqa
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/../dev/sparktestsupport/modules.py", line 112, in <module>
    "sql/test",
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/../dev/sparktestsupport/modules.py", line 74, in __init__
    dep.dependent_modules.add(self)
TypeError: unhashable type: 'Module'

[SparkQA]

Test build #49955 has finished for PR 10885 at commit 00a817a.

• This patch fails executing the dev/run-tests script.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49956 has finished for PR 10885 at commit a17c3d0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Unless anyone has objections, I'm planning to merge this today. If anyone complains about the loss of Python 2.6 support in this script, they can submit a PR to fix compatibility themselves (again, this is only relevant to developers of Spark / maintainers of its test infrastructure, not end users, and I expect that they're using non-ancient Python versions or can easily upgrade).

[SparkQA]

Test build #50113 has finished for PR 10885 at commit 0bc9da0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Going to merge this now; will address any problems in followups.