Skip to content

Conversation

@steveloughran
Copy link
Contributor

This is the patch of PR #10780 applied to branch-1.6, to verify it works there too

…of master PR commit c36555165ce7c0e5da39e6b05863012a874f119c
@SparkQA
Copy link

SparkQA commented Jan 16, 2016

Test build #49511 has finished for PR 10782 at commit 9246e4d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about doing this for all of Spark?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm now thinking all the dependencies should be relocated, with the exception of the jni leveldb stuff —not the thing you can shade; not the thing you can isolate in a classloader hierarchy instead.

What I can do there is add a profile, leveldb-provided which excludes leveldb in this situation. That's important as leveldb does creep onto the classpath from elsewhere —and while the version in Hadoop currently matches that in this JAR, there's no guarantee of that continuing to hold.

@steveloughran
Copy link
Contributor Author

Sean: the PR #10780 is the version against /master; its the one with the full diff. This is just here to show it works for 1.6 too. How about I close this one; the other is used for review, and, when it's good, it can be pulled in to 1.6. So far the dependencies between 1.6 and master are identical as far as the shuffle goes.

@steveloughran steveloughran changed the title SPARK-12807] [YARN] Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3: branch-1.6 patch [SPARK-12807] [YARN] Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3: branch-1.6 patch Jan 19, 2016
@steveloughran
Copy link
Contributor Author

(closing)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants