[SPARK-6511] [Documentation] Explain how to use Hadoop provided builds

[pwendell]

This provides preliminary documentation pointing out how to use the
Hadoop free builds. I am hoping over time this list can grow to
include most of the popular Hadoop distributions.

Getting more people using these builds will help us long term reduce
the number of binaries we build.

[pwendell]

/cc @vanzin and @srowen for any feedback.

[srowen]

To the extent of my knowledge, this is correct. More docs are good.

[vanzin]

Looks good. You can start going into more and more distro-specific details here if you want, but I'm not sure this document should cover those details.

[pwendell]

@vanzin Any specific details in mind? Personally I don't mind having details for different distros. My feeling is if people can't get these hadoop-free builds running easily, people won't use them. So it might be good to include all pertinent information.

[vanzin]

To be fair, any extra configuration also applies to the other builds, not just for the "hadoop-free" one. Each distro may have native libraries in different places, for example, so you may need to configure those paths appropriately.

(A quick look at a CDH config seems to show that's mostly it, aside from what you already documented. But I'm not familiar with other distros.)

[pwendell]

Ah okay - I think it could be good to add those too over time. For instance the MapR one probably wont' work at all unless some native libraries are added since their FS client depends on native code. But sounds like people are okay with this as a first step.

[SparkQA]

Test build #34529 has finished for PR 6729 at commit 1113b76.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[pwendell]

Jenkins, retest this please.

[SparkQA]

Test build #34533 has finished for PR 6729 at commit 1113b76.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.