[SPARK-12084][Core]Fix codes that uses ByteBuffer.array incorrectly

[zsxwing]

ByteBuffer doesn't guarantee all contents in ByteBuffer.array are valid. E.g, a ByteBuffer returned by ByteBuffer.slice. We should not use the whole content of ByteBuffer unless we know that's correct.

This patch fixed all places that use ByteBuffer.array incorrectly.

[zsxwing]

/cc @andrewor14 @JoshRosen @tdas @vanzin @srowen

It's better to have more eyes review this one since it touches a lot of files.

[SparkQA]

Test build #47020 has finished for PR 10083 at commit 93b68de.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #47022 has finished for PR 10083 at commit bfb3360.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

You can't use bufferToArray here too?

[zsxwing]

This is in examples. So I don't want to use private API.

[srowen]

Looking good. Besides those comments, the changes all looked sound to me.

[zsxwing]

Because decompressCache puts ByteBuffer as a key, here should not change the schemaBytes's position, so cannot use ByteBufferInputStream here.

[vanzin]

Not necessary for this change, but at some point it might be worth it to change this to use Kryo's ByteBufferInput.

[zsxwing]

Kryo will use the array as an internal buffer. Why it's not necessary?

[vanzin]

I'm saying that the change I proposed is not necessary, not that your change is not necessary.

[zsxwing]

I'm saying that the change I proposed is not necessary, not that your change is not necessary.

Got it. Sorry for my misunderstanding.

[vanzin]

LGTM.

[SparkQA]

Test build #47077 has finished for PR 10083 at commit a5d965c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #47083 has started for PR 10083 at commit 81d1812.

[zsxwing]

retest this please

[SparkQA]

Test build #47126 has finished for PR 10083 at commit 81d1812.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

Ok, merging to master. I assume we don't want this in 1.6 at this point?

[srowen]

@vanzin @zsxwing it looked like enough of an important fix to go into 1.6.x -- any strong objections to that?

[vanzin]

@srowen I think this is not a bug currently, because the code works based on the buffers being created according to the assumptions being made. But this is needed to unblock SPARK-12060, which breaks the assumption.