[SparkSPARK-9340] - make SparkSQL work with nested types in parquet-protobuf

[dguy]

…rotobuf

[liancheng]

The issue this PR tries to fix should be a special case of the Parquet interoperability issue which has already been fixed in Spark 1.5. Please see my comments in SPARK-9340 for details.

[liancheng]

ok to test

[liancheng]

The first null here should be an empty Seq. The schema of the testing Parquet data file is:

message TestProtobuf.SchemaConverterRepetition {
  optional int32 optionalPrimitive;
  required int32 requiredPrimitive;
  repeated int32 repeatedPrimitive;
  optional group optionalMessage {
    optional int32 someId;
  }
  required group requiredMessage {
    optional int32 someId;
  }
  repeated group repeatedMessage {
    optional int32 someId;
  }
}

As stated by parquet-format spec, repeatedPrimitive should be interpreted as a required list of required elements, so it should never be null.

[SparkQA]

Test build #40294 has finished for PR 8063 at commit fb23f28.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • trait NeedsResetArray
  • trait SchemaConverter

[liancheng]

We can close this one now since #8070 supersedes it.