Skip to content

Conversation

@scwf
Copy link
Contributor

@scwf scwf commented Nov 19, 2014

Set spark.sql.hive.convertMetastoreParquet true now leads to
1 run
create table test_parquet(key int, value string) stored as parquet;
select * from test_parquet;
get error as follow

java.lang.IllegalArgumentException: Could not find Parquet metadata at path file:/user/hive/warehouse/test_parquet
at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$readMetaData$4.apply(ParquetTypes.scala:459)
at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$readMetaData$4.apply(ParquetTypes.scala:459)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.sql.parquet.ParquetTypesConverter$.readMetaData(ParquetTypes.sc

2 run
create table test_parquet(key int, value string) stored as parquet;
insert into table test_parquet select * from src;
select * from test_parquet;
get result as follow

...
282 [B@38fda3b8
138 [B@1407a243
238 [B@12de6fb
419 [B@6c976957
15 [B@48850673
118 [B@156a8d37
72 [B@65d20dd
90 [B@4c18906e
307 [B@60b24cc9
19 [B@59cf51b7
435 [B@39fdf371
10 [B@4f799d75
277 [B@39509516
273 [B@596bf4bf
306 [B@3e915576
224 [B@3781d611
309 [B@2d0d128f

@SparkQA
Copy link

SparkQA commented Nov 19, 2014

Test build #23576 has started for PR 3352 at commit 734e5ef.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 19, 2014

Test build #23576 has finished for PR 3352 at commit 734e5ef.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23576/
Test PASSed.

@scwf
Copy link
Contributor Author

scwf commented Nov 19, 2014

/cc @marmbrus

@marmbrus
Copy link
Contributor

I'd really much rather we try and fix these bugs instead of turning it off by default. A lot of users are getting sub-optimal performance and these both seem like corner cases (either way we will clearly state that this default was changed in the release notes so if people see regressions they know what to turn off). Can you open JIRAs for these two?

@scwf
Copy link
Contributor Author

scwf commented Nov 20, 2014

@marmbrus, sure i will open jiras for this and i am now working on this two issues, here i suggest merge this before fixing them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants