Skip to content

Conversation

@liancheng
Copy link

Major changes:

  • Remove spark.sql.hive.writeDataSourceSchema.
  • Always persist the data source relation in Hive compatible format when possible, and give warning logs to indicate why we can't do this.
  • Now we ony persist non-partitioned HadoopFsRelation with a single input path. The original PR only persists partition columns information without adding individual partitions, so persisting partitioned tables actually doesn't work.
  • Refactor test cases.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this test case since all its contents are covered in refactored DataSourceWithHiveMetastoreCatalogSuite.

@chenghao-intel chenghao-intel merged commit bc3cd73 into chenghao-intel:SPARK-6923 Aug 5, 2015
@chenghao-intel
Copy link
Owner

Thanks, merged. will it trigger the jenkins testing automatically?

@liancheng
Copy link
Author

Yeah, it will.

@liancheng liancheng deleted the spark-6923/refactoring-pr-5733 branch August 5, 2015 14:34
chenghao-intel pushed a commit that referenced this pull request Sep 7, 2015
Fix for incorrect memory in Spark UI as per SPARK-5768

Author: Joshi <[email protected]>
Author: Rekha Joshi <[email protected]>

Closes apache#6972 from rekhajoshm/SPARK-5768 and squashes the following commits:

b678a91 [Joshi] Fix for incorrect memory in Spark UI
2fe53d9 [Joshi] Fix for incorrect memory in Spark UI
eb823b8 [Joshi] SPARK-5768: Fix for incorrect memory in Spark UI
0be142d [Rekha Joshi] Merge pull request apache#3 from apache/master
106fd8e [Rekha Joshi] Merge pull request #2 from apache/master
e3677c9 [Rekha Joshi] Merge pull request #1 from apache/master

(cherry picked from commit 085a721)
Signed-off-by: Kousuke Saruta <[email protected]>
chenghao-intel pushed a commit that referenced this pull request Sep 7, 2015
This makes sure attempts are listed in the order they were executed, and that the
app's state matches the state of the most current attempt.

Author: Joshi <[email protected]>
Author: Rekha Joshi <[email protected]>

Closes apache#7253 from rekhajoshm/SPARK-8593 and squashes the following commits:

874dd80 [Joshi] History Server: updated order for multiple attempts(logcleaner)
716e0b1 [Joshi] History Server: updated order for multiple attempts(descending start time works everytime)
548c753 [Joshi] History Server: updated order for multiple attempts(descending start time works everytime)
83306a8 [Joshi] History Server: updated order for multiple attempts(descending start time)
b0fc922 [Joshi] History Server: updated order for multiple attempts(updated comment)
cc0fda7 [Joshi] History Server: updated order for multiple attempts(updated test)
304cb0b [Joshi] History Server: updated order for multiple attempts(reverted HistoryPage)
85024e8 [Joshi] History Server: updated order for multiple attempts
a41ac4b [Joshi] History Server: updated order for multiple attempts
ab65fa1 [Joshi] History Server: some attempt completed to work with showIncomplete
0be142d [Rekha Joshi] Merge pull request apache#3 from apache/master
106fd8e [Rekha Joshi] Merge pull request #2 from apache/master
e3677c9 [Rekha Joshi] Merge pull request #1 from apache/master

(cherry picked from commit 42d8a01)
Signed-off-by: Sean Owen <[email protected]>
chenghao-intel pushed a commit that referenced this pull request May 25, 2016
## What changes were proposed in this pull request?

This reopens apache#11836, which was merged but promptly reverted because it introduced flaky Hive tests.

## How was this patch tested?

See `CatalogTestCases`, `SessionCatalogSuite` and `HiveContextSuite`.

Author: Andrew Or <[email protected]>

Closes apache#11938 from andrewor14/session-catalog-again.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants