[SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable #18328

dosoft · 2017-06-16T13:41:02Z

What changes were proposed in this pull request?

As described in title

How was this patch tested?

gatorsmile · 2017-06-16T16:48:39Z

sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala

This line exceeds 100 characters.

gatorsmile · 2017-06-16T16:48:52Z

sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala

Combine the two lines together

gatorsmile · 2017-06-16T16:49:22Z

Could you add a test case in org.apache.spark.sql.CachedTableSuite?

gatorsmile · 2017-06-16T16:51:13Z

ok to test

SparkQA · 2017-06-16T16:54:25Z

Test build #78187 has finished for PR 18328 at commit f6dfaab.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

…nMemoryColumnarStorage.level variable

SparkQA · 2017-06-17T05:42:33Z

Test build #78201 has started for PR 18328 at commit 26cfaaa.

AmplabJenkins · 2017-06-17T07:05:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78201/
Test FAILed.

felixcheung · 2017-06-20T03:10:04Z

sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala


+  def cacheQuery(query: Dataset[_], tableName: Option[String] = None): Unit = writeLock {
+    cacheQuery(query, tableName, StorageLevel.fromString(
+      query.sparkSession.sessionState.conf.cacheStorageLevel))


btw, this is a bit odd to roundtrip to string <-> storagelevel...

Could you explain your question in a bit more details?

HyukjinKwon · 2017-07-24T03:18:48Z

@dosoft Is this PR active? then would you mind if I ask to reply to the review comment above?

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017 Closes apache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory … Closes apache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage. Closes apache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation Closes apache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers Closes apache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key… Closes apache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples Closes apache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python Closes apache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Closes apache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins Closes apache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP] Closes apache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job Closes apache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable Closes apache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator Closes apache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset Closes apache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Closes apache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work Closes apache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly Closes apache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column Closes apache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService Closes apache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication Closes apache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone Closes apache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000) Closes apache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos Closes apache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table Closes apache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit… Closes apache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex Closes apache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable Closes apache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting Closes apache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages Closes apache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery Closes apache#18432 - resolve com.esotericsoftware.kryo.KryoException Closes apache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer Closes apache#18585 - SPARK-21359 Closes apache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala Added: Closes apache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I… Closes apache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 Closes apache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to … Closes apache#18667 - Fix the simpleString used in error messages Closes apache#18782 - Branch 2.1 Added: Closes apache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads Added: Closes apache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread Closes apache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable Closes apache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server Added: Closes apache#18827 - Merge pull request 1 from apache/master ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes apache#18780 from HyukjinKwon/close-prs.

gatorsmile reviewed Jun 16, 2017

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala Outdated

Copy link

Member

gatorsmile Jun 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line exceeds 100 characters.

gatorsmile reviewed Jun 16, 2017

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala Outdated

Copy link

Member

gatorsmile Jun 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Combine the two lines together

[SPARK-21121][SQL] Support changing storage level via the spark.sql.i…

26cfaaa

…nMemoryColumnarStorage.level variable

dosoft force-pushed the SPARK-21121 branch from f6dfaab to 26cfaaa Compare June 17, 2017 05:40

felixcheung reviewed Jun 20, 2017

View reviewed changes

HyukjinKwon mentioned this pull request Jul 31, 2017

[INFRA] Close stale PRs #18780

Closed

asfgit closed this in 3a45c7f Aug 5, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable #18328

[SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable #18328

Uh oh!

dosoft commented Jun 16, 2017

Uh oh!

gatorsmile Jun 16, 2017

Uh oh!

gatorsmile Jun 16, 2017

Uh oh!

gatorsmile commented Jun 16, 2017 •

edited

Loading

Uh oh!

gatorsmile commented Jun 16, 2017

Uh oh!

SparkQA commented Jun 16, 2017

Uh oh!

SparkQA commented Jun 17, 2017

Uh oh!

AmplabJenkins commented Jun 17, 2017

Uh oh!

felixcheung Jun 20, 2017 •

edited

Loading

Uh oh!

dosoft Jul 24, 2017

Uh oh!

HyukjinKwon commented Jul 24, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable #18328

[SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable #18328

Uh oh!

Conversation

dosoft commented Jun 16, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

gatorsmile Jun 16, 2017

Choose a reason for hiding this comment

Uh oh!

gatorsmile Jun 16, 2017

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Jun 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Jun 16, 2017

Uh oh!

SparkQA commented Jun 16, 2017

Uh oh!

SparkQA commented Jun 17, 2017

Uh oh!

AmplabJenkins commented Jun 17, 2017

Uh oh!

felixcheung Jun 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dosoft Jul 24, 2017

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jul 24, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

gatorsmile commented Jun 16, 2017 •

edited

Loading

felixcheung Jun 20, 2017 •

edited

Loading