Skip to content

[SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns #16832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
cenyuhai wants to merge 9 commits into from
Closed

[SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns #16832

cenyuhai wants to merge 9 commits into from

Conversation

@cenyuhai
Copy link
Contributor

@cenyuhai cenyuhai commented Feb 7, 2017 •
edited
Loading

What changes were proposed in this pull request?

ignore hive column names case-sensitive when comparing with partition columns

How was this patch tested?

@cenyuhai cenyuhai changed the title [SPARK-19490][SQL] change column names to lower case [SPARK-19490][SQL] change hive column names to lower case Feb 7, 2017
@rxin
Copy link
Contributor

rxin commented Feb 7, 2017

hm is it safe to just do this change?

@SparkQA
Copy link

SparkQA commented Feb 7, 2017

Test build #72501 has finished for PR 16832 at commit efb0b4b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cenyuhai
Copy link
Contributor Author

cenyuhai commented Feb 10, 2017

@rxin I think it is safe, it is only used to check whether the schema contains the columns. Hive columns are not case-sensitive.

@SparkQA
Copy link

SparkQA commented Feb 10, 2017

Test build #72689 has started for PR 16832 at commit 6884ad3.

@cenyuhai cenyuhai changed the title [SPARK-19490][SQL] change hive column names to lower case [WIP][SPARK-19490][SQL] change hive column names to lower case Feb 12, 2017
@cenyuhai cenyuhai changed the title [WIP][SPARK-19490][SQL] change hive column names to lower case [WIP][SPARK-19490][SQL] ignore hive column names case-sensitive when comparing with partition columns Feb 12, 2017
@cenyuhai cenyuhai changed the title [WIP][SPARK-19490][SQL] ignore hive column names case-sensitive when comparing with partition columns [WIP][SPARK-19490][SQL] ignore case-sensitive when filtering hive partition columns Feb 12, 2017
@cenyuhai cenyuhai changed the title [WIP][SPARK-19490][SQL] ignore case-sensitive when filtering hive partition columns [WIP][SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Feb 12, 2017
@SparkQA
Copy link

SparkQA commented Feb 12, 2017

Test build #72750 has finished for PR 16832 at commit 7214b8a.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Feb 12, 2017

Test build #72755 has finished for PR 16832 at commit b734646.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Feb 12, 2017

Test build #72758 has finished for PR 16832 at commit 358769c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Feb 12, 2017

Test build #72763 has started for PR 16832 at commit 5e91877.

@SparkQA
Copy link

SparkQA commented Feb 22, 2017

Test build #73267 has started for PR 16832 at commit f790821.

@SparkQA
Copy link

SparkQA commented Feb 22, 2017

Test build #73275 has finished for PR 16832 at commit c3f7444.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Feb 22, 2017

Test build #73278 has finished for PR 16832 at commit c9dd09b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cenyuhai cenyuhai changed the title [WIP][SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Feb 23, 2017
@janua
Copy link

janua commented Apr 26, 2017

👍
Just bumped into this issue. The error message doesn't make it obvious what's wrong.

@taklwu
Copy link

taklwu commented Jun 9, 2017

any updates on this PR?

@cenyuhai
Copy link
Contributor Author

cenyuhai commented Jun 17, 2017

@taklwu this pr is completed, you can merge this pr by yourself. A committer told me that other pr has fixed this bug, my pr will not be merged......

@taklwu
Copy link

taklwu commented Jun 17, 2017

May I get the link of which commit/pr fix this issue? thanks a lot

@HyukjinKwon HyukjinKwon mentioned this pull request Jul 31, 2017
Closed
@asfgit asfgit closed this in 3a45c7f Aug 5, 2017
@LantaoJin
Copy link
Contributor

LantaoJin commented Sep 15, 2017

Duplicates to SPARK-18572

zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
@HyukjinKwon @zifeif2
[INFRA] Close stale PRs
6639102 
## What changes were proposed in this pull request?

This PR proposes to close stale PRs, mostly the same instances with apache#18017

Closes apache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory …
Closes apache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage.
Closes apache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation
Closes apache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers
Closes apache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key…
Closes apache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples
Closes apache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python
Closes apache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage
Closes apache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins
Closes apache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP]
Closes apache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job
Closes apache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable
Closes apache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator
Closes apache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset
Closes apache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns
Closes apache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work
Closes apache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly
Closes apache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column
Closes apache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService
Closes apache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication
Closes apache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone
Closes apache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000)
Closes apache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos
Closes apache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table
Closes apache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit…
Closes apache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex
Closes apache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable
Closes apache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting
Closes apache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages
Closes apache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery
Closes apache#18432 - resolve com.esotericsoftware.kryo.KryoException
Closes apache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer
Closes apache#18585 - SPARK-21359
Closes apache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala

Added:
Closes apache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I…
Closes apache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0
Closes apache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to …
Closes apache#18667 - Fix the simpleString used in error messages
Closes apache#18782 - Branch 2.1

Added:
Closes apache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads

Added:
Closes apache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread
Closes apache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable
Closes apache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server

Added:
Closes apache#18827 - Merge pull request 1 from apache/master

## How was this patch tested?

N/A

Author: hyukjinkwon <[email protected]>

Closes apache#18780 from HyukjinKwon/close-prs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@cenyuhai @rxin @SparkQA @janua @taklwu @LantaoJin