Skip to content

[SPARK-19970][SQL][BRANCH-2.1] Table owner should be USER instead of PRINCIPAL in kerberized clusters #17363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dongjoon-hyun wants to merge 1 commit into from
Closed

[SPARK-19970][SQL][BRANCH-2.1] Table owner should be USER instead of PRINCIPAL in kerberized clusters #17363

dongjoon-hyun wants to merge 1 commit into from

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Mar 20, 2017

What changes were proposed in this pull request?

In the kerberized hadoop cluster, when Spark creates tables, the owner of tables are filled with PRINCIPAL strings instead of USER names. This is inconsistent with Hive and causes problems when using ROLE in Hive. We had better to fix this.

BEFORE

scala> sql("create table t(a int)").show
scala> sql("desc formatted t").show(false)
...
|Owner:                      |spark@EXAMPLE.COM                                         |       |

AFTER

scala> sql("create table t(a int)").show
scala> sql("desc formatted t").show(false)
...
|Owner:                      |spark                                         |       |

How was this patch tested?

Manually do create table and desc formatted because this happens in Kerberized clusters.

dongjoon-hyun
dongjoon-hyun commented Mar 20, 2017
View reviewed changes
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
}
hiveTable.setPartCols(partCols.asJava)
hiveTable.setOwner(conf.getUser)
hiveTable.setOwner(state.getAuthenticator().getUserName())
Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Mar 20, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vanzin . I made a backport for branch-2.1 of #17311 . This one uses state as you advised.

@SparkQA
Copy link

SparkQA commented Mar 20, 2017

Test build #74891 has finished for PR 17363 at commit 1328f1d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Mar 20, 2017 •
edited
Loading

This is also tested manually in kerberized cluster, @vanzin .

BTW, Spark 1.6 has the same issue at ClientWrapper.

According to the Spark dev email, there exists demands on more Apache Spark 1.6.X. May I create a backport for branch-1.6? How do you think about that?

For Spark 1.6, it happens only with CREATE TABLE .. AS SELECT statement.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Mar 22, 2017

Hi, @vanzin .
Could you review this backport when you have some time?

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Mar 22, 2017

Retest this please

@SparkQA
Copy link

SparkQA commented Mar 22, 2017

Test build #75055 has finished for PR 17363 at commit 1328f1d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@vanzin
Copy link
Contributor

vanzin commented Mar 23, 2017

Merging to 2.1 / 2.0.

I don't really expect any more 1.6 releases at this point, so I wouldn't bother.

asfgit pushed a commit that referenced this pull request Mar 23, 2017
@dongjoon-hyun
[SPARK-19970][SQL][BRANCH-2.1] Table owner should be USER instead of …
af960e8 
…PRINCIPAL in kerberized clusters

## What changes were proposed in this pull request?

In the kerberized hadoop cluster, when Spark creates tables, the owner of tables are filled with PRINCIPAL strings instead of USER names. This is inconsistent with Hive and causes problems when using [ROLE](https://cwiki.apache.org/confluence/display/Hive/SQL+Standard+Based+Hive+Authorization) in Hive. We had better to fix this.

**BEFORE**
```scala
scala> sql("create table t(a int)").show
scala> sql("desc formatted t").show(false)
...
|Owner:                      |sparkEXAMPLE.COM                                         |       |
```

**AFTER**
```scala
scala> sql("create table t(a int)").show
scala> sql("desc formatted t").show(false)
...
|Owner:                      |spark                                         |       |
```

## How was this patch tested?

Manually do `create table` and `desc formatted` because this happens in Kerberized clusters.

Author: Dongjoon Hyun <[email protected]>

Closes #17363 from dongjoon-hyun/SPARK-19970-2.
@vanzin
Copy link
Contributor

vanzin commented Mar 23, 2017

Failed to merge to 2.0. :-/

@vanzin
Copy link
Contributor

vanzin commented Mar 23, 2017

@dongjoon-hyun you'll have to manually close this.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Mar 23, 2017

Oo. Thank you so much, @vanzin !

@dongjoon-hyun dongjoon-hyun mentioned this pull request Mar 23, 2017
Closed
@dongjoon-hyun dongjoon-hyun deleted the SPARK-19970-2 branch January 7, 2019 07:03
@HyukjinKwon HyukjinKwon mentioned this pull request Feb 20, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dongjoon-hyun @SparkQA @vanzin