[K8S] Using hive conf to check whether to apply HIVE_DELEGATION_TOKEN #4835
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
turboFei
changed the title
turboFei
changed the title
turboFei
changed the title
Codecov Report
@@ Coverage Diff @@
## master #4835 +/- ##
============================================
+ Coverage 58.03% 58.07% +0.03%
Complexity 13 13
============================================
Files 583 583
Lines 32561 32570 +9
Branches 4318 4320 +2
============================================
+ Hits 18897 18914 +17
+ Misses 11845 11836 -9
- Partials 1819 1820 +1
... and 6 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Member
Sorry, I don't get your point, w/ apache/spark#37417, Spark downloads and adds those files during spark submit phase, which should happen before SparkContext initialization |
Member
Author
|
Not sure why hive-site.xml is not loaded. 
Anyway, I think |
Member
Author
Seems for spark on k8s, there is no file named |
Member
Author
|
@pan3793 updated the pr description |
pan3793
approved these changes
May 15, 2023
turboFei
added a commit
that referenced
this pull request
May 15, 2023
…ELEGATION_TOKEN
### _Why are the changes needed?_
Now we check the sparkContext.hadoopConfiguration to determine whether to apply HIVE_DELEGATION_TOKEN
Here is the method to create sparkContext hadoopConguration.
And it will add `__spark_hadoop_conf__.xml` to hadoop configuration resource.
```
/**
* Return an appropriate (subclass) of Configuration. Creating config can initialize some Hadoop
* subsystems.
*/
def newConfiguration(conf: SparkConf): Configuration = {
val hadoopConf = SparkHadoopUtil.newConfiguration(conf)
hadoopConf.addResource(SparkHadoopUtil.SPARK_HADOOP_CONF_FILE)
hadoopConf
}
```
```
/**
* Name of the file containing the gateway's Hadoop configuration, to be overlayed on top of the
* cluster's Hadoop config. It is up to the Spark code launching the application to create
* this file if it's desired. If the file doesn't exist, it will just be ignored.
*/
private[spark] val SPARK_HADOOP_CONF_FILE = "__spark_hadoop_conf__.xml"
```
<img width="1091" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/f2a87a23-4565-4164-9eaa-5f7e166519de">
Per the code, this file is only created in yarn module.
#### Spark on yarn
after unzip `__spark_conf__.zip` in spark staging dir, there is a file named `__spark_hadoop_conf__.xml`.
```
grep hive.metastore.uris __spark_hadoop_conf__.xml
<property><name>hive.metastore.uris</name><value>thrift://*******:9083</value><source>programatically</source></property>
```
#### Spark on K8S
Seems for spark on k8s, there is no file named `__spark_hadoop_conf__.xml`
<img width="1580" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/99de73d0-3519-4af3-8f0a-90967949ec0e">
<img width="875" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/f7c477a5-23ca-4b25-8638-4b040b78899d">
We need to check the `hiveConf` instead of `hadoopConf`.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #4835 from turboFei/hive_token.
Closes #4835
7657cbb [fwang12] hive conf
7c0af67 [fwang12] save
Authored-by: fwang12 <[email protected]>
Signed-off-by: fwang12 <[email protected]>
(cherry picked from commit 6b5c138)
Signed-off-by: fwang12 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
Now we check the sparkContext.hadoopConfiguration to determine whether to apply HIVE_DELEGATION_TOKEN
Here is the method to create sparkContext hadoopConguration.
And it will add
__spark_hadoop_conf__.xmlto hadoop configuration resource.Per the code, this file is only created in yarn module.
Spark on yarn
after unzip
__spark_conf__.zipin spark staging dir, there is a file named__spark_hadoop_conf__.xml.Spark on K8S
Seems for spark on k8s, there is no file named
__spark_hadoop_conf__.xmlWe need to check the
hiveConfinstead ofhadoopConf.How was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before make a pull request