Skip to content

HBASE-25326 Allow running and building hbase-connectors with Apache Spark 3.0 #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
saintstack merged 1 commit into from
Feb 1, 2021
Merged

HBASE-25326 Allow running and building hbase-connectors with Apache Spark 3.0 #75

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion spark/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,11 @@ To generate an artifact for a different [spark version](https://mvnrepository.co
$ mvn -Dspark.version=2.2.2 -Dscala.version=2.11.7 -Dscala.binary.version=2.11 clean install
```

See above linked spark version to match spark version and supported scala version.
---
To build the connector with Spark 3.0, compile it with scala 2.12.
Additional configurations that you can customize are the Spark version, HBase version, and Hadoop version.
Example:

```
$ mvn -Dspark.version=3.0.1 -Dscala.version=2.12.10 -Dscala.binary.version=2.12 -Dhbase.version=2.2.4 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.0 -DskipTests clean package
```
8 changes: 5 additions & 3 deletions spark/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ import org.apache.hadoop.hbase.regionserver.{HStore, HStoreFile, StoreFileWriter
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.mapred.JobConf
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.rdd.RDD
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.spark.HBaseRDDFunctions._
Expand Down Expand Up @@ -436,7 +435,10 @@ class HBaseContext(@transient val sc: SparkContext,
classOf[IdentityTableMapper], null, null, job)

val jconf = new JobConf(job.getConfiguration)
SparkHadoopUtil.get.addCredentials(jconf)
val jobCreds = jconf.getCredentials()
UserGroupInformation.setConfiguration(sc.hadoopConfiguration)
jobCreds.mergeAll(UserGroupInformation.getCurrentUser().getCredentials())

new NewHBaseRDD(sc,
classOf[TableInputFormat],
classOf[ImmutableBytesWritable],
Expand Down Expand Up @@ -484,7 +486,7 @@ class HBaseContext(@transient val sc: SparkContext,
Configuration = {

if (tmpHdfsConfiguration == null && tmpHdfsConfgFile != null) {
val fs = FileSystem.newInstance(SparkHadoopUtil.get.conf)
val fs = FileSystem.newInstance(sc.hadoopConfiguration)
val inputStream = fs.open(new Path(tmpHdfsConfgFile))
tmpHdfsConfiguration = new Configuration(false)
tmpHdfsConfiguration.readFields(inputStream)
Expand Down
2 changes: 1 addition & 1 deletion spark/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/datasources/Utils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ object Utils {
case DoubleType => Bytes.toDouble(src, offset)
case DateType => new Date(Bytes.toLong(src, offset))
case TimestampType => new Timestamp(Bytes.toLong(src, offset))
case StringType => UTF8String.fromBytes(src, offset, length)
case StringType => Bytes.toString(src, offset, length)
case BinaryType =>
val newArray = new Array[Byte](length)
System.arraycopy(src, offset, newArray, 0, length)
Expand Down