Add training and test functions to integrate the native XGBoost library #281

maropu · 2016-04-27T01:34:02Z

I'm working on the integration of the XGBoost.
As a first step, each UDTF training worker simply outputs a single xgboost model, and then a testing phase computes prediction values for each built model and averages them.

-- `xgboost_model` outputs (string model_id, byte[] model) where each UDTF process
-- creates a xgboost model with an unique ID.
CREATE TABLE xgboost_models AS
  SELECT train_xgboost(add_bias("features"), "label")
    FROM training_data;

-- prediction
CREATE TABLE test_data_with_id AS
  SELECT rowid(), features
    FROM test_data

SELECT rowid, AVG(predicted)
  FROM (
      SELECT xgboost_predict(rowid, features, model_id, model) AS (rowid, predicted)
        FROM xgboost_models CROSS JOIN test_data_with_id
    )
  GROUP BY rowid

maropu · 2016-04-27T01:41:13Z

core/pom.xml

+			<version>0.5</version>
+			<scope>system</scope>
+			<systemPath>${basedir}/lib/xgboost4j-0.5-jar-with-dependencies.jar</systemPath>
+		</dependency>


Currently, no xgboost4j exists in the maven repository. I'll ask the author later why the library is not uploaded there.

maropu · 2016-04-27T01:59:19Z

An obvious other problem is that the native xgboost library depends on some shared libraries;

$ otool -L libxgboost4j.dylib 
libxgboost4j.dylib:
        jvm-packages/lib/libxgboost4j.so (compatibility version 0.0.0, current version 0.0.0)
        /usr/local/lib/gcc/4.9/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.20.0)
        /usr/local/lib/gcc/4.9/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
        /usr/local/lib/gcc/4.9/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

Also, xgboost4j includes no native library for each platform by default. That is, you need compile it for your own platform. This could make it harder to deploy xgboost4j in a large heterogenous computing cluster. To mitigate the portability and deployment issues, we might need to fix these issues directly in the original xgboost4j codes. Otherwise, we need to fork and fix them.

maropu · 2016-04-27T02:49:47Z

I think snappy-java is a winner example in these kinda issues.

maropu · 2016-04-27T02:56:59Z

I asked xgboost4j guys in here.

myui · 2016-04-27T03:58:17Z

better to create our own package xgboost4j-native including native libraries if they are not positive to make it.

maropu · 2016-04-27T03:59:58Z

okay

myui · 2016-04-27T04:00:35Z

core/src/main/java/hivemall/utils/hadoop/HiveUtils.java

@@ -359,6 +359,21 @@ public static double getAsConstDouble(@Nonnull final ObjectInspector numberOI)
                + TypeInfoUtils.getTypeInfoFromObjectInspector(numberOI));
    }

+    @SuppressWarnings("unchecked")


Please move to Primitives, not related to Hive. Also, I think asFloat is not necessary while asFloatArray is useful.

okay, I'll fix it.

maropu · 2016-05-01T02:41:09Z

I implemented a XGBoost multiclass classifier;

-- `xgboost_model` outputs (string model_id, byte[] model) where each UDTF process
-- creates a xgboost model with an unique ID.
CREATE TABLE xgboost_models AS
  SELECT train_multiclass_xgboost_classifier(add_bias("features"), "label")
    FROM training_data;

-- prediction
CREATE TABLE test_data_with_id AS
  SELECT rowid(), features
    FROM test_data

SELECT rowid, max_row(avg_prob_per_label, label)
  FROM (
      SELECT rowid, label, AVG(probability) AS avg_prob_per_label
        FROM (
            SELECT xgboost_multiclass_predict(rowid, features, model_id, model) AS (rowid, label, probability)
              FROM xgboost_models CROSS JOIN test_data_with_id
          )
        GROUP BY rowid, label
    )
  GROUP BY rowid

maropu · 2016-05-01T02:46:04Z

I did a XGBoost binary classifier;

-- `xgboost_model` outputs (string model_id, byte[] model) where each UDTF process
-- creates a xgboost model with an unique ID.
CREATE TABLE xgboost_models AS
  SELECT train_xgboost_classifier(add_bias("features"), "label")
    FROM training_data;

-- prediction
CREATE TABLE test_data_with_id AS
  SELECT rowid(), features
    FROM test_data

SELECT rowid, CAST((case when AVG(predicted) >= 0.5 then 1.0 else 0.0 end) as FLOAT) as predicted
  FROM (
      SELECT xgboost_predict(rowid, features, model_id, model) AS (rowid, predicted)
        FROM xgboost_models CROSS JOIN test_data_with_id
    )
  GROUP BY rowid

coveralls · 2016-05-10T08:25:12Z

Coverage decreased (-0.8%) to 32.884% when pulling bfe447f on maropu:XgboostIntegration into 8f880ab on myui:master.

myui · 2016-05-10T08:57:38Z

ref. to discussions #251

coveralls · 2016-09-01T05:48:33Z

Coverage decreased (-0.7%) to 35.171% when pulling 4f5706c on maropu:XgboostIntegration into 01b6434 on myui:master.

coveralls · 2016-09-01T06:34:34Z

Coverage decreased (-0.7%) to 35.171% when pulling b57f2d9 on maropu:XgboostIntegration into 01b6434 on myui:master.

maropu · 2016-09-01T06:41:12Z

How to use XGBoost functions on DataFrame/Spark is as follows;

// Load libsvm-formatted training data 
val trainDf = sqlContext.sparkSession.read.format("libsvm").load("E2006.train")

// Load test data
val testDf = sqlContext.sparkSession.read.format("libsvm").load("E2006.test")
  .withColumn("rowid", rowid()).cache

// Set XGBoost options here
val xgbOptions = XGBoostOptions()
  .set("num_round", "10000")
  .set("max_depth", "32")

// Build models with the XGBoost and write them into persistent storages
trainDf.train_xgboost_regr($"features", $"label", s"${xgbOptions}")
  .write.format(xgboost).load("xgboost_models")

// Load models from the storage
val model = sqlContext.sparkSession.read.format(xgboost).load("xgboost_models")

// Do prediction
val predict = model.join(testDf).xgboost_predict($"rowid", $"features", $"model_id", $"pred_model")
  .groupBy("rowid").avg()
  .as("rowid", "predicted")

// Check precision
val result = predict.join(testDf, predict("rowid") === testDf("rowid"), "INNER")
result.select(avg(abs($"predicted" - $"target"))).show

coveralls · 2016-09-05T05:53:46Z

Changes Unknown when pulling 73d8090 on maropu:XgboostIntegration into * on myui:master*.

maropu · 2016-09-05T05:56:04Z

okay, I removed the dependencies (libgcc and libstdc++).
Could you check this again?

myui · 2016-09-05T06:32:59Z

spark/spark-1.6/pom.xml

@@ -9,13 +9,12 @@
 		<relativePath>../../pom.xml</relativePath>
 	</parent>

-	<artifactId>hivemall-spark</artifactId>
-	<name>Hivemall on Spark</name>
+	<artifactId>hivemall-spark_2.11</artifactId>


why 2.11? hivemall-spark-v1 and hivemall-spark-v2 might be good for artifact id.

This means this jar supports spark-2.0 compiled with scala-2.11.
The naming rule is along with sprak jar.

coveralls · 2016-09-05T07:28:35Z

Changes Unknown when pulling 0ad666f on maropu:XgboostIntegration into * on myui:master*.

coveralls · 2016-09-05T12:33:28Z

Changes Unknown when pulling 4c0e33c on maropu:XgboostIntegration into * on myui:master*.

coveralls · 2016-09-05T12:54:12Z

Changes Unknown when pulling 4c0e33c on maropu:XgboostIntegration into * on myui:master*.

coveralls · 2016-09-05T23:30:18Z

Changes Unknown when pulling a8f4cf2 on maropu:XgboostIntegration into * on myui:master*.

coveralls · 2016-09-05T23:30:18Z

Changes Unknown when pulling a8f4cf2 on maropu:XgboostIntegration into * on myui:master*.

coveralls · 2016-09-05T23:46:47Z

Changes Unknown when pulling 3296a82 on maropu:XgboostIntegration into * on myui:master*.

coveralls · 2016-09-06T00:09:56Z

Changes Unknown when pulling 3296a82 on maropu:XgboostIntegration into * on myui:master*.

coveralls · 2016-09-06T00:35:13Z

Changes Unknown when pulling 826b390 on maropu:XgboostIntegration into * on myui:master*.

coveralls · 2016-09-06T03:41:42Z

Changes Unknown when pulling e6889dc on maropu:XgboostIntegration into * on myui:master*.

maropu · 2016-09-06T04:03:55Z

fyi: If you get some errors in a xgboost binary bundled in the jar, you can compile the binary on your platform: mvn -Pcompile-xgboost clean package

myui · 2016-09-06T04:04:10Z

@maropu LGTM. Merged. Well done.

maropu reviewed Apr 27, 2016
View reviewed changes

maropu mentioned this pull request Apr 27, 2016

JVM Package New RoadMap dmlc/xgboost#935

Closed

11 tasks

myui reviewed Apr 27, 2016
View reviewed changes

maropu force-pushed the XgboostIntegration branch 2 times, most recently from 84b4c80 to 6461121 Compare April 29, 2016 02:05

maropu force-pushed the XgboostIntegration branch from 1225226 to 7884c16 Compare May 1, 2016 09:18

maropu changed the title ~~[WIP] Add training and test functions to integrate the native XGBoost library~~ Add training and test functions to integrate the native XGBoost library May 1, 2016

myui added enhancement high-priority pullrequest under-review labels May 2, 2016

myui added this to the v0.4 milestone May 2, 2016

myui self-assigned this May 2, 2016

maropu force-pushed the XgboostIntegration branch from 7884c16 to 9080a9f Compare May 10, 2016 08:01

myui force-pushed the master branch from bd31b3d to ab5c19e Compare June 18, 2016 06:06

maropu force-pushed the XgboostIntegration branch from bfe447f to 88682d0 Compare August 23, 2016 09:53

maropu force-pushed the XgboostIntegration branch from 88682d0 to 4f5706c Compare September 1, 2016 05:29

myui reviewed Sep 5, 2016
View reviewed changes

Move the property of scala.version into topdir/pom.xml

0ad666f

maropu force-pushed the XgboostIntegration branch 2 times, most recently from 377d61c to 4c0e33c Compare September 5, 2016 12:13

maropu force-pushed the XgboostIntegration branch 3 times, most recently from e0cb90d to 0ad666f Compare September 5, 2016 22:55

maropu added 3 commits September 6, 2016 08:00

Fix bugs in bin/build_xgboost.sh

7bf055f

Fix version numbers for xgboost

9b9d440

Add activeByDefault in pom.xml

a8f4cf2

Fix version numbers for spark modules

826b390

maropu force-pushed the XgboostIntegration branch from 3296a82 to 826b390 Compare September 6, 2016 00:10

Add a profile to compile xgboost

e6889dc

maropu force-pushed the XgboostIntegration branch from d13a3cf to e6889dc Compare September 6, 2016 03:19

myui merged commit 4f9037e into myui:master Sep 6, 2016

myui mentioned this pull request Sep 6, 2016

Integrate XGboost into Hivemall #251

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add training and test functions to integrate the native XGBoost library #281

Add training and test functions to integrate the native XGBoost library #281

maropu commented Apr 27, 2016

maropu Apr 27, 2016

maropu commented Apr 27, 2016

maropu commented Apr 27, 2016

maropu commented Apr 27, 2016

myui commented Apr 27, 2016

maropu commented Apr 27, 2016

myui Apr 27, 2016 •

edited

Loading

maropu Apr 27, 2016

maropu commented May 1, 2016

maropu commented May 1, 2016

coveralls commented May 10, 2016 •

edited

Loading

myui commented May 10, 2016

coveralls commented Sep 1, 2016 •

edited

Loading

coveralls commented Sep 1, 2016 •

edited

Loading

maropu commented Sep 1, 2016

coveralls commented Sep 5, 2016 •

edited

Loading

maropu commented Sep 5, 2016

myui Sep 5, 2016

maropu Sep 5, 2016 •

edited

Loading

coveralls commented Sep 5, 2016 •

edited

Loading

coveralls commented Sep 5, 2016

coveralls commented Sep 5, 2016

coveralls commented Sep 5, 2016 •

edited

Loading

coveralls commented Sep 5, 2016

coveralls commented Sep 5, 2016

coveralls commented Sep 6, 2016

coveralls commented Sep 6, 2016

coveralls commented Sep 6, 2016 •

edited

Loading

maropu commented Sep 6, 2016

myui commented Sep 6, 2016

Add training and test functions to integrate the native XGBoost library #281

Add training and test functions to integrate the native XGBoost library #281

Conversation

maropu commented Apr 27, 2016

maropu Apr 27, 2016

Choose a reason for hiding this comment

maropu commented Apr 27, 2016

maropu commented Apr 27, 2016

maropu commented Apr 27, 2016

myui commented Apr 27, 2016

maropu commented Apr 27, 2016

myui Apr 27, 2016 • edited Loading

Choose a reason for hiding this comment

maropu Apr 27, 2016

Choose a reason for hiding this comment

maropu commented May 1, 2016

maropu commented May 1, 2016

coveralls commented May 10, 2016 • edited Loading

myui commented May 10, 2016

coveralls commented Sep 1, 2016 • edited Loading

coveralls commented Sep 1, 2016 • edited Loading

maropu commented Sep 1, 2016

coveralls commented Sep 5, 2016 • edited Loading

maropu commented Sep 5, 2016

myui Sep 5, 2016

Choose a reason for hiding this comment

maropu Sep 5, 2016 • edited Loading

Choose a reason for hiding this comment

coveralls commented Sep 5, 2016 • edited Loading

coveralls commented Sep 5, 2016

coveralls commented Sep 5, 2016

coveralls commented Sep 5, 2016 • edited Loading

coveralls commented Sep 5, 2016

coveralls commented Sep 5, 2016

coveralls commented Sep 6, 2016

coveralls commented Sep 6, 2016

coveralls commented Sep 6, 2016 • edited Loading

maropu commented Sep 6, 2016

myui commented Sep 6, 2016

myui Apr 27, 2016 •

edited

Loading

coveralls commented May 10, 2016 •

edited

Loading

coveralls commented Sep 1, 2016 •

edited

Loading

coveralls commented Sep 1, 2016 •

edited

Loading

coveralls commented Sep 5, 2016 •

edited

Loading

maropu Sep 5, 2016 •

edited

Loading

coveralls commented Sep 5, 2016 •

edited

Loading

coveralls commented Sep 5, 2016 •

edited

Loading

coveralls commented Sep 6, 2016 •

edited

Loading