Skip to content

Latest commit

 

History

History

training_speed

README.md

GBDT speed benchmarks

This benchmark implements optimization algorithm on a pre-defined grid of hyper-parameters for every library. It measures time to perform boosting iteration and metric's value on test set using library primitives. XGBoost benchmark code served as a starting point for this benchmark.

Library dependencies

xgboost==0.80
matplotlib==2.2.2
tqdm==4.26.0
scipy==1.1.0
lightgbm==2.2.1
pandas==0.23.4
catboost==0.10.3
numpy==1.15.2
scikit_learn==0.20.0

TL;DR

How to run all experiments in one line and get table with results. It will run each library on each dataset (from list: abalone, airline, epsilon, higgs, letters, msrank, msrank-classification, synthetic, synthetic-5k-features) with max_depth=6 three times (with learning rate 0.03, 0.05 and 0.15) for 1000 iterations, dump logs to directory 'logs' write all results to file result.json and create table with time per iteration and total learning time (in common-table.txt).

python benchmark.py --use-gpu

For CPU benchmark run without --use-gpu flag.

Run benchmark (how to define your own benchmark parameters)

First of all you need to define a hyperparameters grid in json file, (you may find grid example in the root directory named example_params_grid.json)

Here every line define a hyperparameter name and a list of values to iterate in experiments.

{
    "max_depth": [6, 10, 12],
    "learning_rate": [0.02, 0.05, 0.08],
    "reg_lambda": [1, 10],
    "subsample": [0.5, 1.0]
}

This is example of how you can run an experiments on GPU for all three libraries for dataset airline using grid ''example_params_grid.json'' with 5000 iterations.

python run.py --learners cat lgb xgb --experiment airline --params-grid example_params_grid.json --iterations 5000 --use-gpu

Logs by default will be written to directory 'logs', its compressed version (containing only timestamps and quality values on iteration) will be written to file 'result.json'.

Supported datasets

Higgs, Epsilon, MSRank (MSLR-WEB10K), CoverType, Airline, Synthetic, Abalone, Letters.

Names for script run.py:

abalone,airline,cover-type,epsilon,higgs,letters,msrank,msrank-classification,syntetic,synthetic-5k-features

Draw plots:

It is supported to draw four types of plots:

  1. Learning curves for top N runs between all experiments (parameter name -- best).
  2. Box plot of time per iteration for each library (time-per-iter).
  3. Draw time to achieve percent from best quality (quality-vs-time).
  4. Learning curves for concrete experiments (custom).

Third method calculates the best achieved quality between all experiments. Then fix a grid of relative percents of this value. Then for each level of quality the method filter all experiments from grid search that reach it and compute median (circle point), minimum and maximum time when algorithm achieved that score.

Examples

Draw learning curves for 5 best experiments on dataset MSRank (multiclass mode):

python plot.py --type best --top 5 --from-iter 500 -i result.json -o msrank_mc_plots

Draw time curves with figure size 10x8 for experiments on Higgs dataset starting from 85% of best achieved quality to 100% in 30 points:

python plot.py --type quality-vs-time -f 10 8 -o higgs_plots -i result.json --low-percent 0.85 --num-bins 30

You can pass option --only-min in this mode, to draw only minimum time to achive percent of quality.

Draw time per iteration box plots for every library:

python plot.py --type time-per-iter -f 10 8 -o higgs_plots -i result.json

If you want to draw experiments of concrete method pass option --only method_name. Choices for method_name are cat, xgb, lgb, cpu, gpu and combinations of those.

Experiments

Infrastructure

  1. Used library versions: CatBoost -- 0.10.3, XGBoost -- 0.80, LightGBM -- 2.2.1.
  2. Hardware for CPU experiments: Intel Xeon E312xx (Sandy Bridge) (32 cores).
  3. Hardware for GPU experiments: NVIDIA GTX1080TI + Intel Core i7-6800K (12 cores).

Description

We took the most popular datasets between benchmarks in other libraries and kaggle competitions. Datasets present different task types (regression, classification, ranking), and have different sparsity number of features and samples. This set of datasets cover the majority of computation bottlenecks in gbdt algorithm, for example with large number of samples computation of loss function gradients may become bottleneck, with large number of features the phase of searching of the tree structure may become computational bottleneck. To demonstrate performance of each library in such experiments we created two synthetic datasets -- the first with 10 million objects and 100 features, the second with 100K objects and 5K features.

In such different tasks the optimal hyper parameters, for example maximal tree depth or learning rate, will be different. Thus we choose different hyper parameter grids for different datasets. The selection of the grid for optimization consisted of the following steps:

  1. Run each library with different maximum depth (or number of leaves in case of LightGBM) and learning rate.
  2. Select best tries for each library and run with different sample rate and regularization parameters.
  3. Create grid based on all observations. It must contain all best tries for every algorithm.

Learning curves, distribution of time per iteration and other plots you may find in plots folder.

Datasets charateristics

Table with dataset characteristics. Sparsity values was calculated using the formula sparsity = (100 - 100 * float(not_null_count) / (pool.shape[0] * pool.shape[1])), where not_null_count is number of NaN, None or zero (abs(x) < 1e-6) values in dataset.

Dataset characteristics #samples #features sparsity
Abalone 4.177 8 3.923
Letters 20.000 16 2.621
Epsilon 500.000 2000 3.399
Higgs 11.000.000 28 7.912
MSRank 1.200.192 137 37.951
Airlines 115.069.017 13 7.821
Synthetic (10M samples, 100 features) 10.000.000 100 0.089
Synthetic (100K samples, 5K features) 100.000 5000 0.079

Chosen grid

Parameters grid iterations max_depth learning_rate subsample reg_lambda
Abalone 2000 [6, 8, 10] [.03, .07, .15] [.5, .75, 1] [1]
Letters 2000 [6, 8, 10] [.03, .07, .15] [.5, .75, 1] [1]
Epsilon 5000 [4,6,8] [.09, .15, .25] [.5, 1] [1, 10, 100]
Higgs 5000 [6, 8, 10] [.01, .03, .07, .15, .3] [.25, .5, .75, 1] [2, 4, 8, 16]
MSRank-RMSE 8000 [6, 10, 12] [.02, .05, .08] [.5, 1] [1, 10]
MSRank-MultiClass 5000 [6, 8, 10] [.03, .07, .15] [.5, .75, 1] [1]
Airlines 5000 [6, 8, 10] [.05] default default
Synthetic (10M samples, 100 features) 5000 [6, 8, 10] [.05, .15] default default
Synthetic (100K samples, 5K features) 1000 [6, 8] [.05] default default

Table of quality values

Metric abbreviations: RMSE -- root mean squared error. Error = # incorrect classified samples / # samples.

Dataset Metric CatBoost XGBoost LightGBM
Abalone RMSE 2.154 2.154 2.143
Letters Error 0.020 0.026 0.023
Epsilon Error 0.109 0.111 0.111
Higgs Error 0.227 0.224 0.222
MSRank-RMSE RMSE 0.737 0.738 0.739
MSRank-MultiClass Error 0.424 0.425 0.424
Airlines Error 0.188 0.190 0.176
Synthetic (10M samples, 100 features) RMSE 8.363 8.287 9.354
Synthetic (100K samples, 5K features) RMSE 6.814 10.39 11.060

Table with speed measurements. Here we present only experiments with max_depth = 6 (other parameters were set to default).

CPU time per iteration, seconds:

median time-per-iter, catboost xgboost lightgbm
abalone 0.005 +/- 0.004 0.01 +/- 0.001 0.002
airline 7.08 +/- 2.684 6.293 +/- 0.378 5.002 +/- 0.375
epsilon 0.271 +/- 0.015 9.008 +/- 0.456 1.344 +/- 0.185
higgs 0.723 +/- 0.032 0.75 +/- 0.192 0.691 +/- 0.254
letters 0.265 +/- 0.011 0.125 +/- 0.025 0.074 +/- 0.02
msrank 0.092 +/- 0.004 0.256 +/- 0.09 0.131 +/- 0.027
msrank-classification 0.565 +/- 0.02 1.701 +/- 0.198 0.95 +/- 0.284
synthetic 0.541 +/- 0.079 1.199 +/- 0.23 1.097 +/- 0.242
synthetic-5k-features 0.166 +/- 0.011 12.481 +/- 3.716 1.1 +/- 0.213

CPU total training time 1000 iterations, seconds:

total, sec catboost xgboost lightgbm
abalone 6.57 13.70 6.43
airline 974.107 611.934 526.182
epsilon 266.77 8935.36 1323.40
higgs 733.27 804.18 788.64
letters 259.75 176.82 112.74
msrank 89.72 341.99 185.26
msrank-classification 55.349 174.726 92.670
synthetic 593.87 1327.40 1154.13
synthetic-5k-features 166.57 14951.47 1135.94

GPU time per iteration, seconds:

median time-per-iter catboost xgboost lightgbm
abalone 0.006 ± 0.001 0.005 ± 0.001 0.004 ± 0.016
airline 0.486 ± 0.016 0.932 ± 0.094 3.641 ± 0.612
epsilon 0.045 ± 0.005 2.686 ± 0.283 0.659 ± 0.107
higgs 0.056 ± 0.001 0.111 ± 0.019 0.166 ± 0.06
letters 0.009 0.034 ± 0.008 0.349 ± 0.099
msrank 0.011 ± 0.001 0.039 ± 0.006 0.053 ± 0.013
msrank-classification 0.022 ± 0.001 0.366 ± 0.021 0.191 ± 2.961
synthetic 0.035 ± 0.003 0.209 ± 0.022 0.112 ± 0.03
synthetic-5k-features 0.027 ± 0.002 1.357 ± 0.149 1.51 ± 0.148

GPU total training time 1000 iterations, seconds:

total, sec catboost xgboost lightgbm
abalone 5.763 5.764 23.109
airline 485.825 928.99 3716.344
epsilon 44.971 2661.741 686.4
higgs 55.601 110.485 166.228
letters 9.271 36.79 115.383
msrank 11.057 40.003 60.688
msrank-classification 22.156 372.541 204.634
synthetic 36.658 210.305 123.255
synthetic-5k-features 28.014 1353.212 1557.471

Third Party Licenses

This repository contains modified versions of files released by third parties under other licenses. See the corresponding notifications in these files for the details.