XGBoost 0.90 Roadmap

[hcho3]

This thread is to keep track of all the good things that will be included in 0.90 release. It will be updated as the planned release date (May 1, 2019 as soon as Spark 2.4.3 is out) approaches.

• XGBoost will no longer support Python 2.7, since it is reaching its end-of-life soon. This decision was reached in [VOTE] Ending Python 2.x support in next release (0.83) #4379.
• XGBoost4J-Spark will now require Spark 2.4+, as Spark 2.3 is reaching its end-of-life in a few months ([jvm-packages] support spark 2.4 and compatibility test with previous xgboost version #4377) ([BLOCKING][jvm-packages] upgrade and test with spark 2.4.3 #4409)
• XGBoost4J now supports up to JDK 12 ([jvm-packages] Fixing the NativeLibLoader on Java 9+ #4351)
• Additional optimizations for gpu_hist (Optimisations for gpu_hist. #4248, Further optimisations for gpu_hist. #4283)
• XGBoost as CMake target; C API example (Refactor CMake scripts. #4323, C API example #4333)
• GPU multi-class metrics (GPU multiclass metrics #4368)
• Scikit-learn-like random forest API (Added SKL-like random forest Python API. #4148)
• Bugfix: Fix GPU histogram allocation (Fix Histogram allocation. #4347)
• [BLOCKING][jvm-packages] fix non-deterministic order within a partition (in the case of an upstream shuffle) on prediction [BLOCKING][jvm-packages] fix non-deterministic order within a partition (in the case of an upstream shuffle) on prediction  #4388
• Roadmap: additional optimizations for hist on multi-core Intel CPUs (Optimizations of pre-processing for 'hist' tree method #4310)
• Roadmap: hardened Rabit; see RFC [RFC] harde rabit framework #4250
• Robust handling of missing values in XGBoost4J-Spark [BREAKING][jvm-packages] fix the non-zero missing value handling #4349
• External memory with GPU predictor (initial support for external memory in gpu_predictor #4284, mgpu predictor using explicit offsets #4438)
• Use feature interaction constraints to narrow split search space (Use feature interaction constraints to narrow search space for split candidates #4341)
• Re-vamp Continuous Integration pipeline; see RFC [RFC] Improving continuous integration #4234
• Bugfix: AUC, AUCPR metrics should handle weights correctly for learning-to-rank task (In AUC and AUCPR metrics, detect whether weights are per-instance or per-group #4216)
• Ignore comments in LIBSVM files (Upgrade dmlc-core, to handle comments in LIBSVM files #4430)
• Bugfix: Fix AUCPR metric for ranking (Make AUCPR work with multiple query groups #4436)

[CodingCat]

as we are going to have breaking changes like #4349 and #4377

shall we bump version to 0.9?

[hcho3]

@CodingCat Sure, we can bump to 0.90, if the breaking change is significant. Can you do me a favor and write one-paragraph description of why #4349 was needed?

[CodingCat]

sure,

[alexvorobiev]

* Spark 2.3 is reaching its end-of-life in a few months

Is there an official statement on that? They released 2.2.3 in January and 2.3.3 in February. Our vendor (MapR) still ships 2.3.1.

[CodingCat]

@alexvorobiev #4350, you can check with @srowen from databricks

[srowen]

This is not a question for Databricks but for the Spark project. The default policy is maintenance releases for branches for 18 months: https://spark.apache.org/versioning-policy.html That would put 2.3.x at EOL in about July, so wouldn't expect more 2.3.x releases after that from the OSS project.

[alexvorobiev]

@srowen Thanks!

[hcho3]

@srowen @CodingCat @alexvorobiev Let's also discuss the possibility of supporting Scala 2.12 / 2.13. Right now, XGBoost4J is compiled for Scala 2.11:

xgboost/jvm-packages/pom.xml

Lines 38 to 39 in 2c61f02

	<scala.version>2.11.12</scala.version>
	<scala.binary.version>2.11</scala.binary.version>

A user reported that XGBoost4J JARs compiled for Scala 2.11 is not binary compatible with Scala 2.12.

[srowen]

Yeah, 2.11 / 2.12 are still binary-incompatible, and Spark has two distributions. Both are supported in 2.4.x though 2.12 is the default from here on in 2.4.x. 3.0 will drop Scala 2.11 support.

It may just be a matter of compiling two versions rather than much or any code change. If you run into any funny errors in 2.12 let me know because I stared at lots of these issues when updating Spark.

2.13 is still not GA and think it will be a smaller change from 2.12->2.13 than 2.11->2.12 (big difference here is totally different representation of lambdas).

[CodingCat]

the only issue is that we need to introduce a breaking change to the artifact name of xgboost in maven, xgboost4j-spark => xgboost4j-spark_2.11/xgboost4j-spark_2.12, like spark https://mvnrepository.com/artifact/org.apache.spark/spark-core and we need to double check if we have any transient dependency on 2.11 (I think no)

Hi, @srowen though 2.12 is the default from here on in 2.4.x, I checked branch-2.4 pom.xml, if you don't specify profile scala-2.12, you still get a 2.11 build, no?

[srowen]

You could choose to only support 2.12 in 0.9x, and then you don't have to suffix the artifact name. If you support both, yeah, you'd really want to change the artifact name unfortunately and have _2.11 and _2.12 versions.

Yes the default Spark 2.4.x build will be for 2.11; -Pscala-2.12 gets the 2.12 build.

[CodingCat]

thanks, I'd stay conservative in supporting 2.12 at least for the coming version

as far as I know, most of Spark users are still using 2.11 since they are used to following previous versions of Spark

I may not have bandwidth to go through every test I have for introducing 2.12 support

I would choose to support 2.12 + 2.11 or 2.12 in 1.0 release...

[CodingCat]

@hcho3 FYI, I just removed the dense matrix support from the roadmap given the limited bandwidth

[trivialfis]

@hcho3 Could you take a look at dmlc/dmlc-core#514 when time allows? It might be worth merging before the next release hit.

[hcho3]

@trivialfis Will look at it

[hcho3]

@CodingCat I think we should push back the release date, as Spark 2.4.1 and 2.4.2 have issues. What do you think?

@srowen Do you know when Spark 2.4.3 would be out?

[CodingCat]

I think it’s fine to have some slight delay

[hcho3]

Okay, let’s wait until Spark 2.4.3 is out

[tovbinm]

Would there be the last 0.83 release for Spark 2.3.x?

[hcho3]

@CodingCat What if we make two parallel releases 0.83 and 0.90, where 0.83 includes all commits just before #4377? The 0.83 version would be only released as JVM packages, and Python and R packages would get 0.90. It won't be any more work for me, since I have to write a release note for 0.90 anyway.

One issue though is the user experience with missing value handling. Maybe forcing everyone to use Spark 2.4.x will prevent them from messing up with missing values (the issue which motivated #4349)

[CodingCat]

@hcho3 I am a bit concerned on the inconsistency of different versions in the availability of pkgs.

I can imagine questions like hey, I find 0.83 in maven so I upgrade our Spark pkg, but I cannot use 0.83 in notebook when attempting to explore my new model setup with a small amount of data with python pkg?

I would suggest we either have a full maintenance release on 0.8x branch or nothing

[hcho3]

@CodingCat Got it. We'll do consistent releases for all packages. What's your take on 0.83 release then? Should we do it?

[hcho3]

@CodingCat Actually, this will create work for other maintainers, we'll need to ask them first

[CodingCat]

short answer from a personal view is yes in theory, but it might be more than cutting right before a commit (as you said, it will create work for others as well) (but I am kind of hesitated to do this because of the limited resources in the community...)

here is my 2 cents about how we should think about maintenance release like 0.8x

1. the reason to have a maintenance release is to bring in critical bug fixes, like 2d875ec and 995698b

2. on the other side, to make the community sustainable other than burning out all committers, we should drop support of previous version periodically

3. the innovations and improvements should be brought to the users through a feature release (jump from 0.8 to 0.9)

if we decide to go 0.83, we need to collect opinions from @RAMitchell @trivialfis as well and use their judge to see if we have important (more about correctness) bug fixes which are noticed by them

and then create a 0.83 branch based on 0.82 to cherry-pick commits......a lot of work actually

[RAMitchell]

If I understand correctly, 0.9 will not support older versions of spark, hence the proposal to support a 0.83 version as well as 0.9 to continue support for older spark versions while including bug fixes?

Generally I am against anything that uses developer time. Aren't we busy enough already? I do see some value in having a stable version however.

[hcho3]

@tovbinm You can build XGBoost with commit 711397d to use Spark 2.3.x.

[tovbinm]

Great. So why not make a public release from that commit?

[hcho3]

As @CodingCat said, maintenance releases are not simply a matter of cutting before a commit. Also, making public releases are implicit promises of support. I do not think maintainers are up for supporting two new releases at this point in time.

I'll defer to @CodingCat as to whether we should make a release from 711397d

[hlbkin]

External memory with GPU predictor - this would mean code would not crash with what(): std::bad_alloc: out of memory anymore? (i.e. temporarily swap into RAM?)

related issue I guess #4184 - this was mainly on temporal bursts of memory, the process of fitting itself never require so much memory

[hcho3]

@hlbkin You'll need to explicitly enable external memory, according to https://xgboost.readthedocs.io/en/latest/tutorials/external_memory.html

[CAM-Gerlach]

I assume its not possible to switch otherwise without a major version bump (i.e. 1.0), but when you do, could you consider supporting conformant PEP 440 version numbers (i.e. x.y.z), and preferably semantic versioning? The standard interpretation of 0.90 (rather than 0.9.0) would be that it is the 90th minor release of the major version 0.x (i.e. pre-stable-release) series, and is no more significant than 0.83. Furthermore, this restricts you to a maximum of 9 point releases per minor version, and creates difficulties for some tools (and people) to interpret. Thanks!

[CodingCat]

+1

[hcho3]

@CAM-Gerlach We'll consider it when we release 1.0. On the other hand, we don't want to rush to 1.0. We want 1.0 to be a milestone of some sort, in terms of features, stability, and performance.

[CAM-Gerlach]

Thanks for the explanation, @hcho3 .

You probably want to make sure you set the python_requires argument to '>=3.5' in setup() to ensure users with Python 2 don't get upgraded to an incompatible version accidentally.

[hlbkin]

@hcho3 External memory is not available with GPU algorithms

[hcho3]

@hlbkin You are right. External memory will be available only for GPU predictor, not training.

@rongou @sriramch Am I correct that GPU training isn't available with external memory?

[sriramch]

@hcho3 yes you are correct. we are working on it. the changes are here if you are interested. i'll have to sync this change with master and write some tests.

[hcho3]

@sriramch Awesome! Should we aim to include external memory training in the 0.90 release, or should we come back to it after 0.90?

[CodingCat]

just my two cents, let's reserve a bit on compacting many new features in 0.x (in a rush manner) and consider what is to be put in 1.0 as a milestone version

[hcho3]

@CodingCat I agree. FYI, I deleted distributed customized objective from 0.90 roadmap, since there was substantial disagreement in #4280. We'll consider it again after 0.90.

@sriramch Let's consider external memory training after 0.90 release. Thanks a lot for your hard work.

[RAMitchell]

This might be a good time to release the cuda 9.0 binaries instead of 8.0. I think 9.0 will now be sufficiently supported by users driver version. Additionally the 9.0 binaries will not need to be JIT compiled for the newer Volta architectures.

[CodingCat]

@hcho3 are we ready to go?

[hcho3]

Almost. I think #4438 should be merged.

[hcho3]

All good now. I will go ahead and start working on the next release. ETA: May 16, 2019

• Require Python 3 in setup.py
• Change CI to build CUDA 9.0 wheels ([CI] Build XGBoost wheels with CUDA 9.0 #4459)
• Fix Windows compilation ([CI] Add Windows GPU to Jenkins CI pipeline #4463)
• Set up a minimal viable CI for Windows with GPU ([CI] Add Windows GPU to Jenkins CI pipeline #4463)

[hcho3]

@RAMitchell Should we use CUDA 9.0 or 9.2 for wheel releases?

[RAMitchell]

Lets use 9.2 as that is already set up on CI. The danger is that we require Nvidia drivers that are too new. For reference here is the table showing the correspondence between cuda version and drivers: https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver

As far as I know this should not impact CPU algorithms in anyway. If users begin to report issues then we can address this in future with better error messages around driver compatibility.

[hcho3]

Hmm in that case I can try down-grading one of CI worker to CUDA 9.0. Since we are using Docker containers extensively, it should not be too difficult.

[hcho3]

I'm going to prepare 0.90 release now. My goal is to have the release note complete by end of this week.

[hcho3]

Closed by #4475