Support h2o datatable and numpy types, including for categorical types

[pseudotensor]

https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.Dataset.html

Currently lightgbm consumes a dataframe that is converted to 32-bit or 64-bit floats internally. These values are then binned, but not before having exposed the full data frame as full float values.

h2o-3 and other packages are "lighter" in that they chunk the data further into 1, 2, 4, byte objects. E.g. bools and some floats that only manifest certain number of digits (e.g. 1.1, 1.2, 1.3, 1.4, etc.) take up 4 bytes at least, even if only could have taken 1 byte. A categorical type might have only 10 levels, but still consume a 32-bit object.

Summary

Allow passing data frames with data types less than 32-bit and consume these directly during binning and any categorical handling. Example datasets like Bosch from Kaggle use only 2GB of memory in h2o-3, while consume far more once having to pass a pandas or h2o datatable frame (with more limited types) into lightgbm.

Motivation

Vastly superior memory handling for realistic datasets.

Description

See chunk compression summary at https://www.h2o.ai/wp-content/uploads/2018/01/Python-BOOKLET.pdf

[guolinke]

@pseudotensor seems interesting, I will check it.

[pseudotensor]

Example compression summary from h2o-3 for a version of the Bosch dataset:

I can provide more details if interested.

Note that it allows sparse per column as well. Perhaps lightgbm already supports this?

[guolinke]

@pseudotensor Is it https://github.com/h2oai/datatable ?
BTW, which api can be used to access the raw memory of datatables?

LightGBM currently processes the float data stored in the continued array and pandas.Dataframe will be converted to np array before that.

[pseudotensor]

@pseudotensor Is it https://github.com/h2oai/datatable ?
BTW, which api can be used to access the raw memory of datatables?

Yes, this is what is referred to at: https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.Dataset.html when saying "H2O DataTable's Frame" I believe. If you recall, there was a problem where datatable -> lightgbm caused problems since lightgbm treated it as a buffer type object, but I think you convert it to numpy or pandas inside now.

LightGBM currently processes the float data stored in the continued array and pandas.Dataframe will be converted to np array before that.

Yes, so if types can be persisted, that would save alot.

[StrikerRUS]

Closed in favor of being in #2302. We decided to keep all feature requests in one place.

Welcome to contribute this feature! Please re-open this issue (or post a comment if you are not a topic starter) if you are actively working on implementing this feature.